Multimodal fusion non-invasive aircraft fuel quantity measurement method and system

Abstract

This invention discloses a non-invasive aircraft fuel quantity measurement method and system using multimodal fusion, belonging to the field of aircraft fuel measurement technology. It includes the following steps: acquiring image data and motion data; selecting unobstructed liquid surface areas from the image data; performing 3D reconstruction based on the liquid surface areas in the image data and the motion data to obtain the 3D coordinates of the liquid surface areas in a 3D model of the fuel tank; and obtaining the average fuel level height based on the 3D coordinates to determine the fuel quantity in the tank. Compared to existing technologies, this invention avoids the corrosion and accuracy drift problems caused by long-term immersion of sensors in fuel. Furthermore, by combining accurate image information of the liquid surface area with motion data acquired by the IMU for 3D reconstruction, it accurately simulates the 3D state of the liquid surface inside the fuel tank, thereby achieving precise measurement of the fuel quantity in the aircraft fuel tank.

Description

Technical Field

[0001] This invention relates to the field of aircraft fuel measurement technology, and more specifically, to a non-invasive aircraft fuel quantity measurement method and system using multimodal fusion. Background Technology

[0002] Accurate and stable measurement of aircraft fuel levels is crucial for ensuring aviation safety.

[0003] Current fuel measurement technologies include invasive and non-invasive methods. Invasive methods involve placing sensors inside the fuel tank, which is prone to fuel leaks. Furthermore, sensors immersed in fuel for extended periods are susceptible to corrosion and contamination, leading to measurement accuracy drift and high maintenance costs. Non-invasive technologies are mostly based on single-sensory or inertial measurement units (IMUs). Visual measurements are easily affected by fuel sloshing and changes in aircraft attitude, resulting in feature extraction biases. IMUs lack calibration based on actual fuel surface conditions, leading to cumulative errors. Neither method can simultaneously guarantee measurement accuracy and stability. Therefore, there is an urgent need for a technical solution that can accurately measure aircraft fuel levels. Summary of the Invention

[0004] Therefore, it is necessary to provide a non-invasive aircraft fuel quantity measurement method and system with multimodal fusion to address the above problems and achieve accurate measurement of the fuel quantity in the aircraft fuel tank.

[0005] This application provides a non-intrusive aircraft fuel quantity measurement method based on multimodal fusion, including:

[0006] Acquire image data and motion data; the image data is obtained by the image acquisition device from the liquid level inside the fuel tank; the image acquisition device is set at a designated location inside the fuel tank of the target aircraft; the motion data is obtained by the inertial sensors of the target aircraft. Filter out the liquid surface areas that are not obscured by obstacles from the image data; Based on the liquid surface area and motion data in the image data, a three-dimensional reconstruction is performed to obtain the three-dimensional coordinate values ​​of the liquid surface area in the three-dimensional model of the oil tank. The average fuel level is obtained by using three-dimensional coordinates to determine the amount of fuel in the tank.

[0007] Furthermore, the image data is filtered to identify liquid surface areas that are not obscured by obstacles, specifically including: Local and global features of image data are extracted and fused to obtain a fused feature map of the image data; each element of the fused feature map corresponds to a pixel in the image data; the value of the element represents the confidence level that the corresponding pixel belongs to the liquid surface region; Filter the corresponding pixels whose element values ​​are greater than or equal to the confidence threshold to form the liquid surface area.

[0008] Furthermore, local and global features of the image data are extracted and fused to obtain a fused feature map of the image data, specifically including: Image data is input into a feature extraction network; the feature extraction network includes CNN branches and ViT branches; Local features of image data are extracted using CNN branches; global features of image data are extracted using ViT branches. By using a feature extraction network to perform attention-weighted fusion of local and global features, a fused feature map of the image data is obtained.

[0009] Furthermore, the image data consists of at least one pair of image data, which is obtained by a binocular image acquisition device located at a designated position inside the fuel tank of the target aircraft, capturing the liquid level inside the fuel tank.

[0010] Furthermore, based on the liquid surface area and motion data in the image data, a 3D reconstruction is performed to obtain the 3D coordinate values ​​of the liquid surface area in the 3D model of the oil tank, specifically including: A cost function is constructed based on at least one pair of image data. The cost function includes a matching cost term and a smoothing term. The matching cost term is used to characterize the difference between a pair of image data calculated at each pixel according to the target disparity. The smoothing term is used to suppress the disparity between adjacent pixels. With the goal of minimizing the cost function value, the target disparity of each pixel is optimized to obtain a disparity map composed of the target disparities of each pixel. A 3D model of the fuel tank is obtained by performing 3D reconstruction based on the disparity map. Based on the three-dimensional model of the fuel tank and motion data, the three-dimensional coordinates of the liquid surface area in the three-dimensional model of the fuel tank are calculated.

[0011] Furthermore, the fuel tank is a regular cubic fuel tank; the average fuel level is obtained based on the three-dimensional coordinate values ​​to determine the amount of fuel in the tank, specifically including: The liquid surface area is divided into several blocks; the average fuel level is calculated based on the three-dimensional coordinates of each block in the liquid surface area. Obtain the internal length and width of the fuel tank; calculate the fuel quantity based on the average fuel level, internal length, and internal width.

[0012] Furthermore, the fuel tank is an irregularly shaped tank; the average fuel level is obtained based on three-dimensional coordinate values ​​to determine the amount of fuel in the tank, specifically including: The fuel tank is divided into multiple cylindrical sections along the horizontal direction, and the cross-sectional area of ​​each cylindrical section is obtained. The oil level height of each columnar section is determined based on the three-dimensional coordinate values ​​of each area block in the liquid surface region within each columnar section. The amount of fuel is calculated based on the cross-sectional area and oil level of each column section.

[0013] Furthermore, this application also provides a multimodal fusion non-invasive aircraft fuel quantity measurement system for implementing the aforementioned multimodal fusion non-invasive aircraft fuel quantity measurement method. The system includes: Data acquisition module: used to acquire image data and motion data; the image data is obtained by the image acquisition device from the liquid level inside the fuel tank; the image acquisition device is set at a designated location inside the fuel tank of the target aircraft; the motion data is obtained by the inertial sensors of the target aircraft. Region filtering module: used to filter out liquid surface areas that are not obscured by obstacles from image data; 3D Reconstruction Module: Used to perform 3D reconstruction based on the liquid surface area and motion data in the image data, and obtain the 3D coordinate values ​​of the liquid surface area in the 3D model of the oil tank; Fuel level calculation module: used to obtain the average fuel level height based on three-dimensional coordinate values ​​in order to determine the amount of fuel in the fuel tank.

[0014] In another aspect, this application also provides an electronic device, including: a memory and a processor, which are communicatively connected to each other. The memory stores computer instructions, and the processor executes the computer instructions to perform the above-mentioned multimodal fusion non-invasive aircraft fuel quantity measurement method.

[0015] In another aspect, this application also provides a computer-readable storage medium storing computer instructions for causing a computer to execute the above-described multimodal fusion non-intrusive aircraft fuel quantity measurement method.

[0016] Compared with the prior art, the advantages of this invention are: (1) This invention acquires fuel image data using an image acquisition device installed inside the fuel tank and acquires fuselage motion data using an inertial sensor. This eliminates the need for sensors to be installed through openings in the sidewalls of the fuel tank. Based on this, the invention first filters out liquid surface areas from the image data that are not obscured by air bubbles, impurities, or other interference. With an accurate liquid surface area obtained, three-dimensional reconstruction is performed based on the image information of the liquid surface area and the motion data to obtain the three-dimensional coordinates of the liquid surface area in the three-dimensional model of the fuel tank. Finally, the average fuel level is calculated based on the three-dimensional coordinates to determine the fuel quantity. This approach avoids the corrosion and accuracy drift problems caused by long-term immersion of sensors in fuel. Furthermore, by combining accurate liquid surface area image information with motion data acquired by the IMU for three-dimensional reconstruction, the three-dimensional state of the liquid surface inside the fuel tank is accurately simulated, thereby achieving precise measurement of the fuel quantity in the aircraft fuel tank.

[0017] (2) The present invention extracts and fuses local and global features of image data to obtain a fused feature map. The value of each element of the fused feature map represents the confidence level of the corresponding pixel point belonging to the liquid surface area. By selecting pixels with a confidence level greater than or equal to the confidence level threshold, the liquid surface area is formed, which realizes the accurate segmentation of the liquid surface area from the tank wall, bubbles and impurities, and solves the problem that it is difficult to segment the real liquid surface area from the image under the influence of fuel sloshing and interference.

[0018] (3) This invention uses CNN branches and ViT branches to extract local and global features of the liquid surface, respectively, and achieves adaptive fusion of local and global features through attention weighting. When there is severe oil mist or shaking in the fuel tank, the weight of local features is increased; when the liquid surface is evenly illuminated and the features are sparse, the weight of local features is decreased. This process achieves accurate segmentation of the liquid surface area and solves the problem of difficulty in segmenting the real liquid surface area from the image under the influence of fuel shaking and interference.

[0019] (4) The present invention uses a binocular image acquisition device to acquire at least one pair of image data, uses the principle of binocular vision to obtain liquid surface depth information through parallax calculation, and combines motion data to perform attitude calibration. Compared with the monocular vision scheme, this process can obtain more accurate three-dimensional spatial coordinates. Attached Figure Description

[0020] Figure 1 This is a flowchart of a non-invasive aircraft fuel quantity measurement method using multimodal fusion in one embodiment of the present invention; Figure 2 This is a schematic diagram of the structure of a non-intrusive aircraft fuel quantity measurement system with multimodal fusion according to an embodiment of the present invention; Figure 3 This is a schematic diagram of the structure of an electronic device according to an embodiment of the present invention. Detailed Implementation

[0021] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments. The technical solution of the present invention is applicable to various aircraft fuel quantity measurement scenarios, aiming to solve the problems of fuel leakage risk and sensor corrosion associated with existing invasive measurement methods, as well as the inaccuracy of fuel quantity measurement in non-invasive methods under the influence of fuel sloshing and interference. The technical solution of the present invention will be described in detail below with reference to specific embodiments.

[0022] like Figure 1 As shown, this embodiment discloses a non-intrusive aircraft fuel quantity measurement method based on multimodal fusion, including the following steps: S1. Acquire image data and motion data.

[0023] Specifically, the image data is obtained by an image acquisition device capturing the liquid level inside the fuel tank. The image acquisition device is positioned at a designated location inside the target aircraft's fuel tank.

[0024] More specifically, the image data is at least one pair of image data. For example, it can be binocular image data obtained by a binocular image acquisition device set at a designated location inside the fuel tank of the target aircraft to collect the liquid level inside the fuel tank, which is used to reflect the dynamic changes in the fuel level inside the fuel tank.

[0025] The designated location is a measurement mounting position pre-reserved on the inner wall of the target aircraft's fuel tank, free from fuel obstruction and severe vibration. The lens of the binocular image acquisition device is pointed towards the fuel surface to ensure complete coverage of the fuel surface area in the acquisition field of view.

[0026] The binocular image acquisition device includes an industrial-grade left and right eye camera. The optical axes of the left and right eye cameras are parallel, the resolution is no less than 1080P, and the frame rate is no less than 20fps, so as to ensure that the binocular image data (left and right eye images) acquired by the binocular camera can clearly capture the dynamic changes of the fuel level.

[0027] The binocular image acquisition device is also equipped with an anti-oil fog lens cover and an adaptive area array LED fill light unit. The anti-oil fog lens cover has a built-in micro positive pressure anti-oil fog unit to prevent oil fog from adhering to the lens surface; the adaptive area array LED fill light unit automatically adjusts the brightness according to the light intensity in the oil tank, thereby avoiding the loss of feature points caused by liquid surface reflection.

[0028] Specifically, the motion data is acquired by the inertial sensors of the target aircraft. The motion data includes the angular rate and acceleration of the target aircraft, which are used to subsequently calculate the aircraft's pitch and roll angles.

[0029] The inertial sensor is fixedly connected to the stereo camera to ensure there is no relative movement between them. The inertial sensor uses temperature compensation to eliminate the impact of ambient temperature changes on the accuracy of the acquired data, and outputs motion data including timestamps.

[0030] Preferably, the sampling frequency of the inertial sensor (e.g., IMU) is higher than that of the binocular camera. For example, when the acquisition trigger frequency of the binocular vision module is 30Hz, the sampling frequency of the inertial measurement unit can be set to 50Hz.

[0031] To ensure precise temporal alignment between image and motion data, this step utilizes an FPGA hard clock synchronization module to achieve hardware-level clock trigger synchronization. The FPGA hard clock synchronization module sends rising-edge valid acquisition trigger signals simultaneously to the image acquisition device and the inertial sensor via GPIO level triggering, setting the acquisition trigger frequency to 30Hz to achieve hardware-level acquisition frequency synchronization. Specifically, the left and right images of a single-frame binocular image share the same timestamp. This ensures a strict temporal correspondence between image and motion data in subsequent multimodal data fusion.

[0032] In a specific embodiment, the acquired image data and motion data are preprocessed, and the specific process is as follows: (1) Registration and screening.

[0033] Set the timestamp matching threshold to ±5ms. Iterate through the acquired image data sequences and motion data sequences with timestamps. Single-frame stereo images with a timestamp difference ≤5ms and single sets of motion data are identified as synchronized data pairs. If the timestamp difference >5ms, it is identified as abnormal data and discarded.

[0034] (2) Preprocessing of image data.

[0035] The binocular images are converted into grayscale images of level 0 to 255, and then their size is standardized and uniformly scaled to 224×224 pixels to obtain standardized binocular images.

[0036] Because Gaussian noise and salt-and-pepper noise are easily introduced during image data acquisition and transmission, Gaussian filtering is used for noise reduction to preserve as much of the liquid surface area's edge (such as the boundary between the liquid surface and the tank wall) and the liquid surface texture information as possible. The process is as follows: ; In the formula: The standard deviation of the Gaussian filter is 0.8 to 1.2, and its value ranges from 0.8 to 1.2. In this embodiment... ; It is a natural constant; These are pixel coordinate values; This represents the Gaussian template. Compared with the left eye image respectively And right eye image Perform separately Convolution operation yields the denoised left eye image. And right eye image .

[0037] Zhang's calibration method was used to perform offline calibration on the left and right cameras respectively, obtaining the intrinsic parameter matrix (including focal length and center coordinates) and distortion coefficients (including radial distortion coefficients). and tangential distortion coefficient Then, the denoised binocular images... and Radial distortion correction and tangential distortion correction are performed. Radial distortion correction corrects the radial offset of edge pixels, while tangential distortion correction corrects the distortion caused by the lens not being parallel to the imaging plane. The process is as follows: Radial distortion correction: ; Tangential distortion correction: ; In the formula: These are the pixel coordinates of the distorted image. and The corrected pixel coordinates; This represents the radial distance from the pixel to the center. The radial distortion coefficient is... denoted as the tangential distortion coefficient.

[0038] The coordinates of the corrected image are normalized to map to the resolution of the original image, resulting in a distortion-free binocular image as the final image data.

[0039] (3) Preprocessing of motion data.

[0040] Kalman filtering is used to denoise and estimate motion data, suppressing high-frequency noise while preserving the true motion changes. State variables are defined. For a moment The estimated values ​​(in this embodiment, the state variables refer to the filtered angular velocity and acceleration).

[0041] Kalman filtering consists of a prediction phase and an update phase. The prediction phase estimates the current state based on the state at the previous time step, while the update phase refines the predicted state using the current observations. The prediction phase process is represented as follows: ; In the formula: for State variables after filtering at time step; for Predict state variables at any time; This is the state transition matrix; for The state covariance matrix at any given time; To predict the covariance matrix; The process noise covariance matrix; Represents the state transition matrix The transpose of .

[0042] The update phase process is represented as follows: ; In the formula: Kalman gain; for IMU observations at time points; The observation matrix; It is the identity matrix; To observe the noise covariance matrix; Represents the observation matrix The transpose of .

[0043] The angular velocity and acceleration are obtained after noise reduction by Kalman filtering.

[0044] Thus, through the above preprocessing steps, the final image data and motion data were obtained.

[0045] S2. Filter out the liquid surface area that is not obscured by obstacles in the image data.

[0046] The optical axes of the left and right cameras in the binocular image acquisition device are parallel, and the captured areas of the oil tank almost completely overlap, with highly consistent liquid surface features. In practical applications, adhering to the principle of lightweight model design and facilitating deployment on edge devices, only the image data acquired by one camera can be processed to obtain a fused feature map. When one camera fails, the image data from the other camera can be used for processing, reducing processing time and memory usage. This embodiment takes filtering the liquid surface area not obscured by obstacles in the left-eye image data as an example. The specific steps are as follows: Specifically, image data is input into a feature extraction network to extract and fuse local and global features of the image data. The feature extraction network includes a CNN branch and a ViT branch. The CNN branch extracts local features from the image data; the ViT branch extracts global features.

[0047] The CNN branch uses a ResNet-50 network to extract local features from the left eye image through five convolutional stages, focusing on capturing the edges of the liquid surface, subtle textures of the oil surface, and local sloshing features, and outputting a local feature map. ;in, For feature map size, The number of channels in the feature map is calculated using the following formula for the convolutional layer: ; In the formula: This indicates the location of the output feature map of the convolutional layer. Pixel value at; In this embodiment, the kernel size is... ; These are the kernel weights; For bias, It is the ReLU activation function. ; and The parameters required for the traversal operation. This indicates the corresponding position of the convolution kernel.

[0048] The ViT branch employs the encoder portion of the Transformer architecture to segment and linearly embed the left-eye image, adding category embeddings and learnable positional encodings to extract global features from the left-eye image. It focuses on capturing the overall morphology of the liquid surface, the trend of large-scale sloshing, and the spatial relationship between the liquid surface and the inner wall of the tank, outputting a global feature map. The global feature map and the local feature map have the same size, and the multi-head attention formula is as follows: ; ; ; ； ; In the formula: For query vector; The key vector; It is a value vector; This represents the feature matrix obtained after the input image is segmented and linearly embedded. , , These are the learnable weight matrices corresponding to the query, key, and value, respectively. Represents the key vector matrix The transpose of the matrix; For the dimension of a single attention head, ;in For the total feature dimension, For the number of heads; This is a normalized exponential function used to convert attention weights into a probability distribution; This is a single-head attention computation function used to output weighted features; For multi-head attention computation function, the outputs of multiple single-head attention are concatenated along the channel dimension; Indicates a splicing operation; They represent the 1st to the 1st. The output of each attention head.

[0049] Specifically, a feature extraction network is used to perform attention-weighted fusion of local and global features to obtain a fused feature map of the image data.

[0050] The bi-branch features are dimensionally aligned, and the local and global features are then fused using an attention-weighted fusion method via a feature extraction network. Since the local and global feature maps have the same spatial dimensions and number of channels, they can be directly fused. The process is as follows: ; In the formula: To fuse feature maps; For automatically learning attention weighting coefficients, .

[0051] When there is severe oil mist and shaking inside the fuel tank, the feature extraction network increases in size. Weights are used to enhance local texture feature extraction; when the liquid surface is evenly illuminated and features are sparse, the feature extraction network is reduced. Weights enhance global feature extraction.

[0052] Specifically, each element of the fused feature map corresponds to a pixel in the image data; the value of the element represents the confidence that the corresponding pixel belongs to the liquid surface region; and the corresponding pixels with element values ​​greater than or equal to the confidence threshold are selected to form the liquid surface region.

[0053] The feature extraction network outputs mask features of the liquid surface region through a convolutional layer and a sigmoid activation function. The liquid surface area is determined based on the mask features, achieving precise segmentation of the liquid surface from the tank walls, air bubbles, and impurities. The process is represented as follows: ; when At that time, it was determined to be the liquid surface area. At that time, it was determined to be a non-liquid surface area.

[0054] Specifically, the confidence threshold ranges from (0, 1), and in this embodiment, the confidence threshold is preset to 0.5. The area is identified as the liquid surface area and used for subsequent 3D reconstruction and oil volume calculation.

[0055] Similarly, the embodiments of this application can also process the right eye image in the above manner to obtain the liquid surface area in the right eye image.

[0056] S3. Based on the liquid surface area and motion data in the image data, perform three-dimensional reconstruction to obtain the three-dimensional coordinate values ​​of the liquid surface area in the three-dimensional model of the oil tank.

[0057] Specifically, a cost function is constructed based on at least one pair of image data. The cost function includes a matching cost term and a smoothing term; the matching cost term is used to characterize the difference calculated at each pixel point according to the target disparity of a pair of image data; the smoothing term is used to suppress the disparity between adjacent pixels.

[0058] In one specific embodiment, pixel blocks are used as matching units. The optimal matching block is found for each pixel block of the left image within the right-view image. The disparity value is determined by calculating the matching cost. Represented as: ; In the formula: For the parallax to be detected, This is the weighting coefficient, with a value ranging from 0.6 to 0.8. This is the sum of absolute differences, used to measure the grayscale similarity between pixel blocks; This is a gradient interpolation operator used to measure texture similarity; To match the cost function, a weighted sum of the absolute value difference and the gradient difference is used, which can simultaneously take into account grayscale information and edge texture information, effectively suppressing distortion problems caused by liquid surface reflection, fluctuations, bubbles, etc.

[0059] A cost function is constructed that combines the cost of single-pixel matching with the cost of disparity smoothing between neighboring pixels. The optimal disparity is obtained by minimizing this cost function, thus avoiding getting trapped in local optima. The cost function is expressed as: ; ; In the formula: Indicates the matching cost term; This represents a smoothing term that penalizes the disparity difference between adjacent pixels, used to suppress disparity between adjacent pixels; express 8-neighborhood pixels ; and These represent penalty coefficients, used to maintain smoothness. Protect surface feature information of tilted or curved surfaces. Maintain boundary information, ; This is an indicator function that assigns a value of 1 when the condition is met.

[0060] Specifically, with the goal of minimizing the cost function, the target disparity of each pixel is optimized to obtain a disparity map composed of the target disparities of each pixel. The process is represented as follows: ; In the formula: Preset parallax range; and These are the minimum and maximum parallax, respectively, set by the calibration parameters of the binocular camera. To indicate that The variable corresponding to the minimum value The value of . For each pixel Traverse the preset parallax range The disparity value that minimizes the cost function is taken as the target disparity.

[0061] In one specific embodiment, to eliminate parallax noise caused by air bubbles and reflections inside the fuel tank, the acquired parallax map is optimized: the parallax map calculated based on the left-eye image is compared with the parallax map calculated based on the right-eye image. Inconsistent abnormal parallax (considered as occluded areas) is removed; linear interpolation is performed on the empty parallax of occluded areas; for a pixel marked as an occluded area, the weighted average of the parallax values ​​of its nearest unoccluded pixels on its left and right sides is used as the parallax value of that point, thereby completing the dense parallax map; a 3×3 window median filter is used to take the median of the neighborhood of each pixel's parallax value to eliminate isolated noise points and ensure the smoothness of the parallax map.

[0062] Specifically, based on the principle of binocular vision 3D reconstruction, a 3D model of the fuel tank is obtained by performing 3D reconstruction according to the disparity map. Based on the 3D model of the fuel tank and motion data, the 3D coordinates of the liquid surface area in the 3D model of the fuel tank are calculated.

[0063] First, combine the disparity map Calculate the original three-dimensional coordinates of the liquid surface region , represented as: ; In the formula: The depth distance from the spatial point corresponding to the pixel on the liquid surface to the binocular camera; For camera focal length; The optical center pixel coordinates of the camera are all pre-calibrated according to Zhang's calibration method; Baseline distance, This represents the disparity value.

[0064] Then, based on the motion data, the attitude is calculated using the quaternion method to obtain the pitch and roll angles of the target aircraft fuselage. Based on the pitch and roll angles of the target aircraft fuselage, an attitude rotation matrix is ​​constructed, and the original three-dimensional coordinates are rotated and calibrated to obtain the three-dimensional coordinate values ​​of the liquid surface area in the three-dimensional model of the fuel tank after eliminating attitude deviations.

[0065] In a specific embodiment, let the quaternion be... , .in, For the real part, The imaginary part satisfies Then, the quaternion is converted into a real-time attitude angle, the process of which is expressed as: ; In the formula: This is the roll angle. The pitch angle, For yaw angle, only select , Used for subsequent attitude deviation compensation.

[0066] Since quaternions can only solve for the current time step. The real-time attitude angle needs to be updated using the quaternion from the previous time step.

[0067] Motion data is captured based on clock signal determination during data acquisition, therefore, motion data acquisition has... In this embodiment, the quaternion for the next time step is updated using the first-order Euler integral, and the process is expressed as follows: ; In the formula: Indicates the current time Quaternions; Indicates the next moment Quaternions; Quaternion The derivative with respect to time, i.e., the rate of change of a quaternion; This represents the sampling time interval.

[0068] in, Depends on the measured current body acceleration Driver update, the process is represented as follows: ; That is, it can be expressed as: ; In the formula: Represents quaternion multiplication; It is a pure imaginary quaternion composed of three-axis angular velocity components, where the real part is 0 and the imaginary part is composed of the angular velocities around the x-axis, y-axis and z-axis of the aircraft.

[0069] After the update is completed, the resulting quaternions are normalized to ensure that the quaternions always satisfy the unit constraint and avoid numerical deviation, thereby ensuring the legality of the rotation attitude representation.

[0070] To eliminate attitude deviations, an attitude rotation matrix is ​​constructed. , represented as: ; ; ; In the formula: Here is the attitude rotation matrix; This is the roll rotation matrix; This is the pitch and rotation matrix; The original 3D coordinates are rotated and calibrated using an attitude rotation matrix to obtain the 3D coordinates of the liquid surface area in the 3D model of the oil tank after eliminating attitude deviations. , represented as:

[0071] S4. Obtain the average fuel level height based on the three-dimensional coordinate values ​​to determine the amount of fuel in the fuel tank.

[0072] Specifically, the fuel tank is a regular cubic fuel tank. Based on the three-dimensional coordinates of the fuel surface area in the three-dimensional model of the fuel tank, the average fuel level is obtained to determine the amount of fuel in the tank. This includes: The liquid surface area is divided into several blocks; based on the three-dimensional coordinates of each block within the liquid surface area, the average fuel level is calculated. The process is as follows: ; In the formula: This represents the number of blocks in the region. For the first The height coordinates of each region block; This refers to the average fuel level. Obtain the internal length and width of the fuel tank; calculate the fuel quantity based on the average fuel level, internal length, and internal width. The process is as follows: ; In the formula: This refers to the amount of fuel. This refers to the internal length of the fuel tank. This refers to the internal width of the fuel tank.

[0073] Specifically, the fuel tank is an irregularly shaped tank. The average fuel level is obtained based on three-dimensional coordinates to determine the amount of fuel in the tank, including: The fuel tank is divided into multiple cylindrical sections horizontally, and the cross-sectional area of ​​each section is obtained. Based on the three-dimensional coordinates of each region within the liquid level area of ​​each cylindrical section, the fuel level height of each section is determined. The fuel quantity is calculated based on the cross-sectional area and fuel level height of each cylindrical section. The process is as follows: ; In the formula: This refers to the amount of fuel. For the first The cross-sectional area of ​​each columnar section is pre-calibrated by the fuel tank geometry model; For the first The oil level height of each columnar section. For the first... Each columnar section has an oil level height. The height coordinates of all valid pixels (or their 8-neighborhood) within the columnar partition. The weighted average.

[0074] This invention addresses irregularly shaped fuel tanks by dividing the tank horizontally into multiple cylindrical sections and pre-obtaining the cross-sectional area of ​​each section. The fuel level of each section is determined based on its three-dimensional coordinates, and the total fuel volume is calculated by combining the cross-sectional area and fuel level of each section. This piecewise integration method adapts to the complex variations in the internal structure of the fuel tank, solving the problem of accurately calculating fuel volume using a single average height for irregularly shaped tanks. This makes the solution applicable to different types of aircraft fuel tanks.

[0075] In one specific embodiment, the measurement error of the fuel quantity is further corrected through zero-point calibration and error compensation. The specific process is as follows: (1) Zero point calibration In practical applications, since it is impossible to guarantee that the bottom of the fuel tank is absolutely level, or that the fuel tank will be completely emptied when released, this embodiment recalibrates the zero point of calculation.

[0076] Under the condition of emptying the fuel tank as much as possible, the average height of the empty fuel tank level was obtained by extracting feature regions using CNN-ViT and then performing scale calibration. Combining the fuselage attitude data (including pitch and roll angles) calculated in step S3, it is converted into oil level height in an absolute horizontal reference coordinate system to eliminate the effects of tilt. The process is expressed as follows: ; In the formula: This represents the fuselage attitude data obtained after quaternion attitude calculation; The compensation liquid level height obtained after zero-point calibration is expressed as: .

[0077] (2) Error compensation In practical applications, since aircraft cannot always remain stationary or maintain the same stable attitude during flight, significant errors can occur during large attitude changes (dives, sharp turns, etc.). Therefore, this embodiment applies attitude compensation. Simultaneously, as the aircraft's altitude changes, the temperature also changes, causing the liquid in the fuel tank to expand or contract in stages. Although these changes are slight, they have a significant impact on accurate measurements. Therefore, this embodiment applies temperature compensation, the process of which is represented as follows: ; ; In the formula: For attitude compensation; Temperature compensation can be obtained by function fitting based on a specific dataset.

[0078] Based on the above zero-point calibration and error compensation steps, the final measurement results for each region are obtained as follows: ,in: The weighting coefficients for attitude compensation are identified in the measurement results. The weighting coefficient for temperature compensation in the measurement results is indicated, and the values ​​are all (0, 1).

[0079] Mean squared error is used as the cost function: The cost function is minimized using gradient descent, and its parameters are updated until 90% of the data in the measurement results meet the requirements. Once the predicted output after training meets the preset accuracy requirements, the converted liquid level height can be output. .

[0080] This embodiment also discloses a multimodal fusion non-intrusive aircraft fuel quantity measurement system, used to implement the above-mentioned multimodal fusion non-intrusive aircraft fuel quantity measurement method, including: Data acquisition module: used to acquire image data and motion data; the image data is obtained by the image acquisition device from the liquid level inside the fuel tank; the image acquisition device is set at a designated location inside the fuel tank of the target aircraft; the motion data is obtained by the inertial sensors of the target aircraft. Data processing module: used to filter out liquid surface areas that are not obscured by obstacles from image data; Based on the liquid surface area and motion data in the image data, a three-dimensional reconstruction is performed to obtain the three-dimensional coordinate values ​​of the liquid surface area in the three-dimensional model of the oil tank. Fuel level calculation module: used to obtain the average fuel level height based on three-dimensional coordinate values ​​in order to determine the amount of fuel in the fuel tank.

[0081] like Figure 2 As shown, this application also provides a multimodal fusion non-invasive aircraft fuel quantity measurement system to implement the above embodiments and preferred embodiments, and details already described will not be repeated. As used below, the term "module" can be a combination of software and / or hardware that performs a predetermined function. Although the apparatus described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0082] This application provides a multimodal fusion non-intrusive aircraft fuel quantity measurement system, including: Data acquisition module: used to acquire image data and motion data; the image data is obtained by the image acquisition device from the liquid level inside the fuel tank; the image acquisition device is set at a designated location inside the fuel tank of the target aircraft; the motion data is obtained by the inertial sensors of the target aircraft. Region filtering module: used to filter out liquid surface areas that are not obscured by obstacles from image data; 3D Reconstruction Module: Used to perform 3D reconstruction based on the liquid surface area and motion data in the image data, and obtain the 3D coordinate values ​​of the liquid surface area in the 3D model of the oil tank; Fuel level calculation module: used to obtain the average fuel level height based on three-dimensional coordinate values ​​in order to determine the amount of fuel in the fuel tank.

[0083] As shown in Figure 3, this is a schematic diagram of an electronic device provided in an optional embodiment of the present invention. This electronic device can be a computer device used to execute the above-described methods. The electronic device includes one or more processors 10, a memory 20, and interfaces for connecting the various components, including high-speed interfaces and low-speed interfaces. The various components are interconnected via different buses and can be mounted on a common motherboard or otherwise installed as needed. The processor can process instructions executed within the electronic device, including instructions stored in or on the memory to display graphical information of a GUI on an external input / output device (such as a display device coupled to the interface).

[0084] The processor 10 may further include a hardware chip. This hardware chip may be an application-specific integrated circuit (ASIC), a programmable logic device (PLD), or a combination thereof. The programmable logic device may be a complex programmable logic device (CAMP), a field-programmable gate array (FPGA), a general-purpose array logic (GDA), or any combination thereof.

[0085] The memory 20 stores instructions executable by at least one processor 10 to cause the at least one processor 10 to perform the method shown in the above embodiments.

[0086] The memory 20 may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created by the use of the electronic device based on the display of a mini-program landing page. Furthermore, the memory 20 may include high-speed random access memory (RAM), and may also include non-transient memory, such as at least one disk storage device, flash memory device, or other non-transient solid-state storage device. The memory 20 may include volatile memory, such as RAM; the memory may also include non-volatile memory, such as flash memory, hard disk, or solid-state drive; the memory 20 may also include combinations of the above types of memory.

[0087] The electronic device also includes a communication interface 30 for communicating with other devices or communication networks.

[0088] This invention also provides a computer-readable storage medium. The methods described above according to embodiments of the invention can be implemented in hardware or firmware, or implemented as computer code that can be recorded on a storage medium, or implemented as computer code downloaded via a network and originally stored on a remote storage medium or a non-transitory machine-readable storage medium and then stored on a local storage medium. Thus, the methods described herein can be processed by software stored on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. The storage medium can be a magnetic disk, optical disk, read-only memory, random access memory, flash memory, hard disk, or solid-state drive, etc.; further, the storage medium can also include combinations of the above types of memory. It is understood that computers, processors, microprocessor controllers, or programmable hardware include storage components capable of storing or receiving software or computer code, which, when accessed and executed by the computer, processor, or hardware, implements the methods shown in the above embodiments.

[0089] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0090] The embodiments described above are merely illustrative of several implementations of the present invention, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these all fall within the protection scope of the present invention. Therefore, the protection scope of this invention patent should be determined by the appended claims.

Claims

1. A multi-modal fused non-intrusive aircraft fuel quantity measurement method, characterized in that, include: Acquire image data and motion data; the image data is obtained by an image acquisition device from the liquid level inside the oil tank. The image acquisition device is installed at a designated location inside the fuel tank of the target aircraft; the motion data is acquired by the inertial sensors of the target aircraft. Select the liquid surface area that is not obscured by obstacles from the image data; Based on the liquid surface area in the image data and the motion data, a three-dimensional reconstruction is performed to obtain the three-dimensional coordinate values ​​of the liquid surface area in the three-dimensional model of the oil tank. The average fuel level is obtained based on the three-dimensional coordinate values ​​to determine the amount of fuel in the fuel tank.

2. The multi-modal fused non-intrusive aircraft fuel quantity measurement method of claim 1, wherein, The step of filtering out unobstructed liquid surface areas from the image data specifically includes: Local and global features of the image data are extracted and fused to obtain a fused feature map of the image data; each element of the fused feature map corresponds to a pixel of the image data; the value of each element represents the confidence level that the corresponding pixel belongs to the liquid surface region; The corresponding pixels whose element values ​​are greater than or equal to the confidence threshold are selected to form the liquid surface area.

3. The multi-modal fused non-intrusive aircraft fuel quantity measurement method of claim 2, wherein, The step of extracting and fusing local and global features of the image data to obtain a fused feature map of the image data specifically includes: The image data is input into a feature extraction network; the feature extraction network includes a CNN branch and a ViT branch; Local features of the image data are extracted using the CNN branch; global features of the image data are extracted using the ViT branch; The image data is fused by performing attention-weighted fusion of local and global features through a feature extraction network.

4. The non-invasive aircraft fuel quantity measurement method based on multimodal fusion according to any one of claims 1 to 3, characterized in that, The image data consists of at least one pair of image data, which is obtained by a binocular image acquisition device located at a designated position inside the fuel tank of the target aircraft, capturing the liquid level inside the fuel tank.

5. The multi-modal fused non-intrusive aircraft fuel quantity measurement method of claim 4, wherein, The step of performing three-dimensional reconstruction based on the liquid surface area in the image data and the motion data to obtain the three-dimensional coordinate values ​​of the liquid surface area in the three-dimensional model of the fuel tank specifically includes: A cost function is constructed based on at least one pair of image data; the cost function includes a matching cost term and a smoothing term; the matching cost term is used to characterize the difference between a pair of image data calculated at each pixel according to the target disparity; the smoothing term is used to suppress the disparity between adjacent pixels; With the goal of minimizing the function value of the cost function, the target disparity of each pixel is optimized to obtain a disparity map composed of the target disparities of each pixel. A 3D model of the fuel tank is obtained by performing 3D reconstruction based on the disparity map. Based on the three-dimensional model of the fuel tank and the motion data, the three-dimensional coordinates of the liquid surface area in the three-dimensional model of the fuel tank are calculated.

6. The multi-modal fused non-intrusive aircraft fuel quantity measurement method of claim 1, wherein, The fuel tank is a regular cubic fuel tank; the step of obtaining the average fuel level height based on the three-dimensional coordinate values ​​to determine the amount of fuel in the fuel tank specifically includes: The liquid surface area is divided into several regions; the average fuel level is calculated based on the three-dimensional coordinates of each region in the liquid surface area. Obtain the internal length and internal width of the fuel tank; calculate the fuel quantity based on the average fuel level, the internal length, and the internal width.

7. The multi-modal fused non-intrusive aircraft fuel quantity measurement method of claim 1, wherein, The fuel tank is an irregularly shaped fuel tank; the step of obtaining the average fuel level based on the three-dimensional coordinate values ​​to determine the amount of fuel in the fuel tank specifically includes: The oil tank is divided into multiple columnar sections along the horizontal direction, and the cross-sectional area of ​​each columnar section is obtained. The oil level height of each columnar section is determined based on the three-dimensional coordinate values ​​of each area block in the liquid surface region within each columnar section. The amount of fuel is calculated based on the cross-sectional area and oil level of each column section.

8. A multimodal fusion non-invasive aircraft fuel quantity measurement system, used to implement the multimodal fusion non-invasive aircraft fuel quantity measurement method according to any one of claims 1 to 7, characterized in that, The system includes: Data acquisition module: used to acquire image data and motion data; the image data is obtained by an image acquisition device capturing the liquid level in the fuel tank; the image acquisition device is located at a designated position inside the fuel tank of the target aircraft; the motion data is obtained by the inertial sensors of the target aircraft. Region filtering module: used to filter out liquid surface areas that are not obscured by obstacles from the image data; 3D Reconstruction Module: Used to perform 3D reconstruction based on the liquid surface area in the image data and the motion data, and obtain the 3D coordinate values ​​of the liquid surface area in the 3D model of the oil tank; Fuel quantity calculation module: used to obtain the average fuel level height based on the three-dimensional coordinate values, so as to determine the amount of fuel in the fuel tank.

9. An electronic device, characterized in that, include: A memory and a processor are communicatively connected, the memory stores computer instructions, and the processor executes the computer instructions to perform the non-intrusive aircraft fuel quantity measurement method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions for causing the computer to execute the multimodal fusion non-invasive aircraft fuel quantity measurement method according to any one of claims 1 to 7.

Images