Flight parameter multi-task identification method and system based on progressive extended one-time network

Abstract

The present application relates to the field of general aviation flight parameter identification, and particularly relates to a flight parameter multi-task identification method and system based on a progressive expansion one-off network. The scheme comprises collecting cockpit video containing an electronic integrated display through a cockpit camera device, obtaining a standardized electronic display image sequence after preprocessing, constructing a progressive expansion one-off super network with four dimensions of variable support network depth, width, convolution kernel size and input resolution, combining knowledge distillation to train a shared backbone network, setting a flight parameter display area detection head, a flight parameter value identification head and a timing consistency modeling head on the backbone network, automatically selecting a target sub-network meeting different airborne platform computing power constraints to deploy in an airborne inference module, using a multi-task loss function for joint optimization, and realizing unified identification of flight parameters such as airspeed, attitude, heading, altitude and vertical speed. The present application is suitable for general aviation flight parameter identification.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of civil aviation aircraft flight parameter identification, and in particular to a flight parameter multi-task identification method and system based on a progressive expansion one-off network. BACKGROUND

[0002] With the development of the general aviation industry, more and more light aircraft and trainer aircraft are equipped with electronic flight display systems represented by primary flight displays and multifunction displays for displaying key flight parameters such as airspeed, altitude, attitude, heading, and vertical speed. The existing flight parameter acquisition method relies on the internal bus signals of the avionics system and various sensor data, which requires deep modification or reservation of standard interfaces of the airborne avionics system. The modification cycle is long, the cost is high, and it is strongly dependent on the aircraft model, which is not conducive to rapid promotion in the existing general aviation fleet. In contrast, using a camera device aimed at the electronic display in the cockpit to non-invasively collect video images and identify flight parameters from the images is a technical path with lower modification cost and less invasion to the original avionics system, but it is still a challenge to ensure the accuracy and stability of identification in complex lighting, strong reflection and cockpit vibration environments.

[0003] In the field of image-based cockpit information identification, existing work attempts to detect symbols and icons in the aircraft cockpit display for software verification or human-machine interface evaluation. For example, a cockpit display graphical symbol detection method based on deep learning realizes automatic detection of multiple types of graphical symbols for software verification process. However, this method mainly identifies the presence or absence of graphical symbols or signs, does not construct a special reading model for airspeed, altitude and other continuous numerical flight parameters, and does not consider the cross-frame temporal consistency and multi-parameter joint identification problems.

[0004] In summary, the existing technology lacks an end-to-end multi-task identification method that can output multiple key flight parameters in complex lighting, strong reflection and cockpit vibration environments without modifying the avionics system, and is suitable for general aviation aircraft electronic displays. On the other hand, the existing technology cannot meet the airborne computing power constraints while considering identification accuracy, temporal consistency and system deployment flexibility. SUMMARY

[0005] The purpose of the present application is to overcome the shortcomings of the prior art and provide a flight parameter multi-task identification method and system based on a progressive expansion one-off network, which realizes the identification of multiple flight parameters of a general aviation aircraft in complex situations and improves the accuracy of general aviation aircraft flight parameter identification.

[0006] The present application achieves the above-mentioned purpose by adopting the following technical solutions. The present application provides a flight parameter multi-task identification method based on a progressive expansion one-off network, comprising:

[0007] S1, cockpit data acquisition and preprocessing, obtaining standardized flight parameter display sequence image for training and inference;

[0008] S2, construction and training of progressive expansion one-off network;

[0009] The super network supporting network depth, network width, convolution kernel size and input resolution four-dimensional flexibility is constructed as a shared backbone network, and a progressive expansion training strategy is adopted. Based on the standardized flight parameter display sequence image, training is started from the preset minimum configuration. On the basis of maintaining the learned parameters, expansion is carried out in the depth dimension, width dimension, convolution kernel size dimension and input resolution dimension in turn, and the network of the previous stage is used as the teacher network at each expansion stage. The output of the new stage network is constrained by knowledge distillation, so that the shared backbone network has flight parameter recognition ability under multiple structure configurations;

[0010] S3, construction and joint training of multi-task flight parameter recognition network;

[0011] On the output features of the shared backbone network, a multi-task network including a flight parameter display area detection head, a flight parameter value recognition head and a timing consistency modeling head is constructed;

[0012] The flight parameter display area detection head is used to position and detect the airspeed, altitude, attitude, heading and vertical velocity parameter display area in the electronic integrated display;

[0013] The flight parameter value recognition head is used to perform fine-grained recognition on the scale, digital character or pointer position in the flight parameter display area, and output the corresponding flight parameter value;

[0014] The timing consistency modeling head is used to model and constrain the timing features of each frame of flight parameter recognition result based on adjacent multiple frames of standardized flight parameter display sequence images, and suppresses the instantaneous abnormal jump caused by strong light reflection and cockpit shaking;

[0015] Based on the multi-task loss function, the shared backbone network and each detection head are jointly trained, so that the detection task, recognition task and timing consistency task are cooperatively optimized in the shared feature space;

[0016] S4, multi-objective structure search and target sub-network selection;

[0017] on the progressively expanded one-shot network that is completed training, taking the flight parameter identification accuracy or mean absolute error as the first optimization target, taking the inference delay, floating point operation amount and / or model parameter amount on the airborne computing platform as the second optimization target, using a multi-objective evolutionary search algorithm based on an accuracy prediction model and a hardware time delay lookup table, without retraining the network weight, screening a flight parameter identification target sub-network from the structure space of the super network that satisfies the preset resource constraint condition;

[0018] S5, airborne deployment and real-time flight parameter identification;

[0019] The obtained flight parameter identification target sub-network is deployed in the airborne computing platform of the general aviation aircraft, the preprocessing process of step S1 is performed on the real-time collected cockpit data in the flight process, and the obtained standardized flight parameter display sequence image is input into the target sub-network, sequentially passing through the flight parameter display area detection head, the flight parameter value identification head and the time sequence consistency modeling head, and outputting the real-time identification results of the airspeed, attitude, heading, altitude and vertical speed flight parameters.

[0020] Further, step S1 specifically comprises:

[0021] capturing cockpit videos in the flight process through a camera device installed in the cockpit of the general aviation aircraft and aligned with the electronic integrated display, to obtain an original video stream containing an electronic integrated display picture;

[0022] Performing distortion correction, de-jittering, brightness and contrast normalization, cropping the electronic integrated display area, and enhancement processing based on random overexposure, local highlight, reflective patch, local information loss and motion blur convolution on the original video stream to obtain standardized flight parameter display sequence images for training and inference.

[0023] Further, the reflective patch enhancement processing specifically comprises: pseudo-reflective patch enhancement based on random area brightness enhancement and Gaussian blur superposition, for simulating cockpit glass reflection scenarios;

[0024] The motion blur enhancement processing specifically comprises: directional blur filtering based on a convolution kernel, for simulating motion blur caused by aircraft vibration and rapid attitude change;

[0025] The random overexposure enhancement processing specifically comprises: strong light overexposure and backlight scene enhancement based on gamma correction and local contrast stretching;

[0026] The local information loss enhancement processing specifically comprises: local information loss enhancement based on random occlusion, for improving the robustness of the flight parameter identification network to partial occlusion.

[0027] Further, in step S2, the progressive expansion training strategy specifically comprises:

[0028] In the initial stage, only the network structure corresponding to the minimum network depth, the minimum network width, the minimum convolution kernel size and the minimum input resolution is activated, and the network with the minimum configuration is pre-trained;

[0029] In the depth expansion stage, network layers with a depth greater than the set depth threshold are introduced, and on the basis of keeping the original shallow layer parameters learnable, only the newly added network parameters are updated, and the shallow network trained in the previous stage is used as the teacher network to perform knowledge distillation constraint on the output of the newly added network parameters;

[0030] In the width expansion stage, the number of channels is expanded or expanded by a certain ratio, and part or all of the channels are enabled in the form of channel subsets, and the weight parameters of the newly added channels are updated preferentially during training;

[0031] In the convolution kernel size expansion stage, the small convolution kernel is expanded to a large convolution kernel by adding trainable weights around the small convolution kernel, and the kernel weight is frozen during the expansion process, and only the outer weight parameter is updated;

[0032] In the input resolution expansion stage, multiple input resolution candidate sets are set, and different size input images are randomly used for training during training, so that the progressively expanded one-time network can adapt to multiple resolution configurations.

[0033] Further, in step S3, the multi-task loss function includes a detection loss, a numerical recognition loss, a time smoothing loss and a physical constraint loss, and the total loss satisfies:

[0034] ;

[0035] In the formula, Ltotal represents the total loss, Ldet represents the flight parameter display area detection loss, a weighted sum of the bounding box regression loss and the category cross-entropy loss, Lnum represents the flight parameter numerical recognition loss, a regression loss or a classification loss of each flight parameter value, Lsmooth represents the time smoothing loss, used to constrain the change amplitude of the adjacent frame flight parameter recognition result;

[0036] satisfies:

[0037] ;

[0038] In the formula, N represents the number of frames in the time window, Lsmooth represents the time smoothing loss, a flight parameter identification vector of the frame;

[0039] a physical constraint loss, used to constrain the flight parameter change to comply with the aircraft dynamics characteristics, and a penalty term is applied when the flight parameter change of adjacent frames exceeds a preset threshold, satisfying:

[0040] ;

[0041] a threshold set according to the target aircraft flight parameter change limit, a weight coefficient of each loss.

[0042] Further, in step S3, the multi-task flight parameter identification network task embedding and feature modulation module is used to enhance the task relevance of the intermediate features output by the shared backbone network, specifically including:

[0043] a task embedding vector is set for each of the flight parameter display region detection task, the flight parameter value identification task, and the time sequence consistency modeling task;

[0044] On each feature layer of the shared backbone network, a feature modulation subnetwork driven by the task embedding vector is introduced, which maps the task embedding vector to a channel-level or spatial-level weight mask of the corresponding feature layer;

[0045] By applying the weight mask to the corresponding feature layer, the different task features are weighted and amplified or suppressed, so as to enhance the sharing of positive correlation information between tasks and weaken the conflict interference between tasks.

[0046] Further, the task embedding and feature modulation module further includes a task relationship modeling sub-module, which is used to generate a task relationship matrix according to the similarity and difference between each task embedding vector, and use the task relationship matrix as input to modulate the feature weight mask corresponding to each task. The feature weights of the detection task and the value identification task have high consistency, while the feature weights of the time sequence consistency modeling task have strong smoothness in the time dimension.

[0047] Further, the multi-objective evolutionary search algorithm is a multi-objective evolutionary search algorithm with non-dominated sorting and crowded distance calculation mechanism, which performs mutation and crossover operations based on subnetwork structure encoding in the iteration process, and retains subnetwork structures with superior accuracy and delay performance in each generation.

[0048] Further, step S5 further includes a target subnetwork dynamic switching step based on the flight phase and the on-board computing power state, and the specific process is as follows:

[0049] obtain the current aircraft flight phase information and the current load state of the on-board computing platform;

[0050] The plurality of target sub-networks obtained in step S4 are divided into a first sub-network, a second sub-network and a third sub-network;

[0051] In the take-off phase with high requirements for real-time performance and stability, the third sub-network or the second sub-network with low delay is preferentially selected to perform flight parameter identification;

[0052] In the cruise phase with low requirements for real-time performance and stability, the first sub-network with high identification accuracy is preferentially selected to perform flight parameter identification;

[0053] According to the real-time load state of the airborne computing platform, the sub-network with low complexity is automatically switched in the high-load state, and the sub-network with high complexity is automatically switched in the low-load state, so as to realize dynamic balance between identification accuracy and computing power consumption on the premise of ensuring flight safety.

[0054] The application further provides a flight parameter multi-task identification system based on the progressive extended one-off network, which is used for realizing the flight parameter multi-task identification method based on the progressive extended one-off network.

[0055] An image acquisition module is configured to acquire a cockpit video in a flight process by a camera device installed in a cockpit of a navigable aircraft and aligned with an electronic integrated display, and obtain an original video stream containing a picture of the electronic integrated display;

[0056] A preprocessing module is configured to perform distortion correction, de-jittering, brightness and contrast normalization, cropping of the electronic integrated display region, and enhancement processing based on random overexposure, local highlight, reflective patch, local information loss and motion blur convolution on the original video stream, and obtain a standardized flight parameter display sequence image for training and reasoning.

[0057] A progressive extended one-off network training module is configured to construct a super network supporting network depth, network width, convolution kernel size and input resolution as a shared backbone network, adopt a progressive extension training strategy, and train the standardized flight parameter display sequence image for training and reasoning, starting from a preset minimum configuration, extending in the depth dimension, the width dimension, the convolution kernel size dimension and the input resolution dimension in turn on the basis of maintaining learned parameters, and using the network of the last stage as a teacher network at each extension stage to constrain the output of the new stage network through knowledge distillation, so that the shared backbone network has flight parameter identification capability under multiple structure configurations.

[0058] A multi-task flight parameter identification network module is configured to construct a multi-task network including a flight parameter display region detection head, a flight parameter value identification head and a time sequence consistency modeling head on the output features of the shared backbone network.

[0059] The flight parameter display area detection head is used for positioning detection of airspeed, altitude, attitude, heading and vertical speed parameter display areas in an electronic integrated display;

[0060] The flight parameter value recognition head is used for fine-grained recognition of scales, digital characters or pointer positions in the flight parameter display area, and outputs corresponding flight parameter values;

[0061] The time sequence consistency modeling head is used for time sequence feature modeling and consistency constraint of each frame of flight parameter recognition result based on adjacent multiple frames of standardized flight parameter display sequence images, so as to suppress instantaneous abnormal jumps caused by strong light reflection and cabin shaking;

[0062] The shared backbone network and each detection head are jointly trained based on a multi-task loss function, so that the detection task, the recognition task and the time sequence consistency task are cooperatively optimized in the shared feature space;

[0063] The multi-objective structure search and sub-network management module is used for screening a flight parameter recognition target sub-network that meets a preset resource constraint condition from a structure space of the super network by using a multi-objective evolutionary search algorithm based on an accuracy prediction model and a hardware time delay lookup table, with the flight parameter recognition accuracy or the mean absolute error as a first optimization target and the inference delay, the floating point operation amount and / or the model parameter amount on the airborne computing platform as a second optimization target, without retraining network weights.

[0064] The airborne inference module is used for deploying the obtained flight parameter recognition target sub-network in an airborne computing platform of a general aviation aircraft, performing a preprocessing process of the data preprocessing module on real-time collected cabin data in a flight process, inputting the obtained standardized flight parameter display sequence image into the target sub-network, sequentially passing through the flight parameter display area detection head, the flight parameter value recognition head and the time sequence consistency modeling head, and outputting real-time recognition results of airspeed, attitude, heading, altitude and vertical speed flight parameters.

[0065] The beneficial effects of the present application are:

[0066] The present application can stably and real-timely obtain multiple key flight parameters such as airspeed, attitude, heading, altitude and vertical speed from a general aviation electronic display image without changing the existing avionics system, and can simultaneously consider recognition accuracy, time sequence consistency and deployment efficiency under the conditions of complex illumination, strong reflection, cabin vibration and limited airborne computing power. BRIEF DESCRIPTION OF DRAWINGS

[0067] Figure 1 It is a flight parameter multi-task recognition method flowchart based on a progressively expanded one-time network provided by the present application.

[0068] Figure 2 is a structural block diagram of a flight parameter multi-task identification system based on a progressive expansion one-off network provided by the application;

[0069] Figure 3 is a structural schematic diagram of a progressive expansion one-off super network provided by the application;

[0070] Figure 4 is a multi-task flight parameter identification network structural schematic diagram provided by the application;

[0071] Figure 5 is a multi-target structure search and target sub-network selection process schematic diagram provided by the application. DETAILED DESCRIPTION

[0072] In order to make the purpose, technical scheme and advantages of the embodiments of the application clearer, the technical scheme in the embodiments of the application will be described clearly and completely below with reference to the drawings in the embodiments of the application.

[0073] The application provides a flight parameter multi-task identification method based on a progressive expansion one-off network, as shown in Figure 1 , specifically comprising:

[0074] S1, cabin data acquisition and preprocessing;

[0075] A camera device installed in the cabin of a navigable aircraft and aligned with an electronic integrated display is used to collect cabin videos during flight to obtain a raw video stream. Each frame of image is subjected to distortion correction, de-jittering, brightness and contrast normalization, electronic display area cropping, and simulation of strong reflections and vibration blur in the cabin through random local highlights, motion blur, gamma transformation, etc. to obtain a standardized electronic display image sequence for subsequent network training and reasoning.

[0076] Preferably, the resolution of the camera device is not less than 1920x1080, the frame rate is not less than 25fps, and a fixed bracket is used to reduce the impact of mechanical vibration on the picture.

[0077] S2, progressive expansion one-off network construction and training;

[0078] Based on the standardized electronic display image sequence obtained in step S1 and its corresponding flight parameter value label, display area label and other supervision information, a progressive expansion one-off super network with variable support network depth, network width, convolution kernel size and input resolution is constructed as a shared backbone network, as shown in Figure 3 .

[0079] This invention employs a progressive expansion training strategy starting from the minimum configuration to perform staged training on the supernetwork, enabling subnetworks under various structural configurations to share the same set of weights and all possess good flight parameter feature extraction capabilities.

[0080] The structural configuration of a one-time hypernetwork is determined by the following four parameters:

[0081] Depth Collection ;

[0082] Width (channel spread factor) set ;

[0083] Convolution kernel size set ;

[0084] Input resolution set .

[0085] Any subnetwork structure can be derived from vectors , , , , For standardized electronic display images In structural configuration and shared weight The features extracted through the shared backbone network can be represented as follows:

[0086] ;

[0087] in, For the first Feature map corresponding to the frame image.

[0088] Progressive scaling training from minimal architecture configuration Initially, only the minimum structure configuration is enabled. Forward inference and backpropagation are performed on all training samples to minimize the joint loss function consisting of task loss and distillation loss, and obtain the initial shared weights. Then, the system sequentially enters the stages of depth expansion, width expansion, convolutional kernel size expansion, and input resolution expansion. In each stage, the system iteratively updates according to the preset number of training rounds or validation set performance threshold, so that the new structure can be fully trained without destroying the performance of the existing sub-network.

[0089] Furthermore, progressive scaling training includes the following stages:

[0090] Depth Expansion Phase: Only minimum depth is enabled. After training the initial network with the given configuration, a deeper network is introduced. For newly added deep layers, parameters are updated only when they are activated; for existing shallow layers, parameters are fine-tuned. The previous stage shallow network is used as the teacher network, and a knowledge distillation loss is applied to the output of the current deep network.

[0091] ;

[0092] in, The knowledge distillation loss represents the knowledge distillation stage (subscript KD indicates knowledge distillation, and superscript D indicates the deep expansion stage). This indicates the number of samples in the current training batch;

[0093] These represent the student network and the teacher network respectively in the [number]th [year]. Features of each sample; This is the projection function used to align the feature dimensions.

[0094] Width Expansion Phase: While keeping the number of network layers unchanged, a wider configuration is introduced by expanding the number of channels or the expansion factor. During training, different width configurations are randomly sampled to give more update opportunities to the weights of newly added channels, and the consistency of features across different width configurations is constrained by the following distillation loss:

[0095] ;

[0096] in, The knowledge distillation loss during the width expansion phase is represented by the superscript. (Indicates width or channel dimension). This indicates the number of time steps or batch samples used in training sampling. For width is Subnetwork characteristics, This is a mapping function used to align channel dimensions.

[0097] Kernel size expansion stage: A "kernel sharing + outer ring expansion" approach is adopted, using a smaller convolutional kernel... To larger convolution kernels During the expansion process, the kernel weights remain unchanged, and only the newly added weights on the outer ring are updated. Feature consistency under different convolutional kernel sizes is constrained by the following formula:

[0098] ;

[0099] In the formula, The knowledge distillation loss during the kernel size expansion stage (superscript) represents the loss due to knowledge distillation. (representing the kernel size dimension).

[0100] respectively denote the network mapping function under the current extended convolution kernel and the minimum convolution kernel configuration; denotes the number of samples in the current training batch.

[0101] Input resolution expansion stage: randomly sampling different input resolutions during training to improve the network's adaptability to different input sizes through multi-resolution training while maintaining weight sharing. In each stage of progressive expansion, the optimization objective is the weighted sum of the task loss and each distillation loss:

[0102] ;

[0103] wherein, is the multi-task loss, is the weight coefficient of each distillation loss. By repeatedly performing forward inference and backpropagation on all training samples in each expansion stage, is minimized until the validation set performance converges, and the trained shared backbone network weight is obtained for use by the multi-task flight parameter recognition network in step S3.

[0104] S3, multi-task flight parameter recognition network construction and forward inference;

[0105] As shown in Figure 4 , based on the shared backbone network obtained in step S2, the multi-task flight parameter recognition network is constructed, including a flight parameter display area detection head, a flight parameter value recognition head, and a temporal consistency modeling head, and optionally inserting a task embedding and feature modulation module in the intermediate feature layer.

[0106] Specifically, for each standardized electronic display image , feature extraction is performed to obtain a feature map , and on this basis, a flight parameter display area detection head, a flight parameter value recognition head, and a temporal consistency modeling head are constructed:

[0107] Flight parameter display area detection head: output a set of candidate bounding boxes and their class probabilities, denoted as:

[0108] ;

[0109] wherein, denotes the flight parameter display area detection result set of the th frame image, denotes the number of candidate targets detected in the frame; is the a position parameter of a bounding box, representing the horizontal coordinate of the center of the bounding box, the vertical coordinate of the center of the bounding box, the width of the bounding box, and the height of the bounding box, respectively; is the probability vector that the frame belongs to a certain flight parameter display area (airspeed, altitude, etc.).

[0110] Flight parameter value recognition head: extract regional features on the detected flight parameter display area and output flight parameter value vector , wherein is the number of flight parameter categories, for example , corresponding to airspeed, attitude, heading, altitude, and vertical speed.

[0111] Temporal consistency modeling head: take the feature sequence or the preliminary recognition result sequence as input, introduce a self-attention or Transformer-based temporal network, calculate the representation fused with temporal information , and output the final flight parameter.

[0112] To make full use of the relationship between tasks, a task embedding and feature modulation module is introduced on part of the feature layers of the shared backbone network: for each task , a task embedding vector is defined, and a channel-level weight mask is generated through a small network:

[0113] ;

[0114] wherein, is a Sigmoid activation function, is a parameter on the th feature layer.

[0115] The shared feature layer is weighted by task to obtain:

[0116] ;

[0117] Thus, different tasks can make different use of shared features.

[0118] In the formula, represents the feature map of the th frame image on the th layer feature after modulation for task ; represents the feature weight mask of task on the th layer; represents the element-wise multiplication operation (Hadamard Product); ​The original intermediate feature representing the shared backbone network output. Thus, different tasks can utilize the shared feature differently.

[0119] In the inference stage, the feature extraction, display area detection, numerical value recognition and time series modeling are sequentially performed according to the flow shown in Figure 1 , to obtain the flight parameter output corresponding to the current time.

[0120] S4, multi-task loss function design and joint optimization;

[0121] In the model training stage, the network structure provided in steps S2 and S3 is combined to perform multi-task loss function design and joint optimization. The multi-task loss function adopted by the present application is composed of detection loss, numerical value recognition loss, time series smoothing loss and physical constraint loss , which is in the form of:

[0122] ;

[0123] In the formula, Ltotal represents the total loss, Ldisplay represents the flight parameter display area detection loss, which is a weighted sum of the bounding box regression loss and the class cross-entropy loss, Lnum represents the flight parameter numerical value recognition loss, which is a regression loss or a classification loss of each flight parameter value, Lsmooth represents the time series smoothing loss, which is used to constrain the change amplitude of the adjacent frame flight parameter recognition results.

[0124] Flight parameter display area detection loss : After matching the detection results of each frame with the real labels, the detection loss is defined as the weighted sum of the classification loss and the position regression loss:

[0125] ;

[0126] wherein, Ppos represents the positive sample set, Npos represents the number of positive samples, Ptrue represents the real class distribution, Btrue represents the real bounding box, which can be a cross-entropy loss, which can be a smooth loss or an IoU loss, and a balance coefficient.

[0127] Flight parameter numerical value recognition loss : used to measure the difference between the recognized flight parameter value vector and the real flight parameter vector The difference between them is in the form of absolute error:

[0128] ;

[0129] Temporal smoothing loss : To constrain the smoothness of the change of flight parameter recognition results between adjacent frames, reduce unnecessary jitter caused by noise and transient interference, defined as:

[0130] ;

[0131] Wherein, is the flight parameter vector output by the temporal consistency modeling head.

[0132] Physical constraint loss : According to the dynamic characteristics of the target aircraft, set a maximum reasonable change threshold for each flight parameter , if the change of adjacent frame recognition results exceeds the threshold, a penalty is imposed, defined as:

[0133] ;

[0134] By minimizing the total loss , the shared backbone network of step S2 and each task head of step S3 are trained end-to-end, so that spatial detection, numerical recognition and temporal constraint work together under a unified goal.

[0135] In the specific training process, the standardized electronic display image sequence obtained in step S1 and its supporting label constitute a training sample set, and optimization algorithms such as small batch random gradient descent are used. The joint loss function composed of the above multi-task loss and the distillation loss in step S2 is used to jointly update the shared backbone network and the multi-task head. Through multiple rounds of training, a multi-task recognition model with high accuracy and robustness under multiple flight parameters and multiple interference scenarios is obtained.

[0136] S5, multi-objective structure search and target subnetwork selection;

[0137] As Figure 5 shown, after the above network training is completed, the embodiment performs multi-objective structure search in the structure space of the progressive expansion one-off network to balance the flight parameter recognition performance and the algorithmic constraint of the airborne platform.

[0138] First, sample several subnetworks with different structure configurations in the training phase, evaluate their recognition performance on the validation set, and measure their inference time on the target airborne platform or estimate their complexity according to FLOPs, parameter quantity, etc. The precision prediction model between the structure of the trained subnetwork and the recognition error, and the time delay lookup table between the structure and the inference delay are obtained.

[0139] Then, the flight parameter identification error and the onboard inference latency are taken as two optimization objectives, and the network structure configuration vector is taken as the search variable. A multi-objective evolutionary algorithm with non-dominated sorting and crowded distance calculation mechanism is used to iteratively search and evolve the structure space. Through multiple generations of evolution, crossover and mutation operations, a Pareto solution set composed of a group of candidate sub-networks with good trade-off performance in both identification error and inference latency dimensions is obtained.

[0140] Finally, according to the computing power constraints of different aircraft models and onboard computing platforms, a target sub-network that meets the preset latency or complexity upper limit is selected from the Pareto solution set, and is exported as a network structure and weight file for actual deployment, which is used for the onboard inference module to load and run.

[0141] Specifically, for any structure configuration Definition:

[0142] : flight parameter identification error (such as mean absolute error MAE or its weighted form) on the validation set;

[0143] : inference latency on the target onboard platform, or a complexity indicator converted from FLOPs and parameter quantity.

[0144] The multi-objective optimization problem can be expressed as:

[0145] ;

[0146] In the actual search process, to reduce the evaluation cost, a partial sampling sub-network training accuracy prediction model is used in advance , and a hardware latency lookup table is constructed on the target platform . In the search stage, the is used instead of the real indicator for rapid evaluation. Through a multi-objective evolutionary algorithm with non-dominated sorting and crowded distance calculation mechanism, a group of structures that cannot be dominated by other structures in the space is searched and obtained , that is, a Pareto solution set, from which one or more target sub-networks can be automatically selected according to the computing power constraints of different onboard platforms and task scenarios for deployment.

[0147] The application also provides an aircraft electronic display flight parameter multi-task identification system corresponding to the method, which at least comprises: an image acquisition module, a preprocessing module, a progressive expansion one-off network training module, a multi-task flight parameter identification network module, a multi-target structure search and sub-network management module, and an airborne reasoning module. The image acquisition module is used to acquire cockpit video containing an electronic comprehensive display screen; the preprocessing module is used to complete distortion correction, de-jittering, cropping, normalization and enhancement processing; the progressive expansion one-off network training module is used to build and train a shared backbone network supporting multiple structure configurations; the multi-task flight parameter identification network module is used to build a flight parameter display area detection head, a flight parameter value identification head and a timing consistency modeling head based on the shared backbone network, and perform joint training based on multi-task loss; the multi-target structure search and sub-network management module is used to perform multi-target structure search in a structure space and manage target sub-networks; and the airborne reasoning module is used to deploy the target sub-networks on an airborne computing platform, perform reasoning on the real-time acquired and preprocessed electronic display image sequence, and output identification results of flight parameters such as airspeed, attitude, heading, altitude and vertical speed.

[0148] As shown in Figure 2 The application also provides a flight parameter multi-task identification system based on a progressive expansion one-off network, which is used to implement the flight parameter multi-task identification method based on the progressive expansion one-off network.

[0149] The image acquisition module is used to acquire cockpit video in a flight process through a camera device installed in a cockpit of a navigable aircraft and aligned with an electronic comprehensive display, to obtain an original video stream containing an electronic comprehensive display screen;

[0150] The preprocessing module is used to perform distortion correction, de-jittering, brightness and contrast normalization, cropping of the electronic comprehensive display area, and enhancement processing based on random overexposure, local highlight, reflective patch, local information loss and motion blur convolution on the original video stream, to obtain standardized flight parameter display sequence images for training and reasoning;

[0151] The progressive expansion one-off network training module is used to build a hypernetwork supporting network depth, network width, convolution kernel size and input resolution four-dimensional flexibility as a shared backbone network, adopt a progressive expansion training strategy, and train and reason the standardized flight parameter display sequence images, starting from a preset minimum configuration, expanding in the depth dimension, width dimension, convolution kernel size dimension and input resolution dimension in turn on the basis of maintaining learned parameters, and using the network of the last expansion stage as a teacher network to constrain the output of the new stage network through knowledge distillation, so that the shared backbone network has flight parameter identification capability under multiple structure configurations;

[0152] A multi-task flight parameter recognition network module is configured to build a multi-task network including a flight parameter display area detection head, a flight parameter value recognition head, and a time sequence consistency modeling head on the output features of the shared backbone network;

[0153] The flight parameter display area detection head is configured to detect the display areas of airspeed, altitude, attitude, heading, and vertical speed parameters in the electronic integrated display.

[0154] The flight parameter value recognition head is configured to perform fine-grained recognition on scales, digital characters, or pointer positions in the flight parameter display area, and output corresponding flight parameter values.

[0155] The time sequence consistency modeling head is configured to model and constrain the time sequence features of each frame of flight parameter recognition result based on adjacent multiple frames of standardized flight parameter display sequence images, and suppress transient abnormal jumps caused by strong light reflection and cabin shaking.

[0156] The shared backbone network and each detection head are jointly trained based on a multi-task loss function, so that the detection task, the recognition task, and the time sequence consistency task are cooperatively optimized in the shared feature space.

[0157] A multi-objective structure search and sub-network management module is configured to use a multi-objective evolutionary search algorithm based on an accuracy prediction model and a hardware time delay lookup table to screen one or more flight parameter recognition target sub-networks that satisfy a preset resource constraint condition from a structure space of the super network without retraining network weights, with the flight parameter recognition accuracy or the mean absolute error as the first optimization target and the inference delay, the floating point operation amount, and / or the model parameter amount on the airborne computing platform as the second optimization target, and to store and manage the target sub-network.

[0158] An airborne inference module is configured to deploy the obtained flight parameter recognition target sub-network in the airborne computing platform of the general aviation aircraft, perform a preprocessing process of a data preprocessing module on the real-time collected cabin data in the flight process, input the obtained standardized flight parameter display sequence image into the target sub-network, sequentially pass through the flight parameter display area detection head, the flight parameter value recognition head, and the time sequence consistency modeling head, and output real-time recognition results of airspeed, attitude, heading, altitude, and vertical speed flight parameters.

[0159] Preferably, the multi-task flight parameter recognition network module further includes a task embedding and feature modulation sub-module configured to generate task embedding vectors for different tasks and generate corresponding feature weight masks on one or more feature layers of the shared backbone network to perform weighted modulation on the feature channels of different tasks.

[0160] The foregoing is considered as illustrative only of the principles of the application. Further, since numerous modifications and changes will readily occur to those skilled in the art, it is not desired to limit the application to the exact construction and practice described. Accordingly, all such variations are intended to be included within the scope of the present application as defined in the following claims, along with full authority to use equivalent methods and materials such variations are covered by such claims.

Claims

1. A method for flight parameter multi-task identification based on progressive extended one-off network, characterized in that, The method comprises the following steps: S1, cockpit data acquisition and preprocessing, obtaining standardized flight parameter display sequence images for training and inference; S2, construction and training of a progressive expansion one-off network; An ultra-network supporting four-dimensional flexibility of network depth, network width, convolution kernel size and input resolution is constructed as a shared backbone network, a progressive expansion training strategy is adopted, and based on the standardized flight parameter display sequence images, training is started from a preset minimum configuration, and on the basis of maintaining the learned parameters, expansion is sequentially performed in the depth dimension, the width dimension, the convolution kernel size dimension and the input resolution dimension, and in each expansion stage, the network of the previous stage is used as a teacher network, the output of the new stage network is constrained through knowledge distillation, so that the shared backbone network has flight parameter recognition capability under multiple structural configurations; S3, construction and joint training of a multi-task flight parameter recognition network; On the output features of the shared backbone network, a multi-task network including a flight parameter display area detection head, a flight parameter value recognition head and a temporal consistency modeling head is constructed; The flight parameter display area detection head is used for positioning detection of the airspeed, altitude, attitude, heading and vertical velocity parameter display areas in the electronic integrated display; The flight parameter value recognition head is used for fine-grained recognition of the scale, digital character or pointer position in the flight parameter display area, and outputs the corresponding flight parameter value; The temporal consistency modeling head is used for temporal feature modeling and consistency constraint of each frame of flight parameter recognition result based on adjacent multiple frames of standardized flight parameter display sequence images, and inhibits the instantaneous abnormal jump caused by strong light reflection and cockpit shaking; The shared backbone network and each detection head are jointly trained based on a multi-task loss function, so that the detection task, the recognition task and the temporal consistency task are cooperatively optimized in the shared feature space; S4, multi-objective structure search and target sub-network selection; On the trained progressive expansion one-off network, the flight parameter recognition accuracy or the mean absolute error is taken as the first optimization target, the inference delay, the floating point operation amount and / or the model parameter amount on the airborne computing platform are taken as the second optimization target, and a multi-objective evolutionary search algorithm based on an accuracy prediction model and a hardware time delay lookup table is used to select a flight parameter recognition target sub-network satisfying a preset resource constraint condition from the structure space of the ultra-network without retraining the network weight; S5, airborne deployment and real-time flight parameter recognition; The obtained flight parameter recognition target sub-network is deployed in the airborne computing platform of a general aviation aircraft, the preprocessing process of step S1 is performed on the real-time collected cockpit data in the flight process, the standardized flight parameter display sequence images obtained are input into the target sub-network, the flight parameter display area detection head, the flight parameter value recognition head and the temporal consistency modeling head are sequentially passed through, and the real-time recognition results of the airspeed, attitude, heading, altitude and vertical velocity flight parameters are output.

2. The flight parameter multi-mission identification method based on the progressive expansion one-off network according to claim 1, characterized in that, Step S1 specifically comprises: A cockpit video during flight is collected by a camera installed in a cockpit of a navigable aircraft and aligned with an electronic integrated display, to obtain an original video stream containing a picture of the electronic integrated display; The original video stream is subjected to distortion correction, de-jittering, brightness and contrast normalization, cropping of the electronic integrated display region, and enhancement processing based on random overexposure, local highlight, reflective patch, local information loss, and motion blur convolution, to obtain a standardized flight parameter display sequence image for training and reasoning.

3. The flight parameter multi-mission identification method based on the progressive expansion one-off network according to claim 2, characterized in that, The reflective patch enhancement processing specifically includes pseudo-reflective patch enhancement based on superposition of brightness enhancement and Gaussian blur in a random region, for simulating a cockpit glass reflection scenario; The motion blur enhancement processing specifically includes directional blur filtering based on a convolution kernel, for simulating motion blur caused by aircraft vibration and rapid attitude change; The random overexposure enhancement processing specifically includes light overexposure and backlight scene enhancement based on gamma correction and local contrast stretching; The local information loss enhancement processing specifically includes local information loss enhancement based on random occlusion, for improving the robustness of the flight parameter recognition network to partial occlusion.

4. The method of claim 1, wherein the method is a progressive expansion one-time network based flight parameter multi-task identification method. In step S2, the progressive expansion training strategy specifically includes: In the initial stage, only the network structure corresponding to the minimum network depth, minimum network width, minimum convolution kernel size, and minimum input resolution is activated, and the network with the minimum configuration is pre-trained; In the depth expansion stage, network layers with a depth greater than a set depth threshold are introduced, the original shallow layer parameters are kept learnable, only the newly added network parameters are updated, and the shallow network trained in the previous stage is used as a teacher network to perform knowledge distillation constraint on the output of the newly added network parameters; In the width expansion stage, the number of channels is expanded or the expansion ratio is expanded, part or all of the channels are enabled in a channel subset manner, and the weight parameters of the newly added channels are updated preferentially during training; In the convolution kernel size expansion stage, the convolution kernel size is progressively expanded from small to large by adding trainable weights to the periphery of the small convolution kernel to form a large convolution kernel, and the kernel weight is frozen during the expansion process, only the outer weight parameter is updated; In the input resolution expansion stage, multiple input resolution candidate sets are set, and different sizes of input images are randomly used for training during training, so that the progressively expanded one-time network adapts to multiple resolution configurations.

5. The method of claim 1, wherein the method is a progressive expansion one-time network based flight parameter multi-task identification method. In step S3, the multi-task loss function includes a detection loss, a numerical recognition loss, a time sequence smoothing loss, and a physical constraint loss, and the total loss satisfies: ； In the formula, represents the total loss, represents the flight parameter display area detection loss, a weighted sum of the bounding box regression loss and the category cross-entropy loss, a flight parameter value identification loss, a regression loss or a classification loss of each flight parameter value, a time sequence smoothing loss, used for constraining the change amplitude of the adjacent frame flight parameter identification results; satisfies: ； In the formula, is the number of frames in the timing window, is the flight parameter identification vector of the frame is the flight parameter identification vector of the frame For physical constraint loss, used to constrain the flight parameter change to conform to the aircraft dynamics characteristics, a penalty term is applied when the flight parameter change of adjacent frames exceeds the preset threshold, satisfying: ； thresholds set in accordance with target aircraft flight parameter variation limits, are weight coefficients for each loss.

6. The method of claim 1, wherein the method is a progressive expansion one-time network based flight parameter multi-task identification method. In step S3, the multi-task flight parameter recognition network task embedding and feature modulation module is used to enhance the task relevance of the intermediate features output by the shared backbone network, specifically including: Task embedding vectors are set for the flight parameter display region detection task, the flight parameter numerical recognition task, and the time sequence consistency modeling task, respectively; On each feature layer of the shared backbone network, a feature modulation subnetwork driven by the task embedding vector is introduced to map the task embedding vector to a channel-level or spatial-level weight mask of the corresponding feature layer. By applying the weight mask to the corresponding feature layer, the different task features are weighted, amplified or suppressed to enhance the sharing of positive correlation information between tasks and weaken the interference between tasks.

7. The flight parameter multi-mission identification method based on the progressive expansion one-off network according to claim 6, characterized in that, The task embedding and feature modulation module further comprises a task relationship modeling submodule, which is configured to generate a task relationship matrix according to the similarities and differences between the task embedding vectors, and modulate the feature weight mask corresponding to each task by taking the task relationship matrix as input, so that the feature weights of the detection task and the numerical recognition task have high consistency, and the feature weights of the time sequence consistency modeling task have strong smoothness in the time dimension.

8. The method of claim 1, wherein the method is a progressive expansion one-time network based flight parameter multi-task identification method. The multi-objective evolutionary search algorithm is a multi-objective evolutionary search algorithm with non-dominated sorting and crowded distance calculation mechanism, which performs mutation and crossover operations based on sub-network structure coding in the iteration process, and retains sub-network structures with superior accuracy and delay performance in each generation.

9. The method of claim 1, wherein the method is a progressive expansion one-time network based flight parameter multi-task identification method. Step S5 further comprises a target sub-network dynamic switching step based on the flight phase and the on-board computing power state, and the specific process is as follows: Obtain the current flight phase information of the aircraft and the current load state of the on-board computing platform; Divide the multiple target sub-networks obtained in step S4 into a first sub-network, a second sub-network and a third sub-network; In the take-off phase with high requirements for real-time performance and stability, the third sub-network or the second sub-network with low delay is preferentially selected to perform flight parameter identification; In the cruise phase with low requirements for real-time performance and stability, the first sub-network with high identification accuracy is preferentially selected to perform flight parameter identification; According to the real-time load state of the on-board computing platform, the sub-network with low complexity is automatically switched when the load is high, and the sub-network with high complexity is automatically switched when the load is low, so as to realize the dynamic balance between identification accuracy and computing power consumption under the premise of ensuring flight safety.

10. A flight parameter multitask recognition system based on progressive extended one-shot network, for implementing the flight parameter multitask recognition method based on progressive extended one-shot network as claimed in any one of claims 1-9, characterized in that, The system comprises: An image acquisition module is configured to acquire cockpit videos in a flight process by a camera device installed in a cockpit of a navigable aircraft and aligned with an electronic integrated display, to obtain an original video stream containing a picture of the electronic integrated display; A preprocessing module is configured to perform distortion correction, de-jittering, brightness and contrast normalization, cropping of the electronic integrated display region, and enhancement processing based on random overexposure, local highlight, reflective patch, local information loss, and motion blur convolution on the original video stream, to obtain standardized flight parameter display sequence images for training and inference; A progressive expansion one-off network training module is configured to construct a super network supporting four-dimensional flexibility of network depth, network width, convolution kernel size and input resolution as a shared backbone network, and adopt a progressive expansion training strategy to train the standardized flight parameter display sequence images, starting from a preset minimum configuration, expanding in the depth dimension, width dimension, convolution kernel size dimension and input resolution dimension in turn based on the learned parameters, and using the network of the previous stage as a teacher network at each expansion stage to constrain the output of the new stage network through knowledge distillation, so that the shared backbone network has flight parameter identification capability under multiple structure configurations. The system comprises: The multi-task flight parameter recognition network module is configured to build a multi-task network including a flight parameter display area detection head, a flight parameter value recognition head, and a time sequence consistency modeling head on the output features of the shared backbone network. The flight parameter display area detection head is configured to detect the display areas of airspeed, altitude, attitude, heading, and vertical speed parameters in the electronic integrated display. The flight parameter value recognition head is configured to perform fine-grained recognition on scales, digital characters, or pointer positions in the flight parameter display areas and output corresponding flight parameter values. The time sequence consistency modeling head is configured to perform time sequence feature modeling and consistency constraint on the flight parameter recognition results of each frame based on adjacent multi-frame standardized flight parameter display sequence images, and suppresses instantaneous abnormal jumps caused by strong light reflection and cabin shaking. The multi-task loss function is used to jointly train the shared backbone network and each detection head, so that the detection task, the recognition task, and the time sequence consistency task are cooperatively optimized in the shared feature space. The multi-objective structure search and sub-network management module is configured to use a multi-objective evolutionary search algorithm based on an accuracy prediction model and a hardware time delay lookup table to screen one or more flight parameter recognition target sub-networks that satisfy preset resource constraints from the structure space of the super network without retraining network weights, and store and manage the target sub-networks, with the flight parameter recognition accuracy or mean absolute error as the first optimization target and the inference delay, floating-point operation amount, and / or model parameter amount on the airborne computing platform as the second optimization target. The airborne inference module is configured to deploy the obtained flight parameter recognition target sub-network in the airborne computing platform of a general aviation aircraft, perform a preprocessing process of the data preprocessing module on the real-time collected cabin data in the flight process, input the obtained standardized flight parameter display sequence image into the target sub-network, sequentially pass through the flight parameter display area detection head, the flight parameter value recognition head, and the time sequence consistency modeling head, and output real-time recognition results of airspeed, attitude, heading, altitude, and vertical speed flight parameters.

Images