Multi-modal data fusion flight space cognitive test platform and method

Abstract

The invention provides a flight space cognition test platform and method based on multi-modal data fusion, and relates to the technical field of data processing, an evaluation task program module is obtained, real-time synchronous data sampling is performed through an eye movement tracking module and an fNIRS monitoring module, flight space environment multi-modal data is transmitted to a data fusion and analysis module, and the flight space cognition test platform and method based on multi-modal data fusion are obtained. And according to the data fusion and analysis module, designing a data synchronization and fusion algorithm model, determining a multi-modal data machine learning model, carrying out synchronous fusion on the flight space multi-modal feature data, evaluating the flight multi-modal feature fusion data, and obtaining a flight space cognition test result. The technical problems of inaccurate flight space cognitive test result and insufficient coverage in the prior art are solved. The technical effect of improving the accuracy and reliability of the flight space cognitive test result is achieved.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing, and in particular to a flight space cognition test platform and method for multi-modal data fusion. Background Art

[0002] Spatial cognitive ability refers to an individual's comprehensive ability to perceive, understand, remember and operate the spatial environment, which is crucial in pilot testing. Traditional spatial cognitive testing methods usually rely on single-modal data (such as two-dimensional images, text descriptions, etc.). This information has great limitations in characterizing the pilot's cognitive response in a dynamic flight space, and it is difficult to fully reflect the complexity and variability of the multi-dimensional spatial environment. With the development of the Internet of Things, artificial intelligence and big data technology, multimodal data fusion technology has gradually become a key means to improve the accuracy and depth of spatial cognitive testing. However, the application of multimodal data fusion for flight space cognitive testing in the existing technology still has certain limitations, such as high complexity of the fusion algorithm, poor data consistency, and insufficient real-time performance, which cannot meet the needs of real-time, efficient and accurate flight space test cognitive evaluation.

[0003] In the existing technology, there are technical problems such as high complexity of the fusion algorithm of flight space cognition test data and poor data real-time performance, which lead to inaccurate flight space cognition test results and insufficient coverage. Summary of the invention

[0004] The present application provides a flight space cognition test platform and method for multimodal data fusion, which is used to solve the technical problems in the prior art that the flight space cognition test data has high fusion algorithm complexity and poor data real-time performance, resulting in inaccurate flight space cognition test results and insufficient coverage.

[0005] In view of the above problems, the present application provides a flight space cognition test platform and method with multimodal data fusion.

[0006] In a first aspect of the present application, a flight space cognition test platform with multimodal data fusion is provided, the platform comprising: an evaluation task program execution module, the evaluation task program execution module is used to obtain an evaluation task program module, the evaluation task program module comprises a static space cognition test task, a dynamic space cognition test task and a flight space cognition test task; a data sampling module, the data sampling module is used to perform a spatial cognition task test according to the evaluation task program module, and perform real-time synchronous data sampling through an eye tracking module and an fNIRS monitoring module to obtain multimodal data of the flight space environment; a data preprocessing module, the data preprocessing module is used to transmit the multimodal data of the flight space environment to a data fusion and analysis module, and based on the data fusion and analysis module The module pre-processes the multimodal data of the flight space environment to obtain multimodal feature data of the flight space; the model design module is used to design a data synchronization and fusion algorithm model according to the data fusion and analysis module, and determine a multimodal data machine learning model based on the data synchronization and fusion algorithm model; the data synchronization fusion module is used to synchronize and fuse the multimodal feature data of the flight space based on the multimodal data machine learning model to obtain flight multimodal feature fusion data; the data evaluation module is used to evaluate the flight multimodal feature fusion data using a feedback and training suggestion module to obtain a flight space cognition test result, and visualize the flight space cognition test result.

[0007] A second aspect of the present application provides a flight space cognition test method with multimodal data fusion, the method comprising: obtaining an evaluation task program module, the evaluation task program module comprising a static space cognition test task, a dynamic space cognition test task and a flight space cognition test task; performing a spatial cognition task test according to the evaluation task program module, and performing real-time synchronous data sampling through an eye tracking module and an fNIRS monitoring module to obtain multimodal data of a flight space environment; transmitting the multimodal data of the flight space environment to a data fusion and analysis module, and preprocessing the multimodal data of the flight space environment based on the data fusion and analysis module to obtain multimodal feature data of the flight space; designing a data synchronization and fusion algorithm model according to the data fusion and analysis module, and determining a multimodal data machine learning model based on the data synchronization and fusion algorithm model; synchronously fusing the multimodal feature data of the flight space based on the multimodal data machine learning model to obtain flight multimodal feature fusion data; evaluating the flight multimodal feature fusion data using a feedback and training suggestion module to obtain a flight space cognition test result, and visually displaying the flight space cognition test result.

[0008] One or more technical solutions provided in this application have at least the following technical effects or advantages:

[0009] 1. Through multimodal data fusion technology, real-time data from different sensors such as eye tracking modules and fNIRS monitoring modules are integrated to achieve full coverage of pilot spatial cognition tests, improve the multi-dimensional characteristics and comprehensiveness of the test results, and ensure the richness and completeness of cognitive assessment.

[0010] 2. The collaborative monitoring and data fusion processing of multiple sensors improves the data coverage of the flight space, effectively reduces the potential errors and uncertainties brought by single-modal data, significantly enhances the accuracy and reliability of test results, and provides a more solid foundation for the comprehensive assessment of pilot cognitive abilities.

[0011] 3. Through automated and intelligent data processing and fusion algorithm models, the cognitive testing process is greatly simplified, the real-time response capability of the platform is improved, the user interaction experience is improved, and the efficiency and real-time performance of flight space cognitive testing are ensured, so that it can better support the actual needs of pilot training and cognitive ability improvement. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0013] Figure 1 A schematic diagram of the structural flow of the multimodal data fusion flight space cognition test platform provided in this application;

[0014] Figure 2 A schematic diagram of the framework flow of the multimodal data fusion flight space cognition test platform provided in this application;

[0015] Figure 3 A schematic diagram of the flow chart of the flight space cognition testing method with multimodal data fusion provided in this application.

[0016] Explanation of the accompanying drawings: evaluation task program execution module 11, data sampling module 12, data preprocessing module 13, model design module 14, data synchronization fusion module 15, data evaluation module 16. DETAILED DESCRIPTION

[0017] This application provides a flight space cognition test platform and method for multimodal data fusion, which is used to solve the technical problems in the prior art that the flight space cognition test data has high fusion algorithm complexity and poor data real-time performance, resulting in inaccurate flight space cognition test results and insufficient coverage. It achieves the technical effect of improving the accuracy of multimodal data fusion, enhancing the real-time performance and coverage of flight space cognition tests, and thus improving the accuracy and reliability of test results.

[0018] Below, the technical solutions in the present invention will be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all of the embodiments of the present invention. It should be understood that the present invention is not limited to the example embodiments described herein. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention. It should also be noted that, for the convenience of description, only the parts related to the present invention are shown in the accompanying drawings, rather than all of them.

[0019] Embodiment 1, as Figure 1 , 2 As shown, the present application provides a flight space cognition test platform for multimodal data fusion, the platform comprising:

[0020] The evaluation task program execution module 11 is used to obtain the evaluation task program module, and the evaluation task program module includes a static space cognition test task, a dynamic space cognition test task and a flight space cognition test task.

[0021] Specifically, the flight space cognition test platform is used to test and evaluate the pilot's spatial cognition ability. The flight space cognition test platform includes an evaluation task program execution module 11. The evaluation task program execution module 11 is mainly used to obtain the evaluation task program module to manage and execute the spatial cognition test task. The difficulty of the test task is divided into 3 levels, from easy to difficult: static spatial cognition test task, dynamic spatial cognition test task and flight space cognition test task. The evaluation task program execution module 11 first obtains the evaluation task program module through an interface or preset instructions to ensure that the platform can load appropriate cognitive test tasks according to different task requirements. Among them, the static spatial cognition test task refers to a spatial cognition test conducted in a static or unchanged environment, which is usually used to evaluate the spatial cognition ability of the subject in a fixed environment, such as mental rotation tasks, viewpoint conversion tasks, spatial cueing tasks, etc. The dynamic spatial cognition test task involves cognitive tests conducted in a changing environment, simulating dynamic changing scenes during flight, such as relative arrival time judgment tasks, spatial positioning dynamic tasks, etc. The flight spatial cognition test task is more complex, simulating the scene in actual flight, and its task combines the spatial cognition needs of the pilot when operating the aircraft, such as simulated flight landing tasks and simulated formation flight tasks, etc. The simulated flight landing tasks are such as simulating normal daytime aircraft landing or simulating black approach aircraft landing. The dimensions of the test task involve 6 core abilities of spatial cognition, mainly including: static spatial rotation ability, static visual orientation ability, ability to process and adjust spatial information, dynamic spatial perception ability based on time and speed, dynamic visual spatial orientation ability, and comprehensive spatial cognition ability based on flight. The evaluation task program module is obtained through the evaluation task program execution module 11, which can schedule and manage different types of spatial cognition tests, and ensure that the pilot's cognitive ability can be fully evaluated in static, dynamic and real flight environments by reasonably designing test scenarios and tasks.

[0022] The data sampling module 12 is used to perform spatial cognitive task tests according to the assessment task program module, and to perform real-time synchronous data sampling through the eye tracking module and the fNIRS monitoring module to obtain multimodal data of the flight space environment.

[0023] Specifically, the data sampling module 12 starts the relevant spatial cognitive task test according to the evaluation task program module instruction obtained by the evaluation task program execution module 11, ensuring that the task scenario of data collection matches the test requirements. In the specific test process, the eye movement data of the pilot completing the above spatial cognitive test tasks is collected through the eye tracking module, mainly including the position of the gaze point, the change of pupil diameter, the scanning path and other indicators, and the fNIRS monitoring module (functional near infrared spectroscopy monitoring module) is used to monitor and sample the pilot's brain activity in real time. The fNIRS monitoring module analyzes the changes in brain oxygenation status through infrared light penetrating the scalp and reflecting the signal back, so as to judge the pilot's brain cognitive load and emotional state at different task stages. The data sampling module 12 integrates the multimodal data collected by the eye tracking module and the fNIRS monitoring module to obtain the flight space environment multimodal data, which includes the pilot's visual perception, brain cognitive activities and other aspects of information, and can fully reflect the pilot's state when performing spatial cognitive tasks, providing reliable data support for subsequent data analysis.

[0024] The data preprocessing module 13 is used to transmit the flight space environment multimodal data to the data fusion and analysis module, and preprocess the flight space environment multimodal data based on the data fusion and analysis module to obtain flight space multimodal feature data.

[0025] Specifically, after the data sampling module 12 obtains the pilot's eye movement and fNIRS data, the data preprocessing module 13 transmits the flight space environment multimodal data to the data fusion and analysis module through a reliable transmission method, such as Bluetooth technology, and preprocesses the eye movement data and fNIRS data in the flight space environment multimodal data respectively according to the data fusion and analysis module. Eye movement data preprocessing includes sampling rate matching and data cleaning. The sampling rate matching ensures that the sampling rate of the eye tracker is above 120HZ to match the requirements of subsequent analysis. Data cleaning removes invalid or erroneous eye movement data points, such as data points that cannot be identified as left or right eyes, and data points without position information. fNIRS data preprocessing includes data cleaning, filtering, baseline correction and data segmentation. Data cleaning removes abnormal data points caused by motion artifacts, instrument noise or other interference factors, applies bandpass filtering or other filtering techniques to reduce high-frequency noise and low-frequency drift, and compares the fNIR signal with the baseline level to eliminate the influence of baseline drift. According to the requirements of different spatial cognitive test tasks, the data is divided into different time periods or trials. By performing feature extraction on the preprocessed multimodal data, flight space multimodal feature data is obtained. The flight space multimodal feature data can effectively reflect the pilot's performance in spatial cognitive tasks, such as the changing trajectory of eye gaze points, fluctuations in brain cognitive load, etc., and provide accurate and reliable data information for subsequent data synchronization fusion and machine learning modeling, thereby improving flight space cognitive results.

[0026] Furthermore, the obtaining of the flight space multimodal feature data in the data preprocessing module 13 includes: setting a data anomaly evaluation benchmark according to the flight space environment multimodal data; performing anomaly identification on the flight space environment multimodal data according to the data anomaly evaluation benchmark to obtain abnormal interference flight data; performing data cleaning and data preprocessing on the abnormal interference flight data to obtain standard flight space environment multimodal data; and performing feature extraction on the standard flight space environment multimodal data based on the data fusion and analysis module to obtain the flight space multimodal feature data.

[0027] Specifically, according to the collected multimodal data of the flight space environment, the data anomaly evaluation benchmark is set in combination with historical data, empirical rules or algorithms. The data anomaly evaluation benchmark refers to the standard for distinguishing normal and abnormal data during the flight process. For example, for eye movement data, if the eye movement trajectory of a certain period of time obviously deviates from the expected fixation point, or abnormal brain activity fluctuations appear in the fNIRS monitoring data, these anomalies can be identified according to the set evaluation benchmark. After setting the data anomaly evaluation benchmark, the flight space environment multimodal data is anomaly identified according to the data anomaly evaluation benchmark, and data that does not conform to normal flight cognitive behavior, that is, abnormal interference flight data, is screened out, and the identified abnormal interference flight data is cleaned and preprocessed to obtain standard flight space environment multimodal data. Data cleaning mainly includes noise removal, outlier filling and data format conversion, and data preprocessing includes data standardization, data integration, etc. Finally, the standard flight space environment multimodal data is feature extracted through the data fusion and analysis module, and the feature data of eye movement trajectory and brain activity are extracted by analyzing and extracting to obtain flight space multimodal feature data. By setting evaluation benchmarks for data anomaly identification, data cleaning and feature extraction, we ensure that the multimodal data of the flight space can provide accurate and effective feature data for subsequent analysis after processing.

[0028] Furthermore, the obtaining of the flight space multimodal feature data includes: performing associated cognitive feature extraction on the standard flight space environment multimodal data to obtain an associated flight multimodal cognitive feature set; performing principal component analysis based on the associated flight multimodal cognitive feature set to obtain a reduced-dimensionality flight multimodal cognitive feature set; and performing normalization processing and data alignment on the reduced-dimensionality flight multimodal cognitive feature set to obtain the flight space multimodal feature data.

[0029] Specifically, first, feature information associated with the pilot's flight space cognition is extracted from the standard flight space environment multimodal data. For example, when the pilot performs a complex space task, the frequent jumps of the eye movement trajectory may be associated with the increase of the brain cognitive load. Such associated features can reflect the pilot's reaction behavior and cognitive pressure in the task. In this way, the extracted feature data can more accurately describe the pilot's cognitive state and form a set of associated flight multimodal cognitive features. Then, the principal component analysis is performed on the geometry of the associated flight multimodal cognitive features. Principal component analysis is a commonly used dimensionality reduction technique. By linearly transforming the high-dimensional features in the data, multiple related variables are converted into a few unrelated principal components to reduce the dimension of the data. Through principal component analysis, a set of reduced-dimensional flight multimodal cognitive features is obtained, which can reduce redundant features and extract representative cognitive features with strong correlation. Finally, the reduced-dimensional flight multimodal cognitive feature set is normalized and data aligned to ensure that all feature data are in the same numerical range and that the multimodal data are consistent in time or feature dimensions. Through normalization processing and data alignment, multimodal feature data of flight space are obtained. The multimodal feature data of flight space is more accurate and representative, ensuring that the platform can accurately analyze and provide feedback on the pilot's spatial cognition ability, thereby improving the accuracy of flight space cognition test results.

[0030] The model design module 14 is used to design a data synchronization and fusion algorithm model according to the data fusion and analysis module, and determine a multimodal data machine learning model based on the data synchronization and fusion algorithm model.

[0031] Furthermore, the design of the data synchronization and fusion algorithm model in the model design module 14 includes: obtaining a data fusion algorithm model library according to the data fusion and analysis module; performing application scenario analysis and data characteristic analysis on the flight space multimodal feature data to obtain multimodal data fusion characteristic information; and obtaining the data synchronization and fusion algorithm model by matching and optimizing the multimodal data fusion characteristic information with the data fusion algorithm model library.

[0032] Specifically, the model design module 14 obtains a data fusion algorithm model library based on the feature data provided by the data fusion and analysis module. The data fusion algorithm model library is a pre-designed model set, including a variety of algorithms designed for different data scenarios, such as weighted fusion, sequence fusion, time series synchronization, etc. Then, the obtained flight space multimodal feature data is subjected to application scenario analysis and data characteristic analysis to obtain multimodal data fusion characteristic information. Application scenario analysis refers to determining which tasks and operations need to be processed first according to the actual application environment of the current flight cognitive task, so as to ensure that the model can be optimized for a specific scenario. Data characteristic analysis refers to analyzing the collection frequency, data type and correlation between features of different modal data, such as changes in oxygenated hemoglobin in fNIR, sequence of eye gaze points, etc. Then, according to the multimodal data fusion characteristic information, the algorithm characteristics in the data fusion algorithm model library are matched to obtain a data synchronization and fusion algorithm model, and the relationship between different types of data is analyzed by applying machine learning or statistical methods. The data synchronization and fusion algorithm model can efficiently and accurately process and fuse multimodal data. Finally, based on the determined data synchronization and fusion algorithm model, a multimodal data machine learning model is obtained to provide intelligent support for the evaluation and feedback of flight space cognition tasks. Through multimodal data fusion, information from different sensors and sources is integrated to achieve comprehensive coverage of spatial cognition tests, which can improve the richness and completeness of test results, and thus improve the accuracy of flight space cognition test results.

[0033] Furthermore, the determination of the multimodal data machine learning model includes: selecting a machine learning model architecture based on the data synchronization and fusion algorithm model; using the machine learning model architecture to perform feature fusion training on the flight space multimodal historical data to obtain an initial multimodal data fusion model; verifying and evaluating the initial multimodal data fusion model to obtain model performance parameters, and obtaining a model optimizer based on the model performance parameters; iteratively optimizing the initial multimodal data fusion model based on the model optimizer to determine the multimodal data machine learning model.

[0034] Specifically, based on the data synchronization and fusion algorithm model, the machine learning model architecture is selected. For example, machine learning algorithms (such as cluster analysis, decision tree, etc.) are used to identify the eye movement patterns of pilots in the immobile spatial cognitive task. According to the characteristics of fNIR data, algorithm models such as general linear model (GLM), support vector machine (SVM), random forest (RF) or neural network (NN) are selected to perform statistical analysis on key fNIR data to quantify the relationship between the pilot's brain function activity state and spatial cognitive ability. Fusion algorithms such as weighted fusion, common spatial pattern analysis (CSP), and multimodal fusion network in deep learning are used to integrate data of different modalities. The selected machine learning model architecture is combined with the multimodal features in the multimodal historical data of flight space to perform feature fusion training. The multimodal historical data of flight space can not only reflect the cognitive behavior of pilots in different task scenarios, but also provide a large number of training samples to help the model capture the complex relationship between different modal data. During the training process, the model will gradually adjust the parameters to maximize the fusion of the pilot's cognitive behavior data, thereby obtaining the initial multimodal data fusion model. The initial multimodal data fusion model is tested for performance. According to the model performance parameters, such as accuracy, recall rate, F1 value and other evaluation criteria, the model prediction accuracy, generalization ability, etc. are evaluated to obtain the model performance parameters. And according to the model performance parameters, a model optimizer is selected to optimize the model parameters, such as gradient descent, Adam optimizer, etc. By introducing the optimizer, the platform can make subtle adjustments and optimizations to the parameters of the initial model based on the feedback of the model performance to further improve the model performance. Finally, according to the model optimizer, the initial multimodal data fusion model is iteratively optimized. Each iterative optimization will continuously adjust the parameter configuration in the model according to the feedback of the performance parameters to reduce the model error and improve its processing accuracy and efficiency of the multimodal data of the flight space. After multiple iterations, the final multimodal data machine learning model is obtained. The multimodal data machine learning model has a high degree of accuracy and reliability, and can effectively support cognitive evaluation and feedback in complex flight space missions.

[0035] Furthermore, the platform includes: constructing a data model loss function according to a model fusion optimization objective; initializing the architecture parameters of the initial multimodal data fusion model based on the model optimizer to obtain model architecture parameter information; minimizing and updating the data model loss function based on the model architecture parameter information to determine the multimodal data machine learning model.

[0036] Specifically, according to the optimization goal of model fusion, a data model loss function is constructed, and the data model loss function is used to measure the gap between the model prediction output and the actual target value. According to the optimization goal of model fusion, the data model loss function can adapt to the characteristics of multimodal data at the same time, ensuring that when fusing multiple data sources, the model can effectively learn the relationship between each modality and reduce the prediction error. According to the model optimizer, the architecture parameters of the initial multimodal data fusion model are initialized, and an initial value is assigned to each weight parameter in the model. For example, through Xavier initialization or He initialization, the initial values ​​of the parameters in the model are reasonably allocated to prevent problems such as gradient disappearance or gradient explosion. After initialization, the model architecture parameter information is obtained, and the model architecture parameter information is the initial parameter information of the model architecture, which provides a basis for the training and optimization of subsequent models. Finally, based on the model architecture parameter information, the constructed data model loss function is minimized and solved. By continuously adjusting the parameters in the model, the error between the prediction result and the actual result of the model is gradually reduced, and the weight parameters in the model are iteratively updated to gradually approach the optimal solution. In each iteration, the model parameters are updated according to the current loss value until the loss function converges to a minimum value or reaches the preset optimization goal. Through this process of minimization and parameter updating, a fully optimized multimodal data machine learning model is finally determined. The multimodal data machine learning model can achieve synchronous fusion when processing different modal data, with small prediction errors, high performance and stability, and improves the accuracy and reliability of flight space cognition test data analysis.

[0037] The data synchronization fusion module 15 is used to synchronize and fuse the flight space multimodal feature data based on the multimodal data machine learning model to obtain flight multimodal feature fusion data.

[0038] Specifically, the data synchronization fusion module 15 receives the flight space multimodal feature data collected from the eye tracking module, fNIRS monitoring module and other modules, and based on the synchronization algorithm in the multimodal data machine learning model, automatically aligns the data from different modalities, and synchronizes the data collected by each modality to form flight multimodal feature fusion data. The flight multimodal feature fusion data includes the independent features of each modality and the correlation features between modal data. The data synchronization fusion module 15 generates more accurate flight multimodal feature fusion data by synchronously processing the flight space data of different modalities in time and feature dimensions, combined with the multimodal data machine learning model, to further improve the accuracy and effectiveness of flight space cognitive task evaluation.

[0039] The data evaluation module 16 is used to evaluate the flight multimodal feature fusion data using the feedback and training suggestion module, obtain the flight space cognition test results, and visualize the flight space cognition test results.

[0040] Specifically, the data evaluation module 16 receives the flight multimodal feature fusion data generated from the data fusion and analysis module, and uses the feedback and training suggestion module to analyze and evaluate the flight multimodal feature fusion data. The feedback and training suggestion module generates personalized feedback and training suggestions based on the pilot's performance, and obtains the flight space cognition test results, which include the pilot's performance in different cognitive tasks, including the pilot's task performance, eye movement results, fNIR results visualization, pilot spatial cognitive ability comprehensive evaluation results, and feedback and training suggestion reports based on the pilot's spatial cognitive ability evaluation results. Then, the flight space cognition test results are visualized using charts, heat maps, three-dimensional reconstruction and other means. Through visual display, it is convenient for pilots, coaches or other relevant personnel to quickly understand the performance of pilots. For example, the platform can display the changes in the pilot's cognitive load during the execution of the task through a line chart, and display the ability distribution of different cognitive dimensions through a radar chart. The data evaluation module 16 uses the multimodal feature fusion data for comprehensive evaluation through the feedback and training suggestion module to generate accurate and reliable flight space cognition test results, and makes the test results more intuitive through visual display.

[0041] Furthermore, the data evaluation module 16 obtains the flight space cognition test result, including: using the feedback and training suggestion module to perform cognitive ability evaluation on the flight multimodal feature fusion data to obtain flight space cognition ability feedback information; performing improvement strategy analysis based on the flight space cognition ability feedback information to obtain cognitive ability improvement optimization strategy; generating a test report based on the flight space cognition ability feedback information and the cognitive ability improvement optimization strategy to obtain the flight space cognition test result.

[0042] Specifically, the cognitive ability evaluation of the flight multimodal feature fusion data is carried out through the feedback and training suggestion module to comprehensively analyze the cognitive performance of the pilot in performing the flight mission. The process of cognitive ability evaluation includes a comprehensive analysis of the pilot's attention, reaction speed, decision-making ability and other dimensions. Based on the features extracted from the multimodal data, combined with historical data and standard requirements, the cognitive state of the pilot in different mission stages is evaluated. After obtaining the evaluation results, the flight space cognitive ability feedback information is generated, and the flight space cognitive ability feedback information can reflect the cognitive advantages and disadvantages of the pilot. Then, based on the flight space cognitive ability feedback information, the improvement strategy analysis is carried out. The platform analyzes the problems in the flight space cognitive ability feedback information and combines the training experience to obtain personalized cognitive ability improvement optimization strategies. For example, if the flight space cognitive ability feedback information evaluates that the pilot has difficulty concentrating in dynamic tasks, the improvement strategy may include improving the pilot's attention maintenance ability through special cognitive training tasks. Finally, the flight space cognitive ability feedback information and the cognitive ability improvement optimization strategy are combined to obtain the current pilot's flight space cognitive test report, which includes the pilot's performance in the task and targeted improvement suggestions and future training directions. Based on the test report, the flight space cognition test results are obtained, which are not only used for the summary and feedback of the current flight space cognition test tasks, but also provide clear guidance for subsequent training and performance improvement, and comprehensively enhance the evaluation and optimization of pilots' cognitive abilities.

[0043] Embodiment 2, as Figure 3 As shown, the present application provides a flight space cognition test method of multimodal data fusion, wherein the method comprises:

[0044] An evaluation task program module is obtained, wherein the evaluation task program module includes a static spatial cognition test task, a dynamic spatial cognition test task and a flight spatial cognition test task; a spatial cognition task test is performed according to the evaluation task program module, and real-time synchronous data sampling is performed through an eye tracking module and an fNIRS monitoring module to obtain multimodal data of a flight space environment; the multimodal data of the flight space environment is transmitted to a data fusion and analysis module, and the multimodal data of the flight space environment is preprocessed based on the data fusion and analysis module to obtain multimodal feature data of the flight space; a data synchronization and fusion algorithm model is designed according to the data fusion and analysis module, and a multimodal data machine learning model is determined based on the data synchronization and fusion algorithm model; the flight space multimodal feature data is synchronously fused based on the multimodal data machine learning model to obtain flight multimodal feature fusion data; a feedback and training suggestion module is used to evaluate the flight multimodal feature fusion data to obtain a flight spatial cognition test result, and the flight spatial cognition test result is visualized.

[0045] Furthermore, the obtaining of flight space multimodal feature data includes: setting a data anomaly evaluation benchmark according to the flight space environment multimodal data; performing anomaly identification on the flight space environment multimodal data according to the data anomaly evaluation benchmark to obtain abnormal interference flight data; performing data cleaning and data preprocessing on the abnormal interference flight data to obtain standard flight space environment multimodal data; and performing feature extraction on the standard flight space environment multimodal data based on the data fusion and analysis module to obtain the flight space multimodal feature data.

[0046] Furthermore, the obtaining of the flight space multimodal feature data includes: performing associated cognitive feature extraction on the standard flight space environment multimodal data to obtain an associated flight multimodal cognitive feature set; performing principal component analysis based on the associated flight multimodal cognitive feature set to obtain a reduced-dimensionality flight multimodal cognitive feature set; and performing normalization processing and data alignment on the reduced-dimensionality flight multimodal cognitive feature set to obtain the flight space multimodal feature data.

[0047] Furthermore, the design of the data synchronization and fusion algorithm model includes: obtaining a data fusion algorithm model library according to the data fusion and analysis module; performing application scenario analysis and data characteristic analysis on the flight space multimodal feature data to obtain multimodal data fusion characteristic information; and obtaining the data synchronization and fusion algorithm model by matching and optimizing the multimodal data fusion characteristic information with the data fusion algorithm model library.

[0048] Furthermore, the determination of the multimodal data machine learning model includes: selecting a machine learning model architecture based on the data synchronization and fusion algorithm model; using the machine learning model architecture to perform feature fusion training on the flight space multimodal historical data to obtain an initial multimodal data fusion model; verifying and evaluating the initial multimodal data fusion model to obtain model performance parameters, and obtaining a model optimizer based on the model performance parameters; iteratively optimizing the initial multimodal data fusion model based on the model optimizer to determine the multimodal data machine learning model.

[0049] Furthermore, the method also includes: constructing a data model loss function according to the model fusion optimization objective; initializing the architecture parameters of the initial multimodal data fusion model based on the model optimizer to obtain model architecture parameter information; minimizing and updating the data model loss function based on the model architecture parameter information to determine the multimodal data machine learning model.

[0050] Furthermore, obtaining the flight space cognition test result includes: using a feedback and training suggestion module to perform a cognitive ability assessment on the flight multimodal feature fusion data to obtain flight space cognition ability feedback information; performing an improvement strategy analysis based on the flight space cognition ability feedback information to obtain a cognitive ability improvement optimization strategy; generating a test report based on the flight space cognition ability feedback information and the cognitive ability improvement optimization strategy to obtain the flight space cognition test result.

[0051] The above description is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

[0052] This specification and the drawings are merely exemplary illustrations of the present application and are deemed to cover any and all modifications, variations, combinations or equivalents within the scope of the present application. Obviously, a person skilled in the art may make various modifications and variations to the present application without departing from the scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the present application and its equivalents, the present application intends to include these modifications and variations.

Claims

1. The flight space cognition test platform with multi-modal data fusion is characterized by: The platform includes: An evaluation task program execution module, the evaluation task program execution module is used to obtain an evaluation task program module, the evaluation task program module includes a static space cognition test task, a dynamic space cognition test task and a flight space cognition test task; A data sampling module, wherein the data sampling module is used to perform a spatial cognitive task test according to the assessment task program module, and to perform real-time synchronous data sampling through an eye tracking module and an fNIRS monitoring module to obtain multimodal data of a flight space environment; A data preprocessing module, the data preprocessing module is used to transmit the flight space environment multimodal data to the data fusion and analysis module, and preprocess the flight space environment multimodal data based on the data fusion and analysis module to obtain flight space multimodal feature data; A model design module, wherein the model design module is used to design a data synchronization and fusion algorithm model according to the data fusion and analysis module, and determine a multimodal data machine learning model based on the data synchronization and fusion algorithm model; A data synchronization fusion module, wherein the data synchronization fusion module is used to synchronously fuse the flight space multimodal feature data based on the multimodal data machine learning model to obtain flight multimodal feature fusion data; A data evaluation module, wherein the data evaluation module is used to evaluate the flight multimodal feature fusion data using a feedback and training suggestion module, obtain a flight space cognition test result, and visualize the flight space cognition test result.

2. The multimodal data fusion flight space cognition test platform according to claim 1, characterized in that: The obtaining of flight space multimodal feature data comprises: Setting a data anomaly evaluation benchmark according to the multimodal data of the flight space environment; Performing anomaly identification on the multimodal data of the flight space environment according to the data anomaly evaluation benchmark to obtain abnormal interference flight data; Performing data cleaning and data preprocessing on the abnormal interference flight data to obtain multimodal data of a standard flight space environment; Based on the data fusion and analysis module, feature extraction is performed on the standard flight space environment multimodal data to obtain the flight space multimodal feature data.

3. The multimodal data fusion flight space cognition test platform as claimed in claim 2, characterized in that: The obtaining of the flight space multimodal feature data comprises: Extracting associated cognitive features from the standard flight space environment multimodal data to obtain an associated flight multimodal cognitive feature set; Performing principal component analysis based on the associated flight multimodal cognitive feature set to obtain a reduced-dimensional flight multimodal cognitive feature set; The dimension-reduced flight multimodal cognitive feature set is normalized and data aligned to obtain the flight space multimodal feature data.

4. The multimodal data fusion flight space cognition test platform according to claim 1, characterized in that: The design data synchronization and fusion algorithm model includes: According to the data fusion and analysis module, a data fusion algorithm model library is obtained; Performing application scenario analysis and data characteristic analysis on the flight space multimodal feature data to obtain multimodal data fusion characteristic information; Based on the multimodal data fusion characteristic information and the data fusion algorithm model library, matching and optimization are performed to obtain the data synchronization and fusion algorithm model.

5. The multimodal data fusion flight space cognition test platform according to claim 1, characterized in that: Determining the multimodal data machine learning model includes: Select a machine learning model architecture based on the data synchronization and fusion algorithm model; The machine learning model architecture is used to perform feature fusion training on the flight space multimodal historical data to obtain an initial multimodal data fusion model; Performing verification and evaluation on the initial multimodal data fusion model to obtain model performance parameters, and obtaining a model optimizer based on the model performance parameters; The initial multimodal data fusion model is iteratively optimized based on the model optimizer to determine the multimodal data machine learning model.

6. The multimodal data fusion flight space cognition test platform according to claim 5, characterized in that: The platform includes: Construct the data model loss function according to the model fusion optimization goal; Initializing the architecture parameters of the initial multimodal data fusion model based on the model optimizer to obtain model architecture parameter information; Based on the model architecture parameter information, the data model loss function is minimized and the parameters are updated to determine the multimodal data machine learning model.

7. The multimodal data fusion flight space cognition test platform according to claim 1, characterized in that: The obtaining of flight space cognition test results includes: Using a feedback and training suggestion module to evaluate the cognitive ability of the flight multimodal feature fusion data to obtain flight space cognitive ability feedback information; Performing an improvement strategy analysis based on the flight space cognitive ability feedback information to obtain a cognitive ability improvement optimization strategy; A test report is generated based on the flight space cognitive ability feedback information and the cognitive ability improvement optimization strategy to obtain the flight space cognitive test result.

8. A flight space cognition test method based on multimodal data fusion, characterized in that: The method comprises: Acquire an assessment task program module, wherein the assessment task program module includes a static spatial cognition test task, a dynamic spatial cognition test task, and a flight spatial cognition test task; The spatial cognition task test is performed according to the evaluation task program module, and real-time synchronous data sampling is performed through the eye tracking module and the fNIRS monitoring module to obtain multimodal data of the flight space environment; The flight space environment multimodal data is transmitted to a data fusion and analysis module, and the flight space environment multimodal data is preprocessed based on the data fusion and analysis module to obtain flight space multimodal feature data; According to the data fusion and analysis module, a data synchronization and fusion algorithm model is designed, and based on the data synchronization and fusion algorithm model, a multimodal data machine learning model is determined; Synchronously fusing the flight space multimodal feature data based on the multimodal data machine learning model to obtain flight multimodal feature fusion data; The flight multimodal feature fusion data is evaluated using a feedback and training suggestion module to obtain flight space cognition test results, and the flight space cognition test results are visualized.

Images