Extended reality-based simulation training intelligent guidance and holographic annotation system

Abstract

The application discloses a kind of based on extended reality's simulation training intelligent guidance and holographic marking system, it is related to aviation flight simulation training technical field, including: XR head-mounted display, multiple depth cameras, cabin positioning mark, flight simulator, perception computer, data processing and storage server;Wherein, XR head-mounted display includes multiple data acquisition sensors and multiple sensory feedback units;Flight simulator includes simulation cabin, cabin data acquisition computer, flight simulation computer, computer-generated force computer and image rendering computer;XR head-mounted display, multiple depth cameras, cabin positioning mark are all set in simulation cabin.The present application alleviates the technical problems of low information fusion level and low intelligent degree of guidance and marking in the prior art.

Description

Technical Field

[0001] This invention relates to the field of aviation flight simulation training technology, specifically to an intelligent guidance and holographic annotation system for simulation training based on extended reality. Background Technology

[0002] Flight trainees have a limited understanding of flight, possessing almost no actual flying experience. They only know the basics of airborne equipment and flight operation. Flight training requires stages such as cockpit practice, simulated flight, and instructor flights. Flight instructors need to guide trainees in areas such as hand gestures, commands, and attention allocation. Traditional simulated training for flight trainees requires on-site demonstrations and guidance from instructors, who must promptly correct, prompt, and warn trainees of improper operating behaviors. When encountering difficult problems during simulated training, trainees need to use electronic devices such as knee boards to find information. The simulated training process is time-consuming and labor-intensive, making it difficult to identify trainees' erratic movements and bad habits. Data and knowledge management is inefficient and cannot meet the diverse training needs of flight trainees.

[0003] The application of Extended Reality (XR) technology in flight simulation training is mainly used to provide trainees with visual displays based on XR head-mounted displays, to achieve the fusion of virtual scenes outside the cockpit and the real cockpit, and to realize functions such as head tracking and eye tracking for trainees. Most of these applications focus on simple interactive information prompts and attention allocation, with low levels of integration of multi-sensor elements and information fusion, and low levels of intelligent guidance and annotation. They cannot achieve intelligent perception, intelligent guidance, and holographic annotation in the simulated training process for trainees, nor can they provide personalized training guidance and resource information services based on knowledge graphs for user profiles. They also do not fulfill the role of intelligent virtual flight instructors. Summary of the Invention

[0004] The purpose of this invention is to provide an extended reality-based simulation training intelligent guidance and holographic annotation system to solve at least one of the above-mentioned technical problems.

[0005] In a first aspect, embodiments of the present invention provide an intelligent guidance and holographic annotation system for simulated training based on extended reality, comprising: an XR headset, multiple depth cameras, an in-cabin positioning marker, a flight simulator, a perception computer, and a data processing and storage server; wherein, the XR headset includes multiple data acquisition sensors and multiple sensory feedback units; the flight simulator includes a simulated cockpit, a cockpit data acquisition computer, a flight simulation computer, a computer-generated force computer, and an image rendering computer; the XR headset, the multiple depth cameras, and the in-cabin positioning marker are all disposed within the simulated cockpit; the XR headset is used to collect video information, voice information, head displacement data, and eye movement data within the simulated cockpit, and to display virtual-real fusion holographic images, intelligent annotation visual information, prompting guidance sounds, and vibration prompts; the multiple depth cameras are used to capture the hand gestures and facial images of trainees within the simulated cockpit; The in-cabin positioning marker is used to locate the trainee's head displacement and posture; the cockpit data acquisition computer is used to collect the trainee's operational behavior data in the simulated cockpit; the flight simulation computer is used to calculate the position and attitude of the aircraft; the computer-generated force computer is used to generate and calculate virtual combat forces, providing force confrontation situation data in the battlefield environment; the image rendering computer is used to render the external visual images in real time and draw holographic annotation visual information; the perception computer is used to output the trainee's personal perception results; the personal perception results include head position tracking, eye tracking, face recognition, voice recognition, hand shape recognition, control behavior perception, aircraft posture perception, and battlefield situation perception; the data processing and storage server is used to form a knowledge graph and personnel profile of the trainee based on data knowledge processing, and to store the intelligent perception result data of the flight training process.

[0006] Optionally, the data acquisition sensor includes a camera, a sound acquisition sensor, an inertial measurement unit, and an eye-tracking acquisition device; the sensory feedback unit includes a visual display screen, an auditory feedback headset, and a tactile vibration feedback device.

[0007] Optionally, the depth camera includes an RGBD camera; the plurality of depth cameras are respectively disposed on the left, right and front sides of the simulated cockpit.

[0008] Optionally, it also includes a network switch, through which the data processing server communicates with the image rendering computer and the perception computer.

[0009] Optionally, the data processing and storage server includes a data processing server, an intelligent processing server, and a data storage server; wherein, the data processing server is used to form a knowledge graph and a personnel profile of the trainees based on data knowledge processing; the knowledge graph includes aircraft equipment knowledge, operating standard knowledge, personnel behavior knowledge, and instructor experience knowledge; the personnel profile includes a static information profile and a dynamic behavior profile of the trainees formed based on the training program, training plan, personal information, and the personal perception results; the intelligent processing server is used to perform intelligent guided reasoning based on the personal perception results, the knowledge graph, and the personnel profile, and to generate intelligent reminders and intelligent annotations during the training process; the data storage server is used to store the intelligent perception result data of the flight training process.

[0010] Optionally, the data processing and storage server includes: a knowledge graph module, a personnel profiling module, an intelligent sensing module, an intelligent guidance module, and an interactive function module; wherein, the knowledge graph module is used to construct a knowledge graph based on aircraft equipment knowledge, operating standard knowledge, personnel behavior knowledge, and instructor experience knowledge, using ontology technology; the personnel profiling module is used to create static information profiles and dynamic behavior profiles of the trainees based on training programs, training plans, personal information, and the results of the trainees' perception; the intelligent sensing module is used to detect the trainees' face, head position, and eye movements based on the multiple data acquisition sensors and the multiple depth cameras. The system tracks and recognizes voice and hand gesture features, and intelligently perceives the trainee's operational behavior, aircraft position and attitude, and combat situation based on cockpit equipment status, aircraft flight status, and computer-generated force status. The intelligent guidance module is used to intelligently remind, prompt, and generate standard information about the trainee's attention allocation, operation process, behavior, and command pronunciation based on the knowledge graph and the individual's perception results, by integrating logical rules, vector representation, and graph structure features, and using the relationships in the graph structure for logical reasoning. The interactive function module is used to provide intelligent guidance information feedback to the trainee based on the XR headset.

[0011] Optionally, the personnel profiling module is further configured to: construct typical trainee pilot tags based on knowledge in the flight instructor domain; perform field filtering and preprocessing based on personal academic information, exam scores for each course, practical flight training evaluation information, training operation data, and personal activity data to obtain data fields reflecting the static information and training behavior of the trainees; construct a knowledge graph for trainee profiling based on the data fields; represent the corpus generating the knowledge graph as word vectors using a semantic model; generate a tag table for the target personnel based on the typical trainee pilot tags according to the calculation of the correlation and similarity of the word vectors; determine the static information profile and dynamic behavior profile of the trainees; generate corresponding personalized training plans and training content based on the static information profile and the dynamic behavior profile; find profile data that matches the trainees and simulated training activities; and remind the trainees of training priorities and bad habits.

[0012] Optionally, the intelligent guidance module is further configured to: automatically generate synthesized text and image information based on the knowledge graph; establish a connection with a preset rule base through recorded training data and operational behavior processes; construct a TransE KGE model consisting of entity embedding vectors and relation embedding vectors, mapping entities and relations to a low-dimensional vector space; optimize the entity embedding vectors and relation embedding vectors by minimizing the loss function of incorrect predictions; access the preset rule base in conjunction with Jena inference; compare the personal perception result with the beginning of the total rules of the preset rule base; and finally obtain the inference result; convert the inference result into a synthesized text and image format that can be used for intelligent behavior guidance and push it to the XR headset.

[0013] Optionally, the interactive function module is also used to: mark the target device switch, and display visual prompts that are virtual and real in a holographic display format on the virtual sign in the distant scene outside the cabin and the real device switch panel inside the cabin, and provide auditory feedback prompts in the form of warning sounds and voice, while providing vibration tactile feedback in combination with the vibration sensor of the head-mounted display.

[0014] This invention provides an intelligent guidance and holographic annotation system for simulated training based on extended reality. Building upon a knowledge graph of flight simulation training and a trainee profile, it employs intelligent perception and fusion of multi-sensor and multi-element data. This provides online real-time intelligent guidance and multi-sensory intelligent annotation prompts for flight simulation training based on an XR head-mounted display. It can be applied not only to aircraft cockpit practice, flight procedure training, emergency procedure training, tactical confrontation simulation training, debriefing and analysis, but also to providing flight trainees with personalized simulation training plans and intelligent information services. This enhances the effectiveness of simulation training and promotes the integrated development of intelligent simulation training. The system features high levels of intelligent perception fusion, comprehensive intelligent guidance functions, multi-sensory holographic annotation, and support for personalized training, alleviating the technical problems of low information fusion levels and insufficient intelligence in guidance and annotation found in existing technologies. Attached Figure Description

[0015] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0016] Figure 1 A schematic diagram of an intelligent guidance and holographic annotation system for simulation training based on extended reality provided in an embodiment of the present invention; Figure 2 A schematic diagram of the functional modules of a data processing and storage server provided in an embodiment of the present invention; Figure 3 This is a schematic diagram illustrating the data processing process of a data processing and storage server provided in an embodiment of the present invention. Detailed Implementation

[0017] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0018] Figure 1 This is a schematic diagram of an extended reality-based simulation training intelligent guidance and holographic annotation system according to an embodiment of the present invention. Figure 1As shown, the system includes: an XR headset, multiple depth cameras, an in-cabin positioning marker, a flight simulator, a perception computer, and a data processing and storage server; wherein, the XR headset includes multiple data acquisition sensors and multiple sensory feedback units; the flight simulator includes a simulated cockpit, a cockpit data acquisition computer, a flight simulation computer, a computer-generated force computer, and an image rendering computer; the XR headset, multiple depth cameras, and in-cabin positioning marker are all located inside the simulated cockpit.

[0019] Specifically, the XR headset is used to collect video information, voice information, head displacement data, and eye movement data within the simulated cockpit, and to display holographic images that blend virtual and real elements, intelligently labeled visual information, and prompting and guiding sounds and vibrations.

[0020] Specifically, the data acquisition sensors include a camera, a sound acquisition sensor, an inertial measurement unit, and an eye-tracking acquisition device, which respectively capture video, voice calls, head movement, and binocular eye movement data in real time from the simulated cockpit.

[0021] The sensory feedback unit includes a visual display screen, auditory feedback headphones, and tactile vibration feedback devices, which respectively enable the display of holographic images that blend virtual and real elements and intelligently labeled visual information, as well as intelligently guided sound reminders and vibration prompts.

[0022] Specifically, multiple depth cameras are used to capture the hand gestures and facial images of trainees inside the simulated cockpit. Preferably, such as Figure 1 As shown, the depth cameras include RGBD cameras; multiple depth cameras are respectively set on the left, right and front sides of the simulated cockpit.

[0023] The in-cabin positioning markers are used to locate the head displacement and posture of the trainees. Preferably, the in-cabin positioning markers are arranged on the front side of the simulated cockpit, and can be in the form of multiple black and white positioning QR codes distributed in a dispersed manner. By analyzing the positioning markers in the video images captured by the built-in camera of the XR head-mounted display, the inertial measurement unit can be used to achieve accurate positioning of the trainees' head displacement.

[0024] Specifically, the simulated cockpit is a semi-physical cockpit modeled after the aircraft, including control mechanisms, equipment switch panels, and instrument displays; the cockpit data acquisition computer is used to collect data on the trainees' operational behavior in the simulated cockpit, providing data on the trainees' operational behavior for intelligent perception; the flight simulation computer is used to calculate the position and attitude of the aircraft, providing flight attitude data for intelligent perception; the computer-generated forces computer is used to generate and calculate virtual adversarial forces (CGF), providing force confrontation situation data in the battlefield environment; and the image rendering computer is used to render and draw holographically annotated visual information of the external view images in real time, and to realize the playback of intelligent guidance sounds and vibration prompt control.

[0025] The perception computer is used to output the trainee's individual perception results. Specifically, the perception computer receives multi-source sensor data, including RGBD camera video streams, inertial measurement data from the XR headset, eye-tracking data from the XR headset, audio streams from the XR headset, camera video streams from the XR headset, trainee's operational behavior data, the aircraft's flight posture data, and CGF (Combat Global Area) situational awareness data. It then performs head tracking, eye tracking, facial recognition, speech recognition, hand gesture recognition, perception of maneuvering behavior, perception of the aircraft's posture, and battlefield situational awareness. The computer outputs the individual perception results to the intelligent processing and data processing server, and records the perception results through the data storage server.

[0026] The data processing and storage server is used to generate knowledge graphs and personnel profiles based on data knowledge processing, and to store intelligent perception results data of the flight training process.

[0027] Specifically, data processing and storage servers include data processing servers, intelligent processing servers, and data storage servers; among them, The data processing server is used to generate knowledge graphs and personnel profiles based on data knowledge processing. The knowledge graph is obtained by offline processing of standard specifications such as equipment user manuals, flight manuals, training outlines, operating procedures, emergency procedures, and fault handling cases. It includes aircraft equipment knowledge, operating standard knowledge, personnel behavior knowledge, and instructor experience knowledge, which is the knowledge foundation for intelligent guided reasoning. The personnel profile includes static information profiles and dynamic behavioral profiles of trainees formed based on training programs, training plans, personal information, and personal perception results. The intelligent processing server is used to perform intelligent guided reasoning based on personal perception results, knowledge graphs and personnel profiles, and to generate intelligent reminders and intelligent annotations during the training process, and supports non-real-time intelligent question answering and intelligent explanation; The data storage server stores the intelligent sensing results data from the flight training process, supporting debriefing analysis and deepening the dynamic behavioral profiles of personnel.

[0028] Preferably, such as Figure 1 As shown, the system provided in this embodiment of the invention also includes a network switch, through which the data processing server communicates with the image rendering computer and the sensing computer. Specifically, the network switch is a wired high-speed network device used to achieve high-speed, low-latency data transmission between the intelligent data processing and storage server and the intelligent sensing computer and flight simulator in the system.

[0029] Figure 2 This is a schematic diagram of the functional modules of a data processing and storage server according to an embodiment of the present invention. Figure 2 As shown, the data processing and storage server includes: a knowledge graph module, a personnel profiling module, an intelligent sensing module, an intelligent guidance module, and an interactive function module; among them, The knowledge graph module is used to construct a knowledge graph based on aircraft equipment knowledge, operating standard knowledge, personnel behavior knowledge, and instructor experience knowledge, using ontology technology. The personnel profiling module is used to create static information profiles and dynamic behavioral profiles of trainees based on training programs, training plans, personal information, and personal perception results. The intelligent perception module is used to track and recognize the face, head position, eye movement, voice and hand features of trainees based on multiple data acquisition sensors and multiple depth cameras, and to intelligently perceive the trainees' operational behavior, aircraft position and attitude and combat situation based on the status of cockpit equipment, aircraft flight status and computer-generated force status. The intelligent guidance module is used to provide intelligent reminders, prompts and generate standard information for trainees’ attention allocation, operation process and behavior, and verbal commands based on knowledge graphs and personal perception results. It integrates logical rules, vector representation and graph structure features, and uses the relationships in the graph structure to perform logical reasoning. The interactive function module is used to provide intelligent guidance and feedback to trainees based on the XR headset.

[0030] Figure 3 This is a schematic diagram illustrating the data processing process performed by a data processing and storage server according to an embodiment of the present invention. Figure 3As shown, the knowledge graph module is obtained through data knowledge processing. Based on scattered and low-correlation data such as aircraft equipment related data, operation and use procedures, personnel training data, personnel operation behavior standards and requirements, and instructor historical evaluation and scoring data, ontology technology is used to build the core layer of the intelligent guidance and holographic annotation knowledge graph schema, realizing the definition and constraint of concepts, attributes, relationships, etc. in the knowledge framework.

[0031] Specifically, the knowledge graph module analyzes knowledge related to flight simulation training to determine the scope of application, expected goals, and applicable range of intelligent guidance and holographic annotation ontology. Based on the aforementioned standards and relevant specifications related to aircraft models in the flight simulation training field, it summarizes relevant concepts suitable for intelligent guidance and holographic annotation, as well as their expression forms in the ontology. Then, referring to the suggestions of flight instructors, it uses association rule mining algorithms to mine the concepts and relationships between them in the knowledge of operational standard requirements, scoring and evaluation criteria related to attention allocation, gestures and hand gestures, standard commands, take-off and landing flight, tactical combat flight, and emergency procedure flight, etc., to obtain the flight instructors' experience knowledge. This knowledge is then hierarchically defined in the form of classes, and the data attributes and object attributes of the concepts are determined.

[0032] Then, Protégé, an ontology editing and knowledge acquisition tool developed in Java, was selected for ontology construction, and the consistency and completeness of the ontology were checked and verified. Data extraction based on the schema layer ultimately resulted in four types of knowledge representations: aircraft equipment knowledge, operating standard knowledge, personnel behavior knowledge, and instructor experience knowledge. These were stored in a Neo4j graph database with a framework defined according to the schema layer. Cypher was used for querying and other operations, supporting various knowledge base-based retrieval methods. The constructed intelligent guidance and holographic annotation knowledge graph provides the data foundation for intelligent guidance and knowledge retrieval for trained pilots.

[0033] Specifically, the personnel profiling module is also used to: construct typical trainee pilot labels based on knowledge in the field of flight instructors; and perform field filtering and preprocessing based on personal academic information, exam scores of various courses, actual flight training evaluation information, training operation data and personal activity data to obtain data fields that reflect the static information and training behavior of trainees. A knowledge graph for trainees is constructed based on data fields. Then, the corpus of the generated knowledge graph is represented as word vectors through a semantic model. Based on the calculation of the relevance and similarity of the word vectors, a label table for the target personnel is generated on the basis of the labels of typical trainees, thus determining the static information profile and dynamic behavior profile of the trainees. Based on static information profiles and dynamic behavioral profiles, corresponding personalized training plans and training content are generated. Profile data that matches the trainees and simulation training activities is found, providing trainees with targeted personal information and simulation training services. The training focus and bad habits of trainees are reminded, reflecting the differentiation and personalization of intelligent guidance and annotation in simulation training.

[0034] Specifically, the intelligent perception module comprehensively utilizes the sensors embedded in the XR headset and the RGBD depth cameras deployed in the simulated cockpit. Through eight categories of input data, it can track and recognize the trainee's facial features, head position, eye movements, voice, hand gestures, and other physical characteristics. Furthermore, by collecting data on the status of cockpit equipment, the flight status of the aircraft, and the status of CGF forces on the battlefield, it can achieve intelligent perception of the trainee's operational behavior, the aircraft's position and attitude, and the combat situation.

[0035] In practice, the XR headset can be the Varjo XR-3 or XR-4 or the Huijing XR 2nd generation headset, which has embedded binocular VST (Video See Through) cameras and microphones and other sensors, enabling head position, eye tracking and virtual-real mixed display; the RGBD depth camera can be the Intel D435i with binocular vision technology, which can achieve high-precision motion tracking of hand gestures and depth measurement.

[0036] The intelligent perception module utilizes deep learning to automatically acquire the hand positions of trainee pilots in multi-sensor images. It then uses this hand position information to determine the operated equipment, identifying the pickup relationship between the equipment and the hand in consecutive single frames, accurately perceiving the hand gestures used to operate the equipment. Simultaneously, it detects cockpit positioning markers from images acquired from the XR headset and RGBD cameras. Using these markers, it calculates the relative pose of the cameras, fuses this with XR inertial measurement data, and uses this to detect and track head position. It can also detect and acquire area information of the cockpit control panel and equipment switch positions, providing precise positioning for the annotation of virtual holographic display information on physical objects such as simulated cockpit equipment switches. Face and voice recognition employ lightweight network architectures such as MobileNet, reducing the number of parameters and computational complexity while retaining key feature extraction capabilities. The approximate nearest neighbor ANN algorithm improves search efficiency, achieving high efficiency and accuracy in face and voice recognition. The intelligent perception results also record training process data through a training data recording function to support data playback, debriefing, and review.

[0037] Specifically, the intelligent guidance module is also used to: automatically generate text-image synthesis information based on knowledge graphs, establish a connection with the preset rule base through recorded training data and operation behavior process, construct a TransE KGE model consisting of entity embedding vectors and relation embedding vectors, and map entities and relations to a low-dimensional vector space. The entity embedding vector and relation embedding vector are optimized by minimizing the loss function of incorrect predictions. Combined with Jena inference, the pre-defined rule base is accessed. The personal perception result is compared with the beginning of the total rules in the pre-defined rule base, and finally the inference result is obtained. The reasoning results are converted into a synthetic graphic format that can be used for intelligent behavior guidance and then pushed to the XR headset.

[0038] Specifically, the intelligent guidance module, based on the constructed knowledge graph and intelligent perception results, integrates logical rules, vector representations, and graph structure features. It utilizes the relationships within the graph structure for logical reasoning to generate intelligent reminders, prompts, and annotations regarding trainees' attention allocation, operational processes, actions, and verbal commands. It can also provide intelligent question-and-answer and explanations based on speech recognition and synthesis technologies. Based on knowledge graph technology, it automatically generates synthesized text and graphics information. Through recorded training data and operational processes, it establishes a connection with the rule base, constructing a TransE KGE model consisting of entity embedding vectors and relation embedding vectors. Entities and relations are mapped to a low-dimensional vector space. These vectors are optimized by minimizing the loss function for incorrect predictions. Combined with Jena reasoning, it accesses the rule base, comparing the intelligent perception results with the beginning of the rule base's overall rules. Finally, the obtained reasoning results are converted into a synthesized text and graphics format suitable for intelligent behavioral guidance and pushed to the XR headset.

[0039] Specifically, the interactive function module is also used to: mark the target device switch, and display visual prompts that are virtual and real in a holographic display format on the virtual sign in the distant scene outside the cabin and the real device switch panel inside the cabin, and provide auditory feedback prompts in the form of warning sounds and voice prompts, while providing tactile feedback through the vibration sensor of the head-mounted display.

[0040] Specifically, the interactive module uses an XR head-mounted display to provide intelligent guidance information feedback to trainee pilots. It employs the DeepLab semantic segmentation algorithm based on deep learning to perform pixel-level classification of cockpit images, enabling the labeling of key equipment switches. This improves the trainee pilots' understanding of specific operating equipment and key operating procedures provided by the intelligent guidance. Combining visual, auditory, and tactile elements, it displays holographic information on virtual markers in distant scenes outside the cockpit and on real equipment switch panels inside the cockpit, using overlays of virtual and real images, animation effects, and text symbols. This results in a fusion of virtual and real visual cues with precise positioning. Auditory feedback is provided through warning sounds and voice prompts, while tactile feedback is enhanced by vibration sensors in the head-mounted display. This multi-sensory intelligent feedback system provides intelligent guidance for operating procedures and intelligent reminders for maneuvering.

[0041] As described above, the embodiments of the present invention provide an intelligent guidance and holographic annotation system for simulated training based on extended reality. Based on the construction of a knowledge graph for flight simulation training and a profile of the trainee, it employs intelligent perception and fusion of multi-sensor and multi-element data to provide online real-time intelligent guidance and multi-sensory intelligent annotation prompts for flight simulation training based on an XR head-mounted display. This system can be applied not only to aircraft cockpit practice, flight procedure training, emergency procedure training, tactical confrontation simulation training, debriefing and analysis, but also to providing flight trainees with personalized simulation training plans and intelligent information services. This enhances the effectiveness of simulation training and promotes the integrated construction of intelligent simulation training. It features a high degree of intelligent perception fusion, comprehensive intelligent guidance functions, multi-sensory holographic annotation, and support for personalized training.

[0042] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the invention can be implemented in other specific forms without departing from its spirit or essential characteristics. Therefore, the embodiments should be considered in all respects as exemplary and non-limiting, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be included within the present invention. No reference numerals in the claims should be construed as limiting the scope of the claims.

[0043] Furthermore, it should be understood that although this specification describes embodiments, not every embodiment contains only one independent technical solution. This narrative style is merely for clarity. Those skilled in the art should consider the specification as a whole, and the technical solutions in each embodiment can also be appropriately combined to form other embodiments that can be understood by those skilled in the art.

Claims

1. A simulation training intelligent guidance and holographic annotation system based on extended reality, characterized in that, include: The system comprises an XR headset, multiple depth cameras, an in-cabin positioning marker, a flight simulator, a perception computer, and a data processing and storage server. The XR headset includes multiple data acquisition sensors and multiple sensory feedback units. The flight simulator includes a simulated cockpit, a cockpit data acquisition computer, a flight simulation computer, a computer-generated force computer, and an image rendering computer. The XR headset, the multiple depth cameras, and the in-cabin positioning marker are all located within the simulated cockpit. The XR headset is used to collect video information, voice information, head displacement data and eye movement data in the simulated cockpit, and to display virtual and real holographic images, intelligently labeled visual information, prompting and guiding sounds and vibration prompts. The multiple depth cameras are used to capture the hand gestures and facial images of the trainees inside the simulated cockpit; The in-cabin positioning marker is used to locate the head displacement and posture of the trainee; The cockpit data acquisition computer is used to collect data on the trainee's operational behavior in the simulated cockpit. The flight simulation computer is used to calculate the position and attitude of the aircraft; The computer-generated force computer is used for generating and calculating virtual combat forces, and provides force confrontation situation data in the battlefield environment; The image rendering computer is used to render the external view images in real time and draw holographic annotation visual information. The perception computer is used to output the trainee's personal perception results; the personal perception results include head position tracking, eye tracking, face recognition, speech recognition, hand shape recognition, manipulation behavior perception, machine posture perception, and battlefield situation perception. The data processing and storage server is used to form a knowledge graph and a personnel profile of the trainees based on data knowledge processing, and to store the intelligent perception results data of the flight training process. The data processing and storage server includes: a knowledge graph module, a personnel profiling module, an intelligent sensing module, an intelligent guidance module, and an interactive function module; wherein, The knowledge graph module is used to construct a knowledge graph based on aircraft equipment knowledge, operating standard knowledge, personnel behavior knowledge, and instructor experience knowledge, using ontology technology. The personnel profiling module is used to form static information profiles and dynamic behavioral profiles of the trainees based on training programs, training plans, personal information, and the personal perception results. The intelligent perception module is used to track and recognize the face, head position, eye movement, voice and hand features of the trainee based on the multiple data acquisition sensors and the multiple depth cameras, and to intelligently perceive the trainee's operating behavior, aircraft position and attitude and combat situation based on the cockpit equipment status, aircraft flight status and computer-generated force status. The intelligent guidance module is used to provide intelligent reminders, prompts and generate standard information for the trainee's attention allocation, operation process and behavior, and voice command pronunciation based on the knowledge graph and the personal perception results, by integrating logical rules, vector representation and graph structure features, and using the correlation in the graph structure to perform logical reasoning. The interactive function module is used to provide intelligent guidance information feedback to the trainee based on the XR headset; The intelligent guidance module is also used for: Based on the knowledge graph, text-image synthesis information is automatically generated. By recording training data and operational behavior processes, a connection with a preset rule base is established, and a TransEKGE model consisting of entity embedding vectors and relation embedding vectors is constructed, mapping entities and relations to a low-dimensional vector space. The entity embedding vector and the relation embedding vector are optimized by minimizing the loss function of the incorrect prediction, and the preset rule base is accessed by Jena inference. The personal perception result is compared with the beginning of the total rules of the preset rule base, and finally the inference result is obtained. The reasoning results are converted into a synthetic graphic format that can be used for intelligent behavior guidance and pushed to the XR headset.

2. The extended reality-based simulation training intelligent guidance and holographic annotation system according to claim 1, characterized in that: The data acquisition sensors include a camera, a sound acquisition sensor, an inertial measurement unit, and an eye-tracking acquisition device; The sensory feedback unit includes: a visual display screen, auditory feedback headphones, and tactile vibration feedback device.

3. The extended reality-based simulation training intelligent guidance and holographic annotation system according to claim 1, characterized in that: The depth camera includes an RGBD camera; the multiple depth cameras are respectively located on the left, right and front sides of the simulated cockpit.

4. The extended reality-based simulation training intelligent guidance and holographic annotation system according to claim 1, characterized in that: It also includes a network switch, through which the data processing server communicates with the image rendering computer and the perception computer.

5. The extended reality-based intelligent training guidance and holographic annotation system according to claim 1, characterized in that: The data processing and storage server includes a data processing server, an intelligent processing server, and a data storage server; wherein... The data processing server is used to form a knowledge graph and a personnel profile of the trainees based on data knowledge processing; the knowledge graph includes aircraft equipment knowledge, operating standard knowledge, personnel behavior knowledge, and instructor experience knowledge; the personnel profile includes a static information profile and a dynamic behavior profile of the trainees formed based on the training program, training plan, personal information, and the personal perception results. The intelligent processing server is used to perform intelligent guided reasoning based on the personal perception results, the knowledge graph, and the personnel profile, and to generate intelligent reminders and intelligent annotations during the training process; The data storage server is used to store the intelligent sensing results data of the flight training process.

6. The extended reality-based simulation training intelligent guidance and holographic annotation system according to claim 1, characterized in that: The personnel profiling module is also used for: Based on knowledge in the field of flight instructors, typical trainee pilot tags are constructed. Based on personal academic information, examination results of each course, actual flight training evaluation information, training operation data, and personal activity data, field filtering and preprocessing are performed to obtain data fields that reflect the static information and training behavior of the trainees. Based on the data fields, a knowledge graph for trainee profiles is constructed. Then, the corpus that generates the knowledge graph is represented as word vectors through a semantic model. Based on the calculation of the relevance and similarity of the word vectors, a tag table for the target personnel is generated on the basis of the typical trainee pilot tags, and the static information profile and dynamic behavior profile of the trainee are determined. Based on the static information profile and the dynamic behavior profile, corresponding personalized training plans and training content are generated. Profile data that matches the trainees and simulated training activities is found, and the training focus and bad habits of the trainees are reminded.

7. The extended reality-based simulation training intelligent guidance and holographic annotation system according to claim 1, characterized in that: The interactive function module is also used to: mark the target device switch, and display visual prompts that are virtual and real in a holographic display format on the virtual sign in the distant scene outside the cabin and the real device switch panel inside the cabin, and provide auditory feedback prompts in the form of warning sounds and voice prompts, while providing tactile feedback in combination with the vibration sensor of the head-mounted display.

Images