Xr based evaluation system and method

Abstract

A system for XR-based evaluation of an operator in a mission, comprising: at least one controller; at least one audiovisual display means; at least one input means designated to be controlled by the operator; and at least one sensing means configured to sense various parameters related to the operator. Sensed parameters, including eye movements, are integrated into a means of evaluating the performance of the operator.

Description

[0001] XR BASED EVALUATION SYSTEM AND METHOD

[0002] FIELD OF THE INVENTION

[0003] The present invention relates to the field of evidence-based training, specifically to automatic evaluation systems and methods designated to assess mental capabilities, such as situational awareness and cognitive load management of human operators of various tasks in various operational environments.

[0004] BACKGROUND OF THE INVENTION

[0005] Automatic evaluation systems designated to evaluate operator's mental capabilities such as situational awareness and cognitive skills are beneficial for many reasons. One such reason is the sheer complexity of modem operational environments and the variety of tools and technologies that operators use, which require a high level of situational awareness and cognitive skills. Such skills are essential for operators to make effective decisions, take appropriate actions, and manage risks in a timely and effective manner.

[0006] The nature of many operational environments, such as those experienced by pilots, drivers, soldiers, surgeons, police officers, and other professionals, can be characterized as having high-stress and high-risk circumstances. Under such circumstances, situational awareness and cognitive skills may be impaired by stress, fatigue, and other affecting factors, which may lead to errors, accidents, and even fatalities.

[0007] Common practices and knowledge disclose methods of evaluating operator performance by means of observation by supervisors or peers who are inherently subject to biases and may not provide accurate, objective assessments of operators' mental capabilities and especially those pertaining to situational awareness and other cognitive skills. Example 1: Pilot Training

[0008] Training a flight cadet to operate an aircraft safely and efficiently is a complex and demanding process that involves addressing various difficulties and challenges, including:

[0009] • Technical complexity: Aircraft are intricate machines with numerous systems, controls, and instruments. Learning to operate and manage these systems effectively requires an understanding of aerodynamics, propulsion, avionics, and navigation.

[0010] • Safety procedures and emergency situations: Cadets must be trained to handle various emergency scenarios, from engine failures and instrument malfunctions to weather- related challenges and mid-air collisions. • Simulation vs. real-world experience: Flight simulators are valuable tools for training, but they cannot fully replicate the real-world experience of flying an aircraft. Cadets need to transition from simulators to actual flight, adjusting to differences in sensory input, control responsiveness, and environmental factors.

[0011] • Adapting to new aircraft: Pilots often transition to different types of aircraft throughout their careers, requiring the learning of the nuances of each new aircraft, including its unique systems, controls, and performance characteristics.

[0012] • Physical demands: Flying an aircraft requires precise motor skills and physical endurance to handle the demands of flight, including G-forces, rapid changes in altitude, and challenging weather conditions. • Mental and physical demands: Flying an aircraft requires acquiring and maintaining complex situational awareness in all spatial dimensions and time, quick and accurate decision-making, and the ability to handle high-pressure dynamic situations.

[0013] • Cognitive load management: Flying requires multitasking and managing a significant cognitive load. Cadets must learn to simultaneously monitor instruments, communicate with air traffic control, navigate, and respond to changing conditions while making quick decisions.

[0014] • Spatial awareness and navigation: Developing a strong sense of spatial awareness and effective navigation skills is essential for safe flight. This includes understanding maps, charts, and navigation systems to ensure accurate positioning and route planning.

[0015] • Stress management: The aviation industry can be stressful due to time pressure, long hours, and high levels of responsibility. Flight cadets need to develop effective stress management techniques to maintain their mental and emotional well-being.

[0016] • Communication skills: Effective communication with air traffic control, fellow crew members, and passengers is crucial for safe operations. Cadets must learn and master aviation- specific communication protocols and phraseology.

[0017] • Weather challenges: Weather conditions can change rapidly and significantly impact flight operations. Cadets need to learn how to interpret weather information, make decisions based on weather forecasts, and adapt their flight plans accordingly.

[0018] Instructors and flight training programs may address these challenges by providing comprehensive and structured training that combines theoretical knowledge, hands-on experience, simulator training, and supervised flight. The goal is to gradually build cadets' skills, confidence, and situational awareness to ensure they can operate an aircraft safely and efficiently in various scenarios. Cadet (operator) overall performance monitoring and assessment is key to meeting such challenges. It is appreciated that continued training and practice, even for experienced pilots, require meeting the same or similar challenges.

[0019] Example 2: Firearms Training

[0020] Another example is firearms fundamentals training, aimed at obtaining fast and accurate shooting results and adherence to various safety considerations as presented by law enforcement, military, and / or civilian agencies. Firearm training requires a systemic approach to develop proper visuo-motor mechanics, target perception, and target acquisition skills. Challenges to the training process include:

[0021] • Safety: Safety is paramount in firearm training. Trainees must develop a deep respect for firearms and understand the importance of following safety protocols at all times, as accidental discharges, mishandling, or negligence can have severe consequences.

[0022] • Psychological factors: Handling firearms can induce anxiety, stress, and even fear, particularly for beginners. Overcoming these psychological barriers is crucial to maintain composure and focus during firearm practice.

[0023] • Mental focus and concentration: Shooting accurately demands mental focus, concentration, and mindfulness. Trainees need to learn techniques to maintain mental clarity even during high- stress situations.

[0024] • Physical coordination and motor skills: Shooting accurately demands precise motor skills, hand-eye coordination, and fine motor control. Developing muscle memory for proper stance, grip, sight alignment, and trigger control requires consistent practice. • Recoil management: Managing recoil is essential for maintaining accuracy and control when firing multiple rounds. Trainees need to learn how to anticipate and mitigate recoil while maintaining their shooting stance and sight alignment.

[0025] • Aiming and sight picture: Achieving accurate shots relies on proper aiming techniques and sight alignment. Trainees must understand how to align the front and rear sights with the target and maintain focus on the sight picture.

[0026] • Breathing and trigger control: Controlling breathing and applying steady trigger pressure are critical for accurate shooting. Trainees must learn to time their shots between breaths and smoothly press the trigger without disturbing the sights.

[0027] • Stress and pressure: Trainees need to learn how to shoot accurately under stress and pressure, simulating real-world situations. This could involve scenarios that replicate the pressure of combat, competition, or emergency situations.

[0028] • Dynamic shooting situations: Shooting accurately while moving, transitioning between targets, and engaging multiple threats requires specialized training. Trainees must practice these dynamic shooting skills to be effective in various scenarios.

[0029] • Judgment and decision-making: In law enforcement and self-defense scenarios, trainees must develop the ability to make split-second decisions about whether to shoot, hold fire, or disengage based on the situation's threat level and legality.

[0030] • Environmental factors: Wind, lighting, and other environmental conditions can affect accuracy. Trainees must learn how to adjust their shooting techniques based on these factors. Equipment familiarity: Different firearms have unique features, triggers, and handling characteristics. Trainees must become familiar with the firearms they are using to shoot accurately and safely.

[0031] Effective firearm training may address these and other challenges through a combination of classroom instruction, practical exercises, range practice, scenario -based training, and ongoing skill development. Safety, responsible firearm handling, and ethical considerations remain central to the training process to ensure that trainees can shoot accurately and safely in various contexts. Trainee (operator) overall performance monitoring and assessment is key to meeting such challenges. It is appreciated that continued training and practice, even for experienced firearms operators, requires meeting the same or similar challenges.

[0032] Other operational environments, such as those experienced by drivers, surgeons, firefighters, heavy equipment operators, industrial process operators, and other professionals, present similar challenges. Commonly, each such field is characterized by different methods of monitoring, assessing, and evaluating operator performance, most of which are manual and are carried out by means of observation by supervisors or peers who are susceptible to personal or inherent limitations and biases affecting their assessments of trainees' performance. Such inherent limitations and biases especially affect the observation, monitoring, and thereby assessment and evaluation of the operator's situational awareness and cognitive skills, which in many events may be expressed by minute effects in the performance indicators and parameters of participants such as operators, supervisors, trainees, instructors, assessment specialists, and the like.

[0033] Automatic evaluation systems may address the supervisor's and peer observations' biases and limitations discussed above by providing sustainable feedback to operators, supervisors, trainees, instructors, assessment specialists, and others based on their actual performance. These systems can also help operators learn from their mistakes and develop their skills over time. In addition, automatic evaluation systems may help organizations improve safety and reduce the risk of accidents or errors. Such automatic systems can identify areas where an operator may be struggling and needs to improve and also enable the screening of less capable operators. Thus, such systems may help organizations develop targeted training and support programs to improve overall performance as well as augment operators' personal situational awareness and cognitive skills.

[0034] Some such automatic systems are known in the art. For instance, US 11,287,848 discloses a training apparatus that has an input device and a wearable computing device with a bio-signal sensor and a display to provide an interactive virtual reality ("VR") environment for an operator. The operator is designated to interact with content that is presented in the VR environment. Feedback is given to the operator based on the scores in furtherance of training. The feedback may update the VR environment and may trigger additional VR events to continue training. Another example, US 10,551,826 discloses a method and system designated to increase operator awareness for process control applications providing operator's team realtime information on controlled processes and equipment, immersing them into augmented and virtual reality by a computing system that collects and integrates data from the process and equipment.

[0035] However, the known prior art does not address the automatic evaluation of the mental capabilities, situational awareness, and cognitive skills of operators, supervisors, trainees, instructors, and specialists. Current systems do not conduct assessments of various and complex parameters, or combinations thereof, of operators of a system or of a trainee, which are otherwise undetectable by a human observer such as an instructor, supervisor, or a peer. Thus, there is a need to provide a system and method designated to track various sets of operators' performance, indicators, and parameters, whether minute or in combinations, otherwise not perceivable by a human observer such as an instructor, supervisor, or a peer, providing a platform that can exploit data otherwise not available for assessment and evaluation.

[0036] SUMMARY OF THE INVENTION

[0037] The present invention provides an Extended Reality (XR) based evaluation system and method designated to assess, monitor, and record various data parameters of an operator's visual and / or motor and / or cognitive behavior while taking part in an XR training simulation scenario conducted in an operational environment. According to some embodiments, the XR based evaluation system is further designated to interpret the processed data parameters of specific time intervals and areas of interest that relate to the operational situation and context of the XR scenario. According to some embodiments, the invention may also provide an autonomous evaluation of an operator's minute physiological and / or motor signals pertaining to situational awareness and cognitive skills. According to some embodiments, the said motor signals are eye movement, which are interpreted to autonomously evaluate an operator's perception of the operational environment and thereby assess the operator's situational awareness and cognitive skills.

[0038] The following embodiments and aspects thereof are described and illustrated in conjunction with systems, devices, and methods which are meant to be exemplary and illustrative and not limiting in scope. In various embodiments, one or more of the abovedescribed drawbacks or deficiencies have been reduced or eliminated, while other embodiments are directed to other advantages or improvements. According to one aspect, there is provided an XR based evaluation system, comprising: (i) at least one controller, (ii) at least one display means, (iii) at least one input means designated to be controlled by an operator, (iv) at least one sensing means configured to sense various parameters related to the operator, wherein the system is configured to assess, monitor, and record various data parameters of the operator's visuo-motor behavior while taking part in an XR training simulation scenario conducted in an operational environment, and wherein the system is configured to interpret the data parameters of specific time intervals and areas of interest that relate to the operational situation and context of the XR scenario.

[0039] According to some embodiments, said XR based evaluation system is configured to provide an assessment of the operator's cognitive and visuo-motor skills along with workload or stress-related influences.

[0040] According to some embodiments, said XR based evaluation system's operational environment is configured to be conducted in a full motion 6 degrees of freedom simulator environment or alternatively simulate the operation of aerial, ground, or maritime vehicles or weapons and command and control systems thereof.

[0041] According to some embodiments, said XR based evaluation system's data parameters are utilized to generate a cognitive skill and situational awareness profile of the operator.

[0042] According to some embodiments, said XR based evaluation system's sensing means are at least one of an eye-tracking, heart rate, or EEG sensor(s) integrated within the XR based evaluation system.

[0043] According to some embodiments, said XR based evaluation system's controller is configured to retrieve data signals from the sensing means, classify them using designated algorithms, and process the retrieved data. According to some embodiments, said XR based evaluation system is configured to generate a real-time analysis of the situational awareness and visuo-motor performance parameters of the operator in order to provide immediate feedback. According to some embodiments, said XR based evaluation system is further configured to generate a post-mission analysis of the situational awareness and visuo-motor performance parameters of the operator. Wherein, according to some embodiments, the situational awareness and visuo-motor performance data parameters of the operator are aggregated and compared to previously stored historical data.

[0044] According to some embodiments, said XR based evaluation system's sensing means is configured to sense minute physiological and / or anatomical signals related to visuo-motor performances and / or situational awareness of an operator, such as an eye-tracking sensor(s) configured to capture eye-related parameters of the minute physiological and / or anatomical signals.

[0045] According to some embodiments, said XR based evaluation system operates various simulation scenarios such as a flight simulation designated to evaluate an operator's central and peripheral attention selectivity during flight or a flight simulation designated to evaluate an operator's aerial vehicle control parameters. Wherein, according to some embodiments, the various evaluations aggregated in the various simulations are merged to an overall performance evaluation for each user.

[0046] According to some embodiments, said XR based evaluation system's display means is an XR headset designated to provide an immersive 3 -dimensional experience to the operator. Such immersive experience may, according to some embodiments, be a shooting practice scenario designated to evaluate an operator's sight and gaze dynamics and / or body posture.

[0047] According to some embodiments, as part of said simulation, a at least one audiovisual indication is altered in response to certain sight and gaze dynamics and / or body posture detected by the XR based evaluation system.

[0048] According to some embodiments, the indicia is an at least one target image altered or removed in accordance with the XR based evaluation system configured to interpret non- optimal sight and gaze dynamics and / or body posture. According to some embodiments, the indicia is a firearm sight image altered or removed in accordance with the XR based evaluation system configured to interpret non-optimal sight and gaze dynamics and / or body posture. According to some embodiments, following a shot in a firearm training simulation, indicia is a feedback containing various parameters related to the operator's performance and is presented to the operator. According to some embodiments, the feedback is a gaze distribution indication of a particular operator's gaze point at the time of shooting. According to some embodiments, the feedback is a field of sight indication presenting an image captured by each eye of an operator. According to some embodiments, the feedback is a head position indication presenting data regarding the head position of the operator before, during, and / or after shooting. According to some embodiments, the feedback is a combined snapshot of data gathered regarding the operator's hands, eyes, firearm, and head relative positions.

[0049] According to a second aspect, there is provided an XR based evaluation method, comprising the steps of: (i) utilizing at least one display means in order to present a particular XR scenario to an operator in an operational environment, (ii) providing at least one input means designated to be controlled by the operator, (iii) providing at least one sensing means configured to sense various parameters related to the operator, (iv) utilizing at least one controller to assess, monitor, and record various data parameters of the operator's visuo-motor behavior while taking part in the XR training simulation scenario in an operational environment, wherein the system is configured to interpret the data parameters of specific time intervals and areas of interest that relate to the operational situation and context of the XR scenario.

[0050] BRIEF DESCRIPTION OF THE FIGURES

[0051] Some embodiments of the invention are described herein with reference to the accompanying figures. The description, together with the figures, makes apparent to a person having ordinary skill in the art how some embodiments may be practiced. The figures are for the purpose of illustrative description and no attempt is made to show structural details of an embodiment in more detail than is necessary for a fundamental understanding of the invention.

[0052] In the Figures: FIG. 1A constitutes a schematic perspective view of a general XR evaluation system, according to some embodiments of the invention.

[0053] FIG. IB constitutes a schematic perspective view of a task-specific simulation XR evaluation system, according to some embodiments of the invention.

[0054] FIG. 1C constitutes a schematic perspective view of a task-specific combat training simulation XR evaluation system, according to some embodiments of the invention.

[0055] FIG. ID constitutes a schematic perspective view of a flight simulation XR evaluation system, according to some embodiments of the invention.

[0056] FIG. 2 constitutes a schematic flow chart of the various operations designated to be conducted by an XR based evaluation system according to some embodiments of the invention. FIG. 3 constitutes a schematic flow chart depicting the information processing operations conducted by an XR based evaluation system, according to some embodiments of the invention.

[0057] FIG. 4 constitutes a schematic flow chart depicting the assessment of various domain expertise conducted by the XR based evaluation system, according to some embodiments of the invention.

[0058] FIGS. 5A-5B constitute a perspective view of an XR based evaluation system designated to simulate flight sessions, according to some embodiments of the invention.

[0059] FIGS. 6A-6B constitute a perspective view of a shooting practice training simulation that uses an XR based evaluation system, according to some embodiments of the invention.

[0060] FIG. 7 constitutes a perspective view of a shooting practice training simulation that uses an XR based evaluation system, according to some embodiments of the invention.

[0061] FIGS. 8A-8D constitute a perspective view of a shooting practice training simulation that uses an XR based evaluation system, according to some embodiments of the invention.

[0062] DETAILED DESCRIPTION OF SOME EMBODIMENTS

[0063] In the following detailed description, numerous specific details are set forth in order to provide a thorough understanding of the invention. However, it will be understood by those skilled in the art that the present invention may be practiced without these specific details. In other instances, well-known methods, procedures, and components, modules, units, and / or circuits have not been described in detail so as not to obscure the invention. Some features or elements described with respect to one embodiment may be combined with features or elements described with respect to other embodiments. For the sake of clarity, discussion of same or similar features or elements may not be repeated.

[0064] Although embodiments of the invention are not limited in this regard, discussions utilizing terms such as, for example, "controlling," "processing," "computing," "calculating," "determining," "establishing," "analyzing," "checking," "setting," "receiving," or the like, may refer to operation(s) and / or process(es) of a controller, a computer, a computing platform, a computing system, or other electronic computing device, that manipulates and / or transforms data represented as physical (e.g., electronic) quantities within the computer's registers and / or memories into other data similarly represented as physical quantities within the computer's registers and / or memories or other information non-transitory storage medium that may store instructions to perform operations and / or processes.

[0065] Unless explicitly stated, the method embodiments described herein are not constrained to a particular order or sequence. Additionally, some of the described method embodiments or elements thereof can occur or be performed simultaneously, at the same point in time, or concurrently.

[0066] The term "Controller," as used herein, refers to any type of computing platform or component that may be provisioned with a Central Processing Unit (CPU) or microprocessors, and may be provisioned with several input / output (VO) ports, for example, a general-purpose computer such as a personal computer, laptop, tablet, mobile cellular phone, controller chip, SOC, or a cloud computing system.

[0067] The term "XR (simulation, evaluation, and system)," as used herein, refers to a general term generally known as Extended Reality that encompasses augmented reality (AR), virtual reality (VR), and mixed reality (MR). While all three 'realities' share common overlapping features and requirements, each has different purposes and underlying technologies. XR encompasses a spectrum of technologies including virtual reality (VR), augmented reality (AR), and mixed reality (MR), aiming to create a seamless and dynamic environment that engages operators and enables them to interact with digital objects and information as if they were part of their physical surroundings. Real-time XR may refer to an immersive and interactive digital experience that blends elements of the physical and virtual worlds, where computer-generated content, such as 3D graphics, sounds, and other sensory stimuli, are overlaid onto an operator's perception of reality in real-time. This technology has applications in various fields, from entertainment and gaming to education, training, healthcare, and beyond, enhancing the way people perceive and interact with their surroundings.

[0068] The term "Sensor," as used herein, refers to any sort of sensing device able to sense, detect, or measure a physical property or otherwise respond to it, including the tracking of the participant's bearings or capturing images thereof or detecting any other kind of parameters during an XR scenario which can be a VR / AR scenario or mixture thereof.

[0069] The term "Eyetracker," as used herein, refers to a sensor specifically designated to detect and measure the position and orientation of eyeball anatomy, human or otherwise, separately for each eye.

[0070] The term "Fixation" refers to the stable period during which the eyes remain relatively still and focused on a specific object or point in the visual field. During fixation, the eyes are able to gather detailed visual information from the selected target. This is crucial for tasks such as reading, observing fine details, focusing on a target, and maintaining steady vision on a specific location.

[0071] The term "Eye saccades" or "saccades," as used herein, usually refers to a rapid movement of the eyes from one point to another, commonly between one fixation to another. This movement may serve the purpose of redirecting the line of sight to focus on different objects or points of interest. Saccades are essential for exploring the visual environment and for gathering information from various parts of a scene. These movements are controlled by a complex interplay of neural circuits involving the brain and the muscles that control eye movement.

[0072] The term “mission” as used herein, refers to a single specific training program in which an operator uses the system of the claimed invention to complete a series of specific tasks within a specific environment.

[0073] The term “XR” as used herein, refers to “extended reality”, a class of technologies including AR (augmented reality), VR (virtual reality), and MR (mixed reality), all of which relate to virtually creating or adapting three dimensional scenarios in which an operator operates.

[0074] The alternation between saccades and fixations is a fundamental aspect of how humans perceive the visual world. Saccades rapidly shift the gaze from one point to another, while fixations provide the necessary time for the brain to process and analyze the visual information obtained during these periods of stability. For instance, eye movement would range in amplitude from small minute movements while reading to much larger movements while gazing, whereas in both events, saccades and fixation will be employed by the brain. This combination of rapid eye movements and stable fixation contributes to an ability to navigate, explore, attack, navigate, and interpret the surrounding visual environment. In the claimed invention, the distinction between saccades and fixations represents meaningful data relating to the performance of the operator in comprehending the task at hand.

[0075] Persons skilled in the art will appreciate that other eye movements are also common amongst users, in particular: smooth pursuit movements; vergence movements; and vestibuloocular movements, which relate to: smooth tracking of a moving object; alignments of the fovea of each eye on a single object at different distances from each eye (i.e. to enable stereo vision); and stabilization of vision to compensate for head movement. Both vergence and vestibulo-ocular movements represent entirely involuntary reflexes to maintain visual comprehension with the dynamic optical conditions imposed upon eyes by the human body. Smooth pursuit movements represent voluntary eye movements that can, in some limited cases to which the claimed invention applies, represent a special case of fixations, because the eyes are voluntarily fixated on an object, even though said object is itself moving.

[0076] According to some embodiments, the claimed invention discloses an XR evaluation system for assessing and evaluating the situational awareness and cognitive skills of operators in various operational environments.

[0077] According to some embodiments, said system and method are configured to interact with mission-oriented operators that may be using operational systems such as screen-based, VR / AR / XR (Extended Reality), or full-motion six degrees of freedom (6-DOF) training simulators, vehicles, aircrafts, shooting ranges, and others, as broadly disclosed below.

[0078] According to some embodiments, the claimed invention provides real-time or postmission feedback / analysis to operators (such as trainee or supervisors) on their situational awareness and cognitive and operational skills during training or operational activities and may assist operators in improving their performance and increasing their effectiveness in high-stress situations. The system can be applied to a wide range of operators, including pilots, drivers, soldiers, surgeons, police officers, and other professionals who operate in high-stress environments.

[0079] According to some embodiments, by closely monitoring the operator's performance and various parameters and providing feedback, the claimed invention can help identify areas where the operator needs to improve and provide specific guidance. According to some embodiments, the claimed invention can provide accurate feedback and guidance to operator(s) not otherwise observable by them or by supervisors or peers, Said feedback is based on input data not available by simple observation. Said input data for the invention includes saccades, which can provide insights into a number of parameters relating to operator performance, for example: the levels of specific and generalized stress; situational awareness; cognitive load and fatigue, among others. In embodiments of the invention relating to marksmanship training missions, eye tracking data can be used to determine performance such as in : target identification; target discrimination; target acquisition (including sight alignment; sight picture; and sight focus) quiet eye; visual target transition, among many more. Providing such new, until now unavailable information can be of high significance in improving training and guidance processes.

[0080] According to some embodiments, the claimed invention may also be used to evaluate operators' cognitive skills during operations involving a number of operators operating together in stressful environments, which can be discernible through measurement and evaluation of many features of saccade-related eye movements, for example erratic scanning or tunnel vision. For example, the system may monitor a group of police officers having an interaction with a suspect or multiple suspects during a stressed operational situation and provide feedback on their communication skills and ability to remain calm under pressure.

[0081] According to some embodiments, there is provided an XR based evaluation system and method comprising at least one controller designated to control and operate the evaluation process, analyze results, and suggest recommendations as further disclosed below and at least one display means designated to provide a visual input to an operator who is being evaluated. According to some embodiments, the controller may be designated to retrieve data signals from a sensing means, classify it using designated algorithms, and process the retrieved data. According to some embodiments, various visual display devices may be configured to present visual data to operators operating the XR based evaluation system. For example, various monitors which may be used to display simulation output data such as videos, images, charts, graphs, etc., may be used. According to some embodiments, designated projectors may also be used in larger-scale simulations or virtual reality environments in order to project images onto a screen or a physical space to a single or multiple operators and or evaluators / peers. According to some embodiments, XR head-mounted displays (HMDs or headset) may be used and utilized in order to allow operators to view or be immersed in a simulated environment and / or manipulate objects in a more immersive / realistic manner.

[0082] According to some embodiments, XR (or VR / AR) headsets may be utilized in order to create a fully immersive 3D virtual environment / scenario for an operator. For example, XR devices may overlay digital information onto the operator's real-world view, allowing said operator to experience both the physical world and the simulated objects or data at the same time.

[0083] According to some embodiments, the XR based evaluation system comprises at least one input means designated to be controlled by an operator. For example, a joystick or a real- life driving / navigating simulation means may be used in simulators. For instance, for flight simulation, a joystick can allow the operator to control the pitch, roll, and yaw of the simulated aircraft; a steering wheel and pedals may be used in driving simulators to control the steering, acceleration, and braking of a simulated vehicle; keyboard and mouse may be used in computer-based simulators for a variety of purposes, such as controlling the camera view, selecting menu options, and entering text input; or a gamepad may include buttons, triggers, touchscreens, and thumbsticks for controlling various aspects of the simulated environment, and more. Voluntary smooth pursuit movements can, in limited cases, represent a special case of fixations in the claimed invention. One of said limited cases is the smooth pursuit movements of the eye following a moving object in the operators view, wherein said object is either directly controlled by an input means, or wherein said object is of operational interest.

[0084] According to some embodiments, the simulated environment may be configured to be experienced as the actual operational environment such as aerial, ground, or marine mediums.

[0085] According to some embodiments, the XR based evaluation system is further designated to generate a real-time analysis of the situational awareness and visuo-cognitive performance data parameters of the operator in order to provide immediate feedback.

[0086] According to some embodiments, a variety of other devices may be used as part of the XR based evaluation system in order to monitor and assess the operator's bearings and operations. For example, motion capture devices may be used to track the operator's movements and translate them into the simulation. For example, in a virtual reality simulation, a motion capture device could track the operator's hand movements in order to control the position of virtual objects, etc.

[0087] According to some embodiments, the devices and technologies specified above represent just a few examples of input technologies that may be used during a simulation in a simulator. The specific input devices used will depend on the nature of the system being simulated and the goals of the simulation. According to some embodiments, control inputs may be among the input means that may be used during a simulation. Control inputs are inputs that may be used to control the behavior of the system being simulated, such as in a flight simulator, where the control inputs could be the position of the joystick, throttle, and rudder pedals, and others. According to some embodiments, control inputs may be environmental inputs that may represent the environmental conditions that affect the system being simulated. For example, in a traffic simulation, the environmental inputs could be the traffic volume, weather conditions, and road conditions.

[0088] Other input means that may be used during a simulation may be configured to represent events that may occur and trigger specific behaviors expected or not expected to be conducted by the operator. For example, in a healthcare simulation, an event trigger could be a patient's vital signs reaching a critical level, which would trigger a response from the medical team. According to some embodiments, input means may be operator inputs that may be provided by the operator interacting with the simulator. For example, in a virtual reality simulation, the operator's inputs could be the movements and actions of the operator within the virtual environment. According to some embodiments, stimulation means may affect the operator's response to various stimulations that may be provided through haptic sensation means. In embodiments of the invention related to marksmanship training missions, haptic means might include recoil simulation on a simulated weapon input means. In embodiments of the invention relating to operating a vehicle, haptic means might include simulations of resistance and vibration on a simulated steering wheel or joystick. In some embodiments of the invention, haptic means might include gloves that provide resistance to simulate holding an item. Further specific inputs may be used depending on the nature of simulated operation and the goal(s) of the simulation.

[0089] According to some embodiments, the XR based evaluation system comprises at least one sensing means configured to sense various parameters related to the operator. Among the sensing means that may be used during a simulation can include heart rate monitor that may be used to monitor the operator's heart rate and detect any changes in their cardiovascular system during the simulation; electroencephalogram (EEG) sensors may be used to monitor the operator's brain activity and detect any changes in their cognitive function or level of alertness; eye-tracking sensors that may be used to monitor the operator's eye movements and determine where he / she is looking during a simulation; respiratory rate monitor that may be used to monitor the operator's breathing rate and detect any changes in their respiratory function during the simulation; skin conductance sensor that may measure the electrical conductivity of the skin and can be used to monitor the operator's emotional response to the simulation; motion sensors that may be used to monitor the operator's movements and posture during the simulation and assist in identifying areas where the operator may be experiencing discomfort or struggling to maintain a proper position; temperature sensor that may be used to monitor the operator's body temperature and detect any changes in their thermoregulatory system during the simulation; blood pressure monitor that may be used to monitor the operator's blood pressure and detect any changes in their cardiovascular function during the simulation; oxygen saturation sensor(s) that may measure the amount of oxygen in the operator's blood and can be used to monitor their respiratory function during the simulation; galvanic skin response sensor that may measure the electrical conductivity of the skin and can be used to monitor the operator's stress levels during the simulation. Such and other sensing means may be employed to obtain an overall comprehensive scope of operators' parameters and provide for the identification of areas where the operator may be struggling or missing important information. According to some embodiments, the selection of specific measured parameters can suffice to derive conclusions relevant to certain task-specific operations.

[0090] Specifically, tracking eye movements and where the eyes linger can provide insights into a person's visual attention, gaze patterns / dynamics, and overall visual behavior. Eye tracker devices may be used in fields like psychology, human-computer interaction, market research, usability testing, training, and more. According to some embodiments, the XR based evaluation system is designated to assess, monitor, and record various data parameters of the operator's visual, motor, and cognitive behavior while taking part in an XR training simulation scenario conducted in an operational environment.

[0091] According to some embodiments, the XR based evaluation system is further configured to interpret the processed data parameters in view at specific time intervals and of specific areas of interest that relate to the operational situation and context of the XR scenario. According to some embodiments, the system also provides an automatic evaluation of operator situational awareness and cognitive skills.

[0092] According to some embodiments, various tasks may be designated to be conducted by an operator operating the XR based evaluation system. For example, mission-oriented tasks pertaining to various professionals, such as pilots, drivers, soldiers, surgeons, police officers, operators of machinery, workers in control rooms, flight, marine, terrestrial traffic controllers, and others, etc., may include the operator interacting with an XR based evaluation system employing operational simulation means including screen-based presentation configuration, VR / AR / XR headset, and / or full-mission 6-DOF training simulators configured to simulate various operational environments in various mediums.

[0093] According to some embodiments, the system may also be applied to cognitive skills' assessment of an operator's social interactions and mutual tasks evaluation in stressful environments such as the environments disclosed above.

[0094] According to some embodiments, the system and method disclosed and cited in the claimed invention may include automatically collecting and processing data collected by various sensors integrated into the XR based evaluation system, synchronizing the processed data with data from the operational system and the environment, and generating a cognitive skill and situational awareness profile of the operator. According to some embodiments, data may also be collected from various integrated bio-sensors such as eye-tracking sensors, heart rate, or EEG monitors that are integrated into the XR based evaluation system, or coupled or thereto.

[0095] According to some embodiments, the invention comprises a controller which can comprise an analytics system, an integration system, and an interface system, or a combination thereof. According to some embodiments, the integration system is in communication (such as by a network) with the interface system, the processing system, and the analytics system. According to some embodiments, the processing system is configured to retrieve the signal data from the integrated bio-sensors and to filter them using advanced classification algorithms (algorithms that may be designated to classify or categorize the recorded signals or patterns into specific classes or categories).

[0096] According to some embodiments, the analytics system is configured to interpret the processed signal data in view of specific time intervals and areas of interest that relate to the operational situation and context. According to some embodiments, the situational awareness and cognitive performance assessment data gathered during a session / task may generate a concluding summary at the end of each session / task for immediate review and feedback purposes. According to some embodiments, such a summary may be generated in real-time during the session or post-mission for adaptive training and enhanced human-machine interfacing.

[0097] According to some embodiments, the situational awareness and cognitive performance data gathered by the XR based evaluation system may be aggregated and compared to previously stored historical data in order to provide track patterns, learning and improvement over time, and for benchmarking purposes, etc. According to some embodiments, the XR based evaluation system is configured to provide an assessment of the operator's cognitive & visio-motor skills and workload-related influences. According to some embodiments, stress-related influences may also be assessed by the XR based evaluation system.

[0098] Reference is made to FIGS. 1A-1D, which constitute a perspective view of a general XR based evaluation system 10, according to some embodiments of the invention. As shown, XR based evaluation system 10 may comprise some main components:

[0099] A controller 100, such as a computer configured to execute an XR scenario and may control various devices forming a part of the XR based evaluation system 10. According to some embodiments, controller 100 may be a stand-alone device or may be integrated into a sensory apparatus 200 (disclosed hereinafter).

[0100] At least one sensory apparatus 200 configured to produce sensory output to be perceived by at least one operator and that may be associated with each one of such operators (for example, participant A depicted in FIG. 1). According to some embodiments, personal sensory apparatus 200 may comprise a head-mounted display (HMD) or headset 202 configured to produce images to be perceived by participant A associated with said HMD 202.

[0101] According to some embodiments, HMD 202 may be VR / MR / XR headset that typically implements advanced technology to create an immersive digital environment for operators. By combining high-resolution displays, motion tracking sensors, and often audio components, HMD 202 simulates the sensation of being present in a computer-generated three-dimensional world.

[0102] Operators can interact with and navigate through this virtual environment, feeling a sense of presence and immersion that can range from mild to intense, depending on the quality and capabilities of the headset 202. According to some embodiments, a VR / MR / XR headset such as Pico Neo Pro 3 Eye VR headset or similar hardware may form a part of XR based evaluation system 10 and provide the immersive experience required for the operation of the system.

[0103] According to some embodiments, HMD 202 may produce 3D images that simulate a space 20. According to some embodiments, space 20 may facilitate the enactment of any type of scenario in which participant A operates a simulation.

[0104] According to some embodiments, personal sensory apparatus 200 may comprise auditory means 204 such as earphones, a microphone, or speakers configured to enable participant A to communicate with other participants of the immersive system 10 and / or with a supervisor / instructor (not shown).

[0105] According to some embodiments, the personal sensory apparatus 200 may be equipped to engage with haptic means 208 (disclosed hereinafter) which may be configured to provide tactile stimuli to participant A in order to simulate an actual situation. According to some embodiments, haptic means 208 may be installed on simulation-related tactic paraphernalia.

[0106] At least one sensing means 206 configured to sense the bearings of participant A. According to some embodiments, said determined bearings may serve in simulating participant A's orientation. According to some embodiments, sensing means 206 may be, for example, a motion or a tracking sensor configured to detect the bearings of participant A or, alternatively, the bearings of some parts of the participant A's body, wherein these bearings are then simulated as part of an XR scenario.

[0107] According to some embodiments, sensing means 206 may sense various parameters associated with participant A. These parameters may include, for example, the body temperature, sweating level, breathing rate, hyperventilation level, brain waves activity, eye movements and patterns, nervous system electric rate, etc. According to some embodiments, sensing means 206 may also sense the vital signs of participant A while immersed in an XR scenario.

[0108] According to some embodiments, controller 100 can calculate and determine, according to the parameters or vitals detected by sensing means 206, the well-being (e.g., physical and mental) status of participant A. According to some embodiments, sensing means 206 may be integrated into the personal sensory apparatus 200 or a sensing device separate thereof.

[0109] According to some embodiments, haptic means 208 may be configured to provide tactile sensation in accordance with a specific XR scenario. According to some embodiments, haptic means 208 may be implemented as part of the personal sensory apparatus 200 and configured to provide tactile stimuli to participant A. According to some embodiments, haptic means 208 may be implemented as part of participant A's clothing. For example, haptic means 208 may be implemented as part of a tactical vest and can, for example, simulate a projectile hitting participant A at his / her upper torso.

[0110] According to some embodiments, haptic means 208 may be implemented in a variety of real-life accessories used by participant A during an XR scenario. Such an accessory can be, for example, an adapted weapon such as an assault rifle, a pistol, etc.

[0111] According to some embodiments, haptic means 208 implemented in real-life accessories such as an assault rifle or a pistol, may provide the participant A with a realistic tactile sensation, for example, a real-life feel of firing live ammunition. This may be achieved by the mimicking of the recoil force associated with the use of live ammunition or by the mimicking of the sensation of a magazine replacement or loading operation. According to some embodiments, XR based evaluation system 10 may be installed in a cockpit of an actual aircraft in order to track, monitor, and evaluate an operator (for example, a pilot or a flying cadet) during real-time flight and / or while performing flight-related tasks. According to some embodiments, eye tracker 210 may be configured to track the operator's eyes and gaze dynamics in order to provide real-time assessment and monitoring as explained above.

[0112] Reference is made to FIG. 2, which schematically illustrates a flow chart depicting the various operations / components designated to be conducted and exercised by the XR based evaluation system 10, according to some embodiments of the invention. As shown, an admin / operator 102, who may be, for example, a pilot, a combatant, a firefighter, an operator of heavy machinery, or a sports instructor, is designated to take part in a simulation of any kind and use dashboard 104, which may provide an interface or an operation platform enabling the admin / operator 102 to control and operate the XR based evaluation system 10. According to some embodiments, admin / operator 102 may be an administrator of the system and its scenario, such as a flight simulation coordinator, a shooting instructor, or other types of administrators otherwise using and facilitating dashboard 104.

[0113] According to some embodiments, Application Programming Interface (API) 106 may comprise a set of protocols, routines, tools, and definitions that allow different software applications to communicate and interact with each other. APIs 106 may define how software components should interact, making it easier for an operator to integrate different operations or tasks, without needing to understand the underlying complex code.

[0114] According to some embodiments, APIs 106 may serve as intermediaries that enable applications to request and exchange data or functionality from external sources, such as web services, databases, or other software components, and provide a standardized way to access specific features or data without needing to know the internal details of how those features may be implemented. APIs 106 may also allow developers to leverage existing functionalities, services, or data from other applications, which may save time, promote code reusability, and enable the creation of more robust and feature-rich software applications by building on top of the work done by others. For example, APIs 106 may provide a convenient platform allowing admin / operator 102 to conduct training of any kind specified above.

[0115] According to some embodiments, gaze tracking 108 may be a process in which a device or system measures and adjusts its tracking mechanisms to accurately determine the direction of a person's gaze dynamics or eye movement. It may involve establishing a mapping between the operator's visual focus and the corresponding point in the digital or physical environment being observed.

[0116] According to some embodiments, gaze tracking 108 may be crucial for ensuring accurate and reliable interactions in various technologies, especially those related to eyetracking systems, virtual reality (VR), augmented reality (AR), and extended reality (XR) simulations.

[0117] According to some embodiments, gaze tracking 108 may be used in applications such as:

[0118] • Eye-tracking Systems: In applications that use eye-tracking technology, such as assistive communication applications or training simulations, gaze tracking 108 may be an essential tool in order to accurately determine where an operator is looking on a screen. This calibration process ensures that the system can interpret the operator's intentions correctly, enabling the operator to select items, navigate menus, communicate, apply various devices, etc., by effectively using their gaze while on the other hand enabling the drawing of conclusions from the operator's gazing regiments. • Virtual Reality (VR), Augmented Reality (AR), and Extended Reality (XR): Gaze tracking 108 may be crucial in VR, AR, and XR environments in order to create a realistic and immersive experience. By calibrating the system to the operator's eye movements, these technologies can accurately render virtual objects or information in alignment with the operator's visual focus. This ensures that virtual objects respond correctly to the operator's gaze, enhancing the sense of presence and interaction within the digital environment.

[0119] • Usability monitoring: Gaze tracking 108 may be used in usability testing scenarios to analyze how operators interact with a user interface, other operators, etc., by tracking where an operator is looking and monitoring an operator's behavior patterns and potential usability issues.

[0120] • Driver / pilot assistance and safety: Gaze tracking 108 may play a role in monitoring operator attention and alertness. According to some embodiments, the system may detect if an operator, for instance, a driver or a pilot, is not alert or is drowsy, potentially triggering warnings or interventions to enhance safety.

[0121] According to some embodiments, gaze and pupil data capture 110 may be a process used to measure and analyze the movements of a person's eyes and the changes in their pupils' size and shape. This technology may be employed in various fields, including psychology, neuroscience, human-computer interaction, training, aviation, and transport in order to gain insights into an individual's visual attention, cognitive processes, and emotional responses.

[0122] According to some embodiments, gaze and pupil data capture 110 may be used in applications such as: Gaze tracking: Gaze tracking involves monitoring the direction and movement of a person's eye gaze as they look at different objects or areas within their field of view. This may be achieved by using specialized devices like eye trackers, which may use infrared cameras and other sensors to monitor the position of the eyes and calculate where a person is looking on a screen or in their environment. Gaze and pupil data capture 110 may provide data providing valuable information about visual attention, task performance, and decision-making processes.

[0123] Pupil dilation analysis: Gaze and pupil data capture 110 may include pupil dilation, which refers to the expansion and contraction of the pupil in response to changes in light levels, cognitive processes, and emotional states. Pupil dilation may be measured and analyzed to infer various aspects of a person's mental and emotional state. For example, larger pupils might indicate heightened interest, cognitive effort, or arousal, while smaller pupils could suggest lower levels of interest or cognitive load.

[0124] According to some embodiments, Gaze and pupil data capture 110 integrated with an XR evaluation system may have various implications in psychology and neuroscience. For example, Gaze and pupil data capture 110 may help researchers study cognitive processes such as attention, perception, memory, and decision-making. It may also provide insights into how the brain processes visual information and responds to different stimuli.

[0125] According to some embodiments, Gaze and pupil data capture 110 may provide enhanced human-computer interaction (HCI) and may be used to develop more intuitive and efficient operation interfaces. For example, Gaze and pupil data capture 110 may enable devices to respond to an operator's eye movements, allowing for hands-free control and enhancing accessibility. According to some embodiments, Gaze and pupil data capture 110 may have implications in driving and aviation safety and / or training in such fields. For example, gaze tracking may be utilized to monitor and improve drivers' attention levels on the road and pilots' focus during flight, contributing to improved safety measures and improved training techniques.

[0126] According to some embodiments, data storage 112 may be configured to store the data collected by the XR based evaluation system 10 and store it on a designated medium and / or on a remote service such as a cloud computing service.

[0127] According to some embodiments, raw data 114 may be collected and stored in data storage 112 and may provide the basis for the operation of XR based evaluation system 10. For example, raw data 114 may comprise data collected from the use and operation of the operator of or in platform simulator 116, which may be, for example, a flight, drive, combat, shooting, or any other simulating device or apparatus that simulates an operational environment experience.

[0128] According to some embodiments, operator 118 may be connected or tracked by various sensors such as eye tracker 120 and additional sensors 122, which may be EEG, ECG, GSR, that may interact with API 106 (comprising a set of protocols, routines, tools, and definitions that allow different software applications to communicate and interact with each other, as elaborated above).

[0129] According to some embodiments, tracker 120 and sensors 122 may be included in sensory apparatus 200 (FIG. 1).

[0130] According to some embodiments, additional data sources 124 may also be employed in order to provide additional data and valuable data having significance to the operation of the XR based evaluation system 10. According to some embodiments, data management infrastructure 126 may be configured to manage and control the various data sources and prioritize various operations' analysis and decisions.

[0131] According to some embodiments, operator 118 may be designated to operate within the XR based evaluation system 10, wherein the operation and sensors' reading of operator 118 are configured to be accumulated in raw data 114 or data storage 112 to be filtered in accordance with various parameters including, for example, time measurements, saccades, fixation optionally with other collected bio-sensory data while employing designated algorithm(s) such as advanced classification algorithms to aggregate data matrices to be used in the evaluation and assessment according to some embodiments.

[0132] According to some embodiments, the above-mentioned matrices are evaluated and interpreted in accordance with various scenarios. For example, data may flow in accordance with the flowchart appearing in FIG. 3 as further disclosed below.

[0133] Reference is made to FIG. 3, which schematically illustrates a flow chart depicting the information processing operations conducted by the XR based evaluation system 10, according to some embodiments of the invention. As shown, data from various bio-sensor(s) 301 is designated to be gathered by the XR based evaluation system 10. For example, in the context of flight evaluation, blood pressure, heart rate, skin conductivity, gaze, and eye movements, etc., may be measured and gathered in order to evaluate the mental and physical state of an operator as well as their cognitive workload 302.

[0134] According to some embodiments, various parameters may also be evaluated, such as various situational awareness parameters 304. For example, contextual focus may be evaluated by XR based evaluation system 10 in order to evaluate patterns and points of attention during a simulation scenario. According to some embodiments, sight scan dynamics and peripheral awareness may also be evaluated. According to some embodiments, central and peripheral attention selectivity of an operator while operating / controlling the aerial vehicle may also be evaluated.

[0135] According to some embodiments, vehicle control parameters 306 may also be evaluated, such as operations regarding altitude, airspeed, altitude, flight path, etc. according to some embodiments, the operator's workload 308 may be accessed and taken into account. For example, in a flight simulation scenario, workload would be determined by tasks such as routine flight checkups and readings, etc.

[0136] According to some embodiments, the above readings and evaluations may be analyzed and merged by the XR based evaluation system 10 in order to determine the overall performance 310 of the operator. For example, the objective completion, danger avoidance, operations order, and tactical behavior may comprise an overall performance numerical score.

[0137] According to some embodiments, such operations and evaluations disclosed above may also be implemented in various professional fields that require training and monitoring in order to improve performance and capabilities.

[0138] Reference is made to FIG. 4, which schematically illustrates a flow chart depicting the assessment of various domain expertise conductible by the XR based evaluation system 10, according to some embodiments of the invention.

[0139] As shown, a raw recording section may be configured to record various logs and recordings of an operator's performance while using the XR based evaluation system 10. According to some embodiments, XR based evaluation system 10 may comprise at least three domains: assessment domain, model training domain, and sanity, validation, and debug domain. According to some embodiments, various commands are designated to enable the operation of the XR based evaluation system 10. For example, TOIs commands are dedicated to calculating the Time of Interest, and the matrices gathered are then used to indicate AOI (Areas of Interest) for each TOI, etc. KPIs are designated to indicate key performance indicators and represent a quantifiable measure of performance over time for a specific objective. According to some embodiments, the commands depicted in FIG. 4 are designated to ultimately provide a detailed user assessment report indicating various user performance parameters during a task.

[0140] According to some embodiments, the commands depicted in FIG. 4 are designated to ultimately provide a cohort report specifying the performance of a group of operators sharing some characteristic. For example, a group of pilot cadets may be evaluated together while performing similar task(s). According to another example, a group of shooting trainees may be evaluated together while performing similar task(s), etc.

[0141] According to some embodiments, a sanity and validity report is designated to assess the integrity and quality of data or the results of an analysis of operator(s) performance. A sanity and validity report may contain a data overview summarizing a dataset, including the size, structure, and key variables or parameters, and a sanity check of said dataset for internal consistency, logical coherence, and basic data integrity. A validity check may assess the validity of the data or analysis in terms of its relevance to particular parameters being evaluated.

[0142] Reference is made to FIGS. 5A-5B, which schematically illustrate a training simulation environment that uses XR based evaluation system 10, according to some embodiments of the invention. As shown, an operator may be located in front of a display means and conduct various operations as part of a training session. According to some embodiments, various third- party hardware components may be used as part of XR based evaluation system 10, among them are: An eye tracker device (505A or 505B): Configured to monitor and analyze the movement and focus of an operator's eyes and may use various methods, such as infrared sensors or cameras 505B, or HMD 505A installed eye-tracking sensors to capture the positions of the eyes' gaze points and the movements of the pupils. According to some embodiments, an integrated eye-tracker by Tobii or a similar device may be used as part of the XR based evaluation system 10 and be mounted, for example, on a standard PC display, as exemplified by 505B.

[0143] Joystick 510 and throttle 520: In the context of PC flight simulations, a joystick and / or throttle are input devices that mimic the controls found in real aircraft cockpits, enhancing the immersion and realism of the simulation experience. According to some embodiments, a joystick (such as the one produced by Thrustmaster, etc.) is a hand-held device with a pivot mechanism that allows operators to simulate control of an aircraft's pitch, roll, and yaw movements. Typically, joysticks also feature buttons, triggers, and switches that can be customized to perform different functions within the simulation, such as firing weapons, managing flaps, or activating communication systems. According to some embodiments, a throttle 520 (such as the one produced by Thrustmaster, etc.) is a separate control unit used to adjust the engine's power output and control the aircraft's speed. It typically resembles the throttle lever found in a real aircraft cockpit and can be pushed forward to increase thrust or pulled back to reduce it. Throttle controls often include additional levers or knobs for managing engine settings like propeller pitch, mixture, and thrust reversers.

[0144] According to some embodiments, joystick 510 and throttle 520 forming a part of an XR based evaluation system 10 may provide an operator with a more tactile and intuitive experience to interact with the simulated aircraft. By closely replicating the controls of a real aircraft, such devices contribute to a more authentic flight simulation experience. Reference is made to FIGS. 6A & 6B, which schematically illustrate a shooting range training simulation environment 60 that uses XR based evaluation system 10, according to some embodiments of the invention. As shown, an operator may use a designated display means such as a standard display or a VR / MR / XR headset (such as HMD 202, FIG. 1), which is configured to simulate a shooting range scenario in a practice arena. According to some embodiments, an operator is designated to conduct shooting training in a certain manner. For example, an operator is designated to look at target(s) 602 while both eyes are open. In such a case, the XR based evaluation system 10 is configured to display clear targets for the operator to shoot at while targets 602 may bear a designated at least one audiovisual indication such as a number or a shape designated to indicate a suitable designated target. According to some embodiments, an eye-tracking device forming a part of the XR based evaluation system 10 is designated to track the operator's gaze and sight during said shooting session and enable the display means to alter the at least one audiovisual indication indicating a possible target in case that any parameter captured by the eye-tracking device is considered lacking or wrong, as indicated by the numbers on the targets shown in FIG. 6A. For example, when the operator shuts one eye instead of leaving both open and aims, the numbers on the targets 602 or a particular single target 502 may disappear as shown on the clear target images shown in FIG. 6B, hence informing the operator to act differently in order to perform correctly.

[0145] Reference is made to FIG. 7, which schematically illustrates a shooting practice training simulation environment 60 that uses XR based evaluation system 10, according to some embodiments of the invention. As shown, an operator may use a designated display means such as a display or a VR / MR / XR headset configured to simulate a shooting practice arena in order to improve and teach a correct posture and sight alignment or an operator during a shooting practice. According to some embodiments, at least one audiovisual indication is designated to disappear and reappear or change its visual characteristics in accordance with an operator performing correct posture and sight alignment. For example, a designated marking 702 on the firearm sight may be altered in accordance with the operator's line of sight identification parameters as detected by the XR based evaluation system 10, hence improving aiming performance and skills which otherwise are not identifiable by an external observer such as a regular shooting instructor.

[0146] Reference is made to FIGS. 8A-8B, which schematically illustrate a shooting range training simulation environment 80 that uses XR based evaluation system 10, according to some embodiments of the invention. According to some embodiments, detailed feedback may be provided for the operator regarding a particular shot or a cluster of shots. For example, data sight alignment, accuracy, stability, etc., may be presented to the operator and bear a numeral score such as data 802. According to some embodiments, after pulling the trigger, gaze distribution indication 804 may present a particular point of gaze at the time of shooting, for example, a point presented on a human figure may indicate a particular gaze point at the time of shooting.

[0147] According to some embodiments, an image that simulates the field of sight or a focus point of an operator may appear in indication 806 and mark a picture captured by each eye of an operator. For example, indication 806 may depict an image seen by the operator a short moment before taking a shot. According to some embodiments, indication 706 may be configured to depict an image seen by the operator using their dominant eye, just before taking a shot in order to have an indication regarding the accuracy alignment, with the shooting point inflicting the efficiency of the shooting. According to some embodiments, a snapshot of hands and / or eyes and / or gun and / or head relative positions as shown in FIG. 8C may be designated to be displayed to the user / trainee. For example, head position indication 810 may be presented to the operator in order to show the exact angles and vectors of the operator's head position before, during, and / or after the shooting as shown in FIG. 8D. According to some embodiments, eye control indication 808 feedback may indicate the quality of eye control of the operator during a particular shot or cluster of shots. According to some embodiments, XR based evaluation system 10 may also provide a combined information snapshot of the head position and eye control, providing the operator with a detailed analysis of its performance.

[0148] According to some embodiments, a combined snapshot of an operator's hands, eyes, firearm, and head relative positions may be captured in order to provide a comprehensive evaluation of an operator's operation.

[0149] According to some embodiments, said shooting practice training simulation environments disclosed above may be displayed to the operator on a regular display means such as a screen or as part of a VR / MR / XR training session using a designated headset, wherein the data disclosed above may be presented to the operator while being immersed in the virtual or extended environment post-training or in real-time as well as or to an observer or evaluator thereby providing perspectives not otherwise available by common regular training apparatus and systems.

[0150] Although the present invention has been described with reference to specific embodiments, this description is not meant to be construed in a limited sense. Various modifications of the disclosed embodiments, as well as alternative embodiments of the invention, will become apparent to persons skilled in the art upon reference to the description of the invention. It is, therefore, contemplated that the appended claims will cover such modifications that fall within the scope of the invention.

Claims

CLAIMS1. An XR based evaluation system, comprising: i. at least one controller; ii. at least one audiovisual display means; iii. at least one input means designated to be controlled by an operator; iv. at least one sensing means configured to sense various parameters related to the operator, wherein the system is configured to assess, monitor, and record various data parameters of the operator's visuo-motor behavior while taking part in mission within an XR training simulation scenario conducted in an operational environment, and wherein the system is configured to interpret the said data parameters of specific time intervals and areas of interest that relate to the operational situation and context of the XR scenario.

2. The system of claim 1, configured to provide an assessment of the operator's cognitive and visuo-motor skills in relation to workload-related influences.

3. The system of claim 1, configured to provide an assessment of the operator's cognitive and visuo-motor skills in relation to stress-related influences.

4. The system of claim 1, wherein the operational environment is configured to be conducted in a full motion 6 degrees of freedom simulator environment.

5. The system of claim 1, wherein the operational environment is configured to simulate the operation of aerial, ground, or maritime vehicles or weapons and command and control systems thereof.

6. The system of claim 1, wherein the data parameters are utilized to generate a cognitive skill and situational awareness profile of the operator.

7. The system of claim 1, wherein the sensing means are at least one of an eye-tracking, heart rate, or EEG sensor(s) integrated within the XR based evaluation system.

8. The system of claim 1, wherein the controller is configured to retrieve data signals from the sensing means, classify them using designated algorithms, and process the retrieved data in relation to known external factors relating to the operator and the mission.

9. The system of claim 1, further configured to generate a real-time analysis of the situational awareness and visuo-motor performance parameters of the operator in order to provide immediate feedback during the mission.

10. The system of claim 1, further configured to generate a post-mission analysis of the situational awareness and visuo-motor performance parameters of the operator.

11. The system of any one of claims 9 or 10, wherein the situational awareness and visuo- motor performance data parameters of the operator are aggregated and compared to previously stored historical data.

12. The system of claim 1, wherein the at least one sensing means is configured to sense minute physiological and / or anatomical signals related to visuo-motor performances and / or situational awareness of an operator.

13. The system of claim 12, wherein at least one of the sensing means is an eye-tracking sensor(s) configured to capture eye -related parameters of the minute physiological and / or anatomical signals.

14. The system of claim 1, wherein the XR simulation scenario is a flight simulation designated to evaluate an operator's central and peripheral attention selectivity during flight.

15. The system of claim 14, wherein the XR simulation scenario is a flight simulation designated to evaluate an operator's aerial vehicle control parameters.

16. The system of any one of claims 14 or 15, wherein various evaluations are merged to an overall performance evaluation for each user.

17. The system of claim 1, wherein the at least one display means is an XR headset designated to provide an immersive 3 -dimensional experience to the operator.

18. The system of claim 1, wherein the XR simulation scenario is a shooting practice designated to evaluate an operator's sight and gaze dynamics and / or body posture.

19. The system of any of claims 14, 15, 17, and 18, wherein as part of said simulation, at least one audiovisual indication provided by the audiovisual display means is altered in response to certain sight and gaze dynamics and / or body posture detected by the XR based evaluation system.

20. The system of claim 19, wherein the at least one audiovisual indication is an at least one altered or removed target image wherein the XR based evaluation system configured to interpret non-optimal sight and gaze dynamics and / or body posture.

21. The system of claim 19, wherein the at least one audiovisual indication is an altered or removed firearm sight image wherein the XR based evaluation system configured to interpret non-optimal sight and gaze dynamics and / or body posture.

22. The system of claim 18, wherein following a shot or a cluster of shots, feedback containing various parameters related to performance of the operator is designated to be presented thereto.

23. The system of claim 22, wherein the feedback is an indication of the operator’s gaze distribution at the time of shooting.

24. The system of claim 22, wherein the feedback is an indication of the field of sight presenting an image captured by each eye of an operator.

25. The system of claim 22, wherein the feedback is an indication of head position presenting data regarding the head position of the operator before, during, and / or after a shooting.

26. The system of claim 22, wherein the feedback is a combined snapshot of data gathered regarding the operator's hands, eyes, firearm, and head relative positions.

27. The system of claim 1, wherein the sensing means is configured to sense the eye movements of the operator, and wherein the data collected thereby is processed to distinguish between an operator’s eyes’ fixations and saccades, in relation to the audiovisual stimuli provided by the audiovisual display means.

28. An XR based evaluation method, comprising the steps of: a) utilizing at least one display means in order to present a particular XR scenario to an operator in an operational environment, b) providing at least one input means designated to be controlled by the operator, c) providing at least one sensing means configured to sense various parameters related to the operator, d) utilizing at least one controller to assess, monitor, and record various data parameters of the operator's visuo-motor behavior while taking part in the XR training simulation scenario in an operational environment, wherein the system is configured to interpret the data parameters of specific time intervals and areas of interest that relate to the operational situation and context of the XR scenario.

29. The method of claim 28, wherein the sensing means is configured to sense the eye movements of the operator, and wherein the data collected thereby is processed to distinguish between an operator’s eyes’ fixations and saccades, in relation to the audiovisual stimuli provided by the audiovisual display means.

Images