Human factor intelligent scene awareness evaluation method and system based on simulated training environment

Abstract

The invention provides a human factor intelligent situational awareness evaluation method and system based on a simulated training environment, and the method comprises the steps: displaying a situational awareness task of a preset dimension in the simulated training environment to a person wearing a virtual reality device at a preset visual angle, the dimensions of the situational awareness task comprise a perception dimension, an understanding task dimension, a prediction task dimension and a decision-making task dimension; receiving evaluation feedback information performed by the personnel according to the received situational awareness task of the preset dimension; and recording the evaluation feedback information, and comparing the evaluation feedback result with an evaluation rule in a preset evaluation system to obtain an evaluation result. According to the invention, the objective evaluation method for the situational awareness of the personnel in combination with the ecological data at the preset view angle can be provided.

Description

Technical Field

[0001] The present application relates to the technical field of situational awareness assessment in virtual scenarios, and in particular to a method and system for assessing personnel situational awareness based on a simulated training environment. Background Art

[0002] At present, domestic researchers have combined virtual reality (VR) technology with player passing awareness training. Through the interactive design of virtual training scenes and training content, they have set up passing training content in a targeted manner to meet the passing awareness training needs of players of all ages, positions, and technical levels; established a variety of virtual player passing awareness training venues and training weather to meet the training needs of players; designed passing awareness training course content to realize the application of virtual reality passing training for players.

[0003] However, passing awareness is not equivalent to situational awareness, and it is even more impossible to evaluate the perception, understanding and prediction of players' situational awareness. In addition, there is currently no research on the application of virtual reality technology in the evaluation and training of personnel's situational awareness. Using virtual reality technology to evaluate and train players' situational awareness, players can be immersed in virtual reality scenes, truly restore the game scene, and evaluate players' situational awareness performance through different scenes. And through continuous practice, the ability to respond on the spot can be improved, so that when encountering problems on the field, decisions can be made quickly and subconsciously, thus saving thinking time.

[0004] At the current stage, there are mainly the following key issues in the assessment of players’ situational awareness:

[0005] (1) Highly subjective. Traditional methods of evaluating players’ situational awareness levels are mainly subjective, conducted through post-match interviews or questionnaires, which cannot objectively reflect the players’ actual on-field situational awareness levels. (2) Lack of first-person perspective situational awareness during the game. Traditional methods of evaluating players’ situational awareness levels, even if combined with videos to review and study the game scenes at the time, only analyze the game from a third-person perspective, which cannot truly reflect the situational awareness during the game. As a result, players cannot experience the situation first-hand and improve their situational awareness during real high-level games, limiting their room for improvement. (3) Low ecological validity. Players cannot systematically evaluate their own situational awareness levels, and cannot systematically evaluate players based on their habits, characteristics, on-field positions, and other factors. Summary of the invention

[0006] To this end, the present application provides an objective evaluation method for human intelligent situational awareness in a simulated training environment, which can provide players with an evaluation environment with a preset perspective and combines ecological data.

[0007] The embodiments of the present application provide a method and system for evaluating human factors intelligent situational awareness based on a simulated training environment to eliminate or improve one or more defects existing in the prior art.

[0008] One aspect of the present application provides a method for evaluating human factors intelligent situational awareness based on a simulated training environment, the method comprising:

[0009] Presenting a situational awareness task of preset dimensions in a simulated training environment to a person wearing a virtual reality device from a preset perspective, wherein the dimensions of the situational awareness task include a perception dimension, an understanding task dimension, a prediction task dimension, and a decision-making task dimension;

[0010] Receiving evaluation feedback information performed by the personnel according to the received situational awareness task of the preset dimension;

[0011] The evaluation feedback information is recorded, and the evaluation result is obtained by comparing the evaluation feedback information with the evaluation rules in the preset evaluation system.

[0012] In some embodiments of the present application, the method further includes: in the process of a person wearing a virtual reality device observing a situational awareness task of a preset dimension and giving evaluation feedback information, using a collection device worn by the person to synchronously collect multimodal data of the person to be evaluated wearing the virtual reality device; wherein the multimodal data includes at least eye movement data and motion capture data; after obtaining the evaluation results, using a pre-trained multimodal data evaluation model to score the collected multimodal data to obtain a multimodal evaluation result of the person to be evaluated.

[0013] In some embodiments of the present application, the acquisition device includes a near-infrared spectral imaging device and / or a physiological data acquisition device, and the physiological data acquisition device includes a heart rate acquisition device and / or a skin charge acquisition device; the multimodal data also includes one or more of brain function data collected by a functional near-infrared spectral imaging device, heart rate data collected by a heart rate acquisition device, and skin charge data collected by a skin charge acquisition device.

[0014] In some embodiments of the present application, the method also includes the step of pre-training a multimodal data evaluation model, and the step of training a multimodal data evaluation model includes: establishing a multimodal data evaluation model based on linear regression analysis and initializing weights and biases; using the multimodal data of multiple personnel collected during the pre-processed historical evaluation process and the corresponding evaluation results as a training set, and inputting them into the linear regression model to calculate the degree of fit between the actual evaluation results and the predicted evaluation results as a loss function; inputting the training set into the initialized multimodal data evaluation model for iterative optimization; wherein the preprocessing includes data cleaning and data enhancement; when the loss function converges, the pre-training of the multimodal data evaluation model is completed.

[0015] In some embodiments of the present application, the method further comprises: pre-inputting a planar video including a stadium environment;

[0016] Analyze frame by frame to identify the positions of each participant and the ball in the court in the two-dimensional video; generate the position coordinates of each participant and the ball in the standard court coordinate system corresponding to each frame in the video; obtain the spatial motion trajectory of each participant and the ball motion trajectory that changes with time based on the corresponding position coordinates in the standard court coordinate system; wherein the participants include multiple players, referees and / or coaches from each team; associate a pre-set virtual participant model with the spatial motion trajectory of each participant and associate a pre-set virtual ball model with the ball motion trajectory, and combine the virtual court model, the virtual participant model and the virtual ball model to build a simulated training environment that matches the two-dimensional video.

[0017] In some embodiments of the present application, the method further comprises: supporting different perspective identity switching in the simulation training environment, wherein participants with different perspective identities pre-establish corresponding different situational awareness tasks;

[0018] The virtual court model, virtual participant model and virtual ball model are created by retrieving a preset virtual scene material library.

[0019] In some embodiments of the present application, the presentation of the situational awareness tasks supports customization of the presentation quantity, dimension, and order.

[0020] In some embodiments of the present application, before presenting the steps of a situational awareness task of a preset dimension in a simulated training environment to the person wearing the virtual reality device at a preset perspective, instructions or prompt information are presented to the person wearing the virtual device.

[0021] In some embodiments of the present application, the method also includes: converting situational awareness tasks of preset dimensions and personnel assessment feedback information in a simulated training environment into a two-dimensional video in a television broadcast perspective version and / or a filming perspective version; wherein, different virtual participant models in the simulated training environment are converted into participant virtual human avatars in a two-dimensional plane, and the virtual ball model in the simulated training environment is converted into a virtual ball avatar in a two-dimensional plane, and the spatial motion trajectory of the virtual participant is converted into a virtual human participant spatial motion trajectory in a two-dimensional plane and associated with the participant avatar, and the ball motion trajectory is converted into a virtual ball motion trajectory in a two-dimensional plane and associated with the ball avatar, thereby converting it into a two-dimensional video in a television broadcast perspective version and / or a filming perspective version.

[0022] Another aspect of the present application provides a human intelligent situational awareness assessment system for a simulated training environment, including a processor and a memory, wherein the memory stores computer instructions, and the processor is used to execute the computer instructions stored in the memory. When the computer instructions are executed by the processor, the system implements the steps of the method described in any one of the above embodiments.

[0023] Another aspect of the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described in any one of the above embodiments.

[0024] The human factors intelligent situational awareness assessment method and system based on a simulated training environment proposed in the present application can receive assessment feedback information of personnel on situational awareness tasks based on VR devices worn by personnel, collect multimodal data of personnel in the situational awareness assessment process based on collection devices worn by personnel, obtain assessment results according to preset scoring rules based on the assessment feedback information, display situational awareness scenes to players at a preset perspective through virtual devices, enhance the sense of immersion, and score the assessment feedback information according to objective preset scoring rules.

[0025] Additional advantages, purposes, and features of the present application will be partially described in the following description, and will become partially apparent to those skilled in the art after studying the following, or may be learned from the practice of the present application. The purposes and other advantages of the present application can be achieved and obtained by the structures specifically pointed out in the specification and the drawings.

[0026] Those skilled in the art will understand that the purposes and advantages that can be achieved by the present application are not limited to the above specific description, and the above and other purposes that can be achieved by the present application will be more clearly understood based on the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] The drawings described herein are used to provide a further understanding of the present application, constitute a part of the present application, and do not constitute a limitation of the present application. In the drawings:

[0028] Figure 1 This is a flow chart of a method for evaluating human factors intelligent situational awareness in a simulated training environment in a specific embodiment of the present application.

[0029] Figure 2 This is a flow chart of the human situational awareness assessment process in a specific embodiment of the present application.

[0030] Figure 3 This is a mode setting interface in a specific embodiment of the present application.

[0031] Figure 4 This is a scene setting interface in a specific embodiment of the present application.

[0032] Figure 5 This is a task setting interface in a specific embodiment of the present application.

[0033] Figure 6 This is a simulation training environment setting interface in a specific embodiment of the present application.

[0034] Figure 7 This is a flow chart of the human situational awareness assessment process in another specific embodiment of the present application.

[0035] Figure 8 This is a schematic diagram of scenario selection in another specific embodiment of the present application. DETAILED DESCRIPTION

[0036] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the implementation modes and the accompanying drawings. Here, the illustrative implementation modes and descriptions of the present application are used to explain the present application, but are not intended to limit the present application.

[0037] It should also be noted here that in order to avoid obscuring the present application due to unnecessary details, only the structures and / or processing steps closely related to the scheme according to the present application are shown in the accompanying drawings, while other details that are not very relevant to the present application are omitted.

[0038] It should be emphasized that the term “include / comprises” when used herein refers to the presence of features, elements, steps or components, but does not exclude the presence or addition of one or more other features, elements, steps or components.

[0039] It should also be noted that, unless otherwise specified, the term “connection” herein may refer not only to a direct connection but also to an indirect connection involving an intermediate.

[0040] Hereinafter, embodiments of the present application will be described with reference to the accompanying drawings. In the accompanying drawings, the same reference numerals represent the same or similar components, or the same or similar steps.

[0041] In order to solve the problems existing in the existing situational awareness assessment methods of personnel in the stadium environment, the present application provides a human-factor intelligent situational awareness assessment method and system based on a simulated training environment. This scheme is based on virtual reality technology, and measures the awareness of football players with the three dimensions of situational awareness (perception, understanding, and prediction) combined with the four dimensions of decision-making as the core. In this application, a football scene is constructed through virtual reality technology, including three-dimensional modeling of football fields, virtual players, situational awareness measurement tools, etc.; virtual reality is combined with eye movement technology and near-infrared spectral imaging technology. Combining virtual reality technology with the situational awareness of football players, introducing virtual reality technology into football training, and at the same time opening up a new path for the measurement and training of football players' situational awareness.

[0042] Figure 1 This is a flow chart of a method for evaluating human factors intelligent situational awareness in a simulated training environment in an embodiment of the present application. The method for evaluating human factors intelligent situational awareness in a simulated training environment includes:

[0043] S100: Presenting a situational awareness task of preset dimensions in a simulated training environment to a person wearing a virtual reality device from a preset perspective, wherein the dimensions of the situational awareness task include a perception dimension, an understanding task dimension, a prediction task dimension, and a decision-making task dimension to simulate the training environment.

[0044] In the specific implementation process, the situational awareness tasks in the simulation training environment and VR scene are pre-built, and the built simulation training environment supports perspective switching, that is, the first perspective can be switched to the player's perspective, or it can be the perspective of the same team members in the same game, or the perspective of the opponent players in the usual game, and it can also be switched to the coach's perspective. Further, in some embodiments, it can also be switched to a third-party live broadcast perspective or a two-dimensional plane perspective, etc., where it can support multiple angles. Any switching or simultaneous display of several angles. The evaluation feedback information of personnel can rely on the handle or gesture recognition used to control the VR device, which is specifically determined by the interaction method supported by the virtual device.

[0045] S200: receiving evaluation feedback information performed by the personnel according to the received situational awareness task of the preset dimension.

[0046] S300: Record the evaluation feedback information, and obtain an evaluation result by comparing the evaluation feedback information with evaluation rules in a preset evaluation system.

[0047] In the specific implementation process, the specific scoring rules are set according to the specific situation of the task setting. For example, the answer feedback mode of the situational awareness task is a multiple-choice question, and the choice is to predict the spatial movement trajectory of a certain athlete at the next moment. The scoring of this answer feedback mode can be directly scored according to the question, or it may be manually selecting an action direction. At this time, a deviation degree needs to be calculated, and the parameters used to characterize the deviation degree are standardized into a consistent score. The above is only an example, and the present application is not limited to this.

[0048] The human factors intelligent situational awareness assessment method and system based on a simulated training environment proposed in the present application can receive assessment feedback information of personnel on situational awareness tasks based on virtual reality equipment worn by personnel, collect multimodal data of personnel in the situational awareness assessment process based on collection equipment worn by personnel, obtain assessment results based on the assessment feedback information according to preset scoring rules, display situational awareness scenes to players at a preset perspective through VR equipment, enhance the sense of immersion, and score the assessment feedback information according to objective preset scoring rules.

[0049] It should be noted that the court includes but is not limited to a football field, a basketball court, a volleyball court, etc. The embodiments listed in this application are mainly based on a football field, but are not limited to this.

[0050] In some embodiments of the present application, during the execution of step S100 and step S200, that is, during the process of the person wearing the virtual reality device observing the situational awareness task of the preset dimension and giving evaluation feedback information, the method further includes: using the acquisition device worn by the person to synchronously acquire multimodal data of the person to be evaluated wearing the virtual reality device; wherein the multimodal data at least includes eye movement data and motion capture data. And, after obtaining the evaluation result, the collected multimodal data is scored using the pre-trained multimodal data evaluation model to obtain the multimodal evaluation result of the person to be evaluated.

[0051] By adopting this embodiment, it is possible to establish an association between evaluation results and multimodal data based on the multimodal data and evaluation results of personnel, thereby training a multimodal data evaluation model for predicting multimodal evaluation results based on the multimodal data and evaluation results in the evaluation process on a large data scale, thereby establishing a new evaluation system for evaluating personnel performance.

[0052] Specifically, the acquisition device includes a near-infrared spectral imaging device and / or a physiological data acquisition device, and the physiological data acquisition device includes a heart rate acquisition device and / or a skin charge acquisition device. The multimodal data also includes one or more of brain function data acquired based on a functional near-infrared spectral imaging device, heart rate data acquired based on a heart rate acquisition device, and skin charge data acquired based on a skin charge acquisition device. The above acquisition devices are only examples, and the present invention is not limited thereto.

[0053] In some embodiments of the present application, the method further includes the step of pre-training a multimodal data evaluation model, which can be understood as a multimodal data evaluation system based on multimodal data and preset evaluation rules, and can be improved and updated through large-scale multimodal data and evaluation result feedback. The training step specifically includes:

[0054] (1) Initialization step: establishing a multimodal data evaluation model based on linear regression analysis and initializing weights and biases;

[0055] (2) The multimodal data of multiple personnel collected during the preprocessed historical evaluation process and the corresponding evaluation results are used as a training set and input into a linear regression model, and an equation for calculating the degree of fit between the actual evaluation results and the predicted evaluation results is used as a loss function. The training set is input into the initialized multimodal data evaluation model for iterative optimization; wherein the preprocessing includes data cleaning and data enhancement.

[0056] (3) When the loss function converges, the pre-training of the multimodal data evaluation model is completed.

[0057] By adopting this embodiment, a prediction model based on linear regression analysis, namely a multimodal data evaluation model, can be trained and obtained, and the multimodal data evaluation model can associate the relationship between multimodal data and evaluation results. The multimodal data evaluation model can analyze the input test data (collected multimodal data) and generate a predicted multimodal evaluation result.

[0058] In some embodiments of the present application, before step S100, the method further includes the step of pre-building a virtual stadium scene, which specifically includes the following steps:

[0059] (1) Input a 2D video containing a court environment, analyze and identify the positions of each participant and the ball in the court in the 2D video frame by frame, generate the position coordinates of each participant and the ball in each frame in a standard court coordinate system, and obtain the spatial motion trajectories of each participant and the ball motion trajectory that change with time based on the corresponding position coordinates in the standard court coordinate system; wherein the participants include multiple players, referees and / or coaches, and different participants, athletes or training personnel can be distinguished based on factors such as gender, color and number.

[0060] Among them, the coordinate system is a three-dimensional stereoscopic coordinate system. Based on the specifications of the ball field, the coordinates of the boundaries are pre-set. Based on the distance of each player, referee, coach and ball in the two-dimensional video screen relative to the field boundary, their relative distances are determined, thereby estimating the position coordinates of each participant and the ball in each frame in the standard court coordinate system, and then splicing each frame to obtain the spatial motion trajectory of each participant and the motion trajectory of the ball that changes with time.

[0061] (2) Establish a VR stadium model, VR participant model, and VR ball model by searching the VR scene material library.

[0062] In simple terms, it means choosing a VR model to reshape the court, participants and ball in the two-dimensional video, and choosing a VR model that is as close as possible to the original objects and characters in the two-dimensional video to restore the two-dimensional video as much as possible.

[0063] (3) Associating a preset virtual participant model with the spatial motion trajectory of each participant and associating a preset virtual ball model with the ball motion trajectory, and combining the virtual court model, the virtual participant model and the virtual ball model to build a simulated training environment that matches the plane video. Specifically, each VR participant model can be bound to the spatial motion trajectory of each participant, the VR ball model can be bound to the ball motion trajectory, and the VR court model, the VR participant model and the VR ball model can be combined to achieve the construction of a simulated training environment that matches the plane video.

[0064] In the specific implementation process, setting a situational awareness task in a simulated training environment is to play a VR video converted from a flat video, set different situational awareness tasks according to different flat videos, and then issue situational awareness tasks to the personnel participating in the evaluation. The personnel provide evaluation feedback information based on the VR control device, and then give the evaluation results based on the preset scoring rules of the situational awareness task. Optionally, the types of feedback for the situational awareness task are diverse, which can be selection, filling in answers, or adjusting the direction angle through the VR handle, and this application does not make specific limitations.

[0065] By adopting this embodiment, it is possible to build a flexible simulation training environment and set different situational awareness tasks. Past game records can be reviewed based on flat videos, or some scenes can be rehearsed. This can be used for evaluating players' abilities, as well as for special training of players' perception, understanding, prediction or decision-making abilities.

[0066] In some embodiments of the present application, the simulation training environment supports switching between different perspectives, wherein participants with different perspectives pre-establish corresponding different situational awareness tasks. In layman's terms, before or during the start of the situational awareness task, personnel can adjust the switching perspective by themselves. The virtual court model, virtual participant model, and virtual ball model are created by retrieving a preset VR scene material library.

[0067] By adopting this embodiment, on the one hand, it is possible to implement training for different roles, not only players, but also coaches and referees. On the other hand, it can also help players understand the perspectives of different roles and understand the entire court in a multi-dimensional and comprehensive manner.

[0068] In some embodiments of the present application, the display of the situational awareness tasks supports customization of the display quantity, dimension and order. In layman's terms, the present application can conduct: (1) all-round testing: sequentially displaying situational awareness tasks in the dimensions of perception, understanding, prediction and decision-making; or (2) single-dimensional testing: directly displaying only the situational awareness tasks in the prediction dimension; or (3) special training: continuously performing the situational awareness tasks in the prediction dimension in three different VR stadium scenes. The above are only examples, and the present application is not limited to this.

[0069] In some embodiments of the present application, the method further includes: before presenting the step of presenting the situational awareness task of preset dimensions in the simulated training environment to the person wearing the virtual reality device, presenting instructions to the person wearing the virtual reality device. This setting is to allow the person to better understand the situational awareness task to be performed. Optionally, the instructions can also be issued in the form of voice broadcast. Examples thereof will be given in the specific embodiments below.

[0070] In some embodiments of the present application, when the simulated training environment is a virtual court environment, for example, the method further includes: converting the situational awareness tasks and personnel assessment feedback information of preset dimensions in the virtual court environment into a two-dimensional video in a television broadcast perspective and / or a shooting perspective. Among them, the different virtual participant models in the virtual court environment are converted into virtual human participant avatars in a two-dimensional plane, the virtual ball model in the virtual court environment is converted into a virtual ball avatar in a two-dimensional plane, the spatial motion trajectory of the participant is converted into a virtual human participant spatial motion trajectory in a two-dimensional plane and associated with the participant avatar, the ball motion trajectory is converted into a virtual ball motion trajectory in a two-dimensional plane and associated with the ball avatar, and thus converted into a two-dimensional video in a television broadcast perspective and / or a shooting perspective. Virtual participants in the court environment refer to the players of both sides, as well as the coaches of both sides that may also exist.

[0071] By adopting this embodiment, a third party can synchronously observe the performance of the situational awareness assessment participants. For example, when a player wears a VR device to participate in the situational awareness assessment, the coach can observe the player's reaction speed and answer feedback results in real time.

[0072] In some embodiments of the present application, the acquisition device also includes a functional near-infrared spectral imaging device and a physiological data acquisition device, the physiological data acquisition device includes a heart rate acquisition device and a skin charge acquisition device, and the multimodal data also includes brain function data acquired based on the functional near-infrared spectral imaging device, heart rate data acquired based on the heart rate acquisition device, and skin charge data acquired based on the skin charge acquisition device. It should be noted that before the model training is used as a training set input, the above-collected data needs to be preprocessed, and the preprocessing at least includes normalization processing. The above-collected data are only examples, and the present application is not limited to this.

[0073] By adopting this embodiment, ecological validity can be increased, the credibility of multimodal evaluation results obtained based on multimodal data can be guaranteed, a more accurate evaluation result can be provided on the other hand, and multimodal data can be used as a reference for further research.

[0074] Taking the situational awareness assessment of personnel in a football stadium environment as an example, the simulation training environment can specifically be a virtual stadium environment. First, the virtual stadium environment is built, and its main features include:

[0075] (1) The measurement index is combined with the situational awareness model to evaluate the players' awareness. The characteristics of football projects (decision-making) are innovatively combined with the scientific three-level model of situational awareness (perception, understanding, and prediction) to evaluate the ability of football players in a multi-dimensional and comprehensive manner. The three levels of situational awareness from low to high are: (1) Perception of external environmental information, that is, the individual's attention and perception of stimulus information in the environment; (2) Understanding of external environmental information, that is, the individual judges, understands and stores the perceived environmental information based on his own knowledge and experience; (3) Prediction of environmental status, that is, the individual predicts and estimates the future development of the environment based on reasonable perception and understanding, combined with the experience in his long-term memory, so as to help the individual choose the best decision-making plan and behavior. This is completely consistent with the cognitive processing of football players on the field, which is conducive to helping improve football players' on-field awareness and thus improve athletes' sports performance.

[0076] (2) Through virtual reality technology, the situational awareness is measured from the player's first-person perspective. This can make the evaluation more objective. Compared with traditional football level assessment forms, virtual reality technology can truly restore the situation on the scene of the game. Through three-dimensional modeling, any football video clip can be modeled and converted into a highly restored virtual reality football scene. Coaches can conduct football situational awareness assessment at any time for players of different levels, different positions on the field, and different technical and tactical characteristics, so as to quickly and specifically understand the players' football situational awareness level. The first-person perspective can be used, which has higher ecological validity. The situational awareness can be evaluated from multiple dimensions of different positions on the field. The football situational awareness assessment project based on the VR environment includes a large number of football scenes, such as attack and defense mode, attack to defense, defense to attack, etc. At the same time, the number of players on both sides, assessment time, assessment difficulty, etc. can be customized. The assessment positions include goalkeeper (not available yet), defender, forward, midfielder, forward, any position or multiple role positions. The corresponding virtual reality environment can be customized according to the needs of coaches and players. The assessment subjects wear VR helmets and immerse themselves in the virtual reality environment to conduct football situational awareness assessment. This system sets up four dimensions of football situational awareness assessment: perception, understanding, prediction, and decision-making. Players can run, dribble, hold the ball in place, run with the ball, pass, and shoot in the virtual reality scene, triggering the corresponding interactive content in the scene. Combined with the specific assessment scene and content, the system accurately collects the evaluation subject's answer options, uploads them to the assessment system, and feedbacks the evaluation status of the evaluation subject. The evaluation subject is highly immersed in the virtual reality environment, and obtains perception, understanding, prediction, and decision-making experiences similar to those in real football game scenes, and obtains real reaction phenomena and results.

[0077] (3) Intelligent extraction of video trajectories: The specific operation is as follows: Based on the uniformly established plane court coordinate system (the upper left corner is (0, 0), the lower right corner of the court is (1, 1), the x-axis is the horizontal direction of the court, and the y-axis is the vertical direction of the court. The numbers in X and Y are relative proportional positions), use image recognition tools to collect the position coordinates of each player, referee, coach and ball in each frame of plane video, use (data sorting software such as Excel) to arrange and combine the frame coordinates of each player and the ball, and use the correction algorithm to correct the data to obtain the video trajectory of each player and the ball.

[0078] (4) Conversion between video and VR environments

[0079] First, the video is converted to a VR environment. According to the real scene of the target video, the closest virtual football field and football player, referee, and coach models are selected and matched from the system VR scene library. According to the initial position of the trajectory intelligently extracted from the plane video, the system places the football player model and matches and modifies the jersey color and number, binds each player's movement trajectory to each model, the football movement trajectory to the football model, the referee movement trajectory to the referee model, and the coach movement trajectory to the coach model. Thus, a football virtual reality scene matching a specific video is established. The system will place hidden cameras in the eyeballs of each player, referee, and coach, as well as on the ceiling of the stadium and at the TV broadcast site, so that users can observe the virtual environment from multiple perspectives.

[0080] Secondly, it is the conversion of VR environment to video. According to the TV broadcast and the perspective of the stadium ceiling, the game scene built in the virtual reality environment can be converted into two versions of two-dimensional game videos according to the use requirements.

[0081] ① TV broadcast perspective (similar to the perspective of current football match videos), record the jersey color and number, running direction, distance and relative position between models in the virtual reality environment. Convert each model in the virtual reality environment into its avatar (two-dimensional man) in a two-dimensional plane, input the recorded data into the corresponding avatar, make the avatar replace the model to perform two-dimensional movement, and thus build a two-dimensional match video as a whole.

[0082] ② Shooting perspective board (similar to chessboard / tactical board), record the jersey color and number, running direction, distance and relative position between models in the virtual reality environment. Convert each model in the virtual reality environment into its player chess piece (or point) in the two-dimensional plane. The chess piece has different colors (for example, red chess piece for team A, blue chess piece for team B, black chess piece for referee), and the number displayed in the middle of the chess piece is the player number. According to the running direction, distance and relative position between models in the virtual reality environment, the movement trajectory and relative position of the chess piece are output. The chess piece moves horizontally in the flat football field captured by the ceiling camera, so as to build a flat game video as a whole.

[0083] (5) Synchronous collection of multimodal data: eye movement, heart rate, electrodermal function, brain function, motion capture and other data can be collected and evaluated synchronously to explore the underlying mechanisms.

[0084] The research goal of this technical solution is to use virtual reality technology to evaluate and train football players' football situational awareness, that is, the players' perception, understanding and prediction ability when performing technical and tactical activities in football matches. Football situational awareness is a key factor affecting players' performance and the basis of football technology and tactics. We want to use virtual reality technology to simulate real football scenes, allowing players to make decisions in different situations and give timely feedback. During the evaluation and training process, eye movement, heart rate, skin electricity, brain function, motion capture and other data can be collected simultaneously to evaluate and improve football situational awareness from multiple angles and levels. These physiological indicators can reflect the players' physiological reactions, cognitive processes, attention allocation, emotional states and other information in different situations. They can also be combined with virtual reality technology to form a closed-loop system to achieve personalized and intelligent training. This technology is to explore the mechanism of football situational awareness, that is, the laws and principles followed by humans when making decisions in complex environments, as well as the factors that affect the efficiency and quality of decisions. Exploring the mechanism of football situational awareness can help understand the nature of human behavior and optimize the effects of human behavior.

[0085] In the process of building a virtual display scene, the computer algorithm is used to convert the video and intelligently extract the player trajectory of the plane video, which specifically includes the following steps:

[0086] (1) This technology first uses image recognition tools to collect the positions of each player, referee, coach, and ball in each frame of the target plane video. Based on the plane court standard coordinate system created by this technology, the position of any person in any frame will be expressed in the form of coordinates. Then, using data sorting software, the coordinates of the person and the ball in the plane video are encoded correspondingly to the video frame. Finally, the data is corrected by combining the correction algorithm to obtain the video trajectory of each person and the ball.

[0087] (2) Based on the original video material, this technology will select and match the football virtual field and football player, referee, and coach models that are closest to the original video from the VR scene material library. Combined with the initial position of the trajectory intelligently extracted from the plane video, this technology will place the football player model and match and modify the jersey color and number, bind each player's movement trajectory to each model, bind the football movement trajectory to the football model, bind the referee's movement trajectory to the referee model, and bind the coach's movement trajectory to the coach model. In this way, a football virtual reality scene matching a specific video is established. The virtual environment constructed by this technology will place hidden cameras in the eyeballs of each player, referee, and coach, as well as in the ceiling of the stadium and the TV broadcast area, so that technology users can observe the virtual environment from multiple perspectives.

[0088] (3) After the three-dimensional model is created, the situational awareness of the football players is measured. The dimensions of the evaluation of the present application technology include four types: perception, understanding, prediction, and decision-making. The situational awareness ability of football players of different levels is evaluated from the three dimensions of situational awareness that are typically measured in the three-level theoretical model of situational awareness and the decision-making dimension that is combined with the characteristics of the football project. The set of original parameters given by the present application technology are all adjustable parameters, and the evaluator can adjust them within the adjustable range as needed. For example, the number of players on both sides in the simulated game, the screen playback speed, etc. can be adjusted. At the same time, by setting different trial schedules, the situational awareness or combination of different types of evaluators can be evaluated. This technology covers evaluation methods from basic to complex, and comprehensively evaluates the situational awareness levels of different groups of people according to the needs of the evaluator.

[0089] On the basis of completing the construction of the virtual reality scene, in the simulated training environment, that is, the virtual reality football field, the process of measuring the situational awareness of football players is added. This application mainly designs two embodiments:

[0090] Example 1: Four-level assessment of situational awareness of football players

[0091] This embodiment includes five stages: assessment introduction, practice, task execution, performance feedback, and overall evaluation. Figure 2 As shown, first enter the experiment introduction part, then the user selects the practice experiment and clicks to start the experiment, and then performs tasks one, two, three and four in sequence, and then evaluates the feedback information one by one, merges the overall feedback, and declares the end of the experiment. Specifically, there are many options for the calculation formula for obtaining the overall feedback from each evaluation feedback information, and the average value can be calculated directly, or the average value can be calculated based on the preset weights. This application does not make specific restrictions on this.

[0092] The specific operation process is as follows:

[0093] (1) Open the system software;

[0094] (2) A pop-up window will pop up to enter your name and age, making it easier to record and count your personal performance;

[0095] (3) Select the mode, such as Figure 3 As shown, you can select the number of our team members, the number of opponents, referee settings and goalkeeper settings

[0096] Select the scene and the location on the field, such as Figure 4 and Figure 8 As shown, for example, the selected scene is an offense, defense, offense-to-defense, or defense-to-attack scene, and the position on the field can be a forward, center, or guard, etc.;

[0097] Select the task presentation mode and tasks to turn on / off, as well as the order of tasks, such as Figure 5 As shown, for example Figure 5 The order of tasks in the test was 2-3-4, that is, no perceptual task was performed, and the task presentation mode was either intermittent or continuous;

[0098] Set the parameter information. Click Next to see the parameter information. The initial parameter information is: To be filled in. The contents that can be set include jersey numbers, jersey colors of both teams, stadium sounds, whether to open the guide line drawing, screen speed and key moment speed, and scene number information, such as Figure 6 As shown;

[0099] (4) If the input is correct, click to start the experiment (the home page includes "Practice" and "Perform the formal experiment");

[0100] (5) Start by doing the practice section and reading the instructions.

[0101] Specifically, the added instructions include:

[0102] Instructions at the beginning: Hello, students. Next, you will complete several tasks on a virtual football field. You will see a series of football match situations from a first-person perspective, and then the situation will be frozen at that moment. At this time, you need to perform corresponding operations according to the task requirements. Please treat yourself as a player participating in a football game and immerse yourself in the situation.

[0103] (6) You can read the formal experimental instructions and conduct a formal trial, following the same process as the practice part, or simply end the experiment to view the results and exit the system.

[0104] For example, the instructions before the task starts can be: Perception task instructions: In the perception task, you need to perceive the position, body orientation and other information of your opponent / teammate. After the game situation develops to a certain critical moment and freezes, the characters on the field will be cleared. You need to drag the little man below to restore it as accurately as possible. Pull the trigger to select the little man to move to the correct position on the football field you think, slide the wheel above to adjust the body orientation, and press the trigger to confirm. Press the trigger again to cancel the confirmation; Understanding task instructions: In the understanding task, we mainly examine your understanding of the current situation. You need to read the game carefully and tell it in an oral report. After the game situation develops to a certain critical moment and freezes. We will provide several football-related words. You need to select the words with the highest correlation with the current situation and verbally report your understanding of the situation; Prediction task instructions: In the prediction task, after the game situation develops to a certain critical moment and freezes, the characters on the field will be cleared. You need to predict the situation of the game at the next moment, including the position, body orientation and other information of your opponent / teammate. You need to drag the little man below to restore the final frozen screen. Pull the trigger to select the little man and move him to the correct position on the football field. Slide the wheel above to adjust the body direction and press the trigger to confirm. Press the trigger again to cancel the confirmation. Instructions for decision-making tasks: In the decision-making task, you need to make decisions such as passing, dribbling, shooting, etc. based on the key moments frozen in the original video material or the replay results of the perception / / prediction task.

[0105] (7) Entering the “formal experiment”;

[0106] (8) The background records and saves the information, type and results of the assessment, such as basic information of the assessor, trial type and number of trials, trial parameters, trial number, reaction time and accuracy of each trial.

[0107] (9) Click to view the results and save the data.

[0108] Example 2: Evaluation of a single level of football players

[0109] This embodiment includes five stages: assessment introduction, practice, task execution, performance feedback, and overall evaluation.

[0110] Figure 7 This is a flowchart of the personnel situational awareness assessment process in another specific embodiment of the present application. First, the experiment is introduced, then the practice experiment is selected, the experiment is started, and the task is selected. For the selected task, an assessment feedback information is provided every time a task is completed, and an overall feedback is generated based on all the assessment feedback information, and finally the experiment is declared over.

[0111] The specific steps are as follows:

[0112] (1) Open the system software;

[0113] (2) Enter your name and age, and select the assessment type: Decision-making ability;

[0114] (3) Click Next to see the parameter information. The initial parameter information is: To be filled in;

[0115] (4) If the input is correct, click to start the experiment (the home page includes "Practice" and "Perform the formal experiment");

[0116] (5) Start by doing the practice section and reading the instructions.

[0117] For example, add instructions to the contact section: Hello, students, you will complete several tasks on the virtual football field. You will see a series of football game situations from the first-person perspective, and then the situation will be frozen at that moment. At this time, you need to perform corresponding operations according to the task requirements. Please treat yourself as a player participating in a football game and immerse yourself in the situation.

[0118] (6) You can read the formal experimental instructions and conduct a formal trial, the process is the same as the practice part, or directly end the experiment to view the results and exit the system. The instructions for the perception, understanding, prediction, and decision-making tasks are the same as in Example 1;

[0119] (7) Entering the “Formal Experiment”: Before the decision-making task begins, choose the situation at this time. The position of the players on the field is based on their own review in the prediction task.

[0120] For example, Figure 8 The middle is an offensive situation, and the players on the field are in a position for the subjects to make a passing decision. The subjects use the handle / trigger to generate a ray to judge the passing angle, and there will be a black auxiliary line on the screen to assist the subjects in making choices (that is, the direction of the black auxiliary angle is the direction in which the subjects make passing decisions through the ray). Specifically, the players give feedback through the "option + button" method.

[0121] (8) The background records and saves the information, type and results of the assessment, such as basic information of the assessor, trial type and number of trials, trial parameters, trial number, reaction time and accuracy of each trial;

[0122] (9) Click to view the results and save the data.

[0123] Specifically, the dedicated venue for the virtual reality evaluation environment is planned as an indoor, 10m*10m venue to measure the situational awareness of athletes. Based on virtual reality technology, football players can watch the game from the first perspective, and can switch to the perspectives of different players on the field to understand the situation on the field from different positions. Its training requirements for players are not affected by the venue, time, personnel, equipment and climate. At the same time, multiple athletes can be evaluated at the same time, thereby effectively improving communication and exchanges between athletes. Evaluation operation of situational awareness of virtual reality football players. When players are conducting situational awareness evaluation, they need to wear "head mounted display" (HMD). The player's vision is completely wrapped and immersed in the virtual reality football scene. When the player moves around, the virtual scene that can be seen will change completely with the position and angle of the eyes, just like in the real world. After watching a video in VR from the first perspective, the player uses the joystick to complete the four tasks of situational awareness mentioned above, and thus evaluates the player's situational awareness.

[0124] The technologies that can be combined for the collection of multimodal data in this scheme include: (1) EEG technology: This technology is very popular in cognitive neuroscience research and can collect neural activity indicators that reflect cognitive processes; it can further collect neural activity indicators behind the four dimensions of football players' situational awareness, thereby further providing physiological dimension indicators for the measurement of situational awareness; (2) Near-infrared technology (fNIRS): This technology can avoid the influence of ERP (such as eye movement) on EEG data and collect brain imaging indicators that reflect the four dimensions of situational awareness; (3) Eye movement equipment: Eye movement indicators can reflect the changes in the subjects' eye movements when performing situational awareness tasks, especially for comparing the eye movement trajectories of people with different levels. In addition, we can understand the eye activities of football players when watching videos in VR and explore the impact of eye activities on the level of situational awareness. (4) Motion capture: athletes can interact with VR scenes during the prediction and decision-making process and collect their corresponding movements. Motion capture can be performed through a device similar to a handle controller to evaluate the athlete's behavior. (5) Other related physiological indicators: Physiological indicators such as skin electricity, heart rate, and skin temperature can be used to reflect the physiological state during the test. They can be used as control indicators of the test process or as evaluation indicators of the test process, thereby further verifying the relationship between the four dimensions of situational awareness and physiological indicators.

[0125] The personnel situational awareness assessment method based on a simulated training environment proposed in the present application can receive personnel's assessment feedback information on situational awareness tasks based on VR devices worn by personnel, collect multimodal data of personnel in the situational awareness assessment process based on collection devices worn by personnel, obtain assessment results based on the assessment feedback information according to preset scoring rules, analyze the multimodal data using a pre-trained multimodal data evaluation model to obtain multimodal assessment scores, present situational awareness scenes to players in a first-person perspective through VR devices to enhance the sense of immersion, score the assessment feedback information through objective scoring rules, and conduct multimodal assessment based on objective multimodal data, thereby ensuring the ecological validity of the assessment results.

[0126] Furthermore, the method obtains the trajectory of the participants and the ball in the court based on the two-dimensional video analysis, and then models it based on the VR scene material library, builds a simulated training environment corresponding to the two-dimensional video, and then issues situational awareness tasks in the simulated training environment. The constructed simulation training environment supports multi-perspective switching, can evaluate different participants, and provides multiple perspectives to understand the entire game globally. In addition, the method supports a variety of settings and customized functions, thereby providing personalized services for users, better helping players and teams develop better situational awareness capabilities, and thus achieve better results.

[0127] From the perspective of the players, not only can they experience the actual situation on the court from a first-person perspective by observing the situational awareness tasks, but they can also switch perspectives to understand the dynamics on the court from multiple perspectives. In addition, through the preset evaluation system and the evaluation model based on multimodal data, the actual performance and status of the players can be understood from the physiological data level, which can make the room for future improvement clearer and prepare for better results.

[0128] Corresponding to the above method, the present application also provides a personnel situational awareness assessment system based on a simulated training environment, the system including a computer device, the computer device including a processor and a memory, the memory storing computer instructions, the processor being used to execute the computer instructions stored in the memory, and when the computer instructions are executed by the processor, the system implements the steps of the method described above.

[0129] In particular, football situational awareness is at the forefront of sports cognitive research. This project adopts the multi-target tracking research method in psychology, using virtual reality combined with eye movement, heart rate, skin conduction, brain function (such as functional near-infrared spectroscopy imaging technology), motion capture and other data, from perception, understanding, prediction to decision-making, from behavioral performance, physiological response eye movement strategy and neural mechanism, to provide a systematic theoretical research framework and effective evaluation and training methods for football situational awareness. On this basis, this project will construct a multi-target tracking model for situational awareness, and the research results can be extended to various fields involving dynamic and complex multi-target situations.

[0130] This project closely combines psychology with football, and uses the theories, methods and techniques of psychology to study important issues in football. It can provide a theoretical framework and research methods for football situational awareness, and apply it to the special assessment and training of football players' situational awareness to improve the level of football. Based on experimental research, this project will construct a multi-target tracking model for situational awareness. The research results are not limited to the field of football, but can be transferred to basketball, volleyball and other collective ball sports, which are also the "three major balls". It can also be extended to a wider range of research and practice involving dynamic and complex multi-target situations, such as transportation, military, human-computer interaction and other fields.

[0131] The embodiment of the present application also provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, the steps of the method described above are implemented. The computer-readable storage medium can be a tangible storage medium, such as a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a floppy disk, a hard disk, a removable storage disk, a CD-ROM, or any other form of storage medium known in the technical field.

[0132] It should be understood by those skilled in the art that the exemplary components, systems and methods described in conjunction with the embodiments disclosed herein can be implemented in hardware, software or a combination of the two. Whether it is specifically performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application. When implemented in hardware, it can be, for example, an electronic circuit, an application specific integrated circuit (ASIC), appropriate firmware, a plug-in, a function card, etc. When implemented in software, the elements of this application are programs or code segments used to perform the required tasks. The program or code segment can be stored in a machine-readable medium, or transmitted on a transmission medium or a communication link via a data signal carried in a carrier.

[0133] It should be clear that the present application is not limited to the specific configuration and processing described above and shown in the figures. For the sake of simplicity, a detailed description of the known method is omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of the present application is not limited to the specific steps described and shown, and those skilled in the art can make various changes, modifications and additions, or change the order between the steps after understanding the spirit of the present application.

[0134] In the present application, features described and / or illustrated for one embodiment may be used in the same manner or in a similar manner in one or more other embodiments, and / or combined with features of other embodiments or replace features of other embodiments.

[0135] The above description is only the preferred embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the embodiments of the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A method for evaluating human factors intelligent situational awareness based on a simulated training environment, characterized in that: The method comprises: Presenting a situational awareness task of preset dimensions in a simulated training environment to a person wearing a virtual reality device from a preset perspective, wherein the dimensions of the situational awareness task include a perception dimension, an understanding task dimension, a prediction task dimension, and a decision-making task dimension; Receiving evaluation feedback information performed by the personnel according to the received situational awareness task of the preset dimension; The evaluation feedback information is recorded, and the evaluation result is obtained by comparing the evaluation feedback information with the evaluation rules in the preset evaluation system.

2. The method according to claim 1, characterized in that The method further comprises: In the process of the person wearing the virtual reality device observing the situational awareness task of the preset dimension and giving the evaluation feedback information, the multimodal data of the person to be evaluated wearing the virtual reality device is synchronously collected by using the collection device worn by the person; wherein the multimodal data includes at least eye movement data and motion capture data; After obtaining the evaluation results, the collected multimodal data is scored using a pre-trained multimodal data evaluation model to obtain the multimodal evaluation results of the person to be evaluated.

3. The method according to claim 2, characterized in that The acquisition device includes a near-infrared spectral imaging device and / or a physiological data acquisition device, and the physiological data acquisition device includes a heart rate acquisition device and / or a skin charge acquisition device; The multimodal data also includes one or more of brain function data collected by a functional near-infrared spectroscopy imaging device, heart rate data collected by a heart rate collection device, and skin electricity data collected by a skin electricity collection device.

4. The method according to claim 2, characterized in that: The method also includes the step of pre-training the multimodal data evaluation model, wherein the step of training the multimodal data evaluation model includes: Establish a multimodal data evaluation model based on linear regression analysis and initialize weights and biases; The multimodal data of multiple personnel collected during the preprocessed historical evaluation process and the corresponding evaluation results are used as training sets and input into the linear regression model to calculate the equation of the degree of fit between the actual evaluation results and the predicted evaluation results as the loss function; Inputting the training set into the initialized multimodal data evaluation model for iterative optimization; wherein the preprocessing includes data cleaning and data enhancement; When the loss function converges, the pre-training of the multimodal data evaluation model is completed.

5. The method according to claim 1, characterized in that The method further comprises: A flat video including the stadium environment is pre-input; Analyzing frame by frame to identify the positions of various players and the ball in the court in the two-dimensional video; Generate the position coordinates of each participant and the ball in each frame of the video in the standard court coordinate system; Based on the corresponding position coordinates in the standard court coordinate system, the spatial motion trajectory of each participant and the ball motion trajectory that change with time are obtained; wherein the participants include multiple players, referees and / or coaches of each team; The preset virtual participant model is associated with the spatial motion trajectory of each participant, and the preset virtual ball model is associated with the ball motion trajectory, and the virtual court model, virtual participant model and virtual ball model are combined to build a simulated training environment matched by the plane video.

6. The method according to claim 5, characterized in that The method further comprises: The simulation training environment supports switching between different perspective identities, wherein participants with different perspective identities pre-establish corresponding different situational awareness tasks; The virtual court model, virtual participant model and virtual ball model are created by retrieving a preset virtual scene material library.

7. The method according to claim 1, characterized in that The presentation of the situational awareness tasks supports customization of the presentation quantity, dimension, and order.

8. The method according to claim 1, characterized in that Before presenting the steps of a situational awareness task of a preset dimension in a simulated training environment to a person wearing the virtual reality device at a preset perspective, instructions or prompt information are presented to the person wearing the virtual device.

9. The method according to claim 1, characterized in that: The method further includes: The situational awareness tasks of preset dimensions and the evaluation feedback information of the personnel in the simulated training environment are converted into two-dimensional videos in the perspective of television broadcast and / or shooting; wherein, the different virtual participant models in the simulated training environment are converted into the virtual human avatars of the participants in the two-dimensional plane, the virtual ball model in the simulated training environment is converted into the virtual ball avatar in the two-dimensional plane, the spatial motion trajectory of the virtual participant is converted into the spatial motion trajectory of the virtual human participant in the two-dimensional plane and associated with the participant avatar, the ball motion trajectory is converted into the virtual ball trajectory in the two-dimensional plane and associated with the ball avatar, thereby converting it into a two-dimensional video in the perspective of television broadcast and / or shooting.

10. A human factor intelligent situational awareness assessment system based on a simulated training environment, comprising a processor and a memory, characterized in that: The memory stores computer instructions, and the processor is used to execute the computer instructions stored in the memory. When the computer instructions are executed by the processor, the system implements the steps of the method as claimed in any one of claims 1 to 9.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the method according to any one of claims 1 to 9 are implemented.

Images