Shooting training method based on mixed reality

Abstract

The invention discloses a mixed reality-based shooting training method, which comprises the following steps of: scanning and superposing surrounding environment characteristics through mixed reality head-mounted equipment to obtain a mixed reality shooting training environment, and configuring a virtual shooting target in the mixed reality shooting training environment; capturing movement track data and view angle parameters of a shooter through the mixed reality head-mounted equipment, predicting a future head movement track of the shooter through the movement track data and the view angle parameters of the shooter, and dynamically correcting a virtual shooting target position through the future head movement track of the shooter; shooting training data are collected by simulating a gun for shooting, the shooting hit effect is displayed through the mixed reality head-mounted device, and a vibration signal generated when the device is attacked is simulated through the vibration vest.

Description

Technical Field

[0001] The present invention relates to the technical field of shooting training, and in particular to a shooting training method based on mixed reality. Background Art

[0002] Shooting training based on mixed reality (MR) is a shooting training method that integrates real and virtual scenes. With the help of mixed reality technology, virtual targets or dynamic scenes are superimposed on the real environment to form a virtual-real and highly interactive training environment.

[0003] Use specialized mixed reality glasses or head-mounted devices to help shooters perceive both the real environment and the superimposed virtual content. Use modified real firearms or laser simulators to control factors such as aircraft trajectory, shooting accuracy, and recoil to provide a realistic firearms experience. Use SLAM (Simultaneous Localization and Mapping) technology or spatial sensors to capture real scenes and generate the location of virtual content in real time, such as generating enemy positions or dynamic target positions in real scenes. Equipped with vibration feedback handles, training gun vibrations, or wearable devices (such as tactical vests) to enhance the on-site experience. For example, when a shooter is "attacked" by a virtual target, a vibration signal can be received.

[0004] MR technology needs to track the shooter's position and posture (such as the direction and angle of head movement) in real time and align virtual objects with the real environment. The system generates inaccuracies when capturing the shooter's position and posture. For example, when the shooter turns his head quickly, the MR system may not be able to update the position of the virtual object in time, causing the virtual object to not match the real scene. When the shooter's head posture (direction or angle) changes dramatically, the MR system may not be able to respond quickly, causing the position of the virtual object to not match the real environment.

[0005] 3D registration refers to the technology of accurately superimposing virtual objects into real scenes. It requires that the position, angle and scale of virtual objects are completely aligned with the real environment. Time delay refers to the time difference between the shooter's movement and the completion of the virtual object adjustment. If the delay is too large, the virtual object seen by the shooter will not "keep up" with his own movements. This delay will not only destroy the sense of immersion, but may also cause dizziness and discomfort (especially in high-intensity applications such as military training).

[0006] The purpose of the present invention is to design a shooting training method based on mixed reality in view of the above-mentioned problems in the prior art. Summary of the invention

[0007] In view of this, the purpose of the present invention is to propose a shooting training method based on mixed reality, which can solve the above-mentioned problems.

[0008] The present invention provides a shooting training method based on mixed reality, according to a shooting training system based on mixed reality, comprising:

[0009] A mixed reality head-mounted device, which is mounted on the shooter's head and is used to display a mixed reality shooting training environment and a virtual shooting target;

[0010] A simulated firearm for providing the form and tactile feedback of a training firearm;

[0011] A vibration vest is worn on the shooter's upper body to simulate the vibration signal when being attacked;

[0012] The method comprises:

[0013] The mixed reality head mounted device is used to scan and overlay the surrounding environment features to obtain a mixed reality shooting training environment, and a virtual shooting target is configured in the mixed reality shooting training environment;

[0014] The shooter's motion trajectory data and viewing angle parameters are captured through a mixed reality head-mounted device, the shooter's future head motion trajectory is predicted through the shooter's motion trajectory data and viewing angle parameters, and the position of the virtual shooting target is dynamically corrected through the shooter's future head motion trajectory;

[0015] Shooting training data is collected by simulating gun shooting, the shooting hit effect is displayed through a mixed reality headset, and the vibration signal when being attacked is simulated through a vibration vest.

[0016] Furthermore, an RGB-D camera is provided on the outside of the mixed reality head mounted device for acquiring color three-dimensional environmental features of the surrounding environment;

[0017] An inertial measurement unit is provided inside the mixed reality head mounted device, which is used to capture the movement trajectory data of the shooter's head and update the position and angle of the virtual object in real time;

[0018] A display screen is arranged inside the mixed reality head mounted device for displaying a mixed reality shooting training environment and a virtual shooting target to the shooter.

[0019] Furthermore, the method of obtaining a mixed reality shooting training environment by scanning and superimposing surrounding environment features through a mixed reality head mounted device includes:

[0020] Scan the surrounding environment with an RGB-D camera to obtain color 3D environmental features and generate a 3D feature point cloud of the environment;

[0021] The SLAM algorithm is used to generate an environmental model from the three-dimensional feature point cloud of the environment. The inertial measurement unit is used to obtain the shooter's head motion trajectory data in real time. The pose is estimated based on the user's head motion trajectory data to obtain the shooter's head pose.

[0022] The shooter's head posture and the environment model are fused through the extended Kalman filter to obtain a real-time environment model;

[0023] The training environment is superimposed on the real-time environment model to obtain a mixed reality shooting training environment, which is then displayed to the shooter through a display screen.

[0024] Further, configuring a virtual shooting target in a mixed reality shooting training environment includes:

[0025] In a mixed reality shooting training environment, analyze the three-dimensional position of shooting training environment features and generate target points;

[0026] Configure virtual shooting targets according to training objective requirements and superimpose the targets onto target points in the mixed reality shooting training environment.

[0027] Furthermore, the method of capturing the shooter's motion trajectory data and viewing angle parameters through the mixed reality head mounted device, predicting the shooter's future head motion trajectory through the shooter's motion trajectory data and viewing angle parameters, and dynamically correcting the virtual shooting target position through the shooter's future head motion trajectory includes:

[0028] The shooter's head motion trajectory data is obtained in real time through the inertial measurement unit, and the shooter's viewing angle parameters are obtained through the RGB-D camera;

[0029] The shooter's head motion trajectory data and viewing angle parameters are time-stamped and synchronized to extract the shooter's head trajectory features and visual features;

[0030] Input the shooter's head trajectory features and visual features into the motion trajectory prediction model to obtain the shooter's future head motion trajectory;

[0031] The virtual shooting target position is rendered in advance to the range of the shooter's future head movement trajectory.

[0032] Further, the extracting of the shooter's head trajectory features and visual features includes:

[0033] Extract the linear acceleration, angular velocity and absolute direction of the shooter's head according to the shooter's head motion trajectory data;

[0034] The key points of the shooter's gaze direction are extracted based on the shooter's head visual parameters.

[0035] Furthermore, the motion trajectory prediction model is obtained through the following steps:

[0036] Collect the shooter's historical head trajectory features and visual features, align the timestamps of the shooter's historical head trajectory features and visual features, and obtain a motion trajectory prediction model sample;

[0037] The motion trajectory prediction model is constructed through the LSTM model. The shooter's historical head trajectory features and visual features are taken as input, and the shooter's historical head trajectory features and visual features at the next time point are taken as output to train the motion trajectory prediction model.

[0038] Furthermore, rendering the virtual shooting target position in advance to the range of the shooter's future head movement trajectory includes:

[0039] Calculate the best target position within the shooter's future field of view based on the shooter's future head movement trajectory;

[0040] The optimal target position is input into the rendering engine to generate an image of the virtual shooting target in advance.

[0041] Furthermore, rendering the virtual shooting target position in advance to the range of the shooter's future head movement trajectory further includes:

[0042] If there is a deviation between the shooter's actual movement trajectory and the shooter's future head movement trajectory, the optimal target position within the shooter's future field of view is recalculated.

[0043] Furthermore, if there is a deviation between the actual motion trajectory of the shooter and the future head motion trajectory of the shooter, recalculating the optimal target position within the future field of view of the shooter includes:

[0044] Calculate the displacement deviation and angle deviation between the shooter's actual motion trajectory and the shooter's future head motion trajectory;

[0045] If the displacement deviation exceeds the first deviation threshold and the angle deviation exceeds the second deviation threshold, the optimal target position within the shooter's future field of view is recalculated according to the shooter's actual motion trajectory.

[0046] Beneficial effects of the present invention:

[0047] First, the RGB-D camera is used to scan the surrounding environment, generate color 3D point cloud data, and capture the spatial features of the shooter's environment in real time. The inertial measurement unit is used to capture the motion trajectory of the shooter's head, including acceleration, rotational angular velocity, and attitude angle information. The IMU data and RGB-D data are fused to generate a real-time environment model synchronized with the shooter's perspective. Virtual shooting targets are configured in the real-time environment model to ensure that the virtual objects are fully aligned with the real environment in terms of position, direction, and scale.

[0048] Second, based on the spatial characteristics of the real environment, consider the position of obstacles in the environment to avoid occlusion of virtual targets or fusion with the background. Extract areas suitable for placing targets in the training environment, and generate plane points or three-dimensional suspended target points as needed. According to the visibility of obstacles and the smoothness of the path, exclude unreasonable virtual target candidate points in the environment map. Virtual targets need to be accurately aligned with target points in the mixed reality environment and match their position, size, direction and other characteristics to achieve seamless integration of virtual targets and real scenes.

[0049] Third, the prediction model is used to achieve high-precision prediction of the shooter's future head movement trajectory. Even in the case of rapid head turns or complex movements, the system can calculate the shooter's future movement state in advance. Provide a basis for early rendering of the target position to ensure that the virtual target is always within the shooter's field of view. According to the shooter's future head movement trajectory and the key points of the gaze direction, predict the shooter's future field of view. Select the best target position within the future field of view to ensure that the target is clearly visible and meets training requirements (such as shooting range, unobstructed occlusion, etc.). BRIEF DESCRIPTION OF THE DRAWINGS

[0050] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0051] Figure 1 It is a flow chart of the method of embodiment 2. DETAILED DESCRIPTION

[0052] To facilitate understanding by those skilled in the art, the structure of the present invention is now further described in detail with reference to the embodiments and the accompanying drawings. It should be understood that the steps mentioned in the present embodiment, unless otherwise specified in their order, can be adjusted in order according to actual needs, and can even be executed simultaneously or partially simultaneously.

[0053] Embodiment 1

[0054] Embodiment 1 provides a shooting training system based on mixed reality, including:

[0055] A mixed reality head-mounted device, which is mounted on the shooter's head and is used to display a mixed reality shooting training environment and a virtual shooting target;

[0056] An inertial measurement unit is provided inside the mixed reality head mounted device, which is used to capture the movement trajectory data of the shooter's head and update the position and angle of the virtual object in real time;

[0057] A display screen is arranged inside the mixed reality head mounted device for displaying a mixed reality shooting training environment and a virtual shooting target to the shooter.

[0058] A simulated firearm for providing the form and tactile feedback of a training firearm;

[0059] A vibration vest is put on the upper body of the shooter to simulate the vibration signal when being attacked; an RGB-D camera is arranged on the outside of the mixed reality head mounted device to obtain the color three-dimensional environmental features of the surrounding environment;

[0060] In this embodiment, during the mixed reality shooting training process, it is necessary to instantly integrate the data streams of multiple sensors (motion trajectory, environmental characteristics, and viewing angle parameters) to generate dynamic responses. It is also necessary to maintain high performance (refresh rate with low latency) to achieve real-time updates. The mixed reality head-mounted device integrates an RGB-D camera (for environmental perception), an inertial measurement unit (IMU) (to capture head movement), and a display screen. By synchronizing multiple sensor data streams, the motion data of the shooter's head and the dynamic position of the virtual target are updated in real time to ensure the reasonable alignment and interaction between the virtual object and the real environment. In a mixed reality environment, the shooting scene requires not only the dynamic generation of virtual targets, but also the synchronization of the target with the actual movement of the shooter to provide an accurate and natural shooting experience. The mixed reality head display displays through the display screen: virtual shooting training targets generated in combination with real space features; virtual targets adaptively adjust their position and rotation angle within the shooter's viewing angle. Combining RGB-D environmental perception (building a real three-dimensional scene) and IMU data (shooter motion trajectory), the virtual target is seamlessly aligned with the real environment.

[0061] Traditional methods in shooting training (such as video demonstrations or simple somatosensory training equipment) often lack the simulation of the shooter's sense of presence and tactile feedback when training gun operation, and lack authenticity. The introduction of simulated guns has the appearance of real guns and tactile feedback, such as the feeling of trigger pull, recoil, etc. The built-in sensor module of the virtual gun can record: shooting direction, shooting time, training gun posture; provide tactile feedback to simulate the real shooting experience, and integrate shooting data into the virtual training scene.

[0062] In shooting training, it is difficult for traditional simulation methods to truly restore the vibration feedback after being attacked or hit by a bullet. This lack will affect the shooter's reaction and adaptability in training. Using a vibration vest, the vibration motor generates vibration signals in different areas to simulate the feedback of being shot: different vibrations are triggered according to the hit location (chest, abdomen, back, etc.); the vibration intensity can simulate the power of different training guns (such as pistols and rifles causing different impacts). With mixed reality training scenarios, shooters can instantly perceive the attacked part in the system response, improving the effectiveness of tactical and reaction training.

[0063] Embodiment 2

[0064] like Figure 1 As shown, the second embodiment provides a shooting training method based on mixed reality. According to the shooting training system based on mixed reality in the first embodiment, the specific method steps are as follows:

[0065] S1 uses a mixed reality head mounted device to scan and superimpose the surrounding environment features to obtain a mixed reality shooting training environment, and configures a virtual shooting target in the mixed reality shooting training environment;

[0066] S101 uses an RGB-D camera to scan the surrounding environment to obtain color 3D environmental features and generate a 3D feature point cloud of the environment;

[0067] S102 generates an environment model from the three-dimensional feature point cloud of the environment through the SLAM algorithm, obtains the shooter's head motion trajectory data in real time through the inertial measurement unit, and estimates the position and posture based on the user's head motion trajectory data to obtain the shooter's head position and posture;

[0068] S103 fuses the shooter's head posture with the environment model through an extended Kalman filter to obtain a real-time environment model;

[0069] S104 superimposes the training environment onto the real-time environment model to obtain a mixed reality shooting training environment, and displays the mixed reality shooting training environment to the shooter through a display screen.

[0070] In this step, rapid changes in the shooter's head position and posture (such as direction and angle) may cause the alignment of virtual objects with the real environment to fail. When the shooter turns his head quickly or moves significantly, the mixed reality system may not respond quickly, causing the virtual object to "drift" or "lag". The surrounding environment is scanned by an RGB-D camera to generate color three-dimensional point cloud data to capture the spatial characteristics of the shooter's environment in real time. The motion trajectory of the shooter's head is captured by an inertial measurement unit, including acceleration, rotational angular velocity, and attitude angle information. The SLAM algorithm (Simultaneous Localization And Mapping, synchronous localization and mapping is a core algorithm framework that mainly solves the problem of how to allow devices (such as robots, drones, mixed reality devices, etc.) to simultaneously locate their own position (Localization) and build a map of the surrounding environment (Mapping) in an unknown environment.) Use RGB-D point cloud and IMU data to dynamically generate an environmental model and calculate the shooter's posture (position and direction) in real time. The extended Kalman filter (EKF: Extended Kalman Filter) is an algorithm for state estimation of nonlinear systems. Through the fusion of sensor data, the current state of the uncertain system is estimated, and the accuracy of state estimation is optimized through recursive calculation to reduce the impact of noise and error. ) The IMU data and RGB-D data are fused to correct the sensor error and ensure high-precision tracking of the shooter's head position and posture. Based on the results of SLAM and EKF fusion, a real-time environment model synchronized with the shooter's perspective is generated. Virtual shooting targets are configured in the real-time environment model to ensure that the virtual objects are fully aligned with the real environment in position, direction and scale. The virtual shooting target can be accurately superimposed on the real scene and seamlessly integrated with the real environment to avoid "suspension" or "penetration". Improve the authenticity and immersion of the virtual scene.

[0071] S105 analyzes the three-dimensional position of the shooting training environment feature in the mixed reality shooting training environment and generates a target point;

[0072] S106 configures a virtual shooting target according to the training target requirements, and superimposes the target onto a target point in the mixed reality shooting training environment.

[0073] In this step, in mixed reality shooting training, the position of the virtual target needs to be generated based on the characteristics of the real environment. If the target point configuration is unreasonable (for example, it is blocked by obstacles or in an unreachable space), the shooter's training efficiency will be greatly reduced. Based on the spatial characteristics of the real environment, consider the position of obstacles in the environment to avoid occlusion of virtual targets or fusion with the background. Extract areas suitable for placing targets in the training environment, such as open areas or positions within the shooter's field of view. Generate plane points or three-dimensional suspended target points according to needs. According to the visibility of obstacles and the smoothness of the path, exclude unreasonable virtual target candidate points in the environment map. The virtual target needs to be accurately aligned with the target point in the mixed reality environment and match its position, size, direction and other characteristics to achieve seamless integration of the virtual target with the real scene. This involves the three-dimensional registration and superposition of virtual targets. Select appropriate virtual shooting targets (such as targets of different sizes, shapes or types) according to training requirements. Superimpose the position, angle and scale of the target onto the calculated three-dimensional target points. Ensure that the display of the virtual target is combined with the physical space through real-time rendering and alignment algorithms to avoid offset or suspension. When training requirements (such as target difficulty, number of training targets) change, virtual targets are configured in combination with training target requirements (such as target size, distance, location distribution difficulty, etc.). For example, the number of targets can be increased or decreased according to training requirements; the target's movement speed can be increased, and the target's visible range can be reduced.

[0074] S2 captures the shooter's motion trajectory data and viewing angle parameters through a mixed reality head-mounted device, predicts the shooter's future head motion trajectory through the shooter's motion trajectory data and viewing angle parameters, and dynamically corrects the virtual shooting target position through the shooter's future head motion trajectory;

[0075] S201 obtains the shooter's head motion trajectory data in real time through the inertial measurement unit and obtains the shooter's viewing angle parameters through the RGB-D camera;

[0076] S202 synchronizes the shooter's head motion trajectory data and the viewing angle parameters with timestamps, and extracts the shooter's head trajectory features and visual features;

[0077] S2021 extracts the linear acceleration, angular velocity, and absolute direction of the shooter's head according to the shooter's head motion trajectory data;

[0078] S2022 extracts key points of the shooter's gaze direction according to the shooter's head visual parameters.

[0079] In this step, during shooting training, the shooter's head movement (such as rapid head turning, pitching, rotation, etc.) will cause rapid changes in perspective. If the virtual target cannot respond to the shooter's movement in time, the target may be out of view, the position may lag, or the alignment may be inaccurate, destroying the immersion and training effect. It is necessary to capture the shooter's movement trajectory in real time and predict his future head movement to adjust the position of the virtual target in advance to ensure that the target is always within the shooter's field of view. Therefore, it is necessary to collect the shooter's head movement trajectory data and the shooter's perspective parameters in real time.

[0080] S203 inputs the shooter's head trajectory features and visual features into a motion trajectory prediction model to obtain the shooter's future head motion trajectory;

[0081] Furthermore, the motion trajectory prediction model is obtained through the following steps:

[0082] Collect the shooter's historical head trajectory features and visual features, align the timestamps of the shooter's historical head trajectory features and visual features, and obtain a motion trajectory prediction model sample;

[0083] The motion trajectory prediction model is constructed through the LSTM model. The shooter's historical head trajectory features and visual features are taken as input, and the shooter's historical head trajectory features and visual features at the next time point are taken as output to train the motion trajectory prediction model.

[0084] In this step, during shooting training, the shooter's head movement (such as turning, pitching, rotating, etc.) is uncertain and may change rapidly. If the shooter's future motion trajectory cannot be accurately predicted, the virtual target may not be able to adjust its position in time, causing the target to lag behind or go out of view. Traditional simple prediction methods (such as linear extrapolation) are difficult to handle complex nonlinear motion trajectories. LSTM (Long Short-Term Memory Network) is a deep learning model suitable for processing time series data, which can capture long-term and short-term dependencies in historical trajectories. The shooter's historical head trajectory features (such as linear acceleration, angular velocity, absolute direction) and visual features (such as gaze direction key points) are used as input to train the LSTM model to predict future head motion trajectories. By aligning historical data with timestamps, high-quality motion trajectory prediction samples are generated to improve the accuracy of the model. A high-precision prediction of the shooter's future head motion trajectory is achieved, and the system can calculate the shooter's future motion state in advance even in the case of fast head turning or complex movements. Provide a basis for early rendering of the target position to ensure that the virtual target is always within the shooter's field of view.

[0085] S204 renders the virtual shooting target position in advance to the range of the shooter's future head movement trajectory.

[0086] S2041 calculates the best target position within the shooter's future field of view based on the shooter's future head movement trajectory;

[0087] S2042 inputs the optimal target position into the rendering engine and generates an image of the virtual shooting target in advance.

[0088] In this step, during shooting training, the virtual target needs to adjust its position in advance according to the shooter's future motion state. If the target position cannot be updated in time, the following problems may occur: the target lags behind the shooter's field of view and is out of the shooter's visible range; the target position does not match the shooter's perspective, affecting shooting accuracy and training effects. Rendering the target in advance requires calculating the optimal target position within the shooter's future field of view, while ensuring the real-time and accuracy of rendering. According to the shooter's future head movement trajectory, combined with the gaze direction key points, the shooter's future field of view is predicted. The optimal target position is selected within the future field of view to ensure that the target is clearly visible and meets the training requirements (such as shooting range, unobstructed occlusion, etc.). The optimal target position is input into the rendering engine, and a virtual target image is generated in advance and displayed within the shooter's future field of view. GPU-accelerated rendering pipelines (such as Vulkan or DirectX 12) are used to ensure real-time rendering. The virtual target can be adjusted to the shooter's future field of view in advance, and the target always remains aligned with the perspective even if the shooter moves quickly or turns his head. Improve the real-time and accuracy of target display and avoid the problem of target lagging or being out of view.

[0089] Furthermore, rendering the virtual shooting target position in advance to the range of the shooter's future head movement trajectory further includes:

[0090] If there is a deviation between the shooter's actual movement trajectory and the shooter's future head movement trajectory, the best target position within the shooter's future field of view is recalculated, specifically:

[0091] Calculate the displacement deviation and angle deviation between the shooter's actual motion trajectory and the shooter's future head motion trajectory;

[0092] If the displacement deviation exceeds the first deviation threshold and the angle deviation exceeds the second deviation threshold, the optimal target position within the shooter's future field of view is recalculated according to the shooter's actual motion trajectory.

[0093] In this step, the actual motion trajectory of the shooter may deviate from the output of the future trajectory prediction model. This deviation may be caused by the following reasons: there is uncertainty in the calculation of the motion trajectory prediction model; the shooter's head motion trajectory contains sudden asymmetric acceleration, prohibited mode or uncaptured changes. If the system always relies on the predicted trajectory to adjust the target position and lacks the ability to respond to the deviation, it may cause the virtual target to be out of the shooter's field of view, affecting the training effect and perceptual experience. The difference in straight-line distance between the actual position and the predicted position of the shooter's head. The angle between the actual direction and the predicted direction of the shooter's head. Set a threshold mechanism (first deviation threshold + second deviation threshold). When the displacement deviation exceeds the first deviation threshold and the angle deviation exceeds the second deviation threshold at the same time, the predicted trajectory is judged to be invalid and the virtual target position needs to be recalculated. Use the actual motion trajectory as a new input to recalculate the optimal target position within the future field of view according to the shooter's latest state. When there is an error in the predicted trajectory, the target position can be adaptively adjusted to avoid the target being out of the shooter's field of view or misjudgment. Build a more robust dynamic target adjustment strategy to enhance the stability and flexibility of training.

[0094] S3 collects shooting training data by simulating gun shooting, displays the shooting hit effect through a mixed reality headset, and simulates the vibration signal when being attacked through a vibration vest.

[0095] In this step, when conducting mixed reality training, it is necessary to accurately collect the shooter's data in shooting training, for example, the relevant parameters when training gun shooting (such as shooting direction, timing, shooting frequency, etc.). Specific effect data on whether the shooting is hit or not. The simulated gun captures shooting-related data (such as shooting angle, direction, strength and shooting time point) through built-in sensors, and transmits it to the mixed reality training system. Real-time interaction is achieved by synchronizing shooting data with environmental target status (target position, hit judgment, etc.), and accurate data integration is performed by combining position tracking and shooting feedback. This part of the content is prior art and will not be repeated in the present invention.

[0096] In order to train the shooter's shooting accuracy and improve the training effect, the hit effect needs to be intuitively presented to the shooter through a mixed reality head-mounted device. After shooting, the MR device can display the animation effect of the target being hit (such as target color change, explosion, score display, etc.). Spatial calculation is used to ensure that the displayed hit point is consistent with the actual shooting direction. For example, the corresponding damage or flash mark is displayed after the target is hit. This part of the content is prior art and will not be repeated in this invention.

[0097] In order to simulate shooting scenes more comprehensively, it is necessary to provide physical feedback (such as tactile vibration) with a sense of presence when being attacked, so that the shooter can feel the "attack" and generate tension or tactical response. The shooting training system determines whether the shooter is hit by a virtual bullet and generates a vibration signal of specified intensity and duration. The vibration vest can trigger vibration feedback in specific areas (such as the chest, back, waist, etc.) to accurately locate the shooter's hit part. This part of the content is prior art and will not be elaborated in the present invention.

[0098] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0099] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0100] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0101] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0102] It should be noted that in the claims, any reference signs placed between brackets shall not be construed as limiting the claims. The word "comprising" does not exclude the presence of components or steps not listed in the claim. The word "a" or "an" preceding a component does not exclude the presence of a plurality of such components. The invention may be implemented by means of hardware comprising several different components and by means of a suitably programmed computer. In a unit claim enumerating several means, several of these means may be embodied by the same item of hardware. The use of the words first, second, and third etc. does not indicate any order. These words may be interpreted as names.

[0103] Although the preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.

[0104] Obviously, those skilled in the art may make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

[0105] In the present invention, unless otherwise clearly specified and limited, the terms "installed", "connected", "connected", "fixed" and the like should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium, it can be the internal connection of two elements or the interaction relationship between two elements. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0106] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms should not be understood as necessarily referring to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, unless they are contradictory.

Claims

1. A shooting training method based on mixed reality, characterized in that: According to a shooting training system based on mixed reality, comprising: A mixed reality head-mounted device, which is mounted on the shooter's head and is used to display a mixed reality shooting training environment and a virtual shooting target; A simulated firearm for providing the form and tactile feedback of a training firearm; A vibration vest is worn on the shooter's upper body to simulate the vibration signal when being attacked; The method comprises: The mixed reality head mounted device is used to scan and overlay the surrounding environment features to obtain a mixed reality shooting training environment, and a virtual shooting target is configured in the mixed reality shooting training environment; The shooter's motion trajectory data and viewing angle parameters are captured through a mixed reality head-mounted device, the shooter's future head motion trajectory is predicted through the shooter's motion trajectory data and viewing angle parameters, and the position of the virtual shooting target is dynamically corrected through the shooter's future head motion trajectory; Shooting training data is collected by simulating gun shooting, the shooting hit effect is displayed through a mixed reality headset, and the vibration signal when being attacked is simulated through a vibration vest.

2. A shooting training method based on mixed reality according to claim 1, characterized in that: The mixed reality head mounted device is provided with an RGB-D camera on the outside thereof for acquiring color three-dimensional environmental features of the surrounding environment; An inertial measurement unit is provided inside the mixed reality head mounted device, which is used to capture the movement trajectory data of the shooter's head and update the position and angle of the virtual object in real time; A display screen is arranged inside the mixed reality head mounted device for displaying a mixed reality shooting training environment and a virtual shooting target to the shooter.

3. A shooting training method based on mixed reality according to claim 2, characterized in that: The mixed reality shooting training environment is obtained by scanning and superimposing the surrounding environment features through the mixed reality head mounted device, including: Scan the surrounding environment with an RGB-D camera to obtain color 3D environmental features and generate a 3D feature point cloud of the environment; The SLAM algorithm is used to generate an environmental model from the three-dimensional feature point cloud of the environment. The inertial measurement unit is used to obtain the shooter's head motion trajectory data in real time. The pose is estimated based on the user's head motion trajectory data to obtain the shooter's head pose. The shooter's head posture and the environment model are fused through the extended Kalman filter to obtain a real-time environment model; The training environment is superimposed on the real-time environment model to obtain a mixed reality shooting training environment, which is then displayed to the shooter through a display screen.

4. The shooting training method based on mixed reality according to claim 1, characterized in that: Configuring a virtual shooting target in a mixed reality shooting training environment includes: In a mixed reality shooting training environment, analyze the three-dimensional position of shooting training environment features and generate target points; Configure virtual shooting targets according to training objective requirements and superimpose the targets onto target points in the mixed reality shooting training environment.

5. A shooting training method based on mixed reality according to claim 4, characterized in that: The method of capturing the shooter's motion trajectory data and viewing angle parameters by using the mixed reality head mounted device, predicting the shooter's future head motion trajectory by using the shooter's motion trajectory data and viewing angle parameters, and dynamically correcting the virtual shooting target position by using the shooter's future head motion trajectory comprises: The shooter's head motion trajectory data is obtained in real time through the inertial measurement unit, and the shooter's viewing angle parameters are obtained through the RGB-D camera; The shooter's head motion trajectory data and viewing angle parameters are time-stamped and synchronized to extract the shooter's head trajectory features and visual features; Input the shooter's head trajectory features and visual features into the motion trajectory prediction model to obtain the shooter's future head motion trajectory; The virtual shooting target position is rendered in advance to the range of the shooter's future head movement trajectory.

6. A shooting training method based on mixed reality according to claim 5, characterized in that: The extracting of the shooter's head trajectory features and visual features comprises: Extract the linear acceleration, angular velocity and absolute direction of the shooter's head according to the shooter's head motion trajectory data; The key points of the shooter's gaze direction are extracted based on the shooter's head visual parameters.

7. The shooting training method based on mixed reality according to claim 5, characterized in that: The motion trajectory prediction model is obtained through the following steps: Collect the shooter's historical head trajectory features and visual features, align the timestamps of the shooter's historical head trajectory features and visual features, and obtain a motion trajectory prediction model sample; The motion trajectory prediction model is constructed through the LSTM model. The shooter's historical head trajectory features and visual features are taken as input, and the shooter's historical head trajectory features and visual features at the next time point are taken as output to train the motion trajectory prediction model.

8. The shooting training method based on mixed reality according to claim 5, characterized in that: The step of rendering the virtual shooting target position in advance to the range of the shooter's future head movement trajectory includes: Calculate the best target position within the shooter's future field of view based on the shooter's future head movement trajectory; The optimal target position is input into the rendering engine to generate an image of the virtual shooting target in advance.

9. The shooting training method based on mixed reality according to claim 5, characterized in that: The step of rendering the virtual shooting target position in advance to the range of the shooter's future head movement trajectory further includes: If there is a deviation between the shooter's actual movement trajectory and the shooter's future head movement trajectory, the optimal target position within the shooter's future field of view is recalculated.

10. A shooting training method based on mixed reality according to claim 9, characterized in that: If there is a deviation between the actual motion trajectory of the shooter and the future head motion trajectory of the shooter, recalculating the optimal target position within the future field of view of the shooter includes: Calculate the displacement deviation and angle deviation between the shooter's actual motion trajectory and the shooter's future head motion trajectory; If the displacement deviation exceeds the first deviation threshold and the angle deviation exceeds the second deviation threshold, the optimal target position within the shooter's future field of view is recalculated according to the shooter's actual motion trajectory.

Images