Display method, device and storage medium based on virtual training scene training

By building a two-dimensional map of the virtual training scene and displaying character logos, the problem of conditions restricted by traditional practical training is solved, the virtual training effect is improved and managed, and the training effect is strengthened.

CN113379870BActive Publication Date: 2024-11-01XIMMERSE LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202110269404.0
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-03-12
Publication Date
2024-11-01
Estimated Expiration
2041-03-12

AI Technical Summary

Technical Problem

Traditional practical training is limited by funding, venue and other conditions, and the training effect is limited. Virtual simulation training, as a future development trend, requires a method that can improve the training effect.

Method used

By obtaining the data information of the virtual training scene, a two-dimensional map is built, and combining the position information of the head-mounted display device, the role logo is displayed in real time, covering it on the two-dimensional map, realizing the display and management of the virtual training scene.

Benefits of technology

Real-time management and improvement of the user's training process is achieved, helping backend personnel understand the training situation, guide improvements, and strengthen training effects.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN113379870B_ABST
    Figure CN113379870B_ABST
Patent Text Reader

Abstract

The present application provides a display method based on virtual training scene training, including: obtaining data information of the virtual training scene, the data information including building information, virtual object information and virtual character information; constructing a two-dimensional map according to the data information; obtaining the position information of the head-mounted display device in the virtual scene, and the role identification of the head-mounted display device; displaying the role identification together with the two-dimensional map, wherein the role identification is overlaid on the two-dimensional map, and the overlaid position of the role identification in the two-dimensional map corresponds to the position information of the terminal device in the virtual scene. The display method provided by the present application enables the backstage manager to observe the actual process of the user's training in real time, and can intuitively understand the quality of the user's training situation, which is helpful to guide and improve the subsequent training and enhance the effect of the training.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of virtual training, and in particular to a display method, device and storage medium based on virtual training scene training. Background Art

[0002] Traditional combat training is limited by funding, venues and other conditions. There are problems such as insufficient training bases, outdated training facilities and equipment, and backward training models. Combat training is significantly affected by objective conditions. Virtual simulation training is a way of training that combines reality and virtuality. It is a future technology development trend. It uses hybrid display technology and relies on simple existing venues to innovate new ideas for combat teaching, continuously broaden the use of training skills, and allow students to experience the handling skills in different situations as if they were there, so as to achieve better learning experience and teaching results, and minimize the impact of objective conditions. Summary of the invention

[0003] The embodiments of the present application provide a display method, device, background computer and storage medium based on virtual training scene training, which can improve the training effect of users.

[0004] In a first aspect, an embodiment of the present application provides a display method based on virtual training scene training, the method comprising: obtaining data information of the virtual training scene, the data information comprising building information, virtual object information and virtual character information; constructing a two-dimensional map according to the data information; obtaining position information of a head-mounted display device in the virtual scene, and a role identifier of the head-mounted display device; comparing the role identifier with the two-dimensional map; Figure 1 The character identification is displayed together, wherein the character identification is overlaid on the two-dimensional map, and the overlaid position of the character identification in the two-dimensional map corresponds to the position information of the terminal device in the virtual scene.

[0005] In a second aspect, an embodiment of the present application provides a display device based on virtual training scene training, the device comprising: a first acquisition module for acquiring data information of the virtual training scene, the data information comprising building information, virtual object information and virtual character information; a construction module for constructing a two-dimensional map according to the data information; a second acquisition module for acquiring position information of a head-mounted display device in the virtual scene, and a role identification of the head-mounted display device; and a display module for comparing the role identification with the two-dimensional map. Figure 1 The character identification is displayed together, wherein the character identification is overlaid on the two-dimensional map, and the overlaid position of the character identification in the two-dimensional map corresponds to the position information of the terminal device in the virtual scene.

[0006] In a third aspect, an embodiment of the present application provides a background computer, comprising: one or more processors; a memory; one or more applications, wherein the one or more applications are stored in the memory and configured to be executed by the one or more processors, and the one or more programs are configured to execute the training method provided in the first aspect above.

[0007] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, in which a program code is stored. The program code can be called by a processor to execute the training method provided in the first aspect above.

[0008] The display method, device, background computer and storage medium based on virtual training scene training provided in this application can enable the background manager to observe the actual process of user training in real time, and can intuitively understand the quality of the user's training situation, which is helpful to provide guidance and improvement for subsequent training and enhance the effect of training. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] In order to more clearly illustrate the technical solution of the present application, the drawings required for use in the implementation manner will be briefly introduced below. Obviously, the drawings described below are only some implementation manners of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0010] Figure 1 A schematic diagram of a training system provided for one embodiment of the present application.

[0011] Figure 2 An overall schematic diagram of a firearm in a training system provided by one embodiment of the present application.

[0012] Figure 3 for Figure 2 An overall schematic diagram of the gun shown in another direction.

[0013] Figure 4 for Figure 2 A framework diagram of the main internal structure of the firearm shown.

[0014] Figure 5 for Figure 4 A schematic diagram of the firearm shown in another state.

[0015] Figure 6 for Figure 2 The overall schematic diagram of the gun and holster combined is shown.

[0016] Figure 7 This is a flow chart of a training method using a training gun according to an embodiment of the present application.

[0017] Figure 8 This is a flowchart of a tactical board display method based on virtual training according to an embodiment of the present application.

[0018] Fig. 9 Illustration of a 2D map displayed on a backend computer for user training.

[0019] Fig.10a , 10b A schematic diagram of a role identifier and a first observation identifier.

[0020] Fig.11a , 11b A schematic diagram of the role identification and the first and second observation identification.

[0021] Fig.12 A schematic diagram of the role identification and the first, second, and third observation identification. DETAILED DESCRIPTION

[0022] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. The components of the embodiments of the present application described and shown in the drawings here can be arranged and designed in various different configurations.

[0023] Therefore, the following detailed description of the embodiments of the present application provided in the accompanying drawings is not intended to limit the scope of the present application for which protection is sought, but merely represents selected embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of the present application.

[0024] See also Figure 1 , a training system provided by an embodiment of the present application is applied to the field of mixed reality, augmented reality and / or virtual reality technology, and the training system includes a training device 100 and a head-mounted display device 200. Among them, the training device 100 can be identified and tracked by a camera set outside, and the user can wear the head-mounted display device 200 and view the virtual training scene through the head-mounted display device 200. The user's operation of using the training device 100 can be mapped to the virtual training scene, so that the user can perform virtual scene training. Among them, the camera that identifies and tracks the training device 100 can be integrated on the head-mounted display device 200, and the head-mounted display device 200 is a device that can simultaneously view the real world scene and the virtual training scene, that is, the user can see the real world scene through the head-mounted display device 200, and can also see the virtual training scene superimposed on the real world scene.

[0025] In some embodiments, the head-mounted display device 200 may include a display and an optical lens. The display may project the displayed virtual training scene onto the optical lens and reflect it through the optical lens to the eyes of a user wearing the head-mounted display device, so that the user can view the virtual training scene. In addition, the wearing user can also view the real scene through the optical lens, thereby allowing the user to visually view the virtual and real scene in which the virtual training scene and the real scene are superimposed and integrated.

[0026] In some embodiments, the training system also includes a reference marker 300. The camera on the head-mounted display device 200 can identify and track the reference marker 300, and use the reference marker as the origin of the relative world coordinate system to construct a virtual training scene that is fixed relative to the real world, and map the position of the head-mounted display device 200 (that is, the user) relative to the reference marker to the virtual training scene, thereby obtaining the user's position in the virtual training scene so that the user can perform virtual training.

[0027] In one embodiment, considering the large spatial range of the virtual training scene, as the user moves in the virtual training scene, if only one reference marker is used, the reference marker may leave the visual range of the camera of the head-mounted display device, resulting in the problem of not being able to obtain the user's position in the virtual training scene. Multiple reference markers with positional associations and different identities can be set in the real scene, so that no matter how the user moves in the virtual training scene, there will be no problem of not being able to obtain the user's position in the virtual training scene.

[0028] In one embodiment, a VIO tracking module may be provided in the head mounted display device. When the marker leaves the visual range of the camera of the head mounted display device, the VIO positioning technology may be used to assist in obtaining the real position of the head mounted display device, thereby obtaining the position of the user in the virtual training scene. In some embodiments, the VIO positioning technology may be directly used to obtain the position of the user in the virtual training scene without using a reference marker.

[0029] In the field of training, there are many types of training equipment. Figure 2-4 An embodiment of the present application provides a training gun 100a that can be used for virtual training, including a main body 10 and a sleeve 20 mounted on the main body 10, wherein the sleeve 20 can slide along a specific direction relative to the main body 10 under the action of an external force.

[0030] The main body 10 includes a handle 11, a gun frame 12 and a trigger 13. One end of the handle 11 is connected to the gun frame 12. The trigger 13 is located at the intersection of the handle 11 and the gun frame 12 and is also connected to the gun frame 12. A power supply 41, a main board 42, a bullet changing module 43, a bullet loading module 44, a shooting module 45 and a chambering module 46 are arranged inside the main body 10. The power supply 41, the bullet changing module 43 and the bullet loading module 44 are arranged in the handle 11, the shooting module 45 is arranged in the gun frame 12, and the main board 42 and the chambering module 46 are arranged above the gun frame 12 and covered by the sleeve 20. A reloading button 111 is provided on one side of the grip 11. When the user is holding the firearm 100, the user's thumb can press the reloading button 111 to activate the reloading module 43, thereby causing the mainboard 42 to trigger a control instruction corresponding to pressing the reloading button 111. The control instruction is to control the virtual firearm in the virtual training scene to detach the magazine from the virtual firearm, that is, a reloading instruction. A loading assembly 113 is provided on one end of the grip 11 away from the gun frame 12. When the user is holding the firearm 100, the user can use the other hand to assist in pressing the loading assembly 113 to activate the loading module 43, thereby causing the mainboard 42 to trigger a control instruction corresponding to pressing the loading assembly 113. The control instruction is to control the virtual firearm in the virtual training scene to load a new magazine, that is, a loading instruction. When holding the gun 100, the user can press the trigger 13 with the index finger to activate the shooting module 45, so that the main board 42 triggers the control instruction corresponding to pressing the trigger 13, and the control instruction is to control the virtual gun in the virtual training scene to shoot. When holding the gun 100, the user can use another hand to assist in sliding the sleeve 20 relative to the body 10 to activate the loading module 46, so that the main board 42 triggers the control instruction to load the virtual gun in the virtual training scene.

[0031] In some embodiments, the grip 11 is provided with a sensor 14 on the side facing away from the muzzle and adjacent to the gun rack 12, and the sensor 14 can be used to sense whether the gun 100a is held by the user's hand. In the actual use scenario of a real gun, there are certain requirements for when the user starts holding the gun. Therefore, the present application provides a sensor 14 on the side facing away from the muzzle and adjacent to the gun rack 12, so that the time period in which the user holds the gun 100a can be sensed, and then feedback can be given to the virtual training scenario, which helps to further improve the user's training effect.

[0032] In some embodiments, a sensor 15 is provided on the side of the gun frame 12 near the trigger 13, and the sensor 15 is used to sense whether the finger of the user holding the gun 100a is on the side of the gun frame. When the user holds a real gun, in a non-shooting scenario, the user's finger is generally not allowed to touch the trigger to avoid accidental discharge. In normal training, the user's finger (usually the index finger) that is used to pressing the trigger is required to contact the side of the gun frame; the present application provides a sensor 15 on the side of the gun frame 12 near the trigger 13, so that it can sense the time periods when the user's finger is placed on the side of the gun frame 12, and then it can be fed back to the virtual training scene, which helps to further improve the user's training effect.

[0033] In some embodiments, a charging port 47, a switch button 48, and a menu button 49 are further disposed inside the main body 10, wherein the charging port 47, the switch button 48, and the menu button 49 are disposed adjacent to each other and are all located at the bottom of the gun frame 12 and at one end away from the grip 11. The charging port 47 can be electrically connected to an external power source to provide power to the power source inside the main body 10; the switch button 48 can control the power switch of the gun 100a, and the menu button 49 can control the software of the virtual training scene corresponding to the gun 100a.

[0034] The sleeve 20 includes a shell 21, a tracking module 22, a rear sight 23 and a front sight 24. The shell 21 covers the gun frame 12, and the shell 21 and the gun frame 12 can slide relative to each other, that is, the shell 21 and the gun frame 12 have a sliding matching structure design, and the specific structure is not described in this application; the tracking module 22 is roughly square plate-shaped, which is located at the end of the shell 21 away from the muzzle (unmarked), and a marker (unmarked) is provided on the surface of the side of the tracking module 22 away from the shell 21 and the main body 10. By analyzing the image of the marker taken, the six-degree-of-freedom information of the camera relative to the marker can be obtained, and then the tracking module 22 can be used to obtain the six-degree-of-freedom information of the camera relative to the marker. The real-time acquisition of the firearm 100a is equivalent to the relative position and rotation relationship of the head-mounted display device 200; the rear sight 23 and the front sight 24 are both arranged on the outer surface of the shell 21 away from the gun frame 12, the rear sight 23 is arranged adjacent to the tracking module 22, and the front sight 24 is arranged adjacent to one end of the muzzle. There are two rear sights 23 and one front sight 24. The two rear sights 23 are relatively parallel and spaced apart. The user's line of sight can pass through the gap between the two rear sights 23 on the side with the tracking module to observe the front sight 24. In the embodiment of the present application, the front sight 24 and the rear sight 23 are respectively provided with light-emitting elements 241 and 231 on the side facing the tracking module 22. The light-emitting elements 231 and 241 will emit light after the firearm 100a is turned on. The light-emitting element 241 on the front sight 24 and the light-emitting element 231 on the rear sight 23 emit different colors and have a certain contrast. For example, the light-emitting element 231 on the rear sight 23 emits green light, and the light-emitting element 241 on the front sight 24 emits red light. By providing light-emitting elements on the front and rear sights, the user can quickly find the front and rear sights, and aim at the target object in the virtual scene through the front and rear sights.

[0035] In some embodiments, the direction parallel to the housing 21 and the gun rack 12 is defined as the horizontal direction, and the surface of the tracking module 22 provided with the marker is tilted to the horizontal direction, so that when the user wearing the head mounted display device 200 holds the gun 100a for training, under various normal operation conditions, the marker of the tracking module 22 on the gun 100a can be recognized and tracked by the head mounted display device 200 including the camera, and then the position and rotation relationship of the gun 100a relative to the head mounted display device 200 is obtained, and then the position and rotation information of the gun 100a in the real world can be mapped to the virtual training scene. Preferably, the angle between the surface of the tracking module 22 provided with the marker and the horizontal direction is between 30-60 degrees.

[0036] Please also read Figure 5The sleeve 20 can slide relative to the main body 10 through the action of the sliding assembly 30, and the sliding assembly 30 includes a sliding member 31, an elastic member 32 sleeved on the sliding member 31, and a receiving groove 33 that accommodates part of the sliding member 31. The sliding member 31 is generally cylindrical, one end of which is fixed in the sleeve 20, and the other end is accommodated in the receiving groove 33; the elastic member 32 is a spring, which is sleeved on the cylindrical surface of the sliding member 31, and one end of the elastic member 31 is fixed on the sleeve 20 together with the sliding member 31, and the other end is against the two side surfaces of the entrance of the receiving groove 33; the component with the receiving groove 33 is fixed relative to the main body 10. The user can hold the main body 10 with one hand and apply external force to the sleeve 20 with the other hand, so that one end of the sliding member 31 accommodated in the receiving groove 33 slides in the receiving groove 33 until it abuts against the bottom of the receiving groove 33, so as to activate the loading module 46 and make the elastic member 31 in a compressed state; at this time, the sleeve 20 as a whole slides in the horizontal direction away from the main body 10 along the tracking module 22 relative to the main body 10. When the user no longer applies force to the sleeve, under the action of the elastic force of the elastic member 31 in a compressed state, the sleeve 20 as a whole can slide in the horizontal direction toward the main body 10 along the tracking module 22 relative to the main body 10, thereby restoring the sleeve 20 to its original state relative to the main body 10.

[0037] Please also see Figure 6 , a sensor 16 is provided on the side of the gun frame 12 of the gun 100a, and a contact element 71 is provided on the side of the holster 70 corresponding to the sensor 16. When the sensor 16 senses that it is aligned with the contact element 71, it means that the gun 100a is accurately placed in the holster 70; when the sensor 16 senses that it is not aligned with the contact element 71, it means that the gun 100a is out of the holster 70. In the actual use scenario of real guns, there are requirements for when the gun needs to be pulled out of the holster and whether the gun is accurately placed in the holster. The present application provides the sensor 16 provided on the side of the gun frame 12 of the gun 100a and the contact element 71 provided on the side of the holster 70, so that it can sense in which time periods the gun 100a is out of the holster 70, and then it can be fed back to the virtual training scenario, which helps to further improve the training effect of the user.

[0038] Based on the above training system, the embodiment of the present application provides a training method using a training gun in a virtual training scene, which is applied to a head mounted display device of the above training system. The specific training method is introduced below.

[0039] See also Figure 7 , an embodiment of the present application provides a training method for using a training gun in a virtual training scene, the method comprising the following contents.

[0040] S710: Calculate first position information of the head mounted display device in the virtual training scene.

[0041] The virtual training scene may be a pre-loaded virtual scene, and the program corresponding to the virtual training scene may be pre-stored in the head-mounted display device. The user may enter the corresponding virtual training scene by operating the interface displayed on the head-mounted display device, so that the user wearing the head-mounted display device can see the virtual training scene superimposed on the real world through the head-mounted display device. In some embodiments, the program corresponding to the virtual training scene may be stored in other terminals, such as a computer, and the terminal may distribute the virtual training scene to the head-mounted display device by wireless transmission, which may simplify the configuration of the head-mounted display device.

[0042] The loaded superimposed virtual training scene is fixed relative to the real world. The virtual training scene can be obtained by taking a certain position in the real world as the origin of the world coordinate system. After obtaining the three-dimensional model of the virtual training scene, the corresponding virtual scene image is rendered in the head-mounted display device in combination with the origin of the world coordinate system and the position of the head-mounted display device in the world coordinate system.

[0043] The origin of the world coordinate system can be set by the user himself, for example, by specifying and selecting the origin of the world coordinate system through an external controller or gestures. The position and orientation of the head-mounted display device (i.e., the user) in the virtual training scene can be obtained through VIO positioning technology, that is, the first position information of the head-mounted display device in the virtual training scene can be obtained.

[0044] In some embodiments, the origin of the world coordinate system can also be determined by placing a marker in the real world. For example, a reference marker (including light spots or pattern information, etc.) that can be located and tracked by the head-mounted display device can be placed in the real world, and the reference marker can be used as the origin of the world coordinate system. The head-mounted display device can obtain the position and rotation relationship of the head-mounted display device relative to the reference marker by visual tracking, and then obtain the position and orientation of the head-mounted display device (i.e., the user) in the virtual training scene, that is, obtain the first position information of the head-mounted display device in the virtual training scene. When the reference marker leaves the visual range of the camera of the head-mounted display device, the VIO positioning technology can also be used to assist in obtaining the real position of the head-mounted display device, and then obtain the position of the user in the virtual training scene, so that the first position information of the head-mounted display device in the virtual training scene can be obtained in real time in a large range of virtual training scenes by using a reference marker.

[0045] S720, calculating the muzzle direction information of the training gun.

[0046] In the actual use of the training system, the user usually wears the head mounted display device on the head and holds the training gun, wherein the training gun is connected to the head mounted display device for communication, and the operation information of the training gun can be sent to the head mounted display device.

[0047] In some embodiments, obtaining the muzzle orientation information of a training gun includes: acquiring image information of a marker set on the training gun in real time; calculating the position and posture information of the training gun relative to the head-mounted display device based on the acquired marker image; calculating the muzzle orientation information of the training gun based on the pre-stored rigid body relationship between the marker and the muzzle on the training gun, and the position and posture information of the training gun relative to the head-mounted display device.

[0048] In some embodiments, the three-degree-of-freedom information of the training gun can also be acquired in real time through an inertial measurement unit disposed in the training gun, that is, the muzzle orientation information of the training gun can be obtained.

[0049] S730, obtaining a correspondence between a position in a pre-stored virtual training scene and a muzzle orientation of a training gun at this position.

[0050] The layout of the virtual training scene is set by the developer according to the training situation. For example, the virtual training scene includes multiple virtual rooms, doors, tables, chairs, hostages, gangsters and other virtual objects or characters. In the process of rushing into the room to rescue the hostages, there are specified requirements for the training guns held in hands at different locations in the virtual scene. For example, before rushing into the room (it can be a large room in the entire virtual scene or a small room in the virtual scene), it is not recommended that the training guns held by the user are pointed horizontally. If the corresponding virtual scene is not confirmed during the rushing process, the user is required to point the training guns held in his hand horizontally so as to be able to respond quickly to emergencies.

[0051] In some embodiments, a first preset area can be pre-set for a virtual training scene. The first preset area can be a position such as the side of the doorway of the room, the corner of the wall, etc., where the user needs to hide in the virtual scene, wait for a certain period of time, or after a partial clearance. The muzzle direction requirement can be set at the first preset position, requiring the muzzle direction of the training gun to be obliquely pointed to the ground or obliquely pointed upward. Therefore, it can be pre-set that when the first preset position is in the first preset position, the angle between the muzzle direction of the training gun and the vertical direction perpendicular to the horizontal ground is less than the first preset angle. The first preset angle can be set to 45 degrees. In some embodiments, the first preset area can also be adjusted according to the change of the situation of the virtual scene. For example, when the gangsters are eliminated and the hostages are rescued in the local virtual room, the local scene is updated and adjusted to the first preset area; for another example, the side of the door before entering the room with the closed door is the first preset area, and after the door is opened, the area is no longer set as the first preset area.

[0052] In some embodiments, a second preset area may be pre-set for the virtual training scene. The second preset area may be the location area where the user performs a task walking inside the room to search for criminals. In the second preset position, a muzzle orientation requirement may be set, requiring the muzzle orientation of the training gun to be horizontal. Therefore, a requirement instruction may be pre-set that the angle between the muzzle orientation of the training gun and the horizontal direction is less than a second preset angle when in the second preset position. The second preset angle may be set to 15 degrees.

[0053] S740: Determine a first matching result with the corresponding relationship based on the first position information and the muzzle orientation information of the training firearm at this time.

[0054] In some embodiments, the first matching result may be the number of times the corresponding relationship requirement is met or not met, that is, during the user's tactical room rushing training process, the muzzle direction information is matched in real time, and the number of times the user's muzzle pointing meets or does not meet the requirement during the training process can be obtained.

[0055] S750: Generate first feedback content for training firearm use according to the first matching result.

[0056] In some embodiments, after the entire tactical room rushing training is completed, the first feedback content can be generated based on the first matching result obtained. The first feedback content can be a training result about the muzzle direction operation during the training process presented to the user or the administrator user, and can be an intuitive result of the number of non-compliance or a scoring result.

[0057] In some embodiments, the training method may further include: collecting finger detection information of a finger sensor; and generating second feedback content for training the use of a firearm according to the muzzle orientation information and the finger detection information. The second feedback content may present a judgment of whether the user's finger is holding the trigger in different situations.

[0058] In one embodiment, a finger sensor may be disposed on the upper side of a trigger of a training gun; when the muzzle is facing horizontally, the finger is required to leave the finger sensor, and a feedback result may be generated based on this requirement; when the muzzle is facing a direction that deviates from the horizontal direction by more than a set angle threshold, the finger is required to be placed on the finger sensor, and another feedback result may be generated based on this requirement; the above two feedback results may be integrated to generate a second feedback content.

[0059] In one embodiment, a finger sensor can be set at the trigger of the training gun to sense whether the user's finger is placed on the trigger; when the muzzle is facing horizontally, the finger is required to be placed on the finger sensor, that is, the finger is required to be placed on the trigger, and a feedback result can be generated according to this requirement; when the muzzle direction deviates from the horizontal direction by more than a set angle threshold, the finger is required to leave the finger sensor, that is, the finger is required not to be placed on the trigger, and another feedback result can be generated according to this requirement; the above two feedback results are integrated to generate a second feedback content.

[0060] Of course, finger sensors can be provided at the trigger of the training gun and on the upper side of the trigger. Through the two finger sensors, it is possible to effectively determine whether the user's finger pressing the trigger is placed in the correct position in the corresponding situation, and to generate feedback content, so that after the training is completed, the finger placement information can be intuitively obtained through the feedback content, which helps to improve the user's training effect.

[0061] When the second feedback content is incorrect operation, the user can be reminded in time during the user training process so that the user can make corrections in time, and the corrections will be reflected in the final training results.

[0062] In some embodiments, the training method may further include obtaining second position information of a teammate's virtual character in a virtual training scene; obtaining position information of the training gun; determining an angle between the orientation of the training gun and a mis-pointing direction based on the position information of the training gun, muzzle orientation information, and the second position information, wherein the mis-pointing direction is the direction from the training gun to the second position; obtaining a pre-stored third preset angle, and generating third feedback content for the use of the training gun based on a comparison result of the angle between the orientation of the training gun and the mis-pointing direction with the third preset angle.

[0063] The virtual training scene can be used by multiple users for training. Each user wears a head-mounted display device and holds a training gun. Multiple users can play the role of teammates and conduct training in the same scene together. During the training process, it is required that no user points the muzzle of the gun at a teammate. In one embodiment, the second position information of the teammate in the virtual training scene can be obtained through the head-mounted display device worn by the teammate; the position information of the training gun can be tracked by the head-mounted display device worn by the user himself to obtain the position and rotation information of the training gun relative to the head-mounted display device worn by the user, and the position information of the training gun and the muzzle orientation information are obtained according to the position and orientation of the head-mounted display device worn by the user in the virtual training scene; according to the position information, muzzle orientation information and second position information of the training gun The angle between the muzzle direction of the training gun and the direction in which the muzzle points to the teammate (i.e., the direction in which the gun is pointing incorrectly) can be determined; a third preset angle can be pre-stored, and a third feedback content can be generated by comparing the angle between the muzzle direction of the training gun and the direction in which the muzzle points to the teammate (i.e., the direction in which the gun is pointing incorrectly) with the third preset angle; when the angle between the muzzle direction of the training gun and the direction in which the muzzle points to the teammate (i.e., the direction in which the gun is pointing incorrectly) is less than the third preset angle, feedback information can be generated to prompt the user that the muzzle may point to the teammate; the third feedback content can be a training result presented to the user or the administrator user during the training process regarding whether the muzzle points to the teammate. The third preset angle can be set to 30 degrees. When the third feedback content is to confirm that the muzzle may point to the teammate, the user can be reminded in time during the user's training process so that the user can make corrections in time and the information will be reflected in the final training result.

[0064] In some embodiments, after determining the first matching result with the corresponding relationship, when the first matching result is a non-compliant result, non-compliant prompt information can be generated in real time to prompt the user to change incorrect operations in time during the training process to improve the training effect.

[0065] The training method provided in the present application can enhance the details of the muzzle direction of the user in the process of using a firearm during virtual training using a training system, thereby helping to improve the user's awareness of firearm use and achieve the purpose of actual combat training through virtual scene training.

[0066] The embodiment of the present application also provides a training device for a virtual training scene, which is applied to a head-mounted display device to perform the above-mentioned training method. The training device may include: a first calculation module, a second calculation module, an acquisition module, a determination module, and a feedback module. It can be understood that each of the above modules can be a program module running in a computer-readable storage medium, and the purpose and operation of each of the above modules are as follows: the first calculation module is used to calculate the first position information of the head-mounted display device in the virtual training scene; the second calculation module is used to calculate the muzzle orientation information of the training gun; the acquisition module is used to obtain the correspondence between the position in the pre-stored virtual training scene and the muzzle orientation of the training gun at this position; the determination module is used to determine the first matching result with the corresponding relationship based on the first position information and the muzzle orientation information of the training gun at this time; and the feedback module is used to generate the first feedback content of the use of the training gun based on the first matching result.

[0067] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the above-described device and module can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here. In several embodiments provided in the present application, the coupling or direct coupling or communication connection between the modules displayed or discussed can be through some interfaces, indirect coupling or communication connection of the device or module, which can be electrical, mechanical or other forms. In addition, each functional module in each embodiment of the present application can be integrated in a processing module, or each module can exist physically separately, or two or more modules can be integrated in one module. The above-mentioned integrated modules can be implemented in the form of hardware or in the form of software functional modules.

[0068] In some embodiments, during the training process using the above training system, relevant training data can be transmitted to the background in real time so that the manager can observe the situation during the training process in real time. In view of this, the embodiment of the present application also provides a tactical board display method based on virtual training, which is applied to a background terminal, and the background terminal can be a computer with a display screen. The specific tactical board display method is introduced below.

[0069] See also Figure 8 , an embodiment of the present application provides a tactical board display method based on virtual training, the method includes the following contents.

[0070] S810, obtaining data information of the virtual training scene, where the data information includes building information, virtual object information, and virtual character information.

[0071] Users can watch the virtual training scene for training by wearing a head-mounted display device and loading the three-dimensional model of the relevant virtual training scene. At the same time, the background computer can also obtain the model of the virtual training, that is, the relevant data information; moreover, the relevant data when the user interacts with the virtual training scene can also be obtained by the background computer to update the data information related to the virtual training scene in real time. Among them, the virtual training scene model can include building information (such as a house, etc.), virtual object information (such as a table, an obstacle, etc.) and virtual character information (such as a hostage, a gangster, etc.).

[0072] S820: construct a two-dimensional map according to the data information.

[0073] Using the optical structure of the head-mounted display device, the user sees a three-dimensional virtual training scene superimposed on the real world through the head-mounted display device. To facilitate the administrator's viewing, the background computer usually uses a display screen to view the training scene, and a two-dimensional map can be constructed based on the data information of the above virtual training scene. For example, see Fig. 9 The constructed two-dimensional map includes a two-dimensional view 901 of the room, an identifier 902 of the virtual object, and a role identifier 903 of the virtual character.

[0074] S830: Obtain position information of a head mounted display device in the virtual scene and a character identifier corresponding to the head mounted display device.

[0075] When a user wears a head mounted display device for virtual training, the user's position information in the virtual scene can be calculated, acquired by the background computer and mapped to the two-dimensional virtual map. At the same time, the character identification of the head mounted display device can also be acquired by the background computer and mapped to the two-dimensional virtual map. The character identification can be generated by data information numbered according to the preset ID information of the head mounted display device. Fig. 9 , the character identification 904 corresponding to the head mounted display device is mapped to the two-dimensional map in real time according to the position of the head mounted display device in the virtual training scene.

[0076] In some embodiments, when there are multiple users participating in virtual training together, the head-mounted display device worn by each user can be numbered, and the corresponding number can be mapped to the character identification of the two-dimensional map so that the training process of each specific user can be clearly observed.

[0077] S840: compare the character identification with the two-dimensional map Figure 1 The character identification is displayed together, wherein the character identification is overlaid on the two-dimensional map, and the overlaid position of the character identification in the two-dimensional map corresponds to the position information of the head mounted display device in the virtual scene.

[0078] The position corresponding to the character identifier corresponding to the training user displayed on the two-dimensional map is obtained according to the position of the head-mounted display device corresponding to the user in the superimposed virtual training scene. Specifically, the position of the head-mounted display device relative to the origin in the real world coordinate system can be first obtained, and then calculated together with the position of the virtual training scene superimposed in the real world relative to the origin of the world coordinate system. When the position of the user wearing the head-mounted display device in the real world changes, the position corresponding to the two-dimensional map changes. Figure 1 The position of the character logo displayed in the two-dimensional map will also change accordingly, thereby realizing a one-to-one correspondence between the user's position change in the real world, the position change in the three-dimensional virtual training scene seen by the user, and the position change of the character logo in the two-dimensional map.

[0079] The character identification corresponding to the training user is displayed overlaid on the two-dimensional map, that is, when the character identification coincides with the building information, virtual object or virtual character information in the two-dimensional map, the character identification will be overlaid on the two-dimensional map information that coincides with the character identification.

[0080] In some embodiments, while obtaining the position information of the head-mounted display device in the virtual training scene, the posture information of the head-mounted display device is also obtained, that is, the six-degree-of-freedom information of the head-mounted display in the virtual training scene can be obtained. The method for obtaining the six-degree-of-freedom information has been described in the above part of this application document. For example, it can be obtained by combining the tracking technology of the head-mounted display device for reference markers with the VIO positioning technology, so that the six-degree-of-freedom information of the head-mounted display device relative to the virtual world can be obtained in real time throughout the virtual training scene.

[0081] When the posture information of the head mounted display device in the virtual training scene is obtained, the yaw information in the posture information can be extracted, and a first observation mark can be generated according to the yaw information, and the first observation mark can be displayed together with the corresponding character mark on the two-dimensional map. The distance between the first observation mark and the corresponding character mark is relatively fixed, and the first observation mark defines an area space. Fig.10a , the role identifier 1001 will change its position in the two-dimensional map accordingly with the position change of the user, and the first observation identifier 1002 will be displayed together with the role identifier 1001. The first observation identifier 1002 corresponds to the horizontal rotation direction of the head mounted display device in the real world, that is, it identifies the observation direction of the user in the virtual training scene; Fig.10bWhen the head of the user wearing the head mounted display device changes in the horizontal direction, the direction of the first observation mark 1002 in the virtual training scene will also change accordingly, but the distance of the first observation mark 1002 relative to the character mark 1001 will not change, so that the administrator can observe the user's observation direction in the two-dimensional virtual scene in real time. In some embodiments, the first observation mark is formed by two line segments at a preset angle, and the two line segments do not intersect, and each line segment is a preset distance away from the corresponding character mark; wherein the preset angle value can be set between 45-70 degrees. By setting the first observation mark, the administrator in the background can clearly understand the situation of the user's observation direction in the horizontal rotation direction, and can timely grasp the user's training situation.

[0082] In some embodiments, when the posture information of the head mounted display device in the virtual training scene is obtained, the pitch information in the posture information can be extracted, and a second observation mark can be generated according to the pitch information, and the second observation mark can be displayed together with the corresponding first observation mark and the character mark on the two-dimensional map. The second observation mark is within the area space defined by the first observation mark. Fig.11a The shaded portion 1101a within the area defined by the first observation mark 1002 is the second observation mark, which corresponds to the pitch of the head-mounted display device relative to the horizontal ground in the real world, that is, the vertical rotation direction; the color depth of the shaded portion indicates the pitch angle information of the user's observation direction. Fig.11a and Fig.11bThe depth of the color is different, indicating that the user's viewing direction is different in the vertical rotation direction. The shallower the depth, the greater the degree of looking down; the deeper the depth, the greater the degree of looking up. In some embodiments, an angle mark can be added to the shaded part, and the angle mark can be an angle relative to the vertical rotation direction. For example, when the user's viewing direction is vertically looking at the ground, it is 0 degrees; when the user's viewing direction is horizontally looking forward, it is 90 degrees; when the user's viewing direction is looking up vertically, it is 180 degrees. In some embodiments, the shaded part can also be set to a gradient mode to indicate the user's viewing direction in the vertical rotation direction. There are many specific gradient modes, which will not be described in detail in this application. In some embodiments, the user's viewing direction in the vertical rotation direction can also be indicated by the color information of the shaded part. For example, when the user's viewing direction is more than 30 degrees in looking up, the shaded part is red; when the user's viewing direction is more than 60 degrees in looking down, the shaded part is blue; when the user's viewing direction is less than 30 degrees in looking up and less than 60 degrees in looking down, the shaded part is green. By setting the second observation mark, the backend administrator can clearly understand the user's observation direction in the vertical rotation direction and timely grasp the user's more detailed training situation.

[0083] In some embodiments, when the posture information of the head mounted display device in the virtual training scene is obtained, the scroll information in the posture information can be extracted, and a third observation mark can be generated according to the scroll information, and the third observation mark can be displayed together with the corresponding first observation mark and the character mark on the two-dimensional map. The third observation mark corresponds to the situation in which the head mounted display device is tilted to the side relative to the horizontal direction in the real world, that is, it indicates the situation of the user's side observation direction. Fig.12 As shown, the third observation mark is an angle mark. When the user observes normally horizontally, the angle mark can be displayed as 0 degrees; when the user turns his head to the left to observe, the angle mark can change to a trend of -90 degrees, that is, the more the user's head is to the left, the closer the angle mark is to -90 degrees; when the user turns his head to the right to observe, the angle mark can change to a trend of 90 degrees, that is, the more the user's head is to the right, the closer the angle mark is to 90 degrees. By setting the third observation mark, the backend administrator can clearly understand the user's side observation situation and timely grasp the user's more detailed training situation.

[0084] In some embodiments, when the position information of the head-mounted display device in the virtual scene is obtained, the posture information of the head-mounted display device in the virtual scene is obtained at the same time; the position of the head-mounted display device is mapped to the position of the user in the virtual training scene, and when the position of the user in the virtual training scene is a preset area and meets the preset state, it can be determined whether the posture information of the head-mounted display device obtained is consistent with the preset posture threshold; when the judgment result is inconsistent, a prompt message is generated and displayed in real time. Among them, the preset area and the preset posture threshold can be customized by the user according to the building information. For example, on the left side of the door of the virtual room, the user's head is required to roll to the right within a preset angle range. When the actual angle of the user's head to the right does not reach the preset angle range, a prompt message can be generated and displayed on the background computer to prompt the administrator that the user's training action at this time does not meet the requirements. For another example, in the area at the entrance of the virtual room, the user is required to first turn his head horizontally to the left to observe in a direction within a preset angle range, and then turn his head horizontally to the right to observe in a direction within a preset angle range. When the actual angle of the user's head turning horizontally to the left or right does not reach the preset angle range, a prompt message can be generated and displayed on the background computer to prompt the administrator that the user's training action at this time does not meet the requirements; in some embodiments, the rotation order and / or rotation speed can also be preset according to the situation of the preset position. When the actual rotation order and / or rotation speed do not meet the preset rotation order and / or rotation speed, a prompt message can also be generated and displayed on the background computer to prompt the administrator that the user's training action at this time does not meet the requirements.

[0085] In some embodiments, the location information of the head-mounted display device and the building information, virtual object information, and virtual character information in the data information of the virtual training scene can also be calculated and analyzed; when the location distance between the location information of the head-mounted display device and the building information, virtual object information, and virtual character information in the data information is less than a preset distance, prompt information is generated and displayed in real time. During the virtual training process, the virtual building information, virtual object information, and virtual character information are all superimposed on the real world by the user's visual perception. These information do not exist in the real world. The user can move around freely in the real world and will not be hindered by the objects in the virtual training scene. In order to improve the user's training efficiency, setting the above steps can effectively further evaluate the user's training. Among them, considering the problem of displacement changes in the virtual training scene observed by the user due to long-term VIO positioning drift, setting a preset distance helps to improve the accuracy of judging the user's training.

[0086] In some embodiments, the relative position information of the training gun connected to the head-mounted display device in the virtual training scene can be obtained. Specifically, the training gun can be visually positioned and tracked through the head-mounted display device to obtain the position information of the training gun relative to the head-mounted display device. Since the position of the head-mounted display device in the virtual training scene can be obtained, the position information of the training gun relative to the head-mounted display device can be mapped to the virtual training scene to obtain the relative position information of the training gun in the virtual training scene; then the user's shooting operation information on the training gun is obtained. When the user presses the trigger in the training gun, corresponding data information can be generated according to the shooting operation and sent to the background computer; the background computer can generate a shooting mark according to the relative position information of the training gun and the data information of the shooting operation, and follow the two-dimensional map Figure 1 The shooting mark is displayed dynamically, that is, the corresponding shooting trajectory.

[0087] The display method for virtual training scene training based on the background computer provided in this application enables the background manager to observe the actual process of user training in real time, and can intuitively understand the quality of the user's training situation, which is helpful to provide guidance and improvement for subsequent training and enhance the effect of training.

[0088] The embodiment of the present application also provides a display device based on virtual training scene training, which is applied to a background computer to execute the above-mentioned training method. The display device may include: a first acquisition module, a construction module, a second acquisition module and a display module. It can be understood that each of the above modules can be a program module running in a computer-readable storage medium, and the purpose and operation of each of the above modules are as follows: the first acquisition module is used to obtain data information of the virtual training scene, and the data information includes building information, virtual object information and virtual character information; the construction module is used to construct a two-dimensional map according to the data information; the second acquisition module is used to obtain the position information of the head-mounted display device in the virtual scene, and the role identification of the head-mounted display device; the display module is used to compare the role identification with the two-dimensional map Figure 1 The character identification is displayed together, wherein the character identification is overlaid on the two-dimensional map, and the overlaid position of the character identification in the two-dimensional map corresponds to the position information of the terminal device in the virtual scene.

[0089] A head-mounted display device provided in an embodiment of the present application may include one or more of the following components: a processor, a memory, and one or more applications, wherein one or more applications may be stored in the memory and configured to be executed by one or more processors, and one or more programs are configured to execute the method described in the aforementioned method embodiment. The processor may include one or more processing cores. The processor uses various interfaces and lines to connect various parts of the entire head-mounted display device, and executes various functions and processes data of the head-mounted display device by running or executing instructions, programs, code sets or instruction sets stored in the memory, and calling data stored in the memory. Optionally, the processor may be implemented in at least one hardware form of digital signal processing (DSP), field programmable gate array (FPGA), and programmable logic array (PLA). The processor may integrate one or more combinations of a central processing unit (CPU), a graphics processing unit (GPU), and a modem. Among them, the CPU mainly processes the operating system, user interface, and application programs; the GPU is responsible for rendering and drawing display content; and the modem is used to process wireless communications. The memory may include a random access memory (RAM) or a read-only memory (ROM). The memory may be used to store instructions, programs, codes, code sets or instruction sets. The memory may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for implementing at least one function (such as a touch function, a sound playback function, an image playback function, etc.), instructions for implementing the following various method embodiments, etc. The data storage area may also store data created by the head mounted display device during use, etc.

[0090] A computer-readable storage medium provided in an embodiment of the present application. The computer-readable storage medium stores program code, and the program code can be called by a processor to execute the method described in the above method embodiment. The computer-readable storage medium can be an electronic memory such as a flash memory, an EEPROM (electrically erasable programmable read-only memory), an EPROM, a hard disk, or a ROM. Optionally, the computer-readable storage medium includes a non-volatile computer-readable medium (non-transitory computer-readable storage medium). The computer-readable storage medium has storage space for program codes that execute any method steps in the above method. These program codes can be read from or written to one or more computer program products. The program code can be compressed, for example, in an appropriate form.

[0091] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

[0092] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A display method based on virtual training scene training, characterized in that: Applied to a background terminal with a display screen, the method comprises: Acquiring data information of the virtual training scene, wherein the data information includes building information, virtual object information, and virtual character information; Constructing a two-dimensional map according to the data information; Acquiring position information and posture information of a head mounted display device in the virtual training scene, as well as a role identification of the head mounted display device, wherein the head mounted display device is worn by a user participating in the virtual training; generating a first observation identifier according to the yaw information in the posture information of the head mounted display device, wherein the first observation identifier is used to identify the observation direction of the user in the virtual training scene; generating a second observation mark according to the pitch information in the posture information of the head mounted display device, wherein the color depth of the shadow part in the second observation mark is positively correlated with the degree of looking up; Generate a third observation mark according to the scroll information in the posture information of the head mounted display device, the third observation mark is used to characterize the side observation direction of the user in the virtual training scene, the third observation mark is expressed as an angle, the third observation mark is 0 degrees when the user is in normal horizontal observation, the third observation mark changes with a trend toward -90 degrees when the user turns his head to the left, and the third observation mark changes with a trend toward +90 degrees when the user turns his head to the right; The first observation identifier, the second observation identifier and the third observation identifier are displayed together with the character identifier and the two-dimensional map, wherein the character identifier is overlaid on the two-dimensional map, and the overlaid position of the character identifier in the two-dimensional map corresponds to the position information of the terminal device in the virtual training scene.

2. The method according to claim 1, characterized in that When acquiring the position information of the head mounted display device in the virtual training scene, the method further includes: Acquiring posture information of the head mounted display device in the virtual training scene; When the position of the head mounted display device in the virtual training scene is in a preset area and meets a preset state, determining whether the posture information of the head mounted display device is consistent with a preset posture threshold; When the judgment result is inconsistent, a prompt message is generated and displayed in real time.

3. The method according to claim 1, characterized in that The method further comprises: Calculate and analyze the position information of the head mounted display device and the building information, virtual object information and virtual character information in the data information of the virtual training scene; When the distance between the position information of the head mounted display device and the building information, virtual object information and virtual character information in the data information is less than a preset distance, prompt information is generated and displayed in real time.

4. The method according to claim 1, characterized in that The method further comprises: Acquiring relative position information of a training gun connected to the head mounted display device in the virtual training scene; Acquiring data information of the shooting operation of the training firearm; A shooting mark is generated according to the relative position information of the training gun and the data information of the shooting operation, and is displayed along with the two-dimensional map.

5. A display device based on virtual training scene training, characterized in that: Applied to a backend terminal with a display screen, the device comprises: A first acquisition module is used to acquire data information of the virtual training scene, wherein the data information includes building information, virtual object information and virtual character information; A construction module, used for constructing a two-dimensional map according to the data information; a second acquisition module, for acquiring position information and posture information of a head-mounted display device in the virtual training scene, and a role identification of the head-mounted display device, wherein the head-mounted display device is worn by a user participating in the virtual training; generating a first observation identification according to yaw information in the posture information of the head-mounted display device, wherein the first observation identification is used to identify the observation direction of the user in the virtual training scene; generating a second observation identification according to pitch information in the posture information of the head-mounted display device, wherein the color depth of the shadow part in the second observation identification is positively correlated with the degree of upward viewing; generating a third observation identification according to rolling information in the posture information of the head-mounted display device, wherein the third observation identification is used to characterize the side observation direction of the user in the virtual training scene, wherein the third observation identification is expressed in an angle, wherein the third observation identification is 0 degrees when the user is in normal horizontal observation, wherein the third observation identification shows a trend change toward -90 degrees when the user turns his head to the left, and wherein the third observation identification shows a trend change toward +90 degrees when the user turns his head to the right; A display module is used to display the first observation identifier, the second observation identifier and the third observation identifier together with the role identifier and the two-dimensional map, wherein the role identifier is overlaid on the two-dimensional map, and the overlaid position of the role identifier in the two-dimensional map corresponds to the position information of the terminal device in the virtual training scene.

6. A background computer, characterized in that: include: one or more processors; Memory; One or more application programs, wherein the one or more application programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs are configured to execute the method as described in any one of claims 1 to 4.

7. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores program code, and the program code can be called by a processor to execute the method according to any one of claims 1 to 4.