A virtual simulation teaching method, system and computer equipment

Abstract

The present invention relates to the field of virtual simulation teaching technology, and in particular to a virtual simulation teaching method, system and computer equipment, the method comprising: generating a virtual teaching scene and its corresponding virtual mapping, and obtaining the manipulation data of the virtual object in the virtual teaching scene, the manipulation data at least including manipulation position data and manipulation gesture data; generating a reality verification scene corresponding to the virtual teaching scene and its corresponding reality mapping; and collecting user position data and user hand data, generating a hand mapping corresponding to the user hand data; combining and constructing a simulation teaching scene, and synchronously loading the hand mapping to the simulation teaching scene; based on the user position data and the manipulation gesture of the user's hand mapping in the simulation teaching scene, controlling the virtual object in the virtual teaching scene to execute a corresponding response. The present invention can enhance the interactivity between the practical teaching of virtual simulation and students, and improve the teaching effect of the practical teaching of virtual simulation on students.

Description

Technical Field

[0001] The present invention relates to the technical field of virtual simulation teaching, and in particular to a virtual simulation teaching method, system and computer equipment. Background Art

[0002] In university teaching, especially in engineering teaching, in addition to teaching students theoretical knowledge, it is also necessary to teach students practical ability in order to train innovative talents with strong comprehensive qualities.

[0003] The practical teaching content and equipment such as experiments, course design, and internships are limited in their updating speed and cannot keep up with the updating of theoretical course teaching content. As a result, there may be a certain gap between the practical teaching content that can be carried out and the latest theoretical knowledge. In addition, some practical teaching venues and supporting equipment require high technical content, which is not convenient to set up and maintain on campus. As a result, some practical teaching content is difficult to carry out immediately on campus.

[0004] In recent years, virtual reality technology has developed rapidly. Applying virtual reality technology to practical teaching can effectively solve the above problems. The existing combination of virtual reality technology and practical teaching usually simulates practical teaching scenes through virtual equipment, and then students use the simulated scenes to perform corresponding simulation operations, thereby realizing virtual simulation practical teaching.

[0005] However, the existing virtual simulation practical teaching has poor interactivity with students. A large number of simulation operation steps are implemented by clicking, pressing, dragging and other operations, which are quite different from the operations in actual scenarios. For example, when performing device picking operations, students usually complete the picking by controlling the device to click on the corresponding device, which is different from the device picking operation in actual scenarios. It is difficult to provide real feedback on the position, force, angle and other information during picking. As a result, the virtual simulation practical teaching has poor teaching effect on students. Summary of the invention

[0006] To solve the above problems, the present invention provides a virtual simulation teaching method, system and computer equipment for enhancing the interactivity between virtual simulation practical teaching and students, and improving the teaching effect of virtual simulation practical teaching on students.

[0007] In order to achieve the above object, the technical solution of the present invention is as follows:

[0008] In a first aspect, a virtual simulation teaching method is provided, comprising the following steps:

[0009] Generate a virtual teaching scene that is not limited by position and its corresponding virtual mapping, and obtain control data of virtual objects in the virtual teaching scene, the control data at least including control position data and control gesture data;

[0010] Generate a real verification scene corresponding to the virtual teaching scene but limited by position and its corresponding real mapping; and collect user position data and user hand data to generate a hand mapping corresponding to the user hand data;

[0011] Extracting a number of sub-virtual mappings and sub-realistic mappings in the virtual teaching scene and the real verification scene respectively, combining the sub-virtual mappings and the sub-realistic mappings to construct a simulation teaching scene, and synchronously loading the hand mapping into the simulation teaching scene;

[0012] Based on the user position data and the manipulation gestures of the user's hand mapped in the simulated teaching scene, the virtual objects in the virtual teaching scene are controlled to execute corresponding responses.

[0013] Furthermore, the virtual mapping and the real mapping are synchronously enlarged or reduced according to the manipulation position data and the user position data, and the overlap between the virtual mapping and the real mapping is detected. When the overlap between the virtual mapping and the real mapping is greater than a threshold, the hand mapping is synchronously loaded into the simulation teaching scene.

[0014] Further, a boundary of the virtual mapping is obtained as a first boundary, a boundary of the real mapping is obtained as a second boundary, and a boundary of the real object is obtained as a third boundary;

[0015] Constructing a virtual space coordinate system, where the virtual space coordinate system is a three-dimensional coordinate system, and assigning coordinates of the first boundary, the second boundary, and the third boundary in the virtual space coordinate system;

[0016] Based on the coordinates of the first boundary, the second boundary and the third boundary in the virtual space coordinate system, the distance between the first boundary and the second boundary is calculated. When the distance between each first boundary and each second boundary is less than a threshold value, it is determined that the overlap between the virtual mapping and the real mapping is greater than the threshold value; and it is detected whether the third boundary is located in the virtual mapping. When the third boundary is located in the virtual mapping, a prompt signal is issued.

[0017] Furthermore, the reality verification scene is displayed through a touch display device and the user's touch signal on the reality verification scene is collected, and the user's control position data is obtained based on the user's touch signal on the reality verification scene; the user's control gestures are collected through a wearable device, and the control gestures include hand shape and applied force; the user's position data and viewing angle information are collected through an image acquisition device, and the reality verification scene position is switched within a limited position range according to the user's position information and viewing angle information.

[0018] Furthermore, after the display device receives the touch signal of the user on the reality verification scene, it determines the first touch mode between the user and the reality verification scene, and verifies the first touch mode in combination with the image acquisition device to obtain the final second touch mode, and generates a corresponding control verification gesture through the second touch mode, and compares the user's control gesture collected by the wearable device with the control verification gesture to obtain the user's control gesture.

[0019] Further, construct a manipulation database and recognition model;

[0020] The control database is used to store the control data of the virtual objects in the virtual teaching scene, obtain the control gestures of the hands mapped in the simulation teaching scene in real time, and retrieve the control gesture data corresponding to the control gestures of the hands mapped in the simulation teaching scene;

[0021] If the manipulation gesture data corresponding to the manipulation gesture of the hand mapped in the simulation teaching scene is retrieved in the manipulation database, the virtual object in the virtual teaching scene is directly controlled to execute a corresponding response;

[0022] If the control gesture data corresponding to the control gesture mapped by the hand in the simulation teaching scene is not retrieved in the control database, the user information is obtained, and the user information is input into the trained recognition model along with the control gesture mapped by the hand in the simulation teaching scene. The recognition model outputs the control gesture data corresponding to the control gesture mapped by the hand in the simulation teaching scene that has not been retrieved in the control database, and then controls the virtual object in the virtual teaching scene to execute a corresponding response.

[0023] Further, the virtual mapping is divided into a plurality of first unit areas, and the real mapping is divided into a plurality of second unit areas, and the number of the first unit areas is the same as the number of the second unit areas;

[0024] The sub-virtual mapping is one or more first unit areas, and the sub-reality mapping is one or more second unit areas; the first unit areas included in the sub-virtual mapping and the areas covered by the second unit areas included in the sub-reality mapping do not overlap, and the total number of sub-virtual mappings and sub-reality mappings is equal to the number of first unit areas.

[0025] Furthermore, each sub-reality mapping includes at least a partial area of ​​a virtual object in the virtual teaching scene.

[0026] In a second aspect, a virtual simulation teaching system is provided, comprising:

[0027] A virtual module, used to generate a virtual teaching scene that is not limited by position and its corresponding virtual mapping, and obtain control data of virtual objects in the virtual teaching scene, the control data at least including control position data and control gesture data;

[0028] A verification module is used to generate a real verification scene corresponding to the virtual teaching scene but limited by position and its corresponding real mapping; and collect user position data and user hand data to generate a hand mapping corresponding to the user hand data;

[0029] A processing module, used to extract a plurality of sub-virtual mappings and sub-real mappings in a virtual teaching scene and a real verification scene respectively, combine the sub-virtual mappings and the sub-real mappings to construct a simulation teaching scene, and synchronously load the hand mapping into the simulation teaching scene;

[0030] It is also used to control virtual objects in a virtual teaching scene to execute corresponding responses based on user position data and manipulation gestures of the user's hand mapped in the simulated teaching scene.

[0031] In a third aspect, a virtual simulation teaching computer device is provided, comprising one or more processors and a memory; the memory stores control instructions corresponding to the above-mentioned virtual simulation teaching method; the processor is used to receive user control signals and execute the control instructions stored in the memory.

[0032] The above scheme has the following beneficial effects:

[0033] 1. The present invention constructs a reality verification scene while constructing a virtual teaching scene. The reality verification scene is constructed based on the virtual teaching scene. The two are identical or enlarged or reduced in proportion, and a visual and touchable teaching scene is constructed; and a new simulation teaching scene is constructed by extracting partial mappings of the virtual teaching scene and the reality verification scene. Different from the existing simulation teaching scene, the simulation teaching scene combines virtual display and reality verification. The virtual display part can well present the virtual objects in the teaching scene to the user, and the reality verification part can make the user's operation closer to the actual operation.

[0034] For example: when the user picks up a device in a virtual teaching scene, the user directly picks it up at the specified position according to the display part and performs the corresponding picking action. The actual verification part detects whether the user's picking action meets the picking requirements of the corresponding device, which includes various information such as the picking position, force and angle. When these verification information correspond, the device can be picked up normally and the corresponding operation can be performed in the simulated teaching scene.

[0035] To sum up, compared with traditional control devices or simple click, press, drag and other operations, it can make the user's control more in line with the situation in the real scene and improve the quality of practical teaching through virtual simulation.

[0036] 2. In the present invention, since the reality verification scene can be presented in real time externally, and the presented content is synchronized with the virtual teaching scene and the simulation teaching scene, the teaching staff can provide corresponding guidance according to the students' operation conditions. For example, before using a certain device, a certain inspection operation needs to be performed first. With the existing independent virtual simulation teaching equipment, the teaching staff cannot obtain the operation performed by the students in real time, and often need to obtain it after the execution steps are completed or an error occurs. The reality verification scene of this solution can not only be used for operation verification, but also for teaching staff to obtain the students' operation conditions in real time, so as to provide timely guidance on the students' operations and improve the teaching effect.

[0037] 3. In the present invention, the virtual mapping and the real mapping will be synchronously enlarged or reduced in real time based on the user's position and controllable position, so that the virtual objects in the final constructed simulation teaching scene will change size synchronously with the user's movement. In addition, the virtual mapping and the real mapping will perform coincidence detection before and after enlargement or reduction. Based on the coincidence detection of the two, on the one hand, it can ensure the stability of each virtual object in the constructed simulation teaching scene, and on the other hand, it can verify the user's position, that is, verify whether the user's current position is suitable for virtual simulation teaching. When it is not suitable for virtual simulation teaching, the user cannot manipulate the virtual object. Compared with the prior art, the authenticity of the simulation teaching scene is improved, and the effective implementation of simulation teaching is ensured.

[0038] 4. The present invention determines the overlap between the virtual mapping and the real mapping based on the first boundary of the virtual mapping and the second boundary of the real mapping; in addition, the third boundary of the real object is introduced. The real object is usually an object in the teaching area that may interfere with the virtual simulation teaching, including tables, chairs, power supply equipment, etc. in the classroom. When the third boundary of these real objects exists in the virtual mapping, it indicates that the user may touch the real object, so the user needs to be given a corresponding prompt, such as a text prompt, a voice prompt, etc. Compared with the existing virtual equipment, it can ensure the safety of the learners in the virtual simulation teaching process.

[0039] 5. The present invention uses a touch display device to display a verification scene, and it is also used to assist in judging the user's operation gestures. For example, when a user presses a switch of a device corresponding to a virtual object and picks up a device corresponding to a virtual object, since the user interacts with the touch display device (touch) when manipulating the virtual object, the touch in the present invention is not the point-to-point touch in the prior art, but is based on the gestures used when actually manipulating the virtual object, which may be fingertips, knuckles, palms, backs of hands, etc. Different touches correspond to different user manipulation gestures; pressing a switch may be an interaction between a fingertip and a touch display device, and picking up may be an interaction between a knuckle or a palm and a touch display device. Based on this, a basic judgment is made on the user's manipulation gesture, and a comprehensive judgment is made in combination with image acquisition. Compared with the prior art, the accuracy of user manipulation gesture recognition is improved while ensuring good interaction.

[0040] 6. The present invention constructs a control database and recognition model. Usually, the industry will stipulate the basic specifications for controlling equipment, but some controls are allowed to be performed by individuals according to their own operating habits. The existing virtual simulation teaching does not accept users to perform control actions other than preset control actions, that is, when users are conducting virtual simulation teaching, they cannot use control actions that are suitable for individuals and allowed by industry specifications. However, the present invention obtains user information, and the recognition model can autonomously identify the user's operating habits. When performing certain operations, the user is allowed to use control actions that are suitable for individuals and allowed by industry specifications, thereby improving the adaptability of virtual simulation teaching and bringing a better experience to the learners.

[0041] Additional aspects and advantages of the present invention will be given in part in the following description and in part will be obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 It is a flow chart of the virtual simulation teaching method of the present invention;

[0043] Figure 2 This is the architecture diagram of the virtual simulation teaching system of the present invention;

[0044] Figure 3 This is a computer device structure diagram of the virtual simulation teaching method of the present invention;

[0045] Figure 4 A schematic diagram of scene interaction of an embodiment of the virtual simulation teaching method of the present invention.

[0046] The reference numerals in the drawings of the specification include: 1. head-worn device; 2. touch display. DETAILED DESCRIPTION

[0047] The technical solution of the present invention will be described clearly and completely below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0048] The following is further described in detail through specific implementation methods:

[0049] Embodiment 1: A virtual simulation teaching method, such as Figure 1 As shown, the following steps are included:

[0050] First, a virtual teaching scene that is not limited by location is generated. The virtual teaching scene that is not limited by location is usually selected according to teaching needs. It is not limited by location (it can be displayed in the user's visual space without location restrictions), and can include virtual teaching scenes in various fields such as mechanical engineering, medicine, architecture, computer, communication, etc. These teaching scenes can be constructed in real time according to the theoretical knowledge being taught, which can effectively solve the problem that the update speed of practical teaching content and equipment is difficult to keep up with the update of theoretical course teaching content.

[0051] At the same time, based on the generated virtual teaching scene that is not limited by position, a corresponding virtual mapping is generated. The virtual mapping corresponds to the content in the virtual teaching scene, that is, the presented teaching scene is a purely virtual teaching scene, and the virtual objects exist in the virtual coordinate system. In this embodiment, the virtual mapping is mainly used for the subsequent construction of simulation teaching scenes.

[0052] And obtain the control data of the virtual object in the virtual teaching scene, the control data at least includes the control position data and the control gesture data, wherein the control position data is the position of all users who can control the corresponding virtual object, and the control gesture data is the gestures of all users who can control the corresponding virtual object. The control position data and control gesture data of this embodiment can be preset according to the actual teaching needs. For example, when the teaching difficulty is greater and stricter, more standard control position data and control gesture data can be preset, and correspondingly, the user can also add other control data according to the actual teaching situation. The control position data and control gesture data are mainly used to verify which virtual object the user's position and gesture are controlling and the type of control.

[0053] Then, a reality verification scene corresponding to the virtual teaching scene but limited by the position is generated. The reality verification scene is limited by the position (it can only be displayed within a preset spatial range). The reality verification scene has the same content as the virtual teaching scene, but it is adapted to the device that presents the reality verification scene. In this embodiment, a touch display device is used to display the reality verification scene. The touch display device of this embodiment is composed of multiple displays, combined with Figure 4As shown, it is arranged around the user and can switch to different angles based on the user's viewing area.

[0054] At the same time, based on the reality verification scene, the corresponding reality mapping is generated. The virtual objects in the reality mapping are the same as the virtual objects in the virtual teaching scene, but they are adapted to the touch display device. The presented teaching scene combines the actual parameters of the touch display device, that is, the presented teaching scene is not a pure virtual teaching scene. The virtual objects in the reality mapping exist in a real coordinate system and have real coordinates. The coordinates vary based on the different touch display devices, and the touch display device assigns coordinates to them, that is, the virtual objects in the reality mapping carry their position information in the touch display device. In this embodiment, the reality mapping is also mainly used for the subsequent construction of simulation teaching scenes.

[0055] The user position data and the user hand data are collected through the touch display device and the image acquisition device integrated on the touch display device; wherein the user position data is directly collected through the image acquisition device, and the image acquisition device can be an infrared camera or a plurality of ordinary cameras integrated on the touch display 2 (the user's location is obtained by multi-camera image acquisition). The touch display device is used to preliminarily determine the user's touch position (which can be used as a control basis for calling the corresponding image acquisition device), convert it into the corresponding control position, and then generate accurate user hand data and control position based on the user's hand image collected by the image acquisition device.

[0056] Among them, the user position data and the user hand data correspond to the control position data and the control gesture data. The user position data and the user hand data are generated in real time with the user, and the control position data and the control gesture data are used to determine which virtual object the real-time generated user position data and user hand data are specifically controlling and the type of control.

[0057] At the same time, a hand mapping corresponding to the user's hand data is generated, and the hand mapping is synchronized with the user position data and user hand data generated by the user in real time. In this embodiment, it is mainly used to synchronously load it into the subsequently constructed simulation teaching scene, so as to realize the manipulation of virtual objects.

[0058] In addition, since the touch display device is composed of multiple touch displays 2 and in order to save the cost of virtual simulation teaching, the touch displays 2 that are not temporarily involved by the user should be in standby mode. Therefore, the user's visual angle information (the area that the user can see in the front direction range) is synchronously collected by the image acquisition device, and the position of the reality verification scene is switched in the multiple touch displays 2 set in combination with the user's position information, ensuring that the reality verification scene can be switched following the user's control. In other embodiments, a single touch display 2 can also be used, but it is equipped with a structure for switching the position and angle of the touch display 2. The single touch display 2 is controlled to switch the position and angle within a limited position range according to the user's position information and visual angle information, so as to achieve a similar effect to setting multiple touch displays 2.

[0059] Then, the virtual mapping is divided into a number of first unit areas, and the real mapping is divided into a number of second unit areas, wherein the sizes of the first unit area and the second unit area can be adjusted according to the actual situation. Since the smaller the sizes of the first unit area and the second unit area, the more verification points there are for detecting user operations in the future, the sizes of the two are inversely proportional to the precision requirements of the manipulation, that is, for virtual simulation practical teaching with high manipulation precision requirements, the sizes of the first unit area and the second unit area should be set relatively small. And it is necessary to ensure that the number of the first unit area and the second unit area are the same. In this embodiment, the number of the first unit area into which the virtual mapping is divided is equal to the number of the second unit area into which the real mapping of the real verification scene that can be presented in the multiple touch displays 2 is divided.

[0060] Several sub-virtual mappings (the sub-virtual mapping is one or more first unit areas) and sub-realistic mappings (the sub-realistic mapping is one or more second unit areas) are extracted from the virtual teaching scene and the real verification scene respectively. The first unit area contained in the sub-virtual mapping and the area covered by the second unit area contained in the sub-realistic mapping should not overlap, and the total number of the two should be equal to the number of the first unit areas, that is, they can be combined to construct a simulation teaching scene that is the same as the virtual teaching scene.

[0061] The sub-virtual mapping and the sub-reality mapping are combined to construct a simulation teaching scene, which combines the virtual objects required to be presented in the virtual teaching scene and the verification points in the reality verification scene. The verification points are used to verify the user's operation to provide feedback that is closer to the actual operation. In order to ensure that each virtual object has the verification capability, each second unit area included in the sub-reality mapping should contain at least part of the virtual object in the virtual teaching scene, so as to ensure that the user can obtain operation verification when manipulating any virtual object.

[0062] Then, the hand mapping is synchronously loaded into the simulation teaching scene. In this embodiment, the simulation teaching scene is finally presented to the user through the head wearable device 1, and the user's hand data is presented on the head wearable device 1 together with the simulation teaching scene. The control gestures in the hand data are collected through the wearable device, specifically the VR gloves; the control gestures include hand shape and applied force, the applied force is mainly collected through the VR gloves, and the hand shape can be assisted by the touch display device and the image acquisition device.

[0063] Specifically, after the display device receives the touch signal of the user on the reality verification scene, it determines the first touch mode between the user and the reality verification scene, and verifies the first touch mode in combination with the image acquisition device to obtain the final second touch mode, generates the corresponding control verification gesture through the second touch mode, and compares the user's control gesture collected by the wearable device with the control verification gesture to obtain the user's control gesture. For example: the user presses the switch of the device in the virtual object. Since the user interacts with the touch display device (touch) when manipulating the virtual object, pressing the switch may be the interaction between the fingertips and the touch display device, thereby determining the first touch mode between the user and the reality verification scene (pressing or pushing a certain component); the image acquisition device is combined to collect the user's hand image, verify and determine the first touch mode to obtain the second touch mode (if it is also a pressing or pushing action, it is determined to be pressing or pushing a certain component). Finally, it is compared and judged with the user's hand shape collected by the wearable device, and then combined with the applied force to generate the control gesture to be executed.

[0064] Finally, based on the user position data and the manipulation gestures of the user's hand mapped in the simulated teaching scene, the virtual objects in the virtual teaching scene are controlled to perform corresponding responses.

[0065] Specifically, combined Figure 4 As shown, assuming that there is a scene composed of six touch display devices, a simulation teaching scene is constructed through the above steps and presented to the user through the head wearable device 1, and the real verification teaching scene involved in the teaching process is presented in each touch display 2. Therefore, when the user operates in the simulation teaching scene, other people except the user can obtain the user's operation status in the touch display 2.

[0066] like Figure 4 As shown in (4a), the user performs a certain operation in the simulation teaching scene. Since the virtual objects controlled by the user belong to a certain sub-reality mapping, when the user manipulates the virtual object, the hand needs to interact (touch) with the touch display 2, so that the corresponding virtual object in the simulation teaching scene will execute a corresponding response. The specific virtual object and the executed response are determined by the user's manipulation position and the manipulation gesture mapped by the hand in the simulation teaching scene. Figure 4The user's position in (4a) is the user's position data, the arrow indicates the user's control position, and the control gestures mapped in the simulation teaching scene correspond to the user's gestures in reality, which are specifically obtained through the wearable device, the touch display 2 and the image acquisition device, which will not be repeated here. Figure 4 As shown in (4b), the user performs a certain operation in the simulation teaching scene, but the control position is relatively Figure 4 (4a) has changed, and the user's visual angle information has changed. Therefore, the touch display 2 in the area corresponding to the visual angle information is switched to present a reality verification teaching scene. Preferably, the touch display 2 in the forward direction of the visual angle information and on the left and right sides remain turned on, and the other touch displays 2 are switched to standby mode, so that the user can complete the complete virtual simulation teaching in this scene.

[0067] In addition, combined Figure 2 As shown, this embodiment also provides a virtual simulation teaching system, which mainly includes a virtual module, a verification module and a processing module.

[0068] The virtual module is used to generate a virtual teaching scene that is not limited by position and its corresponding virtual mapping, and obtain the control data of the virtual object in the virtual teaching scene, and the control data at least includes control position data and control gesture data.

[0069] The verification module is used to generate a real verification scene corresponding to the virtual teaching scene but limited by position and its corresponding real mapping; and collect user position data and user hand data to generate a hand mapping corresponding to the user hand data.

[0070] The processing module is used to extract several sub-virtual mappings and sub-realistic mappings in the virtual teaching scene and the real verification scene respectively, combine the sub-virtual mappings and sub-realistic mappings to construct a simulation teaching scene, and synchronously project the hand mapping to the simulation teaching scene; based on the user position data and the user's hand mapping manipulation gestures in the simulation teaching scene, control the virtual objects in the virtual teaching scene to perform corresponding responses. The specific manipulation logic of the system corresponds to the above-mentioned virtual simulation teaching method, which will not be repeated here.

[0071] In addition, combined Figure 3 As shown, this embodiment also provides a virtual simulation teaching computer device, including one or more processors and a memory; the memory stores control instructions corresponding to the above-mentioned virtual simulation teaching method; the processor is used to receive user control signals and execute the control instructions stored in the memory. The specific control logic of the virtual simulation teaching computer device also corresponds to the above-mentioned virtual simulation teaching method, which will not be repeated here.

[0072] Embodiment 2: The difference from Embodiment 1 is that the virtual mapping and the real mapping are synchronously enlarged or reduced according to the manipulation position data and the user position data, so that the virtual objects in the virtual simulation scene finally constructed can be enlarged or reduced as the user moves, making the virtual objects in the virtual simulation scene closer to the actual situation.

[0073] Before synchronously loading the hand mapping into the simulation teaching scene, the overlap between the virtual mapping and the real mapping is detected. When the overlap between the virtual mapping and the real mapping is greater than a threshold, the hand mapping is synchronously loaded into the simulation teaching scene to ensure that the constructed virtual simulation scene is accurate and stable. When it is operable, the user's hand mapping is synchronously loaded into the simulation teaching scene so that the user can perform corresponding operations.

[0074] The overlap between the virtual mapping and the real mapping is obtained by: obtaining the first boundary of the virtual mapping and the second boundary of the real mapping, constructing a virtual space coordinate system, which is a three-dimensional coordinate system, and assigning coordinates of the first boundary and the second boundary in the virtual space coordinate system.

[0075] Based on the coordinates of the first boundary and the second boundary in the virtual space coordinate system, the distance between the first boundary and the second boundary is calculated. When the distance between each first boundary and each second boundary is less than a threshold, it is determined that the overlap between the virtual mapping and the real mapping is greater than the threshold.

[0076] In addition, the third boundary of the real object (the real object in the teaching scenario) is obtained, the coordinates of the third boundary in the virtual space coordinate system are assigned, and it is detected whether the third boundary is located in the virtual mapping. When the third boundary is located in the virtual mapping, a prompt signal (text prompt, voice prompt, etc.) is issued to prevent the user from coming into contact with it and interfering with the virtual simulation teaching.

[0077] Embodiment 3: The difference from Embodiment 2 is that a manipulation database and a recognition model are constructed (pre-trained based on user information and multiple manipulation gestures of the user); the manipulation database is used to store the manipulation data of virtual objects in the virtual teaching scene, obtain the manipulation gestures of the hands mapped in the simulation teaching scene in real time, and retrieve the manipulation gesture data corresponding to the manipulation gestures mapped in the simulation teaching scene. And perform the following manipulations according to the retrieval results:

[0078] If the manipulation gesture data corresponding to the manipulation gesture mapped by the hand in the simulation teaching scene is retrieved in the manipulation database, the virtual object in the virtual teaching scene is directly controlled to execute a corresponding response.

[0079] If the manipulation gesture data corresponding to the manipulation gesture mapped by the hand in the simulation teaching scene is not retrieved in the manipulation database, the user information is obtained, and the user information is input into the trained recognition model along with the manipulation gesture mapped by the hand in the simulation teaching scene (the recognition model is constructed based on a convolutional neural network, and its training method is a relatively mature existing technology, which will not be repeated here), and the recognition model outputs the manipulation gesture data corresponding to the manipulation gesture mapped by the hand in the simulation teaching scene that is not retrieved in the manipulation database, and then controls the virtual object in the virtual teaching scene to perform the corresponding response. By obtaining user information, the recognition model can autonomously identify the user's operating habits, and allow the user to adopt manipulation actions that are suitable for the individual and allowed by industry regulations when performing certain operations.

[0080] Obviously, the above embodiments are merely examples for the purpose of clear explanation, and are not intended to limit the implementation methods. For those skilled in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the implementation methods here. The obvious changes or modifications derived therefrom are still within the scope of protection of the invention.

Claims

1. A virtual simulation teaching method, characterized in that: The steps include: Generate a virtual teaching scene that is not limited by position and its corresponding virtual mapping, and obtain control data of virtual objects in the virtual teaching scene, the control data at least including control position data and control gesture data; Generate a real verification scene corresponding to the virtual teaching scene but limited by position and its corresponding real mapping; and collect user position data and user hand data to generate a hand mapping corresponding to the user hand data; Extracting a number of sub-virtual mappings and sub-realistic mappings in the virtual teaching scene and the real verification scene respectively, combining the sub-virtual mappings and the sub-realistic mappings to construct a simulation teaching scene, and synchronously loading the hand mapping into the simulation teaching scene; Based on the user position data and the manipulation gestures of the user's hand mapped in the simulated teaching scene, the virtual objects in the virtual teaching scene are controlled to perform corresponding responses; Acquire a boundary of the virtual mapping as a first boundary, a boundary of the real mapping as a second boundary, and a boundary of the real object as a third boundary; Constructing a virtual space coordinate system, where the virtual space coordinate system is a three-dimensional coordinate system, and assigning coordinates of the first boundary, the second boundary, and the third boundary in the virtual space coordinate system; Based on the coordinates of the first boundary, the second boundary and the third boundary in the virtual space coordinate system, the distance between the first boundary and the second boundary is calculated, and when the distance between each first boundary and each second boundary is less than a threshold value, it is determined that the overlap between the virtual mapping and the real mapping is greater than the threshold value; and whether the third boundary is located in the virtual mapping is detected, and when the third boundary is located in the virtual mapping, a prompt signal is issued; Displaying the reality verification scene through a touch display device and collecting the user's touch signal on the reality verification scene, and obtaining the user's control position data based on the user's touch signal on the reality verification scene; collecting the user's control gesture through a wearable device, the control gesture including the hand shape and the applied force; collecting the user's position data and visual angle information through an image acquisition device, and switching the reality verification scene position within a limited position range according to the user's position information and visual angle information; After the display device receives the touch signal of the user on the reality verification scene, it determines the first touch mode between the user and the reality verification scene, and verifies the first touch mode in combination with the image acquisition device to obtain the final second touch mode, and generates a corresponding control verification gesture through the second touch mode, and compares the user's control gesture collected by the wearable device with the control verification gesture to obtain the user's control gesture.

2. The virtual simulation teaching method according to claim 1, characterized in that: The virtual mapping and the real mapping are synchronously enlarged or reduced according to the manipulation position data and the user position data, and the overlap between the virtual mapping and the real mapping is detected. When the overlap between the virtual mapping and the real mapping is greater than a threshold, the hand mapping is synchronously loaded into the simulation teaching scene.

3. The virtual simulation teaching method according to claim 1, characterized in that: Build manipulation database and recognition model; The control database is used to store the control data of the virtual objects in the virtual teaching scene, obtain the control gestures of the hands mapped in the simulation teaching scene in real time, and retrieve the control gesture data corresponding to the control gestures of the hands mapped in the simulation teaching scene; If the manipulation gesture data corresponding to the manipulation gesture mapped by the hand in the simulation teaching scene is retrieved in the manipulation database, the virtual object in the virtual teaching scene is directly controlled to execute a corresponding response; If the control gesture data corresponding to the control gesture mapped by the hand in the simulation teaching scene is not retrieved in the control database, the user information is obtained, and the user information is input into the trained recognition model along with the control gesture mapped by the hand in the simulation teaching scene. The recognition model outputs the control gesture data corresponding to the control gesture mapped by the hand in the simulation teaching scene that has not been retrieved in the control database, and then controls the virtual object in the virtual teaching scene to execute a corresponding response.

4. The virtual simulation teaching method according to claim 1, characterized in that: Dividing the virtual mapping into a plurality of first unit areas, and dividing the real mapping into a plurality of second unit areas, wherein the number of the first unit areas is the same as the number of the second unit areas; The sub-virtual mapping is one or more first unit areas, and the sub-reality mapping is one or more second unit areas; the first unit areas included in the sub-virtual mapping and the areas covered by the second unit areas included in the sub-reality mapping do not overlap, and the total number of sub-virtual mappings and sub-reality mappings is equal to the number of first unit areas.

5. The virtual simulation teaching method according to claim 4, characterized in that: Each sub-reality mapping contains at least a partial area of ​​the virtual object in the virtual teaching scene.

6. A virtual simulation teaching system, characterized in that: include: A virtual module, used to generate a virtual teaching scene that is not limited by position and its corresponding virtual mapping, and obtain control data of virtual objects in the virtual teaching scene, the control data at least including control position data and control gesture data; A verification module is used to generate a real verification scene corresponding to the virtual teaching scene but limited by position and its corresponding real mapping; and collect user position data and user hand data to generate a hand mapping corresponding to the user hand data; A processing module, used to extract a plurality of sub-virtual mappings and sub-real mappings in a virtual teaching scene and a real verification scene respectively, combine the sub-virtual mappings and the sub-real mappings to construct a simulation teaching scene, and synchronously load the hand mapping into the simulation teaching scene; It is also used to control the virtual objects in the virtual teaching scene to perform corresponding responses based on the user position data and the manipulation gestures of the user's hand mapped in the simulated teaching scene; Also used for obtaining a boundary of the virtual mapping as a first boundary, a boundary of the real mapping as a second boundary, and a boundary of the real object as a third boundary; Constructing a virtual space coordinate system, where the virtual space coordinate system is a three-dimensional coordinate system, and assigning coordinates of the first boundary, the second boundary, and the third boundary in the virtual space coordinate system; Based on the coordinates of the first boundary, the second boundary and the third boundary in the virtual space coordinate system, the distance between the first boundary and the second boundary is calculated, and when the distance between each first boundary and each second boundary is less than a threshold value, it is determined that the overlap between the virtual mapping and the real mapping is greater than the threshold value; and whether the third boundary is located in the virtual mapping is detected, and when the third boundary is located in the virtual mapping, a prompt signal is issued; Displaying the reality verification scene through a touch display device and collecting the user's touch signal on the reality verification scene, and obtaining the user's control position data based on the user's touch signal on the reality verification scene; collecting the user's control gesture through a wearable device, the control gesture including the hand shape and the applied force; collecting the user's position data and visual angle information through an image acquisition device, and switching the reality verification scene position within a limited position range according to the user's position information and visual angle information; After the display device receives the touch signal of the user on the reality verification scene, it determines the first touch mode between the user and the reality verification scene, and verifies the first touch mode in combination with the image acquisition device to obtain the final second touch mode, and generates a corresponding control verification gesture through the second touch mode, and compares the user's control gesture collected by the wearable device with the control verification gesture to obtain the user's control gesture.

7. A virtual simulation teaching computer device, characterized in that: It comprises one or more processors and a memory; the memory stores control instructions corresponding to the virtual simulation teaching method described in any one of claims 1-5; the processor is used to receive user control signals and execute the control instructions stored in the memory.

Images