VR teaching processing method and device, electronic equipment and readable storage medium
By obtaining user action and material attribute data in VR teaching and generating multimodal sensory feedback signals, the problem of insufficient sensory feedback in the existing technology is solved, the authenticity and interactivity of VR teaching is improved, and the learning effect is enhanced.
Patent Information
- Application Number
- CN202411132393.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-16
- Publication Date
- 2024-12-06
AI Technical Summary
The sensory feedback experience in existing VR teaching techniques is poor, resulting in poor teaching results.
By obtaining the action data and material attribute data when the user interacts with the target object, a multimodal sensory feedback signal is generated, including touch, hearing, vision and smell, combining deep learning models to simulate material attributes and sensory feedback during action interaction.
It improves the authenticity and interactivity of VR teaching, enhances learners' immersion and memory efficiency, and allows learners to feel and master handicraft skills more comprehensively.
Smart Images

Figure CN119088216A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the field of virtual reality technology, and in particular to a VR teaching processing method, device, electronic device and readable storage medium. Background Art
[0002] With the development of virtual reality (VR) technology, its application in education and training has gradually increased, especially in the teaching of handicraft skills. VR technology provides a new interactive learning method. By simulating the real handicraft environment and process, VR teaching can help learners learn in all aspects of space and vision without physical restrictions.
[0003] In the related art, VR teaching solutions usually include using head-mounted displays to provide visual simulations, and using handheld controllers or somatosensory devices for interaction. For example, in pottery teaching, learners can "touch" and "shape" virtual clay in a VR environment. However, the related art mainly simulates sensory feedback experience through visual and vibration feedback, and its teaching effect is relatively poor. Summary of the invention
[0004] The embodiments of the present disclosure provide a VR teaching processing method, device, electronic device and readable storage medium to solve the problem in the related art that the sensory feedback experience is relatively poor, resulting in relatively poor VR teaching effect.
[0005] In order to solve the above technical problems, the present disclosure is implemented as follows:
[0006] In a first aspect, an embodiment of the present disclosure provides a VR teaching processing method, the method comprising:
[0007] Acquire first data in a VR teaching production process of a target object, wherein the first data includes first action data of a user on the target object and target attribute data, wherein the target attribute data indicates at least one of a material attribute of the target object and a material attribute of a tool that the user acts on the target object;
[0008] Determining sensory data generated when the user interacts with the target object based on the first action data;
[0009] generating, based on the target attribute data and the sensory data, a first sensory feedback signal corresponding to a material attribute indicated by the target attribute data;
[0010] Sensory feedback is performed based on the first sensory feedback signal.
[0011] In a second aspect, an embodiment of the present disclosure provides a VR teaching processing device, the device comprising:
[0012] A first acquisition module is used to acquire first data in a VR teaching production process of a target object, wherein the first data includes first action data of a user on the target object and target attribute data, wherein the target attribute data indicates at least one of a material attribute of the target object and a material attribute of a tool that the user acts on the target object;
[0013] A first determination module, configured to determine sensory data generated when a user interacts with the target object based on the first action data;
[0014] A first generating module, configured to generate, based on the target attribute data and the sensory data, a first sensory feedback signal corresponding to a material attribute indicated by the target attribute data;
[0015] The first sensory feedback module is used to perform sensory feedback based on the first sensory feedback signal.
[0016] In a third aspect, an embodiment of the present disclosure provides an electronic device, comprising: a memory, a processor, and a program stored in the memory and executable on the processor, wherein when the program is executed by the processor, the steps in the VR teaching processing method described in the first aspect are implemented.
[0017] In a fourth aspect, an embodiment of the present disclosure provides a readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps in the VR teaching processing method described in the first aspect are implemented.
[0018] In a fifth aspect, an embodiment of the present disclosure provides a computer program product, including computer instructions, which, when executed by a processor, implement the steps in the VR teaching processing method as described in the first aspect.
[0019] In this embodiment, by acquiring the first data in the VR teaching production process of the target object, the first data includes the first action data of the user on the target object and the target attribute data, the target attribute data indicates at least one of the material attribute of the target object and the material attribute of the tool that the user acts on the target object; based on the first action data, the sensory data generated when the user interacts with the target object is determined; based on the target attribute data and the sensory data, a first sensory feedback signal corresponding to the material attribute indicated by the target attribute data is generated; sensory feedback is performed based on the first sensory feedback signal. Since the sensory feedback combines the user's action on the target object, the material attribute of the target object and the material attribute of the tool, the sensory feedback in the VR teaching production process of the target object can be made more realistic, thereby improving the VR teaching effect. BRIEF DESCRIPTION OF THE DRAWINGS
[0020] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the drawings required for use in the description of the embodiments of the present disclosure will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.
[0021] Figure 1 is a flow chart of a VR teaching processing method provided by an embodiment of the present disclosure;
[0022] Figure 2 is a structural diagram of a VR teaching processing device provided in an embodiment of the present disclosure;
[0023] Figure 3 It is a schematic diagram of the structure of an electronic device provided by an embodiment of the present disclosure. DETAILED DESCRIPTION
[0024] The following will be combined with the drawings in the embodiments of the present disclosure to clearly and completely describe the technical solutions in the embodiments of the present disclosure. Obviously, the described embodiments are part of the embodiments of the present disclosure, rather than all of the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present disclosure.
[0025] It should be noted that the method of the embodiment of the present disclosure is applied to a VR device, which integrates a sensory feedback module for simulating multiple sensory types. For example, the VR device may include: a head-mounted display, a wearable device (such as gloves and clothing), which may have built-in multi-point tactile feedback technology such as a pressure sensor and a vibration motor, a device with an odor release function, and an environmental sound system, etc. These sensory feedback modules and the central controller of the VR device can form a sensory feedback system. At the same time, the VR device can also integrate multiple sensors to collect user data and environmental data. For example, the VR device can also include a camera, a handheld controller, and a somatosensory device.
[0026] See also Figure 1 , Figure 1 is a flow chart of a VR teaching processing method provided by an embodiment of the present disclosure, such as Figure 1 As shown, the following steps are included:
[0027] Step 101, obtaining first data in a VR teaching production process of a target object, wherein the first data includes first action data of a user on the target object and target attribute data, wherein the target attribute data indicates at least one of a material attribute of the target object and a material attribute of a tool applied by the user to the target object.
[0028] In this step, the target object may be any object. In some embodiments, the target object may be a handicraft object, such as ceramics, wood carvings, and the like.
[0029] VR teaching of target objects refers to displaying the production screen of the target object through the head-mounted display in the VR device, so that users can conduct all-round spatial and visual learning of the production process of the target object without physical restrictions.
[0030] During the VR teaching and production process of the target object, controls for users to interact with the target object can be provided. For example, during the teaching and production demonstration of the target object, controls for users to start production of the target object can be provided. The user can trigger the control to enter the stage of simulating the user's production of the target object in the virtual environment.
[0031] Accordingly, the user can interact with the virtual target object in the production screen, such as touching or kneading clay, scratching wood with metal tools, etc. The VR device can collect the user's interaction data on the target object through a camera to obtain the user's first action data on the target object. Among them, the interaction data refers to the image data when the user contacts the virtual target object, and the action data refers to the action performed by the user on the target object. The action data may include the user's action parameters on the target object, such as the gesture and strength of kneading clay, the contact relationship between the hand and the clay (such as contacting the bottom of the clay), etc.
[0032] For example, by collecting images of the user interacting with the target object and analyzing the images, the first action data of the user on the target object can be obtained. In this way, the user's actions on the target object can be obtained through image analysis. For example, the user touches or kneads clay to feel the hardness, texture, elasticity, humidity, etc. of the clay.
[0033] In order to provide better and more realistic action feedback, it is usually necessary to simulate some continuous actions of the user, for example, the user continuously kneads clay to observe the process of clay molding, or the user scratches wood with a metal tool to make a wood carving. In some embodiments, the first action data is obtained by:
[0034] Acquire interaction data between the user and the target object during the VR teaching production process of the target object;
[0035] Inputting the interaction data into a first model for action prediction to obtain the first action data;
[0036] The first model is trained based on second data, and the second data includes second action data captured during the actual production process of the target object.
[0037] In VR teaching of handicraft skills, it is crucial to accurately capture the user's movements, because it directly affects whether learners can accurately learn and imitate complex manual skills. High-resolution motion capture devices in VR devices can be used, such as high-precision motion capture cameras and some sensors to capture images or videos to capture the interaction data between the user and the target object, such as capturing the subtle movements of the user's hands and whole body. The images collected from the motion capture device can be processed and analyzed based on the first model to obtain the user's first action data on the target object, which can improve the accuracy and fluency of action recognition and capture the details of the user's actions.
[0038] Among them, the first model can be a deep learning model, which can capture the motion of craftsmen in the actual production process of the target object, such as a real handicraft production environment, through data collection and analysis, and collect a large amount of second motion data. These second motion data include every small movement of the hand, such as the bending and rotation of the fingers, and the operation of various tools. Through model training and optimization, the first model is trained using the collected second motion data to ensure that the first model can recognize and simulate various complex handicraft movements. The first model will be repeatedly optimized during the model training process to achieve high-precision motion reproduction.
[0039] The VR device can obtain the first action data based on the first model, and then perform real-time action rendering based on the first action data, to ensure that the learner's actions can be rendered accurately in real time and the user actions can be displayed.
[0040] It should be noted that in the stage of simulating the user's production of the target object in the virtual environment, the user's action strength, gestures and other action parameters on the target object are different, and the final product may be different. For example, for wood carving, the user's carving strength on the wood is different, and the resulting finished wood carving will be different in overall shape and detailed features. In some embodiments, after inputting the interaction data into the first model for action prediction and obtaining the first action data, the method further includes:
[0041] Displaying action guidance information, where the action guidance information is used to correct the action corresponding to the first action data and to instruct the user on at least one of the next actions after the action corresponding to the first action data;
[0042] The action guidance information is obtained by comparing the first action data with the second action data.
[0043] On the basis of obtaining and displaying the first action data, the action data in the second action data that matches the first action data can be determined by comparing the first action data with the second action data. Since the matching action data indicates a more precise action of the craftsman, it can be used as a guiding action.
[0044] In some embodiments, the action guidance information can be displayed by comparing the first action data with the action data matching the first action data, so as to correct the action corresponding to the first action data. For example, when kneading clay, by comparing the action parameters, it can be obtained that the curvature of the fingers in the user's hand action corresponding to the first action data is not enough, which will result in insufficient force in kneading the clay. Accordingly, the action guidance information can be displayed to correct the curvature of the fingers in the user's hand action.
[0045] In some embodiments, the second action data includes a series of coherent action data for making a target object. Action data in the second action data that matches the first action data can be obtained, and based on the matched action data, the next action after the action corresponding to the matched action data is obtained from the second action data, and action guidance information is displayed to utilize the next action after the action corresponding to the matched action data to guide the user in the next action after the action corresponding to the first action data.
[0046] For example, when kneading clay, the first action data indicates that the bottom of the clay should be kneaded according to the first curvature of the fingers. By comparing the action parameters, the action data matching the first action data is obtained from the second action data, and other action data associated with the matching action data are obtained from the second action data. The other associated action data indicates that the next action after the action corresponding to the matching action data is to knead the bottom of the clay according to the second curvature of the fingers. Accordingly, action guidance information can be displayed to guide the user to adjust the curvature of the hand and knead the bottom of the clay according to the second curvature.
[0047] In this way, by displaying action guidance information, appropriate guidance and correction can be provided to users, which helps learners to accurately imitate and practice professional skills in a real, comprehensive and interactive virtual learning environment for handicraft learning without actual contact with materials, thereby improving the VR teaching effect of the target object.
[0048] The target attribute data may include material attribute data of the target object. For example, if the target object is made of clay with relatively high hardness, and if the user uses a tool to act on the target object, the target attribute data may also include material attribute data of the tool. For example, if the tool is made of iron metal.
[0049] VR devices can simulate the physical properties of different materials, such as hardness, texture, elasticity, etc. Accordingly, VR devices can determine the target attribute data based on the VR teaching content of the target object and the user's choice. For example, the VR teaching content is to use clay to make pottery bowls. According to the teaching content, it can be determined that the target object is a clay material with relatively high hardness. For another example, the VR teaching content is to make wood carvings. In the process of simulating wood carving, users can choose to use tools made of iron metal materials to carve wood.
[0050] Step 102: Determine sensory data generated when the user interacts with the target object based on the first action data.
[0051] The sensory data may include sensory types and sensory parameters, and the sensory types may include touch, hearing, vision, and smell, etc. The sensory parameters may include sensory intensity, sensory mode, and sensory characteristics, etc. In some embodiments, the sensory data matching the first action data may be determined based on a preset correspondence table between action data and sensory data. For example, the sensory type matching the action identifier of the first action data may be determined based on the action identifier of the first action data, and the sensory intensity, sensory mode, and sensory characteristics may be determined based on the action parameters of the first action data.
[0052] For example, the action identifier of the first action data indicates that the user scratches wood with a metal tool, and the sensory types that match the action identifier include touch (i.e., the touch of scratching wood with a tool), hearing (i.e., scratching sound), etc. The action parameters of the first action data may include the curvature of the finger (the curvature may indicate the scratching strength of the user's hand) and the scratching frequency of the finger, etc., and the tactile intensity and auditory intensity that match the curvature of the finger, as well as the corresponding sensory mode, may be determined. For example, if the curvature of the hand is relatively large, it means that the scratching strength of the user is relatively large. At this time, the sensory mode may be pressure feedback, that is, feedback to the user through a more delicate sense of pressure. If the curvature of the hand is relatively small, it means that the scratching strength of the user is relatively small. At this time, the sensory mode may be vibration feedback. It is also possible to determine the sensory characteristics that match the finger scratching frequency. For example, the faster the finger scratching frequency, the faster the frequency of the sensory characteristics indicating feedback is emitted.
[0053] In addition, in some embodiments, sensory features can also be determined based on action identifiers. For example, an action identifier indicates an action of a user scratching wood with a metal tool, and indicates an action of a user kneading clay. The sensory features are different in hearing. Scratching wood with a metal tool sounds sharper than kneading clay.
[0054] In some embodiments, the first action data may also be input into a pre-trained deep learning model to obtain sensory data generated when the user interacts with the target object.
[0055] Step 103: Based on the target attribute data and the sensory data, generate a first sensory feedback signal corresponding to the material attribute indicated by the target attribute data.
[0056] In some embodiments, the first sensory feedback signal includes at least one of the following:
[0057] The sensory type is the first feedback signal of touch;
[0058] The sensory type is the second feedback signal of auditory;
[0059] The sensory type is the third feedback signal of vision;
[0060] The fourth feedback signal of sensory type is smell.
[0061] A sensory feedback signal can be generated based on the sensory data, and the generated sensory feedback signal can be adjusted in combination with the target attribute data to generate a first sensory feedback signal corresponding to the material attribute indicated by the target attribute data. For example, the signal strength of the sensory feedback signal generated based on the sensory data is A. Since the tactile feedback of scratches by different metal tools can be different, the signal strength of the sensory feedback signal can be adjusted according to the target attribute data to obtain a first sensory feedback signal corresponding to the material attribute indicated by the target attribute data. The target attribute data and the sensory data can also be used to accurately predict and simulate the sensory feedback that should be generated when different actions and materials interact through a deep learning model to obtain a first sensory feedback signal, that is, the first sensory feedback signal indicates the sensory feedback of the user's action on the material of the attribute corresponding to the target attribute information.
[0062] In some embodiments, the target attribute data and the first action data can be input into a pre-trained deep learning model, which can determine the sensory data based on the first action data, and use the target attribute data and the sensory data to output a first sensory feedback signal corresponding to the first data.
[0063] Among them, the first sensory feedback signal may include a first feedback signal, the deep learning model can simulate the physical properties of different materials, such as hardness, texture, elasticity, etc., and convert these properties into tactile feedback signals in combination with the sensory data of the first action data, and the signal strength of the first sensory feedback signal can correspond to the action parameters of the first action data, such as the action intensity indicated by the curvature of the finger, and the signal type of the first sensory feedback signal can correspond to the sensory mode.
[0064] In the related art, although some VR devices try to enhance the experience by adding simple physical interactions such as vibration or thrust, they fail to effectively integrate multiple sensory inputs, such as combining touch, smell, etc., which is particularly important in handicraft learning, because the comprehensive experience of multiple senses is essential for the mastery of skills. Limited tactile feedback: Current VR teaching technology mainly relies on visual and auditory feedback. Although some systems use vibration controllers to simulate tactile experience, this feedback is usually very limited. For example, in learning that requires fine manual skills such as pottery or woodworking, simple vibration cannot accurately convey the texture of the material, pressure changes, and other subtle movements. In addition, it is very important to provide a multimodal learning experience, that is, a teaching environment that integrates vision, hearing, touch, and even smell at the same time. Sensory experience is critical to the acquisition and memory of skills. In addition, the related art lacks the fine simulation of complex handicraft actions and the tactile feedback that should be generated when different actions and materials interact.
[0065] In this embodiment, the sensory data generated when the user interacts with the target object is determined based on the first action data; based on the target attribute data and the sensory data, a first sensory feedback signal corresponding to the material attribute indicated by the target attribute data is generated. In this way, not only can the tactile feedback generated when different actions and materials interact can be simulated, but also a multimodal learning experience can be provided, that is, a teaching environment that integrates vision, hearing, touch and even smell at the same time, thereby improving the VR teaching effect.
[0066] Step 104: performing sensory feedback based on the first sensory feedback signal.
[0067] The generated first sensory feedback signal can be sent to the sensory feedback module through the central controller, and sensory feedback can be provided to the user through the multiple sensory feedback modules using synchronized multiple sensory input technology, thereby ensuring seamless integration of visual, auditory, tactile and olfactory information and rapid response.
[0068] Haptic feedback technology can be integrated into VR devices to ensure timely and accurate feedback. A high-speed microcontroller is used to adjust the response of the sensor to ensure that the tactile feedback felt by the user closely matches the actual action when performing handicraft operations. The tactile feedback module in the VR device can provide corresponding physical feedback based on the interaction in the virtual environment, simulating the real touch of handicraft materials such as the pressure of clay and the scratching of wood. That is, the tactile feedback module can simulate various physical sensations, such as the hardness and texture of the material, and the specific feeling when the tool interacts with the material, which can greatly enhance the realism and interactivity of handicraft VR teaching.
[0069] In this embodiment, by acquiring the first data in the VR teaching production process of the target object, the first data includes the first action data of the user on the target object and the target attribute data, the target attribute data indicates at least one of the material attribute of the target object and the material attribute of the tool that the user acts on the target object; based on the first action data, the sensory data generated when the user interacts with the target object is determined; based on the target attribute data and the sensory data, a first sensory feedback signal corresponding to the material attribute indicated by the target attribute data is generated; sensory feedback is performed based on the first sensory feedback signal. Since the sensory feedback combines the user's action on the target object, the material attribute of the target object and the material attribute of the tool, the sensory feedback in the VR teaching production process of the target object can be made more realistic, thereby improving the VR teaching effect.
[0070] Furthermore, a VR teaching platform that integrates vision, hearing, touch, and even smell can be realized. Through this multimodal learning environment, learners can not only see and hear the tutorial content, but also feel the texture and possible smell of the materials, and more comprehensively simulate the real handicraft environment. That is, this embodiment can solve the shortcomings of related technologies in providing a comprehensive and immersive handicraft VR teaching experience by introducing a multimodal fusion technology. By integrating more sophisticated motion capture technology, tactile feedback equipment, and synchronization of vision and hearing, a real, comprehensive, and interactive virtual learning environment is provided for handicraft learning, allowing learners to feel and learn various handicraft skills more realistically.
[0071] Moreover, it can improve the sense of reality and immersion in learning. Through the tactile feedback system, it can provide more realistic physical sensations, such as the texture of materials, the interaction between tools and materials, etc., which greatly enhances the sense of immersion and authenticity in learning. This helps learners to understand and master handicraft skills more deeply. The all-round sensory stimulation is closer to the real handicraft operation environment, which helps to improve memory and learning efficiency.
[0072] In some embodiments, the generating, based on the target attribute data and the sensory data, a first sensory feedback signal corresponding to the material attribute indicated by the target attribute data comprises:
[0073] generating a first sensory feedback signal corresponding to the material property indicated by the target property data based on the target property data, the sensory data and third data, the third data including at least one of user preference information and user sensory feedback information;
[0074] The third data is used to adjust the signal strength and signal type of the first sensory feedback signal.
[0075] Users can adjust the intensity and method of sensory feedback such as tactile feedback according to their personal preferences, such as choosing a more delicate sense of pressure or a more obvious vibration feedback. In the process of VR teaching production, the sensory effect can be adjusted in real time according to the user's sensory feedback, such as automatically adjusting the pressure sensing intensity according to the force applied by the user.
[0076] A user interface can be designed that allows learners to customize their learning experience, such as adjusting the intensity of the scent, selecting auditory and visual preferences, etc., and provides real-time feedback to optimize the learning process.
[0077] In this way, personalized sensory feedback can be achieved for users, improving user experience.
[0078] In some embodiments, the VR teaching production process of the target object is performed in a VR tutorial environment. Before step 101, the method further includes:
[0079] Acquiring the production environment characteristics of the target object;
[0080] generating a second sensory feedback signal based on the production environment characteristics;
[0081] Sensory feedback is performed based on the second sensory feedback signal to form the VR tutorial environment.
[0082] By simulating the craft environment of the target object, the production environment characteristics of the target object can be obtained, such as the unique smell of the pottery studio (such as the smell of clay and the firing process), the smell and sound of sawdust in the woodworking room, ensuring that the atmosphere of each environment is as real as possible.
[0083] A second sensory feedback signal corresponding to the production environment characteristics can be generated based on the production environment characteristics, and sensory feedback can be performed based on the second sensory feedback signal to form a VR tutorial environment. In this way, specific virtual environments can be designed for different handicraft skills, and each environment is optimized according to the characteristics of the skill to enhance the learning effect and experience.
[0084] In some embodiments, the method further comprises:
[0085] Acquire fourth data in the process of producing VR teaching for the target object, wherein the fourth data includes at least one of a user's learning progress and a teaching difficulty of the target object fed back by the user;
[0086] Based on the fourth data, task information of the VR teaching of the target object is adjusted, the task information including at least one of the task difficulty of the VR teaching of the target object and the signal parameter of the sensory feedback, and the task information is used to indicate the VR teaching content of the target object.
[0087] In this embodiment, the task information of the VR teaching of the target object can be dynamically adjusted according to user feedback and learning progress to adjust the VR teaching content and sensory input of the target object. For example, the task difficulty, the intensity and method of sensory stimulation can be adjusted according to the success rate of the user's action operation during the production process of the target object, making the learning process more personalized and adaptable.
[0088] Among them, the task information is used to indicate the VR teaching content of the target object, and the difficulty of VR teaching production can be adjusted by adjusting the production method of the target object. For example, due to the different production methods of the target object, the task difficulty of VR teaching production of the target object will be different. For example, in order to make more exquisite ceramics or wood carvings, the production method of the target object may be relatively complicated, and the task difficulty of VR teaching production of the target object will be relatively difficult. Correspondingly, the VR teaching content of the target object will be different. For another example, the signal parameters of sensory feedback are different, and the sensory input to the user will be different, so that the VR teaching content of the target object will be adjusted.
[0089] Afterwards, the VR teaching content of the target object corresponding to the adjusted task information can be displayed through the head-mounted display.
[0090] The present disclosure also provides a VR teaching processing device, see Figure 2 , Figure 2 is a structural diagram of a VR teaching processing device provided in an embodiment of the present disclosure, such as Figure 2 As shown, the device comprises:
[0091] A first acquisition module 201 is used to acquire first data in a VR teaching production process of a target object, wherein the first data includes first action data of a user on the target object and target attribute data, wherein the target attribute data indicates at least one of a material attribute of the target object and a material attribute of a tool applied by the user to the target object;
[0092] A first determination module 202, configured to determine sensory data generated when the user interacts with the target object based on the first action data;
[0093] A first generating module 203, for generating a first sensory feedback signal corresponding to a material property indicated by the target property data based on the target property data and the sensory data;
[0094] The first sensory feedback module 204 is configured to perform sensory feedback based on the first sensory feedback signal.
[0095] Optionally, the first sensory feedback signal includes at least one of the following:
[0096] The sensory type is the first feedback signal of touch;
[0097] The sensory type is the second feedback signal of auditory;
[0098] The sensory type is the third feedback signal of vision;
[0099] The fourth feedback signal of sensory type is smell.
[0100] Optionally, the first action data is obtained in the following manner:
[0101] Acquire interaction data between the user and the target object during the VR teaching production process of the target object;
[0102] Inputting the interaction data into a first model for action prediction to obtain the first action data;
[0103] The first model is trained based on second data, and the second data includes second action data captured during the actual production process of the target object.
[0104] Optionally, the device further comprises:
[0105] A display module, used to display action guidance information, where the action guidance information is used to correct the action corresponding to the first action data and to instruct the user on at least one of the next actions after the action corresponding to the first action data;
[0106] The action guidance information is obtained by comparing the first action data with the second action data.
[0107] Optionally, the first generating module 203 is specifically configured to:
[0108] generating a first sensory feedback signal corresponding to the material property indicated by the target property data based on the target property data, the sensory data and third data, the third data including at least one of user preference information and user sensory feedback information;
[0109] The third data is used to adjust the signal strength and signal type of the first sensory feedback signal.
[0110] Optionally, the VR teaching production process of the target object is performed in a VR tutorial environment, and the device further includes:
[0111] A second acquisition module, used to acquire the production environment characteristics of the target object;
[0112] A second generating module, used for generating a second sensory feedback signal based on the characteristics of the production environment;
[0113] The second sensory feedback module is used to perform sensory feedback based on the second sensory feedback signal to form the VR tutorial environment.
[0114] Optionally, the device further comprises:
[0115] A third acquisition module is used to acquire fourth data in the VR teaching production process of the target object, wherein the fourth data includes at least one of a user's learning progress and a teaching difficulty of the target object fed back by the user;
[0116] An adjustment module is used to adjust task information of the VR teaching of the target object based on the fourth data, wherein the task information includes at least one of the task difficulty of the VR teaching of the target object and the signal parameter of the sensory feedback, and the task information is used to indicate the VR teaching content of the target object.
[0117] The VR teaching processing device 200 provided in the embodiment of the present disclosure can implement each process implemented in the above-mentioned VR teaching processing method embodiment, and can achieve the same beneficial effects. To avoid repetition, it will not be described here.
[0118] The present disclosure also provides an electronic device, which may be a VR device. Figure 3 , the figure shows a schematic diagram of the structure of an electronic device provided by an embodiment of the present disclosure. Figure 3 As shown, the electronic device 300 includes: a processor 301 , a memory 302 , a user interface 303 and a bus interface 304 .
[0119] The processor 301 is used to read the program in the memory 302 and execute the following process:
[0120] Acquire first data in a VR teaching production process of a target object, wherein the first data includes first action data of a user on the target object and target attribute data, wherein the target attribute data indicates at least one of a material attribute of the target object and a material attribute of a tool that the user acts on the target object;
[0121] Determining sensory data generated when the user interacts with the target object based on the first action data;
[0122] generating, based on the target attribute data and the sensory data, a first sensory feedback signal corresponding to a material attribute indicated by the target attribute data;
[0123] Sensory feedback is performed based on the first sensory feedback signal.
[0124] Optionally, the first sensory feedback signal includes at least one of the following:
[0125] The sensory type is the first feedback signal of touch;
[0126] The sensory type is the second feedback signal of auditory;
[0127] The sensory type is the third feedback signal of vision;
[0128] The fourth feedback signal of sensory type is smell.
[0129] Optionally, the first action data is obtained in the following manner:
[0130] Acquire interaction data between the user and the target object during the VR teaching production process of the target object;
[0131] Inputting the interaction data into a first model for action prediction to obtain the first action data;
[0132] The first model is trained based on second data, and the second data includes second action data captured during the actual production process of the target object.
[0133] Optionally, the processor 301 is further configured to:
[0134] Displaying action guidance information, where the action guidance information is used to correct the action corresponding to the first action data and to instruct the user on at least one of the next actions after the action corresponding to the first action data;
[0135] The action guidance information is obtained by comparing the first action data with the second action data.
[0136] Optionally, the processor 301 is further configured to:
[0137] generating a first sensory feedback signal corresponding to the material property indicated by the target property data based on the target property data, the sensory data and third data, the third data including at least one of user preference information and user sensory feedback information;
[0138] The third data is used to adjust the signal strength and signal type of the first sensory feedback signal.
[0139] Optionally, the VR teaching production process of the target object is performed in a VR tutorial environment, and the processor 301 is further configured to:
[0140] Acquiring the production environment characteristics of the target object;
[0141] generating a second sensory feedback signal based on the production environment characteristics;
[0142] Sensory feedback is performed based on the second sensory feedback signal to form the VR tutorial environment.
[0143] Optionally, the processor 301 is further configured to:
[0144] Acquire fourth data in the process of producing VR teaching for the target object, wherein the fourth data includes at least one of a user's learning progress and a teaching difficulty of the target object fed back by the user;
[0145] Based on the fourth data, task information of the VR teaching of the target object is adjusted, the task information including at least one of the task difficulty of the VR teaching of the target object and the signal parameter of the sensory feedback, and the task information is used to indicate the VR teaching content of the target object.
[0146] exist Figure 3 In the embodiment, the bus architecture may include any number of interconnected buses and bridges, specifically linking various circuits of one or more processors represented by processor 301 and memory represented by memory 302. The bus architecture may also link various other circuits such as peripherals, voltage regulators, and power management circuits, which are well known in the art and are therefore not further described herein. The bus interface 304 provides an interface. For different user devices, the user interface 303 may also be an interface that can connect external or internal devices, and the connected devices include but are not limited to keypads, displays, speakers, microphones, joysticks, etc.
[0147] The processor 301 is responsible for managing the bus architecture and general processing, and the memory 302 can store data used by the processor 301 when performing operations.
[0148] Preferably, the embodiment of the present disclosure further provides an electronic device, including a processor 301, a memory 302, and a computer program stored in the memory 302 and executable on the processor 301. When the computer program is executed by the processor 301, each process of the above-mentioned VR teaching processing method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be described here.
[0149] The disclosed embodiment also provides a readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, each process of the above-mentioned VR teaching processing method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it is not repeated here. The readable storage medium is, for example, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.
[0150] The disclosed embodiment also provides a computer program product, including computer instructions. When the computer instructions are executed by a processor, each process of the above-mentioned VR teaching processing method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.
[0151] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.
[0152] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present disclosure, or the part that contributes to the relevant technology, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a disk, or an optical disk), and includes a number of instructions for enabling a terminal device (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods described in each embodiment of the present disclosure.
[0153] The embodiments of the present disclosure are described above in conjunction with the accompanying drawings, but the present disclosure is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present disclosure, ordinary technicians in this field can also make many forms without departing from the scope of protection of the purpose of the present disclosure and the claims, all of which are within the protection of the present disclosure.
Claims
1. A VR teaching processing method, characterized in that: The method comprises: Acquire first data in a VR teaching production process of a target object, wherein the first data includes first action data of a user on the target object and target attribute data, wherein the target attribute data indicates at least one of a material attribute of the target object and a material attribute of a tool that the user acts on the target object; Determining sensory data generated when the user interacts with the target object based on the first action data; generating, based on the target attribute data and the sensory data, a first sensory feedback signal corresponding to a material attribute indicated by the target attribute data; Sensory feedback is performed based on the first sensory feedback signal.
2. The method according to claim 1, characterized in that: The first sensory feedback signal includes at least one of the following: The sensory type is the first feedback signal of touch; The sensory type is the second feedback signal of auditory; The sensory type is the third feedback signal of vision; The fourth feedback signal of sensory type is smell.
3. The method according to claim 1, characterized in that: The first action data is obtained by: Acquire interaction data between the user and the target object during the VR teaching production process of the target object; Inputting the interaction data into a first model for action prediction to obtain the first action data; The first model is trained based on second data, and the second data includes second action data captured during the actual production process of the target object.
4. The method according to claim 3, characterized in that After inputting the interaction data into the first model for action prediction to obtain the first action data, the method further includes: Displaying action guidance information, where the action guidance information is used to correct the action corresponding to the first action data and to instruct the user on at least one of the next actions after the action corresponding to the first action data; The action guidance information is obtained by comparing the first action data with the second action data.
5. The method according to claim 1, characterized in that The step of generating a first sensory feedback signal corresponding to the material property indicated by the target property data based on the target property data and the sensory data comprises: generating a first sensory feedback signal corresponding to the material property indicated by the target property data based on the target property data, the sensory data and third data, the third data including at least one of user preference information and user sensory feedback information; The third data is used to adjust the signal strength and signal type of the first sensory feedback signal.
6. The method according to claim 1, characterized in that The VR teaching production process of the target object is performed in a VR tutorial environment. Before obtaining the first data in the VR teaching production process of the target object, the method further includes: Acquiring the production environment characteristics of the target object; generating a second sensory feedback signal based on the production environment characteristics; Sensory feedback is performed based on the second sensory feedback signal to form the VR tutorial environment.
7. The method according to claim 1, characterized in that The method further comprises: Acquire fourth data in the process of producing VR teaching for the target object, wherein the fourth data includes at least one of a user's learning progress and a teaching difficulty of the target object fed back by the user; Based on the fourth data, task information of the VR teaching of the target object is adjusted, the task information including at least one of the task difficulty of the VR teaching of the target object and the signal parameter of the sensory feedback, and the task information is used to indicate the VR teaching content of the target object.
8. A VR teaching processing device, characterized in that: The device comprises: A first acquisition module is used to acquire first data in a VR teaching production process of a target object, wherein the first data includes first action data of a user on the target object and target attribute data, wherein the target attribute data indicates at least one of a material attribute of the target object and a material attribute of a tool that the user acts on the target object; A first determination module, configured to determine sensory data generated when a user interacts with the target object based on the first action data; A first generating module, configured to generate, based on the target attribute data and the sensory data, a first sensory feedback signal corresponding to a material attribute indicated by the target attribute data; The first sensory feedback module is used to perform sensory feedback based on the first sensory feedback signal.
9. An electronic device, characterized in that: include: A memory, a processor, and a program stored in the memory and executable on the processor, wherein when the program is executed by the processor, the steps in the VR teaching processing method according to any one of claims 1 to 7 are implemented.
10. A readable storage medium, characterized in that: The readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in the VR teaching processing method according to any one of claims 1 to 7 are implemented.
11. A computer program product, characterized in that The method comprises computer instructions, which, when executed by a processor, implement the steps in the VR teaching processing method according to any one of claims 1 to 7.
