Multi-sensory simulation method for virtual reality and related device
Through virtual reality devices, the simulated physical location and sensory data of the target object are determined, and the actuator of the robot operates the device for multi-sensory simulation is solved, which solves the problem of the clumsy traditional equipment affecting the experience, realizes wear-free and efficient multi-sensory interaction, and enhances users Experience and versatility.
Patent Information
- Application Number
- CN202411392178.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-30
- Publication Date
- 2024-12-27
AI Technical Summary
Traditional virtual reality multi-sensory interaction devices require users to wear bulky devices, affecting user experience and activity convenience, and lack universality and standardization, resulting in poor hardware compatibility between different platforms.
Through virtual reality devices, the simulated physical position of the target object is determined based on the virtual relative position relationship and physical position between the user and the target object, and multi-sensory simulation is performed through the actuator of the robot operating the device, providing sensory data such as touch and smell to avoid users wearing it Hardware to improve user experience and spatial experience.
Multi-sensory simulation without user wear is realized, user experience and spatial feeling is improved, and actuators can be replaced to provide different sensory effects, improving the versatility and user participation of multi-sensory simulation.
Smart Images
Figure CN119200854A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of data processing, and in particular to a multi-sensory simulation method for virtual reality and related devices. Background Art
[0002] Virtual reality (VR) is a technology that uses computer technology to simulate the environment and create a new, virtual, three-dimensional interactive environment, making users feel as if they are in the real world. VR technology allows users to immerse themselves in a three-dimensional environment generated by a computer through specific equipment (such as VR helmets, handles, etc.), interact with the virtual world, and thus gain a new experience. Traditional VR multi-sensory technology mainly focuses on the presentation of vision and hearing, while multi-sensory interaction creates a more realistic virtual environment by introducing more sensory stimulation, thereby increasing the user's sense of immersion and participation.
[0003] Current multi-sensory interactions simulate touch, smell, etc. for users by installing sensory simulation devices on the user's head, hands or other parts of the body.
[0004] However, these devices with integrated tactile feedback or olfactory rendering functions need to be worn on the user, and the user cannot move easily due to the bulky devices, which affects the user experience. Summary of the invention
[0005] The embodiments of the present application provide a multi-sensory simulation method for virtual reality and related devices, which are used to simulate the physical position of a target object in a real scene through an actuator of a robot operating device, and perform multiple sensory data simulations through the actuator. Various sensory simulation modules are configured by the actuator, and the user does not need to wear them, thereby improving the user experience. The actuator can adjust the physical position based on the virtual relative position relationship and then perform multi-sensory simulation, which can improve the user's spatial perception. The actuator is replaceable and can provide different sensory simulation effects, further improving the user experience.
[0006] In view of this, the present application provides, on one hand, a multi-sensory simulation method for virtual reality, comprising:
[0007] Generate virtual reality scenes;
[0008] Determine the virtual relative position relationship between the user and the target object in the virtual reality scene;
[0009] Determine the simulated physical position of the target object based on the virtual relative position relationship and the user's physical position and physical orientation;
[0010] Determining interactive sensory data according to a virtual state of a target object in a virtual reality scene;
[0011] Sending instructions to the robot operating device, the instructions including the simulated physical position and the interactive sensory data, so that the robot operating device controls the replaceable actuator to move to the simulated physical position, and performs multi-sensory simulation according to the interactive sensory data through the actuator.
[0012] In some embodiments, determining the virtual relative position relationship between the user and the target object in the virtual reality scene includes:
[0013] Obtaining the user's virtual interaction operation on the target object in the virtual reality scene;
[0014] The virtual relative position relationship between the user and the target object is determined according to the virtual interaction operation.
[0015] In some embodiments, determining the interactive sensory data according to the virtual state of the target object in the virtual reality scene includes:
[0016] Determine the user's virtual sense based on the target object according to the virtual state and virtual interactive operation of the target object in the virtual reality scene;
[0017] Generate interactive sensory data based on virtual senses.
[0018] In some embodiments, sending instructions to the robotic manipulation device includes:
[0019] Send instructions to the robot operating device through rosbridge.
[0020] In some embodiments, the virtual sense includes at least one of touch and smell.
[0021] On the other hand, the present application provides a multi-sensory simulation method for virtual reality, comprising:
[0022] Receiving an instruction from a virtual reality device, the instruction including a simulated physical position and interactive sensory data, the simulated physical position being a physical position of a target object simulated based on a virtual relative position relationship between a user and the target object in a virtual reality scene of the virtual reality device, and the interactive sensory data being determined based on a virtual state of the target object in the virtual reality scene;
[0023] The replaceable actuators are controlled to move to simulated physical locations according to instructions, and multi-sensory simulation is performed through the actuators according to interactive sensory data.
[0024] In some embodiments, receiving the instruction from the virtual reality device includes:
[0025] Receive instructions from the VR device via rosbridge.
[0026] In some embodiments, the interactive sensory data includes at least one of tactile interaction data and olfactory interaction data.
[0027] On the other hand, the present application provides a virtual reality multi-sensory simulation device, comprising:
[0028] A processing unit, configured to generate a virtual reality scene; determine a virtual relative position relationship between a user and a target object in the virtual reality scene; determine a simulated physical position of the target object based on the virtual relative position relationship and the user's physical position and physical orientation; and determine interactive sensory data based on a virtual state of the target object in the virtual reality scene;
[0029] The transceiver unit is used to send instructions to the robot operating device, where the instructions include simulated physical positions and interactive sensory data, so that the robot operating device controls the replaceable actuator to move to the simulated physical position and performs multi-sensory simulation according to the interactive sensory data through the actuator.
[0030] On the other hand, the present application provides a virtual reality multi-sensory simulation device, comprising:
[0031] a transceiver unit, configured to receive an instruction from a virtual reality device, the instruction including a simulated physical position and interactive sensory data, the simulated physical position being a physical position of a target object simulated based on a virtual relative position relationship between a user and the target object in a virtual reality scene of the virtual reality device, and the interactive sensory data being determined based on a virtual state of the target object in the virtual reality scene;
[0032] The processing unit is used to control the replaceable actuators to move to the simulated physical positions according to the instructions, and perform multi-sensory simulation according to the interactive sensory data through the actuators.
[0033] In one possible design, in another implementation of another aspect of the embodiment of the present application,
[0034] Another aspect of the present application provides a computer device, comprising:
[0035] memories, transceivers, processors, and bus systems;
[0036] Wherein, the memory is used to store programs;
[0037] The processor is used to execute the program in the memory, including executing the above-mentioned methods;
[0038] The bus system is used to connect the memory and the processor so that the memory and the processor can communicate with each other.
[0039] Another aspect of the present application provides a computer-readable storage medium, in which instructions are stored. When the computer-readable storage medium is run on a computer, the computer is enabled to execute the above-mentioned methods.
[0040] Another aspect of the present application provides a computer program product or a computer program, the computer program product or the computer program includes computer instructions, the computer instructions are stored in a computer-readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the methods provided by the above aspects.
[0041] It can be seen from the above technical solutions that the embodiments of the present application have the following advantages:
[0042] In the embodiment of the present application, the virtual reality device determines the simulated physical position of the target object in the real scene based on the virtual relative position relationship between the user and the target object in the virtual reality scene, combined with the user's physical position and physical orientation, and then determines the interactive sensory data relative to the user based on the virtual state of the target object, and instructs the robot operating device to move the simulated physical position and the interactive sensory data through instructions, and the robot operating device controls the actuator to move to the simulated physical position, and controls the actuator to perform multi-sensory simulation according to the interactive sensory data. In the above manner, the physical position of the target object in the real scene is simulated by the actuator of the robot operating device, and multiple sensory data simulations are performed by the actuator. Various sensory simulation modules are configured by the actuator, which does not need to be worn by the user, thereby improving the user experience, and the actuator can adjust the physical position based on the virtual relative position relationship and then perform multi-sensory simulation, which can improve the user's spatial perception, and the actuator can be replaced, which can provide different sensory simulation effects, further improving the user experience. BRIEF DESCRIPTION OF THE DRAWINGS
[0043] Figure 1 It is a schematic diagram of the architecture of the multi-sensory simulation system in an embodiment of the present application;
[0044] Figure 2 A schematic diagram of a multi-sensory simulation method for virtual reality in an embodiment of the present application;
[0045] Figure 3 A schematic diagram of the interaction between a virtual reality device and a robot operating device in an embodiment of the present application;
[0046] Figure 4 A schematic diagram of virtual reality firefighting training in an embodiment of the present application;
[0047] Figure 5 A schematic diagram of a fire in an embodiment of the present application;
[0048] Figure 6 A schematic diagram of a mechanical arm in an embodiment of the present application;
[0049] Figure 7 A schematic diagram of a mechanical arm in an embodiment of the present application;
[0050] Figure 8 This is a schematic diagram of an example that can be simulated by the robotic arm in the embodiment of the present application;
[0051] Fig. 9 This is a structural schematic diagram of a virtual reality multi-sensory simulation device in an embodiment of the present application;
[0052] Fig.10 Schematic diagram of the structure of another virtual reality multi-sensory simulation device in an embodiment of the present application;
[0053] Fig.11 It is a schematic diagram of the structure of the computer device in the embodiment of the present application. DETAILED DESCRIPTION
[0054] The embodiments of the present application provide a multi-sensory simulation method for virtual reality and related devices, which are used to simulate the physical position of a target object in a real scene through an actuator of a robot operating device, and perform multiple sensory data simulations through the actuator. Various sensory simulation modules are configured by the actuator, and the user does not need to wear them, thereby improving the user experience. The actuator can adjust the physical position based on the virtual relative position relationship and then perform multi-sensory simulation, which can improve the user's spatial perception. The actuator is replaceable and can provide different sensory simulation effects, further improving the user experience.
[0055] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.
[0056] The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein, for example. In addition, the terms "including" and "corresponding to" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.
[0057] The word “exemplary” is used exclusively herein to mean “serving as an example, example, or illustration.” Any embodiment described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other embodiments.
[0058] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program with a predetermined function, and works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuits or memories) or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be part of an overall module or unit that includes the function of the module or unit.
[0059] In addition, in order to better illustrate the present application, numerous specific details are given in the following specific embodiments. It should be understood by those skilled in the art that the present application can also be implemented without certain specific details. In some examples, methods, means, components and circuits well known to those skilled in the art are not described in detail in order to highlight the subject matter of the present application.
[0060] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.
[0061] Multi-sensory interaction of virtual reality (VR) technology involves enhancing the user's immersive experience through multiple senses such as vision, hearing, touch, smell and taste. Traditional VR technology mainly focuses on the presentation of vision and hearing, while multi-sensory interaction creates a more realistic virtual environment by introducing more sensory stimulation, thereby increasing the user's immersion and participation. Current multi-sensory interaction simulates touch, smell, etc. for users by installing sensory simulation devices on the user's head, hands or other parts of the body. However, these multi-sensory interaction devices lack universality and standardization. Different VR applications and platforms may require different hardware support, resulting in a lack of compatibility between devices of different brands and models. In addition, the devices need to be worn by users, resulting in a poor user experience.
[0062] The embodiment of the present application provides a multi-sensory simulation method of virtual reality, which determines the simulated physical position of the target object in the real scene by using a virtual reality device based on the virtual relative position relationship between the user and the target object in the virtual reality scene, combined with the user's physical position and physical orientation, and then determines the interactive sensory data relative to the user based on the virtual state of the target object, and instructs the robot operating device to use the simulated physical position and the interactive sensory data, and the robot operating device controls the actuator to move to the simulated physical position, and controls the actuator to perform multi-sensory simulation according to the interactive sensory data. In the above manner, the physical position of the target object in the real scene is simulated by the actuator of the robot operating device, and multiple sensory data simulations are performed by the actuator, and various sensory simulation modules are configured by the actuator, and the sensory simulation is performed by the robot operating device. It only needs to send instructions to the robot operating device through network communication, and no special hardware support is required, which improves the versatility of multi-sensory simulation. And it does not need to be worn by the user, which improves the user experience, and the actuator can adjust the physical position based on the virtual relative position relationship and then perform multi-sensory simulation, which can improve the user's spatial perception, and the actuator can be replaced, which can provide different sensory simulation effects, further improving the user experience.
[0063] See also Figure 1 , Figure 1 The virtual reality device 11 is connected to the robot operation device 13 via the network 12, and the network 12 can be a wide area network or a local area network, or a combination of the two.
[0064] In some embodiments, the virtual reality device 11 provides a virtual reality scene for the user, and then analyzes the simulated physical position corresponding to the target object based on the user's physical position and physical orientation in the real scene based on the virtual relative position relationship between the target object and the user in the virtual reality scene, and then analyzes the impact of the virtual state of the target object in the virtual reality scene on the user to determine the interactive sensory data. The virtual reality device 11 sends the instruction including the simulated physical position and the interactive sensory data to the robot operating device 13 through the network 12, and the robot operating device 13 moves the actuator to the simulated physical position and controls the actuator to perform multi-sensory simulation based on the interactive sensory data.
[0065] The following is a description of a multi-sensory simulation method for virtual reality provided in an embodiment of the present application. The method is as follows:
[0066] See also Figure 2 ,like Figure 2 The figure is a flow chart of a multi-sensory simulation method of virtual reality provided by an embodiment of the present application, including:
[0067] Step 201: The virtual reality device generates a virtual reality scene.
[0068] In one or more possible implementations, virtual reality devices are key tools for generating and displaying virtual reality scenes, mainly including head-mounted displays (such as VR helmets), handles, joysticks, somatosensory devices, etc. These devices enable users to immerse themselves in a digital space that feels like the real world by blocking the user's visual and auditory input and replacing it with computer-generated stereoscopic images and 3D sound effects. The user wears a VR helmet, and the display screen inside the helmet presents the rendered virtual scene to the user in the form of a stereoscopic image. Sensors such as gyroscopes sense the user's head rotation and position changes, and update the picture content in real time to maintain the continuity and immersion of the scene.
[0069] Step 202: The virtual reality device determines the virtual relative position relationship between the user and the target object in the virtual reality scene.
[0070] In one or more possible implementations, the virtual reality scene is constructed in the virtual space through three-dimensional modeling technology, including the creation and layout of elements such as terrain, buildings, and objects. These elements have their own position coordinates in the virtual space. Through the data of internal sensors and external tracking systems, the system can calculate the position coordinates of the user in the virtual space in real time. Each target object in the virtual scene (such as game characters, items, etc.) also has its fixed position coordinates, which are determined when the scene is constructed and stored in the system's database. By comparing the position coordinates of the user and the target object, the system can calculate the relative position relationship between them, including parameters such as distance, direction, and angle.
[0071] Optionally, in the above Figure 2 On the basis of the corresponding embodiments, in another optional embodiment provided by the embodiments of the present application, determining the virtual relative position relationship between the user and the target object in the virtual reality scene includes: obtaining the virtual interaction operation of the user on the target object in the virtual reality scene; and determining the virtual relative position relationship between the user and the target object based on the virtual interaction operation.
[0072] In one or more possible implementations, the user's virtual interaction operation on the target object can affect the user's position in the virtual reality scene, and the virtual relative position relationship between the user and the target object can be adjusted based on the change in position. Exemplarily, the user can control the position in the virtual reality scene through the buttons on the handle, such as controlling forward movement, backward movement, left movement, right movement, etc., or the virtual reality device can achieve this by identifying the position change of the user in the real scene, which is not limited here. The virtual relative position relationship between the user and the target object is updated in real time to facilitate more accurate position simulation and improve user experience.
[0073] Step 203: The virtual reality device determines the simulated physical position of the target object based on the virtual relative position relationship and the user's physical position and physical orientation.
[0074] In one or more possible implementations, the built-in sensors such as gyroscopes and accelerometers of the VR device can sense the user's head and body movements in real time, thereby determining the user's physical position and physical orientation in the physical space. Based on the relative position of the user and the target object in the virtual reality scene indicated by the virtual relative position relationship, the target object is mapped to the real scene in a simulated manner to obtain the simulated physical position of the target object in the real scene, that is, the distance, direction, and angle between the simulated physical position and the user in the real scene are the same as the distance, direction, and angle between the target object and the user in the virtual reality scene.
[0075] Step 204: The virtual reality device determines interactive sensory data according to the virtual state of the target object in the virtual reality scene.
[0076] In one or more possible implementations, the virtual reality device can sense and identify the virtual state of a target object in a virtual reality scene, and the virtual state may include the position, posture, motion trajectory, physical properties (such as mass, hardness, elasticity, temperature, etc.) and possible interaction rules of the target object. The virtual reality device can analyze the possible sensory effects of the user on the target object based on the virtual state of the target object, and convert various sensory effects into data that can be simulated, and the data that can be simulated can be called interactive sensory data.
[0077] Optionally, in the above Figure 2 On the basis of the corresponding embodiments, in another optional embodiment provided by the embodiments of the present application, determining interactive sensory data according to the virtual state of the target object in the virtual reality scene includes: determining the user's virtual sense based on the target object according to the virtual state and virtual interactive operation of the target object in the virtual reality scene; and generating interactive sensory data based on the virtual sense.
[0078] In one or more possible implementations, the virtual state is the state of the target object in the virtual reality scene, such as size, shape, color, material, motion state (stationary, moving, rotating, etc.) and possible other attributes (such as temperature, etc.). Virtual interactive operations define how users interact with these target objects, including touching, grabbing, pushing, approaching, etc. The virtual reality device can analyze the interaction results between the user and the target object based on the virtual interactive operation, and determine the virtual senses that need to be fed back to the user based on the virtual state of the target object, and then generate corresponding interactive sensory data based on the required feedback effect. Exemplarily, taking the target object as fire as an example, when the virtual interactive operation is approaching, the virtual sense is heat, and the interactive sensory data is the intensity of thermal radiation required to be provided by the robot operating device. When the target object is an ordinary object, when the virtual interactive operation is touching, the virtual sense is touch, and the interactive sensory data is the material, shape, size and other data of the target object.
[0079] Step 205. The virtual reality device sends instructions to the robot operating device, where the instructions include simulated physical position and interactive sensory data.
[0080] In one or more possible implementations, the virtual reality device may send instructions including simulated physical locations and interactive sensory data to the robot operating device via a specific communication protocol and communication interface (such as Wi-Fi, Bluetooth, Ethernet, etc.).
[0081] Optionally, in the above Figure 2 On the basis of the corresponding embodiments, in another optional embodiment provided by the embodiments of the present application, sending instructions to the robot operating device includes: sending instructions to the robot operating device through rosbridge.
[0082] In one or more possible implementations, rosbridge is a bridge connecting a robot operating system (ROS) system with other systems. The robot operating device runs the ROS system. Rosbridge enables non-ROS programs to easily communicate with ROS by providing a JSON API. Rosbridge supports multiple communication protocols (such as transmission control protocol (TCP) / user datagram protocol (UDP) / WebSocket), allowing non-ROS systems to exchange data with the ROS system through these protocols, including topic subscription, message publishing, service call, and parameter setting. Exemplarily, a WebSocket server can be used as the interface of rosbridge, allowing Web browsers to interact, and data is transmitted through a high-bandwidth, low-latency WiFi 6 connection.
[0083] For example, the interaction between the virtual reality device and the robot operation device can refer to Figure 3 As shown, the virtual reality device can run Unity, which is a popular cross-platform game engine widely used to create two-dimensional and three-dimensional video games, simulation training, visualization, etc. The robot operation device runs the ROS system, and rosbridge acts as a bridge between Unity and the ROS system. Among them, rosbridge can be divided into a ROS bridge client (RosBridgeClient) and a ROS bridge server (RosBridgeServer). Through the WebSocket protocol (here specifically the ROS bridge network interface RosBridgeWebSocket, running on the ROS bridge server), the ROS bridge server can communicate with the RosBridgeClient in real time. The ROS bridge client runs in Unity, and the task parameters processed include: Action Name represents the task name, the Fibonacci order FibonacciOrder specifies the order of the Fibonacci sequence to be calculated (Fibonacci calculation is only an example), and the Action Name and FibonacciOrder are notified to the ROS bridge server through the Fibonacci execution client FibonacciActionClient. The ROS bridge server runs in the ROS system, and receives the Action Name and FibonacciOrder from the FibonacciActionClient through the Fibonacci execution server FibonacciActionServer. The parameter Status indicates the task status (such as running, completed, failed, etc.), Feedback provides feedback information during the task execution, and Result is the final calculation result. Among them, ESP8266 can be used as the underlying hardware support of FibonacciActionServer, which may be used to perform specific computing tasks or control external devices. ROS can be run on the Ubuntu operating system.
[0084] Optionally, in the above Figure 2 On the basis of the corresponding embodiments, in another optional embodiment provided by the embodiment of the present application, the virtual sense includes at least one of touch and smell.
[0085] In one or more possible implementations, based on one or more of the possible senses of touch or smell of the virtual senses, the interactive sensory data is data of one or more of the corresponding senses of touch or smell. For example, assuming that the target object is a fire, the senses provided to the user by the target object may include touch and smell in addition to the visual and auditory effects that can be directly provided by the virtual reality device. The touch is the sensation of heat that the user can perceive based on the fire, and the smell is the smoke effect caused by the fire.
[0086] Step 206. The robot operating device controls the replaceable actuator to move to the simulated physical position according to the instruction, and performs multi-sensory simulation according to the interactive sensory data through the actuator.
[0087] In one or more possible implementations, the robot operating device is usually equipped with a position sensor (such as an encoder, a gyroscope, etc.) for detecting the position information of the robot operating device and its actuator in real time. According to the simulated physical position in the instruction, the robot operating device will perform path planning and calculate the optimal path from the current position to the simulated physical position. By controlling the actuator's drive system (such as a motor, hydraulic or pneumatic device), the actuator is moved to the simulated physical position along the planned path. After the actuator reaches the simulated physical position, the robot operating device can control the sensory simulation structure on the actuator to simulate according to the interactive sensory data, and simulate multiple sensory effects based on multiple sensory data. Among them, the actuator is a replaceable structure, which can be replaced by the robot operating device according to the sensory effect of the interactive sensory data, or it can be replaced manually.
[0088] Exemplarily, taking a virtual reality fire scene as an example, the user interacts with the olfactory and thermal actuators integrated on the end effector of the robot operating device, which is the most basic application scenario of olfactory and tactile thermal radiation. The user wearing the headset can only see the fire source and smoke, but not the robot itself. The virtual reality device maps the location of the fire in the virtual reality scene to the real scene, and provides the mapped simulated physical location and the interactive sensory data that the user can perceive to the robot operating device. The robot operating device can move the actuator to place the device with the tactile column precisely at the interactive position corresponding to the simulated physical location, and simulate the thermal radiation and smoke effects according to the interactive sensory data through the actuator, so that the user can interact with the virtual smoke and fire source through smell and touch. Among them, the actuator can also be adjusted according to the sensory effect, such as reducing the simulated output while shortening the interaction distance to achieve the same sensory effect.
[0089] See also Figure 4 Schematic diagram of virtual reality firefighting training shown, the user wears virtual reality equipment, such as Figure 4In the virtual reality helmet, the user controls the interaction with the virtual reality scene through the handle. The virtual reality scene is a fire scene with fire and smoke. The virtual reality device can map the fire source to the physical position in the physical scene based on the position relationship in the virtual reality scene, and then analyze the sensory impact of the fire source and smoke on the user as sensory data, and send the sensory data and physical position to the robot operating device. The robot operating device controls the actuator to move to the corresponding physical position, and simulates the smoke phenomenon through the smoke generator of the actuator, and simulates the fire source phenomenon through the heater, so as to realize the physical interaction with the virtual fire source. For scenes with only fire, you can refer to Figure 5 As shown in the fire diagram, there is only a fire source in the virtual reality scene. Due to the thermal radiation of the fire source, the heater of the actuator in the real scene is a range type, which can simulate the face-blowing effect of thermal radiation.
[0090] In one example, the actuator may be in the form of a robotic arm, see Figure 6 The schematic diagram of the robotic arm shown in the figure shows that the robotic arm is integrated with a variety of sensors and actuators. These components include an electric heating fan, which may be used to provide heat or perform thermal management, an atomization module, which is used to output smoke, a control module, which is the "brain" of the robotic arm and is responsible for processing data and sending instructions, a power supply; and In addition, a 3D printed base is installed under the robotic arm. The picture on the right shows some electronic components and circuit boards of the robotic arm, including an electronic component housing containing a fan, a microcontroller board with a license, which is the control center of the robot system, responsible for executing programs, processing data and communicating with peripherals such as sensors and actuators; a CP2008 model + 4x5 mm LED light, which is used to provide indication of operation; a planar electronic ceramic capacitor to protect other electronic components from voltage spikes or sudden changes; a mode selector or driver circuit board, which selects or switches different working modes according to the needs of the device, or drives other components to perform specific tasks; a power connector + cable, which provides power connection to ensure that the robotic arm can obtain enough power to work normally.
[0091] In another example, see Figure 7 The schematic diagram of the robotic arm shown in the figure shows that the robotic arm can be divided into three main parts: head, arm and wrist. Each part is equipped with a varying number of knobs for precise control. The head has four knobs, showing strong support for complex operations; the two knobs on the arm may be used to adjust the arm length or strength; and the knob on the wrist focuses on fine-tuning fine movements. The robotic arm can change the positional relationship with the user by adjusting the arm length. The end effector of the robotic arm can be loaded with interactive tools and has a replaceable function, and the interactive tools can be automatically replaced through the interactive toolkit.
[0092] Examples of robot arm simulations can be found here Figure 8 As shown, Figure 8 The comparison between the real interaction and the virtual experience is shown. For example, in the real interaction, the end force simulation of the robotic arm, that is, the end of the robotic arm is equipped with a door handle and moves to the corresponding position for the user to grasp, while in the virtual experience scene, the user sees the picture of opening the door through the door handle; the end heat simulation of the robotic arm, that is, the end of the robotic arm is equipped with a heater and moves to the corresponding position for heat radiation, while in the virtual experience scene, the user sees the picture close to the fire source; the end smell simulation of the robotic arm, that is, the end of the robotic arm is equipped with fragrant oil containing floral fragrance, while in the virtual experience scene, the user sees a bouquet of flowers emitting floral fragrance.
[0093] In the embodiment of the present application, the virtual reality device determines the simulated physical position of the target object in the real scene based on the virtual relative position relationship between the user and the target object in the virtual reality scene, combined with the user's physical position and physical orientation, and then determines the interactive sensory data relative to the user based on the virtual state of the target object, and instructs the robot operation device with the simulated physical position and interactive sensory data through instructions, and the robot operation device controls the actuator to move to the simulated physical position, and controls the actuator to perform multi-sensory simulation according to the interactive sensory data. In the above manner, the actuator of the robot operation device simulates the physical position of the target object in the real scene, and performs multiple sensory data simulations through the actuator, and the actuator is configured with various sensory simulation modules, which does not require the user to wear, improves the user experience, and the actuator can adjust the physical position based on the virtual relative position relationship and then perform multi-sensory simulation, which can improve the user's spatial perception, and the actuator is replaceable, which can provide different sensory simulation effects, further improving the user experience. The actuator of the robot operation device simulates the physical position of the target object in the real scene, and performs multiple sensory data simulations through the actuator, and the actuator is configured with various sensory simulation modules, and the robot operation device performs sensory simulation, and only needs to send instructions to the robot operation device through network communication, and does not require special hardware support, which improves the versatility of multi-sensory simulation.
[0094] The multi-sensory simulation method of virtual reality is described above, and the device for executing the method is described below.
[0095] See also Fig. 9 The structure diagram of a virtual reality multi-sensory simulation device is shown, and the device 90 includes:
[0096] Processing unit 901 is used to generate a virtual reality scene; determine the virtual relative position relationship between the user and the target object in the virtual reality scene; determine the simulated physical position of the target object according to the virtual relative position relationship and the user's physical position and physical orientation; determine interactive sensory data according to the virtual state of the target object in the virtual reality scene;
[0097] The transceiver unit 902 is used to send instructions to the robot operating device, where the instructions include simulated physical positions and interactive sensory data, so that the robot operating device controls the replaceable actuator to move to the simulated physical position and performs multi-sensory simulation according to the interactive sensory data through the actuator.
[0098] In some other embodiments of the present application, the processing unit 901 is specifically used for:
[0099] Obtaining the user's virtual interaction operation on the target object in the virtual reality scene;
[0100] The virtual relative position relationship between the user and the target object is determined according to the virtual interaction operation.
[0101] In some other embodiments of the present application, the processing unit 901 is specifically used for:
[0102] Determine the user's virtual sense based on the target object according to the virtual state and virtual interactive operation of the target object in the virtual reality scene;
[0103] Generate interactive sensory data based on virtual senses.
[0104] In some other embodiments of the present application, the transceiver unit 902 is specifically used for:
[0105] Send instructions to the robot operating device through rosbridge.
[0106] In other embodiments of the present application, the virtual sense includes at least one of touch and smell.
[0107] See also Fig.10 The structure diagram of another virtual reality multi-sensory simulation device is shown, and the device 100 includes:
[0108] The transceiver unit 1001 is used to receive an instruction from a virtual reality device, where the instruction includes a simulated physical position and interactive sensory data, where the simulated physical position is a physical position of a target object simulated based on a virtual relative position relationship between a user and the target object in a virtual reality scene of the virtual reality device, and the interactive sensory data is determined based on a virtual state of the target object in the virtual reality scene;
[0109] The processing unit 1002 is used to control the replaceable actuator to move to the simulated physical position according to the instruction, and perform multi-sensory simulation according to the interactive sensory data through the actuator.
[0110] In some embodiments, the transceiver unit 1001 is specifically used for:
[0111] Receive instructions from the VR device via rosbridge.
[0112] In some embodiments, the interactive sensory data includes at least one of tactile interaction data and olfactory interaction data.
[0113] Fig.11 3 is a schematic diagram of the structure of a computer device provided in an embodiment of the present application. The computer device 300 may have relatively large differences due to different configurations or performances, and may include one or more central processing units (CPU) 322 (for example, one or more processors) and memory 332, and one or more storage media 330 (for example, one or more mass storage devices) storing application programs 342 or data 344. Among them, the memory 332 and the storage medium 330 can be short-term storage or permanent storage. The program stored in the storage medium 330 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations in the computer device. Furthermore, the central processing unit 322 can be configured to communicate with the storage medium 330 to execute a series of instruction operations in the storage medium 330 on the computer device 300.
[0114] The computer device 300 may also include one or more power supplies 326, one or more wired or wireless network interfaces 350, one or more input and output interfaces 358, and / or one or more operating systems 341, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, etc.
[0115] The steps performed by the virtual reality device or the robot operation device in the above embodiments can be based on the Fig.11 The computer device structure shown.
[0116] A computer-readable storage medium is also provided in an embodiment of the present application, on which a computer program is stored. When the computer program is executed by a processor, the steps of the methods described in the above embodiments are implemented.
[0117] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.
[0118] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.
[0119] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0120] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.
[0121] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk and other media that can store program code.
[0122] As described above, the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.
Claims
1. A multi-sensory simulation method for virtual reality, characterized in that: include: Generate virtual reality scenes; Determining a virtual relative position relationship between a user and a target object in the virtual reality scene; Determining a simulated physical position of the target object according to the virtual relative position relationship and the physical position and physical orientation of the user; Determining interactive sensory data according to the virtual state of the target object in the virtual reality scene; An instruction is sent to a robot operating device, wherein the instruction includes the simulated physical position and the interactive sensory data, so that the robot operating device controls a replaceable actuator to move to the simulated physical position, and performs multi-sensory simulation according to the interactive sensory data through the actuator.
2. The method according to claim 1, characterized in that Determining the virtual relative position relationship between the user and the target object in the virtual reality scene includes: Acquire a virtual interaction operation of the user on the target object in the virtual reality scene; The virtual relative position relationship between the user and the target object is determined according to the virtual interaction operation.
3. The method according to claim 2, characterized in that Determining the interactive sensory data according to the virtual state of the target object in the virtual reality scene includes: Determining a virtual sense of the user based on the target object according to a virtual state of the target object in the virtual reality scene and the virtual interaction operation; The interactive sensory data is generated according to the virtual sense.
4. The method according to claim 1, characterized in that: The sending of instructions to the robot operating device comprises: Send instructions to the robot operating device via rosbridge.
5. The method according to claim 3, characterized in that: The virtual sense includes at least one of touch and smell.
6. A multi-sensory simulation method for virtual reality, characterized in that: include: Receiving an instruction from a virtual reality device, the instruction including a simulated physical position and interactive sensory data, the simulated physical position being a physical position of a target object simulated based on a virtual relative position relationship between a user and the target object in a virtual reality scene of the virtual reality device, and the interactive sensory data being determined according to a virtual state of the target object in the virtual reality scene; The replaceable actuator is controlled to move to the simulated physical position according to the instruction, and multi-sensory simulation is performed according to the interactive sensory data through the actuator.
7. The method according to claim 6, characterized in that The receiving of the instruction from the virtual reality device comprises: The command from the virtual reality device is received via rosbridge.
8. The method according to claim 6, characterized in that The interactive sensory data includes at least one of tactile interaction data and olfactory interaction data.
9. A collaborative interaction system, characterized in that: Including virtual reality equipment and robotic operation equipment; The virtual reality device is used to generate a virtual reality scene; determine a virtual relative position relationship between a user and a target object in the virtual reality scene; determine a simulated physical position of the target object according to the virtual relative position relationship and the physical position and physical orientation of the user; determine interactive sensory data according to a virtual state of the target object in the virtual reality scene; and send an instruction to a robot operating device, the instruction including the simulated physical position and the interactive sensory data; The robot operating device is used to control the replaceable actuator to move to the simulated physical position according to the instruction, and perform multi-sensory simulation according to the interactive sensory data through the actuator.
10. A virtual reality multi-sensory simulation device, characterized in that: include: A processing unit, for generating a virtual reality scene; Determining a virtual relative position relationship between a user and a target object in the virtual reality scene; Determine the simulated physical position of the target object according to the virtual relative position relationship and the physical position and physical orientation of the user; determine the interactive sensory data according to the virtual state of the target object in the virtual reality scene; The transceiver unit is used to send instructions to the robot operating device, wherein the instructions include the simulated physical position and the interactive sensory data, so that the robot operating device controls the actuator to move to the simulated physical position and performs multi-sensory simulation according to the interactive sensory data through the actuator.
11. A virtual reality multi-sensory simulation device, characterized in that: include: a transceiver unit, configured to receive an instruction from a virtual reality device, wherein the instruction includes a simulated physical position and interactive sensory data, wherein the simulated physical position is a physical position of a target object simulated based on a virtual relative position relationship between a user and the target object in a virtual reality scene of the virtual reality device, and the interactive sensory data is determined according to a virtual state of the target object in the virtual reality scene; The processing unit is used to control the replaceable actuator to move to the simulated physical position according to the instruction, and perform multi-sensory simulation according to the interactive sensory data through the actuator.
12. A computer device, characterized in that: include: memories, transceivers, processors, and bus systems; Wherein, the memory is used to store programs; The processor is used to execute the program in the memory, including executing the method according to any one of claims 1 to 8; The bus system is used to connect the memory and the processor so that the memory and the processor can communicate with each other.
13. A computer-readable storage medium comprising instructions, which, when executed on a computer, causes the computer to perform the method according to any one of claims 1 to 8.