A VR-based high-presence visual perception method and related device

Abstract

The application discloses a VR-based high-immersive visual perception method and related device, and the method comprises the following steps: acquiring depth target information of a target observation object collected by a depth camera and space pose information of a VR head-mounted device, wherein the depth target information comprises a point cloud model and an RGB image, and the space pose information comprises space position and attitude information; performing virtual-real coordinate mapping based on the depth camera, the VR head-mounted device and a three-degree-of-freedom holder to obtain an inverse kinematics transformation model; adjusting the observation angle of the depth camera according to the space pose information through the inverse kinematics transformation model to obtain an angle change amount; and performing proportional scaling on the RGB image according to the angle change amount and the virtual distance between the depth camera and the point cloud model to realize near-far perception of the target observation object. The application can solve the technical problem that the existing VR high-immersive visual perception method based on a mechanical arm is restricted by the motion mechanism of the mechanical arm, it is difficult to realize high-quality real-time perception, and dizziness is caused.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of VR visual perception, and particularly relates to a high-immersion visual perception method based on VR and a related device. BACKGROUND

[0002] Currently, the combination of VR and electric power robots mainly has two applications: one is to use a VR head-mounted device as a video observation screen to observe video data returned by a camera carried on a remote platform (such as a drone or an unmanned vehicle); the other is to combine a VR head-mounted device with a work robot visual perception system, and combine the pose of the VR head-mounted device with a small 6-DOF (degree of freedom) mechanical arm carrying a camera at the end, so that the user controls the movement of the mechanical arm end with the VR head-mounted device, and the user observes the visible light image of the direction pointed by the mechanical arm end through the camera carried at the end of the mechanical arm.

[0003] The existing VR head-mounted device based on a mechanical arm is not only limited to a small mechanical effective working space, but also limited by the dynamics performance of the mechanical arm, and it is difficult to realize high-speed tracking of the head movement by the mechanical arm, resulting in obvious delay and dizziness. In addition, the multi-DOF mechanical arm is expensive, and the overall practical value is low. SUMMARY

[0004] The present application provides a high-immersion visual perception method based on VR and a related device, which is used to solve the technical problem that the existing VR high-immersion visual perception method based on a mechanical arm is limited by the movement mechanism of the mechanical wall, and it is difficult to realize high-quality real-time perception, thereby causing dizziness.

[0005] Therefore, the first aspect of the present application provides a high-immersion visual perception method based on VR, comprising:

[0006] obtaining depth target information of a target observation object collected by a depth camera and spatial pose information of a VR head-mounted device, the depth target information comprising a point cloud model and an RGB image, and the spatial pose information comprising spatial position and attitude information;

[0007] performing virtual-real coordinate mapping based on the depth camera, the VR head-mounted device and a 3-DOF (degree of freedom) holder to obtain an inverse kinematics transformation model;

[0008] adjusting the observation angle of the depth camera according to the spatial pose information through the inverse kinematics transformation model to obtain an angle change amount;

[0009] scaling the RGB image according to the angle change amount and the virtual distance between the depth camera and the point cloud model to realize the near-far perception of the target observation object.

[0010] Preferably, the virtual-real coordinate mapping based on the depth camera, the VR headset and the three-degree-of-freedom holder obtains an inverse kinematics transformation model, and specifically includes:

[0011] The virtual-real coordinate mapping based on the virtual observation point position of the depth camera, the spatial pose information of the VR headset and the holder attitude information of the three-degree-of-freedom holder obtains an inverse kinematics transformation model.

[0012] Preferably, the scaling ratio is:

[0013]

[0014] Wherein, l is the distance between the depth camera and the target observation object, d is the moving distance of the VR headset, and s is the mapping ratio of the VR headset.

[0015] Preferably, the scaling of the RGB image according to the angle change amount and the virtual distance between the depth camera and the point cloud model realizes the near-far perception of the target observation object, and then includes:

[0016] The scaled RGB image is displayed on the built-in screen of the VR headset for the user to watch.

[0017] The second aspect of the application provides a VR-based high-immersive visual perception device, comprising:

[0018] An image acquisition module is configured to acquire depth target information of a target observation object collected by a depth camera and spatial pose information of a VR headset, wherein the depth target information includes a point cloud model and an RGB image, and the spatial pose information includes spatial position and attitude information.

[0019] A model construction module is configured to perform virtual-real coordinate mapping based on the depth camera, the VR headset and a three-degree-of-freedom holder to obtain an inverse kinematics transformation model.

[0020] An angle adjustment module is configured to adjust the observation angle of the depth camera according to the spatial pose information through the inverse kinematics transformation model to obtain an angle change amount.

[0021] A scaling perception module is configured to scale the RGB image according to the angle change amount and the virtual distance between the depth camera and the point cloud model to realize the near-far perception of the target observation object.

[0022] Preferably, the model construction module is specifically configured to:

[0023] mapping virtual-real coordinates based on the virtual observation point position of the depth camera, the spatial pose information of the VR head-mounted device, and the gimbal attitude information of the three-degree-of-freedom gimbal, to obtain an inverse kinematics transformation model.

[0024] Preferably, the scaling ratio is:

[0025]

[0026] wherein, l is the distance between the depth camera and the target observation object, d is the moving distance of the VR head-mounted device, and s is the mapping ratio of the VR head-mounted device.

[0027] Preferably, the method further comprises:

[0028] The display module is configured to display the scaled RGB image on the built-in screen of the VR head-mounted device for a user to view.

[0029] The third aspect of the present application provides a VR-based high-immersive visual perception device, which comprises a processor and a memory;

[0030] The memory is configured to store program code and transmit the program code to the processor;

[0031] The processor is configured to execute the VR-based high-immersive visual perception method according to the instructions in the program code.

[0032] The fourth aspect of the present application provides a computer-readable storage medium for storing program code, which is configured to execute the VR-based high-immersive visual perception method.

[0033] As can be seen from the above technical solutions, the embodiments of the present application have the following advantages:

[0034] In the present application, a VR-based high-immersive visual perception method is provided, which comprises: acquiring depth target information of a target observation object collected by a depth camera and spatial pose information of a VR head-mounted device, wherein the depth target information comprises a point cloud model and an RGB image, and the spatial pose information comprises spatial position and attitude information; mapping virtual-real coordinates based on the depth camera, the VR head-mounted device, and a three-degree-of-freedom gimbal, to obtain an inverse kinematics transformation model; adjusting the observation angle of the depth camera according to the spatial pose information through the inverse kinematics transformation model, to obtain an angle change amount; and scaling the RGB image according to the angle change amount and the virtual distance between the depth camera and the point cloud model, to realize the near-far perception of the target observation object.

[0035] The VR-based high-immersive visual perception method provided in the application adopts a three-degree-of-freedom holder instead of a mechanical arm, can realize radial observation based on an origin, and avoids the inconvenience caused by the limited movement of the mechanical arm; and the virtual RGB image is scaled according to the angle change and the virtual distance instead of tracking and adjusting the actual image or the device, so that time delay can be reduced, reliable visual tracking can be provided, dizziness can be avoided, and better visual perception experience can be provided. Therefore, the application can solve the technical problem that the existing VR high-immersive visual perception method based on a mechanical arm is subject to the movement mechanism of the mechanical wall, it is difficult to realize high-quality real-time perception, and dizziness is generated. BRIEF DESCRIPTION OF DRAWINGS

[0036] Figure 1 A flowchart of a VR-based high-immersive visual perception method provided in an embodiment of the application is shown in the figure.

[0037] Figure 2 A structural diagram of a VR-based high-immersive visual perception device provided in an embodiment of the application is shown in the figure.

[0038] Figure 3 A device diagram of a VR-based high-immersive visual perception system provided in an embodiment of the application is shown in the figure.

[0039] Figure 4 A device coordinate system diagram for constructing an inverse kinematics transformation model provided in an embodiment of the application is shown in the figure.

[0040] Figure 5 A scaling structure diagram provided in an embodiment of the application is shown in the figure.

[0041] Figure 6 A scaling geometric relationship diagram provided in an embodiment of the application is shown in the figure. DETAILED DESCRIPTION

[0042] In order to enable personnel in the technical field to better understand the application scheme, the technical solutions in the embodiments of the application will be clearly and completely described below in conjunction with the drawings in the embodiments of the application. Obviously, the described embodiments are only part of the embodiments of the application, not all embodiments. Based on the embodiments in the application, all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the scope of the application.

[0043] For ease of understanding, please refer to Figure 1 An embodiment of a VR-based high-immersive visual perception method provided in the application includes:

[0044] Step 101, obtaining depth target information of a target observation object collected by a depth camera and spatial pose information of a VR head-mounted device, the depth target information including a point cloud model and an RGB image, and the spatial pose information including spatial position and attitude information.

[0045] Referring to Figure 3 The hardware devices used in the method of the embodiment include a VR head-mounted device, a depth camera, and a three-degree-of-freedom gimbal. The three-degree-of-freedom serial robot mainly changes the observation direction of the depth camera according to the attitude change of the VR head-mounted device; the VR head-mounted device is used to feedback the camera image and reflect the position and attitude of the user's head; the depth camera is installed at the end of the three-degree-of-freedom serial robot and can obtain the depth target information of the observation target, i.e., the point cloud model and the RGB image, based on the TOF ranging principle; the controller includes functions such as controlling the motion of the three-degree-of-freedom serial robot, processing the point cloud image of the depth camera, changing the size of the target image according to the position information reflected by the VR head-mounted device to achieve the effect of observing from a distance and observing closely, and the like.

[0046] As described above, after obtaining the spatial pose information of the VR head-mounted device, the observation angle and observation distance of the VR head-mounted device can be adjusted to maintain a better head attitude and ensure the comfort of observation.

[0047] Step 102, virtual-real coordinate mapping based on the depth camera, the VR head-mounted device, and the three-degree-of-freedom gimbal to obtain an inverse kinematics transformation model.

[0048] The advantage of using a three-degree-of-freedom gimbal instead of a six-degree-of-freedom mechanical arm is that the three-degree-of-freedom gimbal can perform radial observation from a single origin without being affected by the limited effective motion range and structural constraints of the mechanical arm, and the three-degree-of-freedom gimbal has lower cost and higher reliability, making it more practical.

[0049] Further, step 102 specifically includes:

[0050] Virtual-real coordinate mapping based on the virtual observation point position of the depth camera, the spatial pose information of the VR head-mounted device, and the gimbal attitude information of the three-degree-of-freedom gimbal to obtain an inverse kinematics transformation model.

[0051] Referring to Figure 4 Both the VR head-mounted device and the three-degree-of-freedom gimbal have their own coordinate reference systems, while the reference of the depth camera is the virtual observation point position. The construction process of the inverse kinematics transformation model is the mapping process of the coordinate systems, and the coordinate relationship can be constructed according to the coordinate systems:

[0052] q2 = -arccos(cos(ry)cos(rz))

[0053]

[0054]

[0055]

[0056]

[0057] According to the coordinate mapping transformation relationship, the inverse kinematics transformation model can be constructed.

[0058] In step 103, the observation angle of the depth camera is adjusted according to the spatial pose information by the inverse kinematics transformation model, and the angle change amount is obtained.

[0059] The purpose of adjusting the observation angle of the depth camera by the inverse kinematics transformation model is to make the observation angle of the depth camera consistent with the observation angle of the user's eyes, so as to facilitate obtaining the target virtual image.

[0060] In step 104, the RGB image is scaled according to the angle change amount and the virtual distance between the depth camera and the point cloud model, and the near-far perception of the target observation object is realized.

[0061] Further, the scaling ratio is:

[0062]

[0063] Wherein, l is the distance between the depth camera and the target observation object, d is the moving distance of the VR head-mounted device, and s is the mapping ratio of the VR head-mounted device.

[0064] Compared with the VR perception system of the six-degree-of-freedom mechanical arm, the three-degree-of-freedom holder and the depth camera in the embodiment adjust the size of the virtual image according to the virtual observation distance and the angle change amount, and can realize multi-angle three-dimensional scaling adjustment, which is not limited by specific mechanical equipment and will not cause large delay and observation dizziness. However, the VR perception system of the six-degree-of-freedom mechanical arm adjusts the size of the observed object caused by the actual observation distance change adjustment of the mechanical arm end position movement, so it is easy to cause dizziness, and it is also a planar image scaling.

[0065] For the setting of the scaling ratio, the embodiment considers that the camera holder is 3 degrees of freedom, and is only used for mapping the VR head-mounted device posture. For the position change of the VR head-mounted device, the virtual movement of the depth camera is achieved by changing the image size of the VR device built-in screen. Since the target information collected by the depth camera contains both the RGB information and the point cloud model of the target, the image scaling can be achieved by relative position transformation. Assuming that the VR head-mounted device is in the initial position b, the depth camera observes the observation target outside the distance l and displays the actual diameter D of the target display contour circle on the VR built-in screen. When the VR head-mounted device moves forward by a distance d at the current observation angle, the size of the target display contour circle will be scaled relative to the initial state, and the scaling ratio is calculated as follows:

[0066] Let the display ratio of the object in the VR head-mounted device built-in screen at the initial time be k1, then:

[0067]

[0068] When the VR head-mounted device moves d, the display ratio of the object in the VR head-mounted device built-in screen is k2 under the condition that the mapping ratio of the preset camera virtual movement distance to the actual movement distance of the VR head-mounted device is s, then:

[0069]

[0070] When the VR head-mounted device moves d, the display size of the object in the VR head-mounted device built-in screen is scaled by a factor of k. For the specific scaling principle of the virtual image according to the scaling ratio, please refer to Figure 5 and Figure 6 .

[0071] Further, the step 104 further includes:

[0072] The RGB image obtained by scaling the ratio is displayed on the built-in screen of the VR head-mounted device for the user to watch.

[0073] The VR-based high-immersive visual perception method provided by the embodiment adopts a three-degree-of-freedom holder instead of a mechanical arm, can realize radial observation based on the origin, avoids the inconvenience caused by the movement limitation of the mechanical arm, and scales the virtual RGB image according to the angle change and the virtual distance, instead of tracking and adjusting the actual image or the device, so as to reduce the time delay, provide more reliable visual tracking, avoid the generation of dizziness, and provide a better visual perception experience. Therefore, the embodiment can solve the technical problem that the existing VR high-immersive visual perception method based on a mechanical arm is subject to the movement mechanism of the mechanical wall, and it is difficult to realize high-quality real-time perception, and further generates dizziness.

[0074] For ease of understanding, please refer toFigure 2 The application provides an embodiment of a VR-based high-immersive visual perception device, which comprises:

[0075] An image acquisition module 201 is configured to acquire depth target information of a target observation object collected by a depth camera and spatial pose information of a VR head-mounted device, wherein the depth target information comprises a point cloud model and an RGB image, and the spatial pose information comprises spatial position and attitude information;

[0076] A model construction module 202 is configured to perform virtual-real coordinate mapping based on the depth camera, the VR head-mounted device and a three-degree-of-freedom holder to obtain an inverse kinematics transformation model;

[0077] An angle adjustment module 203 is configured to adjust an observation angle of the depth camera according to the spatial pose information through the inverse kinematics transformation model to obtain an angle change amount;

[0078] A scaling perception module 204 is configured to proportionally scale the RGB image according to the angle change amount and a virtual distance between the depth camera and the point cloud model to realize near-far perception of the target observation object.

[0079] Further, the model construction module 202 is specifically configured to:

[0080] perform virtual-real coordinate mapping based on a virtual observation point position of the depth camera, the spatial pose information of the VR head-mounted device and holder attitude information of the three-degree-of-freedom holder to obtain the inverse kinematics transformation model.

[0081] Further, the scaling ratio is:

[0082]

[0083] wherein l is a distance between the depth camera and the target observation object, d is a moving distance of the VR head-mounted device, and s is a mapping ratio of the VR head-mounted device.

[0084] Further, the application further comprises:

[0085] A display module 205 is configured to display the scaled RGB image on a built-in screen of the VR head-mounted device for a user to view.

[0086] The application further provides a VR-based high-immersive visual perception device, which comprises a processor and a memory.

[0087] The memory is configured to store program code and transmit the program code to the processor.

[0088] The processor is configured to execute the VR-based high-immersive visual perception method in the above method embodiment according to instructions in the program code.

[0089] The application further provides a computer-readable storage medium for storing program code for executing the VR-based high-fidelity visual perception method in the method embodiments.

[0090] In several embodiments provided in the application, it should be understood that the disclosed apparatus and method can be implemented in other manners. For example, the described apparatus embodiments are merely schematic. The division of the units is merely a logical function division. There can be another division manner for the actual implementation, for example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the displayed or discussed mutual couplings or direct couplings or communication connections can be indirect couplings or communication connections through some interfaces, devices or units, and can be electrical, mechanical or other forms.

[0091] The units described as separate components can or can not be physically separate, and the components displayed as units can or can not be physical units, i.e., can be located in one place, or can be distributed on multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the embodiment.

[0092] In addition, each functional unit in the various embodiments of the application can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit.

[0093] The integrated unit, if realized in the form of a software functional unit and sold or used as an independent product, can be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the application essentially or the part of the prior art that makes a contribution or the whole or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a plurality of instructions for executing all or part of the steps of the method described in the various embodiments of the application by a computer device (which can be a personal computer, a server, or a network device, etc.). The aforementioned storage medium includes a U disk, a mobile hard disk, a read-only memory (English full name: Read-Only Memory, English abbreviation: ROM), a random access memory (English full name: Random Access Memory, English abbreviation: RAM), a magnetic disk or an optical disk, and various media that can store program codes.

[0094] The above-described embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacements for part of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A VR-based high sense of presence perception method, characterized in that, The method comprises the following steps: acquiring depth target information of a target observation object collected by a depth camera and spatial pose information of a VR head-mounted device, the depth target information comprising a point cloud model and an RGB image, and the spatial pose information comprising spatial position and attitude information; performing virtual-real coordinate mapping based on the depth camera, the VR head-mounted device and a three-degree-of-freedom holder to obtain an inverse kinematics transformation model; adjusting an observation angle of the depth camera according to the spatial pose information through the inverse kinematics transformation model to obtain an angle change amount; scaling the RGB image according to the angle change amount and a virtual distance between the depth camera and the point cloud model to realize near-far perception of the target observation object, the scaling ratio being: ； wherein, is a distance of the depth camera to the target observation object, is a moving distance of the VR head-mounted device, is a mapping scale of the VR head-mounted device.

2. The VR-based high-presence visual perception method of claim 1, wherein, The method further comprises the following steps: displaying the scaled RGB image on a built-in screen of the VR head-mounted device for a user to view.

3. The VR-based high-presence visual perception method of claim 1, wherein, The method comprises the following steps: an image acquisition module, configured to acquire depth target information of a target observation object collected by a depth camera and spatial pose information of a VR head-mounted device, the depth target information comprising a point cloud model and an RGB image, and the spatial pose information comprising spatial position and attitude information; 4. A VR-based high sense of presence perception device, characterized in that, a model construction module, configured to perform virtual-real coordinate mapping based on the depth camera, the VR head-mounted device and a three-degree-of-freedom holder to obtain an inverse kinematics transformation model; an angle adjustment module, configured to adjust an observation angle of the depth camera according to the spatial pose information through the inverse kinematics transformation model to obtain an angle change amount; a scaling perception module, configured to scale the RGB image according to the angle change amount and a virtual distance between the depth camera and the point cloud model to realize near-far perception of the target observation object, the scaling ratio being: The model construction module is specifically configured to: perform virtual-real coordinate mapping based on a virtual observation point position of the depth camera, the spatial pose information of the VR head-mounted device and holder attitude information of the three-degree-of-freedom holder to obtain an inverse kinematics transformation model. ； wherein, is a distance of the depth camera to the target observation object, is a moving distance of the VR head-mounted device, is a mapping scale of the VR head-mounted device.

5. The VR-based high presence visual perception device of claim 4, wherein, The method further comprises the following steps: a display module, configured to display the scaled RGB image on a built-in screen of the VR head-mounted device for a user to view.

6. The VR-based high presence visual perception device of claim 4, wherein, The device comprises a processor and a memory; the memory is configured to store program code and transmit the program code to the processor; 7. A VR-based high-presence visual perception device, characterized by, the processor is configured to execute the instructions in the program code to perform the VR-based high-illusion visual perception method according to any one of claims 1-3. ​ ​ 8. A computer-readable storage medium, characterized in that, The computer readable storage medium is configured to store program code for performing the VR-based high-presence visual perception method according to any one of claims 1-3.

Images