A dynamic exposure method, system, VR virtual reality device and storage medium

Abstract

The present invention relates to the field of spatial positioning and tracking technology, and in particular to a method, system, VR virtual reality device and storage medium for dynamic exposure. The method comprises the following steps: acquiring image data based on a number of shooting modules; calculating the 2D coordinates of a light spot in the image data on each frame of an image based on the image data; calculating the position information of a handle based on the correspondence between the 2D coordinates and the 3D model information; updating the state of the handle based on the position information, and optimizing the position of the handle based on the updated state of the handle; predicting whether each light source on the 3D model of the handle can form a light spot imaging in the image taken by each shooting module in the next frame based on the optimized position; controlling the opening and / or closing of the exposure of each shooting module based on whether there is a light spot in each picture in the predicted image, thereby reducing the power consumption of the VR virtual reality device.

Description

Technical Field

[0001] The present invention relates to the field of spatial positioning and tracking technology, and in particular to a dynamic exposure method, system, VR virtual reality device and storage medium. Background Art

[0002] VR virtual reality equipment is a computer simulation system that can create and experience a virtual world. It uses a computer to generate a virtual environment. It is a system simulation of multi-source information fusion, interactive three-dimensional dynamic vision and entity behavior, which immerses users in the environment. The current virtual reality (Virtual Reality) technology generates an integrated virtual environment through digital information in vision, hearing, touch, etc., so that it has the characteristics of immersion, interactivity, and multi-sensory. In VR virtual reality equipment, at least one handle is included. The handle exists to be able to control or operate the character in the virtual environment. Generally, for the operability of the handle, the handle is generally relatively independent, so there will be problems with the positioning and tracking of the handle.

[0003] In common positioning and tracking scenarios, they are usually divided into outside-in and inside-out categories according to the angle of the observer. Typical outside-in systems include OptiTrack, HTC VIVE, etc., and typical inside-out systems include Microsoft MR, MetaQuest, etc. However, regardless of the classification, the core technology is to use the camera to expose and image the light-emitting device of the handle, and then infer the position of the handle according to the position of the light spot in the image. In general, in order to ensure consistency of observation and maximize coverage, multiple cameras will be configured into a synchronous exposure working mode, that is, all cameras operate at the same time. However, in actual usage scenarios, it is often the case that only a few cameras can observe the handle, that is, some cameras are actually doing useless work. Therefore, the camera that fails to observe the handle at this time is in an invalid working state, so there is a waste of power consumption.

[0004] In summary, the technical problem actually solved by the present invention is how to reduce the power consumption of VR virtual reality devices. Summary of the invention

[0005] In order to overcome the above technical defects, the purpose of the present invention is to provide a dynamic exposure method, system, VR virtual reality device and storage medium, which reduce the power consumption of VR virtual reality device.

[0006] The present invention discloses a dynamic exposure method, comprising the steps of:

[0007] Acquiring image data based on a number of shooting modules;

[0008] Calculate the 2D coordinates of the light spot in the image data on each frame of the image according to the image data;

[0009] Obtain the position information of the handle based on the correspondence between the 2D coordinates and the 3D model information;

[0010] The status information of the handle is fused and tracked based on the posture information and the information of the IMU;

[0011] Predicting whether a light spot will appear in a predicted image corresponding to each shooting module in the next frame according to the updated state information and the optimized posture information;

[0012] The exposure of each shooting module is controlled to be turned on and / or off according to whether there is a light spot in each picture in the predicted image. When there is no light spot in a picture in the predicted image, the exposure of the shooting module corresponding to the picture is turned off. When there is a light spot in a picture in the predicted image, the exposure of the shooting module corresponding to the picture is turned on.

[0013] All camera modules should be jointly calibrated in advance, and the calibration parameters will be saved on the storage medium for software to read.

[0014] Preferably, the image data is one frame and / or multiple frames of pictures taken by several shooting modules at the same time.

[0015] Preferably, when acquiring the position information of the handle according to the 2D coordinates, the position information of the handle is calculated by a PNP algorithm according to the imaging parameters of each module and the 2D coordinates of the light spot in the picture taken by each shooting module and the 3D model information.

[0016] Preferably, the posture information includes 6 degrees of freedom, namely 3 position degrees of freedom and 3 attitude degrees of freedom, and the state information includes motion state information of velocity, acceleration deviation and angular velocity deviation. The spatial motion of the handle can be tracked and predicted based on the posture information and state information.

[0017] Preferably, an inertial sensor unit is provided in the handle, and the inertial sensor unit obtains IMU information of the handle.

[0018] Preferably, the acquired IMU information and the position and state information of the handle are fused using a Kalman filter algorithm to improve the position and state prediction accuracy of the handle;

[0019] The predicted spatial coordinates of the handle light-emitting device are projected onto the focal plane according to the imaging parameters of the shooting module to obtain the 2D coordinates of each light spot in the image corresponding to each shooting module; if the projected 2D coordinates cannot fall within the rectangular range constrained by the width and height of the image, it can be considered that the light spot will not appear in the next frame of the image;

[0020] If the 2D coordinates obtained by projection can fall within the rectangular range constrained by the width and height of the image, the visibility of the light spot needs to be further verified; by calculating the direction cosines of the emission normal of the handle light-emitting device and the normal of the main optical axis of each shooting module, the angle between the two normal directions can be obtained. When the angle is greater than a certain empirical threshold, the light spot will appear in the next frame of the image, otherwise it will still not appear;

[0021] In view of this, a second object of the present invention is to provide a control system using the above control method, comprising:

[0022] Shooting module: used to obtain image data and send it to the processing module;

[0023] Vision module: used to calculate the position information of the handle based on visual image data;

[0024] IMU module: used to obtain IMU data and perform data preprocessing;

[0025] Filter module: fuses and locates the state of the handle based on the visual posture information and IMU information;

[0026] Prediction module: used to determine in advance whether each light source of the handle can form an image in each shooting module when the next frame arrives based on the positioning and tracking information;

[0027] Control module: used to control the exposure of each shooting module to be turned on and / or off according to whether a light spot will appear in the predicted image. When the light spot will not appear in the predicted image, the exposure of the shooting module is turned off, and when the light spot will appear in the predicted image, the exposure of the shooting module is turned on.

[0028] In view of this, a third object of the present invention is to provide a VR virtual reality device, on which at least one control system as mentioned above is provided.

[0029] In view of this, a fourth object of the present invention is to provide a computer-readable storage medium having a computer program stored thereon, which implements the steps of the above-mentioned control method when the computer program is executed by a processor.

[0030] After adopting the above technical solution, compared with the prior art, the beneficial effect of the present invention is that the power consumption of VR virtual reality devices is reduced, thereby extending the battery life, thereby improving the user experience, and reducing the computing power requirements under the premise of reducing the pictures taken by the shooting module. In addition, the prediction module can be implemented by reusing the intermediate results of the visual inertial guidance filter, which almost does not bring additional computing power requirements. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1A schematic diagram of the workflow steps of a dynamic exposure method, system, VR virtual reality device and storage medium of the present invention;

[0032] Figure 2 It is an "inside-out" schematic diagram of a dynamic exposure method, system, VR virtual reality device and storage medium of the present invention;

[0033] Figure 3 This is an “outside-in” schematic diagram of a dynamic exposure method, system, VR virtual reality device, and storage medium of the present invention.

[0034] Reference numerals:

[0035] 1 is a head mounted device, 101 is a shooting module, 2 is a handle, 3 is image data, 4 is a schematic diagram of the predicted position of the light spot in the image, 401 is that the light spot will not appear in the imaging area, and 5 is a positioning base station. DETAILED DESCRIPTION

[0036] The advantages of the present invention are further described below in conjunction with the accompanying drawings and specific embodiments.

[0037] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0038] The terms used in this disclosure are for the purpose of describing specific embodiments only and are not intended to limit the disclosure. The singular forms of "a", "said" and "the" used in this disclosure and the appended claims are also intended to include plural forms unless the context clearly indicates otherwise. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0039] It should be understood that although the terms first, second, third, etc. may be used in the present disclosure to describe various information, such information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of the present disclosure, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0040] In the description of the present invention, it is necessary to understand that the terms "longitudinal", "lateral", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", etc., indicating the orientation or position relationship, are based on the orientation or position relationship shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present invention.

[0041] In the description of the present invention, unless otherwise specified and limited, it should be noted that the terms "installed", "connected" and "connected" should be understood in a broad sense. For example, it can be a mechanical connection or an electrical connection, or it can be the internal connection between two components. It can be a direct connection or an indirect connection through an intermediate medium. For ordinary technicians in this field, the specific meanings of the above terms can be understood according to the specific circumstances.

[0042] In the following description, the suffixes such as "module", "component" or "unit" used to represent elements are only used to facilitate the description of the present invention, and have no specific meanings. Therefore, "module" and "component" can be used interchangeably.

[0043] See also Figures 1 to 3 As shown, this embodiment provides a dynamic exposure method, which is mainly used in VR virtual reality equipment and specifically includes the following steps:

[0044] S100: Acquire image data 3 based on a plurality of shooting modules 101, the shooting modules 101 are generally shooting cameras on the head mounted device 1, or shooting cameras in a plurality of positioning base stations 5 in the environment where the VR virtual reality device is applied. When the shooting modules 101 are shooting cameras on the head mounted device 1, it is "inside-out" at this time; when the shooting modules 101 are a plurality of positioning base stations 5 in the environment where the VR virtual reality device is applied, it is "outside-in" at this time.

[0045] S200: Calculating the 2D coordinates of the light spot in the image data 3 on each frame of the image according to the image data 3. It should be noted that when there is no light spot on a frame of the image in the image data 3, the 2D coordinates of the light spot on the image are not calculated.

[0046] S300: Acquire the position information of the handle 2 according to each 2D coordinate.

[0047] S400: tracing back the state of the handle 2 according to the position information of the handle 2, and determining the position information of the light spot according to the traced state of the handle 2.

[0048] S500: Predicting, based on the position information, whether the light source on the handle 2 will form a corresponding light spot in the picture to be captured by each shooting module 101 when the next frame arrives.

[0049] S600: Controlling the on / off of the exposure of the shooting module 101 according to whether there is a light spot in each picture in the predicted image 4. When there is no light spot in a picture in the predicted image 4, the exposure of the shooting module 101 in the shooting picture without the light spot is turned off. When there is a light spot in a picture in the predicted image 4, the exposure of the shooting module 101 in the shooting picture with the light spot is turned on.

[0050] It should be noted that the general principle of the above steps is that the posture information of the handle 2 is confirmed based on the 2D coordinates of the light spot in the picture. Generally, the shooting module 101 is relatively fixed. Generally, multiple shooting modules 101 in the VR virtual reality device each shoot their own fixed area. If there is no light spot in all the pictures of the next frame, then the shooting module 101 that shoots the picture without the light spot is turned off, thereby reducing the power consumption of the VR virtual reality device. For example: the VR virtual reality device is used in a six-sided space, including the top, bottom, left, right, front, and back. After the 2D coordinates are obtained based on the light spot on any surface to obtain the posture information of the handle 2, the state of the handle 2 in the next second (next frame) will be traced back according to the posture information to determine the position information of the light spot. For example, if the light spot is determined to be on the top, left, and front, then the exposure of the shooting module 101 shooting the bottom, right, and back will be turned off. If it is determined that the light spot is on the bottom, right, and back, then the shooting will be turned off. The exposure of the camera module 101 above, on the left, and in the front will also start the exposure of the camera module 101 below, on the right, and in the back; PNP provides visual posture solution; IMU provides high-frequency motion state perception; the posture is optimized by integrating the information at the visual level and the IMU level; based on the optimized posture, the posture of the next frame is predicted; the predicted 3D model information is projected onto the camera imaging plane; if it exceeds the imaging plane, it is not visible; if the angle between the light emission direction and the camera's direction is too small (close to the same direction), it is also not visible;.

[0051] It should be noted that the image data 3 is one frame and / or multiple frames of pictures taken by several shooting modules at the same time.

[0052] It should be noted that each shooting module 101 is provided with information parameters. When the posture information of the handle 2 is obtained according to each 2D coordinate, the posture information of the handle 2 is calculated through the PNP algorithm based on the information parameters of each shooting module 101 and the 2D coordinates of the light spot of the picture taken by each shooting module 101. This step uses visual information to form a higher-precision observation of the posture of the handle 2.

[0053] It should be noted that the posture information at least includes direction information, and the position information of the handle 2 is tracked based on the direction information. For example, if the posture information of the handle 2 is facing a certain direction, the position of the handle 2 will be tracked based on the direction in which the handle 2 is facing.

[0054] It should be noted that an inertial sensor unit is also provided in the handle 2. The inertial sensor unit obtains the IMU information of the handle 2 and fuses the posture information according to the IMU information.

[0055] See also Figure 2 As shown, in a typical "inside-out" embodiment, it includes a head-mounted device, which includes multiple shooting modules 101, and a handle 2, which acts as a light source to radiate out of the square.

[0056] After the shooting module 101 captures the image data 3, the light spot in the image data 3 is extracted. It should be noted that the light spot extraction is performed in a certain frame of the image. After the light spot in the image is extracted, the 2D coordinates of the light spot on the image can be obtained. The posture information of the handle 2 is calculated by combining the information parameters that have been calibrated in advance for the shooting module 101. Then, the IMU information obtained by the inertial measurement unit configured for the handle 2 is fused to obtain the optimized posture information. Then, according to the state of the handle 2, the position of the light spot on the imaging plane of each shooting module 101 when the next frame arrives can be tracked and predicted. Due to the difference in field of view angles, the light spot at the predicted position will not be imaged within the field of view angle of some shooting modules 101. The prediction can be based on the tracing results to control the exposure of the shooting module 101, so as to achieve the effect of starting and stopping the exposure of the corresponding shooting module 101 at the right time. At the same time, the algorithm links related to the light spot extraction and center calculation will be skipped, thereby saving some computing power. Figure 2 3 represents image data 3 captured by the shooting module 101 of the head mounted device, 4 represents predicted image 4, wherein 401 indicates that the predicted light spot will not appear in the imaging area.

[0057] See also Figure 3As shown, in a typical "outside-in" (positioning base station 5 with handle 2), 5 is the positioning base station 5, and there are several shooting modules 101 in the positioning base station 5 for fill light and image acquisition, 2 is the handle, 3 and 4 are image data 3 and predicted image 4 shot by the shooting module 101, respectively, and 401 indicates that the predicted light spot will not appear in the imaging area. In this embodiment, the exposure control of the shooting module 101 still follows the process provided in the above embodiment. Compared with the above embodiment, the main difference between this embodiment and the above embodiment is that the exposure control of the shooting module 101 of the positioning base station 5 is replaced by the head-mounted device 1 with the driving Hub; the carrier of the shooting module 101 is replaced by the head-mounted device 1 with an external computing platform such as PC and NUC.

[0058] This embodiment also provides a control system using the above control method, the system comprising:

[0059] Shooting module 101: used to obtain image data 3, the shooting module 101 is generally a shooting camera on the head mounted device 1, or a shooting camera in a plurality of positioning base stations 5 in an environment where a VR virtual reality device is applied. When the shooting module 101 is a shooting camera on the head mounted device 1, it is "inside-out" at this time; when the shooting module 101 is a plurality of positioning base stations 5 in an environment where a VR virtual reality device is applied, it is "outside-in" at this time.

[0060] Processing module: used to calculate the 2D coordinates of the light spot in the image data 3 on each frame of the image according to the image data 3. It should be noted that when there is no light spot on a frame of the image in the image data 3, the 2D coordinates of the light spot on the image are not calculated.

[0061] Acquisition module: used to obtain the position information of handle 2 according to each 2D coordinate.

[0062] Tracing module: used to trace the state of handle 2 according to the posture information of handle 2, and determine the position information of the light spot according to the traced state of handle 2.

[0063] Virtual image module: used to generate in advance the picture taken by each shooting module 101 in the next frame according to the position information so as to obtain the predicted image 4.

[0064] Control module: used to control the on / off of the exposure of the shooting module 101 according to whether there is a light spot in each picture in the predicted image 4. When there is no light spot in a picture in the predicted image 4, the exposure of the shooting module 101 in the shooting picture without the light spot is turned off; when there is a light spot in a picture in the predicted image 4, the exposure of the shooting module 101 in the shooting picture with the light spot is turned on.

[0065] This embodiment also provides a VR virtual reality device, which includes at least one of the above-mentioned control systems.

[0066] This embodiment also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the above control method are implemented.

[0067] It should be noted that the embodiments of the present invention have better practicability and do not impose any form of limitation on the present invention. Any technician familiar with the field may use the technical content disclosed above to change or modify it into an equivalent effective embodiment. However, any modification or equivalent changes and modifications made to the above embodiments based on the technical essence of the present invention without departing from the content of the technical solution of the present invention are still within the scope of the technical solution of the present invention.

Claims

1. A dynamic exposure control method, applied to VR virtual reality equipment, characterized in that: Includes steps: Acquiring image data based on a number of shooting modules; Calculate the 2D coordinates of the light spot in the image data on each frame of the image according to the image data; Calculating the position and posture information of the handle according to the correspondence between the 2D coordinates and the 3D model information; Updating the state information of the handle according to the position information, and optimizing the position information of the handle according to the updated state information of the handle; Predicting whether each light spot will appear in the predicted image corresponding to each shooting module in the next frame according to the updated state information and the optimized posture information; The exposure of each shooting module is controlled to be turned on and / or off according to whether there is a light spot in each picture in the predicted image. When there is no light spot in a picture in the predicted image, the exposure of the shooting module corresponding to the shooting of the picture without the light spot is turned off. When there is a light spot in a picture in the predicted image, the exposure of the shooting module corresponding to the shooting of the picture with the light spot is turned on.

2. The control method according to claim 1, characterized in that: The image data is one frame and / or multiple frames of pictures taken by several shooting modules at the same time.

3. The control method according to claim 1, characterized in that: The imaging parameters and 3D model information of the shooting module are preset in the software system. When the position information of the handle is obtained according to each of the 2D coordinates, the position information of the handle is calculated by a PNP algorithm according to the imaging parameters of each of the shooting modules and the 2D coordinates of the light spot of the picture taken by each of the shooting modules and the 3D model information.

4. The control method according to claim 1, characterized in that: The posture information includes 6 degrees of freedom, namely 3 position degrees of freedom and 3 posture degrees of freedom. The state information includes velocity, acceleration deviation, and angular velocity deviation motion state information. The spatial motion of the handle is tracked and predicted based on the posture information and state information.

5. The control method according to claim 1, characterized in that: An inertial sensor unit is provided in the handle, and the inertial sensor unit obtains IMU information of the handle.

6. A control system using the control method according to any one of claims 1 to 5, characterized in that: include: Shooting module: used to obtain image data and send it to the processing module; Vision module: used to calculate the position information of the handle based on visual image data; IMU module: used to obtain IMU data and perform data preprocessing; Filter module: fusing and positioning the state of the handle according to the visual posture information and the IMU information; Prediction module: used to determine in advance whether each light source of the handle can form an image in each of the shooting modules when the next frame arrives according to the positioning tracking information; Control module: used to control the exposure of each shooting module to be turned on and / or off according to whether a light spot will appear in the predicted image. When the light spot will not appear in the predicted image, the exposure of the shooting module is turned off, and when the light spot will appear in the predicted image, the exposure of the shooting module is turned on.

7. A VR virtual reality device, characterized in that: The VR virtual reality device includes at least one control system as claimed in claim 6.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps according to any one of claims 1 to 5 are implemented.

Images