Positioning method of virtual reality equipment and virtual reality equipment

Abstract

The invention relates to the technical field of VR, provides a positioning method of virtual reality equipment and the virtual reality equipment, and is used for improving the robustness and precision of SLAM positioning. According to the method, for a multi-view camera on the virtual reality equipment, the concept of a main camera and a slave camera is introduced to obtain at least one group of main-slave camera pairs, and inter-frame feature point matching is performed on original environment images continuously acquired by each main camera, so that the time and the computing power of feature matching are saved, the main camera and the slave camera can be switched, and the accuracy of feature matching is improved. Therefore, the problem that inter-frame feature point tracking fails is solved, and in consideration of scale difference and rotation difference between images caused by orientation difference of the master camera and the slave camera, epipolar correction is carried out on original environment images collected by the master camera and the slave camera at the same moment in each group of master-slave camera pairs. And intra-frame feature point matching is carried out on the target environment image after epipolar correction, point pairs of binocular feature point matching are increased, and the SLAM positioning precision is improved.

Description

Technical Field

[0001] The present application relates to the technical field of virtual reality (VR), and provides a positioning method for a virtual reality device and a virtual reality device. Background Art

[0002] Currently, when users wear VR devices for an immersive experience, they generally collect information from the real environment and calculate the position and posture of the VR device itself through Simultaneous Localization and Mapping (SLAM) technology.

[0003] SLAM technology relies on tracked feature points for positioning. In order to ensure that enough feature points can be tracked during SLAM operation, it is necessary to keep rich texture information in the camera's field of view as much as possible. Therefore, related technologies usually equip VR devices with multiple cameras, and in order to obtain a larger field of view, these cameras have different orientations. However, if the orientation differences between the cameras are too large, the same feature point may have different scales and rotations in the environmental images captured by different cameras at the same time, thereby affecting the calculation accuracy of the pose. On the other hand, the use of multiple cameras will also increase the computing power of feature point matching between images, reducing the real-time performance of SLAM positioning. Summary of the invention

[0004] The embodiments of the present application provide a positioning method for a virtual reality device and a virtual reality device, which are used to ensure real-time positioning while improving positioning accuracy.

[0005] In one aspect, an embodiment of the present application provides a positioning method for a virtual reality device, wherein the virtual reality device includes at least one master-slave camera pair, and the method includes:

[0006] Acquire the original environment image captured by the master-slave camera pair in real time, and perform epipolar correction on the two original environment images synchronously captured by the master camera and the slave camera in the master-slave camera pair to reduce the scale difference and rotation difference between the environment images, so as to obtain the target environment image;

[0007] Extracting feature points from an original environment image captured by a master camera in the master-slave camera pair, and if the number of the extracted feature points is less than a preset number threshold, switching the slave camera in the master-slave camera pair to the master camera;

[0008] Performing inter-frame matching on feature points extracted from two adjacent original environment images captured by a master camera in the master-slave camera pair to obtain inter-frame matching points;

[0009] Determine, according to the inter-frame matching points, feature points in the target environment image of the master camera and the slave camera in the master-slave camera pair at the same time, and obtain intra-frame matching point pairs;

[0010] The position and posture of the virtual reality device at the current moment are determined according to the intra-frame matching point pairs and in combination with the pre-calibrated intrinsic parameters and extrinsic parameters of the master camera and the slave camera in the master-slave camera pair.

[0011] On the other hand, an embodiment of the present application provides a virtual reality device, including a processor, a memory, a display, and at least one master-slave camera pair;

[0012] The master-slave camera pair is used to collect original environment images in real time;

[0013] The display is used to display a virtual picture;

[0014] The memory stores a computer program, and the processor performs the following operations according to the computer program:

[0015] Acquire the original environment image captured by the master-slave camera pair in real time, and perform epipolar correction on the two original environment images synchronously captured by the master camera and the slave camera in the master-slave camera pair to reduce the scale difference and rotation difference between the environment images, so as to obtain the target environment image;

[0016] Extracting feature points from an original environment image captured by a master camera in the master-slave camera pair, and if the number of the extracted feature points is less than a preset number threshold, switching the slave camera in the master-slave camera pair to the master camera;

[0017] Performing inter-frame matching on feature points extracted from two adjacent original environment images captured by a master camera in the master-slave camera pair to obtain inter-frame matching points;

[0018] Determine, according to the inter-frame matching points, feature points in the target environment image of the master camera and the slave camera in the master-slave camera pair at the same time, and obtain intra-frame matching point pairs;

[0019] The position and posture of the virtual reality device at the current moment are determined according to the intra-frame matching point pairs and in combination with the pre-calibrated intrinsic parameters and extrinsic parameters of the master camera and the slave camera in the master-slave camera pair.

[0020] On the other hand, an embodiment of the present application provides a computer-readable storage medium, which stores computer-executable instructions, and the computer-executable instructions are used to enable a computer device to execute the steps of a positioning method for a virtual reality device provided in an embodiment of the present application.

[0021] In a positioning method of a virtual reality device and a virtual reality device provided by an embodiment of the present application, in order to reduce the scale difference and rotation difference between environmental images caused by the different orientations of the main camera and the slave camera, the two original environmental images synchronously collected by the main camera and the slave camera are subjected to epipolar correction, thereby eliminating the scale difference and rotation difference between binocular feature points, increasing the number of binocular feature point matching, and improving the accuracy of SLAM positioning. The main camera and the slave camera can be switched, thereby avoiding the influence of the camera being blocked or the situation of collecting insufficient texture on feature point extraction, and improving the stability of SLAM positioning. On the other hand, considering the real-time requirements of SLAM positioning, only the main camera is matched between frames, thereby reducing the computing power of feature matching and ensuring the real-time performance of SLAM positioning.

[0022] Other features and advantages of the present application will be described in the following description, and partly become apparent from the description, or understood by practicing the present application. The purpose and other advantages of the present application can be realized and obtained by the structures specifically pointed out in the written description, claims, and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0024] Figure 1 A schematic diagram of a virtual reality device provided in an embodiment of the present application;

[0025] Figure 2 A flowchart of a positioning method for a virtual reality device provided in an embodiment of the present application;

[0026] Figure 3 This is a rendering of the original environment image synchronously captured by two sets of master and slave camera pairs provided in an embodiment of the present application;

[0027] Figure 4 A flow chart of a method for epipolar correction of an original environment image provided in an embodiment of the present application;

[0028] Figure 5 A schematic diagram of the polar line correction principle provided in an embodiment of the present application;

[0029] Fig. 6A This is a comparison diagram of the original environment image and the target environment image corresponding to the main camera before and after epipolar correction;

[0030] Figure 6BThe comparison diagram of the original environment image and the target environment image corresponding to the camera before and after epipolar correction;

[0031] Figure 7 The intra-frame feature point matching process provided in the embodiment of the present application;

[0032] Figure 8 A schematic diagram of a tracking process of feature points during positioning of a virtual reality device provided in an embodiment of the present application;

[0033] Fig. 9 A schematic diagram of the triangulation principle provided in an embodiment of the present application;

[0034] Fig.10 An overall architecture diagram of the virtual reality device positioning provided in the embodiment of the present application;

[0035] Fig.11 The structure of the virtual reality device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0036] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the technical solution of the present application, rather than all of the embodiments. Based on the embodiments recorded in the application documents, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the technical solution of the present application.

[0037] The technical terms of the embodiments of the present application are explained below.

[0038] Inter-frame feature point matching: Match the feature points in images continuously captured by the same camera at different times.

[0039] Intra-frame feature point matching: Match the feature points in the images captured synchronously by the master camera and the slave camera at the same time.

[0040] The following is an overview of the design concepts of the embodiments of the present application.

[0041] The visual features between cameras are one of the main information sources for SLAM positioning. VR devices use visual features to estimate their own position in the real environment at all times. When users wear VR devices for an immersive experience, they will inevitably encounter situations where the camera is blocked or the texture of the acquisition area is relatively simple (such as a large white wall, a smooth desktop, etc.), resulting in a lack of rich visual features and affecting the positioning results. Therefore, in order to avoid these situations as much as possible to ensure that the VR device obtains rich visual features, in addition to being equipped with multiple cameras, the orientation differences between the multiple cameras are also large enough to ensure that when a camera is blocked or the texture of the acquisition area is relatively simple, another camera with a different orientation will not be blocked or the texture of the acquisition area is relatively rich.

[0042] However, although the hardware structure avoids the positioning problems caused by camera occlusion and simple texture areas to the greatest extent, the large differences in the orientations of the cameras may cause scale and rotation differences between different images at the same time (i.e., within the frame), that is, the scale and rotation of the same feature point in the environmental images collected by different cameras at the same time are inconsistent, which reduces the accuracy of feature matching and thus affects the calculation accuracy of the pose. In addition, due to the use of multiple cameras, the computing power brought by feature matching will also increase exponentially, thereby reducing the real-time performance of SLAM positioning and affecting the user experience.

[0043] In view of this, an embodiment of the present application provides a positioning method for a virtual reality device, which divides the multi-cameras with different orientations on the virtual reality device into a main camera and a slave camera, performs intra-frame matching on the images captured by the main camera, and performs inter-frame matching on the images captured by the main camera and the slave camera at the same time, thereby saving the computing power required for feature matching and improving the real-time performance of the virtual reality device posture calculation, wherein, when performing inter-frame matching, the images captured by the main camera and the slave camera are subjected to epipolar correction to overcome the scale difference and rotation difference of the feature points in the image caused by the different orientations of the main camera and the slave camera, improve the accuracy of feature matching, and thus improve the accuracy of the virtual reality device posture calculation. At the same time, in the SLAM positioning process, in order to avoid the influence of camera occlusion and low texture on feature point matching, the main camera and the slave camera can be switched back and forth to ensure the stability of SLAM positioning.

[0044] like Figure 1 As shown, it is a schematic diagram of a virtual reality device provided by an embodiment of the present application, and the virtual reality device is equipped with 4 cameras facing in different directions, and the upper and lower cameras located on the same side form a master-slave camera pair, and each master-slave camera pair includes a master camera and a slave camera. The master camera and the slave camera in the same master-slave camera pair can be switched. Among them, inter-frame feature point matching is applied to the image captured by the master camera, and intra-frame feature point matching is applied to the image synchronously captured by the master and slave camera pairs at the same time.

[0045] In an optional example, the camera on the virtual reality device is a fisheye camera.

[0046] It should be noted that Figure 1 It is only an example of a virtual reality device. The embodiment of the present application does not impose any restrictive requirements on the number of master-slave camera pairs. The selection of the master camera and the slave camera in the master-slave camera pair can be determined based on the information richness contained in the captured image. The higher the information richness, the more feature points it contains.

[0047] For example, the camera corresponding to the image containing a larger number of feature points in the master-slave camera pair is used as the master camera, and the other camera is used as the slave camera.

[0048] Based on a virtual reality device including at least one set of master-slave camera pairs, a method for using SLAM technology to perform self-positioning is as follows: Figure 2 As shown, it mainly includes the following steps:

[0049] S20 1: Obtaining the original environment image collected in real time by the master and slave camera pairs.

[0050] In the SLAM positioning process, in order to obtain a more accurate and robust posture, the virtual reality device is usually equipped with at least one set of master-slave camera pairs to observe the external environment. Generally, in order to expand its own field of view, the orientations of the cameras in the master-slave camera pair are different and vary greatly.

[0051] Take the virtual reality device containing two sets of master-slave camera pairs as an example. Figure 3 As shown, it is a schematic diagram of the original environment image collected by four cameras at the same time.

[0052] S202: performing epipolar correction on two original environment images synchronously captured by a master camera and a slave camera in a master-slave camera pair to reduce scale differences and rotation differences between the environment images, thereby obtaining a target environment image.

[0053] Depend on Figure 3 It can be seen that due to the large difference in the orientation of each camera in the two groups of master-slave camera pairs, the scale and rotation of the original environment images collected by the master and slave camera pairs at the same time are respectively quite inconsistent. In this way, the same feature point may have different scales and different rotations in the original environment images collected by the master and slave camera pairs at the same time. In addition, due to the influence of fisheye camera distortion, even if some feature descriptors with scale invariance and rotation invariance (such as ORB operator) are used for feature matching, it may not be possible to obtain accurate matching point pairs, which will affect SLAM positioning failure or error.

[0054] In order to solve the influence of camera orientation difference on feature point matching, the embodiment of the present application performs epipolar correction on the two original environmental images synchronously captured by the master camera and the slave camera at the same time in each master-slave camera pair. Through epipolar correction, the images captured by two cameras with an angle can be corrected to images captured by two parallel cameras, thereby eliminating the scale difference and rotation difference between the environmental images captured by the master camera and the slave camera to a certain extent, solving the feature matching problem between the master camera and the slave camera with a large rotation angle, increasing the logarithm of binocular feature point matching, and improving the accuracy of SLAM positioning.

[0055] According to Euler's rotation theorem, the rotation of any rigid body in three-dimensional space can be expressed as a certain angle around the rotation axis. Therefore, epipolar correction can be performed by correcting the images collected by two cameras with a rotation angle of θ into images collected by two parallel cameras.

[0056] In one example, for the original environment image captured by each master-slave camera pair at the current moment, the epipolar correction process is as follows: Figure 4 As shown, it mainly includes the following steps:

[0057] S2021: Determine a rotation angle between the master camera and the slave camera in the master-slave camera pair according to pre-calibrated extrinsic parameters between the master camera and the slave camera in the master-slave camera pair.

[0058] S2022: According to the rotation angle, the Euler rotation theorem is used to convert the original environment image captured by the main camera and the original environment image captured by the slave camera into images captured at parallel angles to reduce the scale difference and rotation difference between the environment images, thereby obtaining two target environment images.

[0059] Take a master-slave camera pair as an example, Figure 5 The figure shows the principle diagram of epipolar correction, assuming that the external parameters between the master camera and the slave camera are in, represents the translation vector between the master camera and the slave camera, Represents the rotation matrix between the master camera and the slave camera, according to and The rotation angle between the master camera and the slave camera is determined to be θ, which is equivalent to the master camera and the slave camera rotating around the rotation axis at angles of -θ / 2 and θ / 2, respectively. At this time, the epipolar correction formula of the original environment images collected by the master camera and the slave camera is expressed as:

[0060]

[0061]

[0062] Among them, π src (·) and π tgt(·) Projection transformation of the camera before and after rotation, p src and p tgt represents the original environment image and the target environment image of the main camera before and after epipolar correction, p src′ and p tgt′ Represents the original environment image and the target environment image from the camera before and after epipolar correction.

[0063] like Fig. 6A The following is a comparison of the original environment image and the target environment image corresponding to the main camera before and after epipolar correction. Figure 6B The following is a comparison of the original environment image and the target environment image from the camera before and after epipolar correction. Fig. 6A and Figure 6B It can be seen that the target environment image with epipolar alignment after epipolar correction of the original environment image collected by the same pair of master and slave cameras has high scale consistency and rotation consistency, and better matching results can be obtained when performing intra-frame feature point matching later.

[0064] S203: Extracting feature points from an original environment image captured by a master camera in a master-slave camera pair.

[0065] Since the virtual reality device is equipped with multiple cameras, if the inter-frame feature point matching is performed on the images captured by each camera, the time and computing power of the feature point matching will be increased, and the real-time performance of SLAM positioning will be reduced. Therefore, in order to ensure the real-time performance of the virtual reality device posture calculation, the embodiment of the present application only performs inter-frame feature point matching on the original environment images continuously captured by the master camera at different times in each master-slave camera pair, thereby achieving feature tracking.

[0066] At present, there are two main types of feature tracking commonly used in SLAM positioning, one is the matching method based on the assumption of photometric invariance (i.e., optical flow method), and the other is the matching method based on feature descriptors (such as ORB, SITF, SURF, etc.). Among them, the matching method based on feature point descriptors is slower and matches fewer feature points, but the feature points themselves have a certain degree of scale and rotation invariance, while the matching method based on optical flow method is faster and matches more feature points, but the feature points themselves do not have scale and rotation invariance.

[0067] In one example, based on the requirements of real-time performance and the number of feature points, an optical flow method is used to extract feature points from the original environment images captured by the master and slave cameras in each master-slave camera pair.

[0068] It should be noted that, since the orientation of the main camera at different times is almost unchanged, the scale and rotation differences of the original environment images continuously collected by the main camera at different times are small. Therefore, when performing feature point matching between frames, the feature points extracted from the original environment images can be directly used.

[0069] S204: Determine whether the number of extracted feature points is less than a preset number threshold, if so, execute S205, if not, execute S206.

[0070] When the user wears the virtual reality device for immersive experience, he is in a constant state of motion, and it is inevitable that the camera is blocked or passes through an area with poor texture. In order to save time and computing power for feature matching, the embodiment of the present application only performs optical flow tracking on the feature points in the image continuously collected by the main camera in each group of master-slave camera pairs. When the main camera is blocked or the texture in the collected image is less, the feature points extracted from the image are insufficient. In this way, SLAM lacks the feature points in the previous and next frames, resulting in tracking loss or tracking failure. Even if the occlusion time lasts for a short time or the time in the low-texture area lasts for a short time, some feature points that have been tracked for a long time will be discarded at the moment of being blocked or missing texture, resulting in a decrease in the robustness of SLAM positioning. Therefore, while ensuring the real-time performance of SLAM, it is necessary to solve the problem of camera being blocked or insufficient texture information. In this regard, in the process of SLAM positioning of the virtual reality device, the embodiment of the present application determines whether the main camera is blocked or passes through a low-texture area by the number of feature points extracted from the original environment image collected by the main camera, so as to take corresponding measures to ensure the robustness of SLAM positioning.

[0071] S205: Switching the slave camera in the master-slave camera pair to the master camera, and extracting feature points in the original environment image captured by the master camera after switching.

[0072] When the number of feature points extracted from the original environment image collected by the main camera is less than the preset threshold, it indicates that the main camera may be blocked or pass through a low-texture area, which may cause positioning failure. The probability of both cameras in each master-slave camera pair being blocked at the same time is low. Therefore, when the main camera is blocked or passes through a low-texture area, the slave camera may not be blocked or the texture information in the image is relatively rich. At this time, the slave camera in the same master-slave camera pair can be switched to the master camera, and the original master camera can be downgraded to a slave camera. The optical flow method is used to extract feature points from the original environment image collected by the main camera after the switch. This can solve the problem of feature point tracking loss caused by the main camera being blocked or insufficient texture of the collected image. At the same time, some feature points that have been tracked for a long time are retained to a certain extent, thereby improving the robustness of SLAM positioning.

[0073] S206: performing inter-frame matching on feature points extracted from two adjacent original environment images captured by the master camera in the master-slave camera pair to obtain inter-frame matching points.

[0074] When the number of feature points extracted from the original environment image captured by the main camera is greater than or equal to the preset threshold, it indicates that the main camera is not blocked and the texture in the image is relatively rich, and a sufficient number of feature points can be tracked. Therefore, the feature points extracted from two adjacent original environment images captured by the main camera in the master-slave camera pair can be directly matched between frames to obtain inter-frame matching points.

[0075] Similarly, when the slave camera is switched to the master camera and the original master camera is downgraded to the slave camera, inter-frame matching can be performed on feature points extracted from the original environment images continuously captured by the master camera after the switch.

[0076] S207: Determine feature points of the target environment image of the master camera and the slave camera in the master-slave camera pair at the same time according to the inter-frame matching points, and obtain intra-frame matching point pairs.

[0077] The inter-frame matching points are the tracking results of the monocular feature points of the master camera in each master-slave camera pair. In order to obtain the 6Dof pose of the virtual reality device, it is also necessary to track the feature points successfully tracked by each master camera in the target environment image after epipolar correction to obtain the binocular feature point matching results.

[0078] Take the matching process of one frame between the master camera and the slave camera as an example. Figure 7 The figure shows the intra-frame feature point matching process, which mainly includes the following steps:

[0079] S2071: Mapping the inter-frame matching points of the master camera in the master-slave camera pair in the original environment image to the target environment image corresponding to the master camera after epipolar correction to obtain mapping points of the inter-frame matching points in the target environment image.

[0080] In one example, according to the coordinates of the tracked inter-frame matching points in the original environment image of the main camera, the mapping points thereof in the target environment image after epipolar correction can be determined by formula 1.

[0081] S2072: Mapping points of the main camera in the target environment image and performing feature matching with the target environment image corresponding to the slave camera at the same time to obtain intra-frame matching point pairs.

[0082] In one example, since the original environment images synchronously captured by the master camera and the slave camera at the same time are epipolar corrected, the target environment images of the two cameras have scale consistency and rotation consistency after correction. Therefore, the points mapped by the master camera in the target environment image can be matched with the feature points of the target environment image corresponding to the slave camera at the same time to obtain intra-frame matching point pairs.

[0083] See also Figure 8 , which is a schematic diagram of the tracking process of feature points in the positioning process of a virtual reality device provided in an embodiment of the present application, performs inter-frame feature point matching on the first k frames of original environment images continuously acquired by the main camera to obtain inter-frame matching points, and maps the inter-frame matching points in the first k frames of original environment images continuously acquired by the main camera to the target environment image after epipolar correction to obtain mapping points of inter-frame matching points; then performs intra-frame feature point matching on the mapping points in the first k frames of target environment images after epipolar correction of the main camera and the first k frames of original environment images acquired from the camera at the same time after epipolar correction to obtain intra-frame matching point pairs. When the main camera is blocked in the k+1th frame, the original main camera is downgraded to a slave camera, and the original slave camera is upgraded to the master camera to realize the switching of the master and slave cameras. After the slave camera is switched to the master camera, the k+1th frame original environment image of the switched main camera is matched with the kth frame original environment image. At the same time, the inter-frame matching points in the k+1th frame original environment image of the switched main camera are mapped to the k+1th frame target environment image after epipolar correction. When the original main camera field of view is restored in the k+2th frame, the current main camera is downgraded to a slave camera, and the current slave camera is upgraded to the master camera. The k+2th frame original environment image of the restored main camera is matched with the k+3th frame original environment image. At the same time, the inter-frame matching points in the k+2th frame original environment image of the restored main camera are mapped to the k+2th frame target environment image after epipolar correction, and the k+2th frame target environment image of the restored main camera is matched with the k+2th frame target environment image of the slave camera.

[0084] S208: Determine the position and posture of the virtual reality device at the current moment according to the intra-frame matching point pairs and in combination with the pre-calibrated intrinsic parameters and extrinsic parameters of the master camera and the slave camera in the master-slave camera pair.

[0085] After obtaining the matching point pairs in the frames of the master camera and the slave camera in each master-slave camera pair at the same time, the 6DoF pose of the virtual reality device at the current moment is calculated using the triangulation method.

[0086] like Fig. 9 The figure shows a schematic diagram of the triangulation principle. According to the 2D coordinates of imaging points p1 and p2 of point P in the three-dimensional space in the binocular image, combined with the intrinsic and extrinsic parameters of the binocular camera, the 3D coordinates of point P can be solved by minimizing the squares method.

[0087] In one example, after the slave camera in a master-slave camera pair is switched to the master camera, when the number of feature points extracted from the original environment image captured by the switched main camera is less than a preset number threshold, it indicates that the field of view of the switched main camera is blocked or passes through a low-texture area. At this time, the switched main camera can be restored to the slave camera, and at the same time, the switched slave camera can be restored to the master camera.

[0088] The overall architecture of the above embodiment is as follows: Fig.10 As shown, during the SLAM positioning process, at least one master-slave camera pair on the virtual reality device collects the surrounding original environment images in real time, and performs epipolar correction on the original environment images collected synchronously at the same time by each master-slave camera pair. For the master camera in each master-slave camera pair, extract the feature points in the original environment images collected by it, and determine whether the master camera is blocked or the texture of the collection area is insufficient according to the number of feature points. If so, switch the master camera and the slave camera in the corresponding master-slave camera pair to solve the influence of camera occlusion or insufficient texture on positioning and improve the robustness of SLAM positioning. Otherwise, perform inter-frame feature point matching on the adjacent two frames of original environment images collected by each master camera in each master-slave camera pair to save time and computing power for feature matching and ensure the real-time performance of SLAM positioning. In addition, by performing epipolar correction on the original environment images collected by the master camera and the slave camera in each master-slave camera pair at the same time, the scale difference and rotation difference between the images are eliminated, so that when performing inter-frame feature point matching based on the target environment image after epipolar correction, more intra-frame matching point pairs can be obtained to improve the accuracy of SLAM positioning.

[0089] In the positioning method of a virtual reality device provided in an embodiment of the present application, positioning is performed with the help of at least one group of master-slave camera pairs on the virtual reality device. Considering that the orientation differences of each camera are relatively large, there will be scale differences and rotation differences between images. Therefore, epipolar correction is performed on the original environment images continuously acquired by each group of master-slave camera pairs, thereby eliminating the scale differences and rotation differences between feature points in the images acquired by the master camera and the slave camera at the same time, increasing the number of intra-frame matching point pairs, and improving the positioning accuracy of SLAM; at the same time, considering that matching between multiple cameras will increase the time and computing power of feature matching, for this reason, the embodiment of the present application only performs inter-frame feature point matching on the original environment images acquired by each master camera in each group of master-slave camera pairs, and performs intra-frame feature point matching on the images acquired by the same group of master cameras and slave cameras, thereby improving the real-time performance of SLAM positioning. On the other hand, since only the main camera is matched with feature points between frames, when the main camera is blocked or insufficient texture is collected, feature point tracking will fail. To this end, the embodiment of the present application determines whether the main camera is blocked or insufficient texture is collected by determining the number of feature points in the original environment image collected by the main camera, and switches the main camera and slave camera in the same group when it is blocked or insufficient texture is collected, thereby overcoming the problem of inaccurate positioning caused by insufficient or failed feature point tracking and improving the robustness of SLAM positioning.

[0090] Based on the same technical concept, an embodiment of the present application provides a virtual reality device, which can implement the steps of the positioning method of the above-mentioned virtual reality device and achieve the same technical effect.

[0091] See also Fig.11 , the virtual reality device includes a processor 1101, a memory 1102 and at least one set of master-slave camera pairs 1103, the master-slave camera pairs 1103, the memory 1102 and the processor 1101 are connected via a bus 1104;

[0092] The memory 1102 may be a volatile memory, such as a random-access memory (RAM); the memory 1102 may also be a non-volatile memory, such as a read-only memory, a flash memory, a hard disk drive (HDD) or a solid-state drive (SSD); or the memory 1102 may be any other medium that can be used to carry or store a desired computer program in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory 1102 may be a combination of the above memories;

[0093] The processor 1101 may include one or more central processing units (CPU) or a digital processing unit, etc.;

[0094] The master-slave camera 1103 pair is used to collect the original environment image in real time;

[0095] The memory 1102 stores a computer program, and the processor 1101 performs the following operations according to the computer program:

[0096] Acquire the original environment image captured in real time by the master-slave camera pair 1103, and perform epipolar correction on the two original environment images synchronously captured by the master camera and the slave camera in the master-slave camera pair 1103 to reduce the scale difference and rotation difference between the environment images, so as to obtain the target environment image;

[0097] Extracting feature points from the original environment image captured by the master camera in the master-slave camera pair 1103, and if the number of the extracted feature points is less than a preset number threshold, switching the slave camera in the master-slave camera pair 1103 to the master camera;

[0098] Performing inter-frame matching on feature points extracted from two adjacent original environment images captured by the master camera in the master-slave camera pair 1103 to obtain inter-frame matching points;

[0099] Determine feature points of the target environment image of the master camera and the slave camera in the master-slave camera pair 1103 at the same time according to the inter-frame matching points, and obtain intra-frame matching point pairs;

[0100] According to the matching point pairs within the frame, combined with the pre-calibrated intrinsic parameters and extrinsic parameters of the master camera and the slave camera in the master-slave camera pair 1103, the position and posture of the virtual reality device at the current moment is determined.

[0101] Optionally, the processor 1101 performs epipolar correction on two original environment images synchronously captured by the master camera and the slave camera in the master-slave camera pair 1103 to reduce the scale difference and rotation difference between the environment images to obtain the target environment image. The specific operation is:

[0102] Determine a rotation angle between the master camera and the slave camera in the master-slave camera pair according to pre-calibrated extrinsic parameters between the master camera and the slave camera in the master-slave camera pair 1103;

[0103] According to the rotation angle, the Euler rotation theorem is used to convert the original environment image captured by the main camera and the original environment image captured by the slave camera into images captured at parallel angles to reduce the scale difference and rotation difference between the environment images, and obtain two target environment images.

[0104] Optionally, the epipolar correction formula of the original environment image captured by the main camera is expressed as:

[0105]

[0106] The epipolar correction formula of the original environment image collected from the camera is expressed as:

[0107]

[0108] Among them, -θ / 2 is the angle of rotation of the main camera around the rotation axis, θ / 2 is the angle of rotation of the slave camera around the rotation axis, and π src (·) and π tgt (·) Projection transformation of the camera before and after rotation, p src and p tgt represents the original environment image and the target environment image of the main camera before and after epipolar correction, p src′ and p tgt′ Represents the original environment image and the target environment image from the camera before and after epipolar correction.

[0109] Optionally, the processor 1101 determines the feature points of the master camera and the slave camera in the master-slave camera pair 1103 in the target environment image at the same time according to the inter-frame matching points, and obtains the intra-frame matching point pairs. The specific operation is:

[0110] Mapping the inter-frame matching points of the master camera in the master-slave camera pair 1103 in the original environment image to the target environment image corresponding to the master camera after epipolar correction to obtain mapping points of the inter-frame matching points in the target environment image;

[0111] The mapping points in the target environment image corresponding to the main camera are matched with the feature points of the target environment image corresponding to the slave camera at the same time to obtain the intra-frame matching point pairs.

[0112] Optionally, after switching the slave camera in the master-slave camera pair 1103 to the master camera, the processor 1101 further executes:

[0113] If the number of feature points extracted from the original environment image captured by the switched main camera is less than a preset number threshold, the switched main camera is restored to the slave camera, and at the same time, the switched slave camera is restored to the main camera.

[0114] It should be noted that Fig.11 This is just an example, and provides the necessary hardware for the virtual reality device to execute the steps of the positioning method of the virtual reality device provided in the embodiment of the present application. If not shown, the virtual reality device may also include conventional hardware such as a speaker, a pickup, a display, and a power supply.

[0115] The embodiment of the present application also provides a computer-readable storage medium for storing some instructions, which, when executed, can complete a positioning method for a virtual reality device in the aforementioned embodiment.

[0116] An embodiment of the present application also provides a computer program product for storing a computer program, wherein the computer program is used to execute a positioning method for a virtual reality device in the aforementioned embodiment.

[0117] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0118] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0119] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0120] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0121] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A positioning method for a virtual reality device, characterized in that: The virtual reality device includes at least one master-slave camera pair, including: Acquire the original environment image captured by the master-slave camera pair in real time, and perform epipolar correction on the two original environment images synchronously captured by the master camera and the slave camera in the master-slave camera pair to reduce the scale difference and rotation difference between the environment images, so as to obtain the target environment image; Extracting feature points from an original environment image captured by a master camera in the master-slave camera pair, and if the number of the extracted feature points is less than a preset number threshold, switching the slave camera in the master-slave camera pair to the master camera; Performing inter-frame matching on feature points extracted from two adjacent original environment images captured by a master camera in the master-slave camera pair to obtain inter-frame matching points; Determine, according to the inter-frame matching points, feature points in the target environment image of the master camera and the slave camera in the master-slave camera pair at the same time, and obtain intra-frame matching point pairs; The position and posture of the virtual reality device at the current moment are determined according to the intra-frame matching point pairs and in combination with the pre-calibrated intrinsic parameters and extrinsic parameters of the master camera and the slave camera in the master-slave camera pair.

2. The method according to claim 1, characterized in that The step of performing epipolar correction on two original environment images synchronously captured by a master camera and a slave camera in the master-slave camera pair to reduce scale differences and rotation differences between the environment images to obtain a target environment image includes: Determining a rotation angle between a master camera and a slave camera in the master-slave camera pair according to pre-calibrated extrinsic parameters between a master camera and a slave camera in the master-slave camera pair; According to the rotation angle, the original environment image captured by the main camera and the original environment image captured by the slave camera are converted into images captured at parallel angles using the Euler rotation theorem to reduce the scale difference and rotation difference between the environment images, thereby obtaining two target environment images.

3. The method according to claim 2, characterized in that The epipolar correction formula of the original environment image collected by the main camera is expressed as: The epipolar correction formula of the original environment image collected from the camera is expressed as: Wherein, -θ / 2 is the angle of rotation of the main camera around the rotation axis, θ / 2 is the angle of rotation of the slave camera around the rotation axis, π src (·) and π tgt (·) Projection transformation of the camera before and after rotation, p src and p tgt represents the original environment image and the target environment image of the main camera before and after epipolar correction, p src′ and p tgt′ Represents the original environment image and the target environment image from the camera before and after epipolar correction.

4. The method according to claim 1, characterized in that The step of determining feature points of a master camera and a slave camera in the master-slave camera pair in the target environment image at the same time according to the inter-frame matching points to obtain intra-frame matching point pairs includes: Mapping the inter-frame matching points of the master camera in the master-slave camera pair in the original environment image to the target environment image corresponding to the master camera after epipolar correction to obtain the mapping points of the inter-frame matching points in the target environment image; The mapping points in the target environment image corresponding to the main camera are matched with the target environment image corresponding to the slave camera at the same time to obtain intra-frame matching point pairs.

5. The method according to claim 1, characterized in that After switching the slave camera in the master-slave camera pair to the master camera, the method further includes: If the number of feature points extracted from the original environment image captured by the switched main camera is less than a preset number threshold, the switched main camera is restored to the slave camera, and at the same time, the switched slave camera is restored to the main camera.

6. A virtual reality device, characterized in that: comprising a processor, a memory and at least one master-slave camera pair; The master-slave camera pair is used to collect original environment images in real time; The memory stores a computer program, and the processor performs the following operations according to the computer program: Acquire the original environment image captured by the master-slave camera pair in real time, and perform epipolar correction on the two original environment images synchronously captured by the master camera and the slave camera in the master-slave camera pair to reduce the scale difference and rotation difference between the environment images, so as to obtain the target environment image; Extracting feature points from an original environment image captured by a master camera in the master-slave camera pair, and if the number of the extracted feature points is less than a preset number threshold, switching the slave camera in the master-slave camera pair to the master camera; Performing inter-frame matching on feature points extracted from two adjacent original environment images captured by a master camera in the master-slave camera pair to obtain inter-frame matching points; Determine, according to the inter-frame matching points, feature points in the target environment image of the master camera and the slave camera in the master-slave camera pair at the same time, and obtain intra-frame matching point pairs; The position and posture of the virtual reality device at the current moment are determined according to the intra-frame matching point pairs and in combination with the pre-calibrated intrinsic parameters and extrinsic parameters of the master camera and the slave camera in the master-slave camera pair.

7. The virtual reality device according to claim 6, characterized in that: The processor performs epipolar correction on two original environment images synchronously captured by the master camera and the slave camera in the master-slave camera pair to reduce the scale difference and rotation difference between the environment images to obtain the target environment image, and the specific operation is: Determining a rotation angle between a master camera and a slave camera in the master-slave camera pair according to pre-calibrated extrinsic parameters between a master camera and a slave camera in the master-slave camera pair; According to the rotation angle, the original environment image captured by the main camera and the original environment image captured by the slave camera are converted into images captured at parallel angles using the Euler rotation theorem to reduce the scale difference and rotation difference between the environment images, thereby obtaining two target environment images.

8. The virtual reality device according to claim 7, characterized in that: The epipolar correction formula of the original environment image collected by the main camera is expressed as: The epipolar correction formula of the original environment image collected from the camera is expressed as: Wherein, -θ / 2 is the angle of rotation of the main camera around the rotation axis, θ / 2 is the angle of rotation of the slave camera around the rotation axis, π src (·) and π tgt (·) Projection transformation of the camera before and after rotation, p src and p tgt represents the original environment image and the target environment image of the main camera before and after epipolar correction, p src′ and p tgt′ Represents the original environment image and the target environment image from the camera before and after epipolar correction.

9. The virtual reality device according to claim 6, characterized in that: The processor determines the feature points of the master camera and the slave camera in the master-slave camera pair in the target environment image at the same time according to the inter-frame matching points, and obtains the intra-frame matching point pairs, and the specific operation is: Mapping the inter-frame matching points of the master camera in the master-slave camera pair in the original environment image to the target environment image corresponding to the master camera after epipolar correction to obtain the mapping points of the inter-frame matching points in the target environment image; The mapping points in the target environment image corresponding to the main camera are matched with the target environment image corresponding to the slave camera at the same time to obtain intra-frame matching point pairs.

10. The virtual reality device according to claim 6, characterized in that: After switching the slave camera in the master-slave camera pair to the master camera, the processor further executes: If the number of feature points extracted from the original environment image captured by the switched main camera is less than a preset number threshold, the switched main camera is restored to the slave camera, and at the same time, the switched slave camera is restored to the main camera.

Images