Handle positioning method and virtual reality equipment

Abstract

The invention relates to the technical field of VR, provides a handle positioning method and virtual reality equipment, and is used for improving the positioning precision and robustness of a handle. According to the method, the handle is improved, and a plurality of infrared LED lamps are replaced by an infrared curve light band, so that when the handle is closer to a camera of the virtual reality equipment, the light band is continuously encircled, the camera can be ensured to collect an infrared handle image containing the light band, the tracking range of the handle is effectively expanded, and the tracking accuracy of the handle is improved. The influence of a strip-shaped light source and a strip-shaped light source in the environment is reduced, a sufficient number of 2D central point sets are extracted by utilizing a plurality of contour points for an infrared curve light band, and the target six-degree-of-freedom pose of the handle relative to the virtual reality equipment can be stably and robustly solved by combining a preset 3D central point set and camera internal parameters; and accurate interaction is carried out based on the accurate target six-degree-of-freedom pose, so that the immersive experience of the user is improved.

Description

Technical Field

[0001] The present application relates to the field of virtual reality (VR) technology and provides a handle positioning method and a virtual reality device. Background Art

[0002] VR technology has been widely used in various industries by bringing users an immersive experience. During the experience, users can interact with VR devices through head control and handle control. Among them, the control of the handle to the VR device is as important as the control of the mouse to the PC.

[0003] Natural and smooth interaction is an important indicator to measure the user's satisfaction with the immersive experience, and accurate and robust handle positioning is a key technology to ensure natural and smooth interaction. At present, the handle is usually equipped with multiple infrared LED lights that light up in a flashing manner. When the LED lights are on, the camera on the VR device can collect multiple light spots. Each light spot has a unique ID, and its 3D coordinates in the handle coordinate system are pre-designed and fixed. Combined with the 2D coordinates of each light spot in the image collected by the camera, the PnP algorithm can be used to calculate the position of the handle, thereby completing the positioning of the handle.

[0004] However, the handle is constantly moving during the interaction. When the handle is close to the VR device, the LED light spots may not be captured or the number of LED light spots captured may be less than three. At this time, the handle tracking may fail or the effect may be poor, thereby affecting the calculation results of the handle posture and reducing the user's immersive experience. Summary of the invention

[0005] The embodiments of the present application provide a handle positioning method and a virtual reality device, which are used to improve the accuracy and robustness of handle positioning.

[0006] On the one hand, an embodiment of the present application provides a handle positioning method, which is applied to a virtual reality device, wherein the virtual reality device interacts with a handle, and the handle is equipped with an infrared curved light strip, and the method includes:

[0007] Acquire an infrared image of the handle captured by the camera, and detect an infrared curve light band in the infrared image of the handle;

[0008] According to a plurality of contour points on the infrared curve light band, a 2D center point set of the infrared curve light band is obtained;

[0009] According to the pre-calibrated camera internal parameters, the preset 3D center point set of the infrared curved light band and the 2D center point set, through at least one round of iteration, the target 6-DOF position of the handle relative to the virtual reality device when a preset convergence condition is satisfied is calculated;

[0010] According to the target 6-DOF posture, a three-dimensional model of the handle is displayed in a virtual screen for interaction.

[0011] Optionally, acquiring a 2D center point set of the infrared curve light band according to a plurality of contour points on the infrared curve light band includes:

[0012] For each contour point on the infrared curve light band, perform the following operations:

[0013] Determine a first straight line according to the contour point and the adjacent points of the contour point;

[0014] Obtaining the intersection of a second straight line passing through the contour point and perpendicular to the first straight line with the contour of the infrared curve light band, and determining a third straight line according to the intersection and adjacent points of the intersection;

[0015] Draw a circle passing through the contour point and inscribed in the first straight line and the third straight line respectively, and count the number of contour points contained in the circle;

[0016] If the number of contour points is less than a preset threshold, the center of the circle is taken as a 2D center point.

[0017] Optionally, the calculating, through at least one round of iterations, a target 6-DOF pose of the handle relative to the virtual reality device when a preset convergence condition is satisfied based on pre-calibrated camera internal parameters, a preset 3D center point set of the infrared curve light band, and the 2D center point set, includes:

[0018] Based on a preset initial 6-DOF posture and the camera intrinsic parameters, projecting the preset 3D center point set to obtain a 2D projection point set;

[0019] Matching the 2D projection point set with the 2D center point set to obtain a plurality of initial matching point pairs;

[0020] Calculating an initial reprojection error based on the multiple initial matching point pairs;

[0021] According to the initial reprojection error, the initial 6-DOF posture is optimized until the optimized 6-DOF posture meets the preset convergence condition, thereby obtaining a target 6-DOF posture.

[0022] Optionally, the formula for the reprojection error is expressed as:

[0023]

[0024] Among them, W is the rotation matrix and translation matrix, K is the matching point pair, and are the 2D center point and 2D projection point with the smallest distance, and L is the reprojection error represented by the sum of the squared distances of the matching point pairs.

[0025] Optionally, the infrared curved light band has at least one of the following characteristics:

[0026] The shape of the infrared curved light band is wavy;

[0027] The curve distribution of the infrared curve light band has no periodicity;

[0028] The difference between the number of inflection points in the infrared curve light band and the number of infrared LED lights in other handles is less than a preset difference.

[0029] On the other hand, an embodiment of the present application provides a virtual reality device, including a processor, a memory, a display, a camera, and a communication interface, wherein the communication interface, the camera, the display, the memory, and the processor are connected via a bus;

[0030] The communication interface is used to interact with a handle equipped with an infrared curved light strip;

[0031] The memory stores a computer program, and the processor performs the following operations according to the computer program:

[0032] Acquire an infrared image of the handle captured by the camera, and detect an infrared curve light band in the infrared image of the handle;

[0033] According to a plurality of contour points on the infrared curve light band, a 2D center point set of the infrared curve light band is obtained;

[0034] According to the pre-calibrated camera internal parameters, the preset 3D center point set of the infrared curved light band and the 2D center point set, through at least one round of iteration, the target 6-DOF position of the handle relative to the virtual reality device when a preset convergence condition is satisfied is calculated;

[0035] According to the target 6-DOF posture, a three-dimensional model of the handle is displayed in a virtual picture displayed on the display for interaction.

[0036] Optionally, the processor obtains a 2D center point set of the infrared curve light band according to a plurality of contour points on the infrared curve light band, and the specific operation is:

[0037] For each contour point on the infrared curve light band, perform the following operations:

[0038] Determine a first straight line according to the contour point and the adjacent points of the contour point;

[0039] Obtaining the intersection of a second straight line passing through the contour point and perpendicular to the first straight line with the contour of the infrared curve light band, and determining a third straight line according to the intersection and adjacent points of the intersection;

[0040] Draw a circle passing through the contour point and inscribed in the first straight line and the third straight line respectively, and count the number of contour points contained in the circle;

[0041] If the number of contour points is less than a preset threshold, the center of the circle is taken as a 2D center point.

[0042] Optionally, the processor calculates, through at least one round of iteration, a target 6-DOF pose of the handle relative to the virtual reality device when a preset convergence condition is satisfied according to pre-calibrated camera internal parameters, a preset 3D center point set of the infrared curve light band, and the 2D center point set, and the specific operation is:

[0043] Based on a preset initial 6-DOF posture and the camera intrinsic parameters, projecting the preset 3D center point set to obtain a 2D projection point set;

[0044] Matching the 2D projection point set with the 2D center point set to obtain a plurality of initial matching point pairs;

[0045] Calculating an initial reprojection error based on the multiple initial matching point pairs;

[0046] According to the initial reprojection error, the initial 6-DOF posture is optimized until the optimized 6-DOF posture meets the preset convergence condition, thereby obtaining a target 6-DOF posture.

[0047] Optionally, the formula for the reprojection error is expressed as:

[0048]

[0049] Among them, W is the rotation matrix and translation matrix, K is the matching point pair, and are the 2D center point and 2D projection point with the smallest distance, and L is the reprojection error represented by the sum of the squared distances of the matching point pairs.

[0050] Optionally, the infrared curved light band has at least one of the following characteristics:

[0051] The shape of the infrared curved light band is wavy;

[0052] The curve distribution of the infrared curve light band has no periodicity;

[0053] The difference between the number of inflection points in the infrared curve light band and the number of infrared LED lights in other handles is less than a preset difference.

[0054] On the other hand, an embodiment of the present application provides a computer-readable storage medium, which stores computer-executable instructions, and the computer-executable instructions are used to enable a computer device to execute the steps of a handle positioning method provided in an embodiment of the present application.

[0055] The beneficial effects of the handle positioning method provided by the embodiment of the present application are as follows:

[0056] The infrared curved light strip is used to replace the multiple infrared LED lights on the handle, thereby effectively expanding the tracking range of the handle, and can eliminate the interference of background lights (such as point-shaped or long strip-shaped lights), thereby improving the positioning accuracy of the handle. When positioning the handle containing the infrared curved light strip, the infrared handle image captured by the camera of the virtual reality device is used to obtain the 2D center point set of the infrared curved light strip based on the multiple contour points on the detected infrared curved light strip, and the 6-degree-of-freedom posture of the handle is calculated in combination with the preset 3D center point set camera parameters, thereby avoiding the tracking problem of insufficient or no 2D points collected when the handle is close to the camera, improving the robustness and accuracy of the handle positioning, and then ensuring the accuracy of the interaction between the handle and the virtual reality device, which helps to enhance the user's immersive experience.

[0057] Other features and advantages of the present application will be described in the following description, and partly become apparent from the description, or understood by practicing the present application. The purpose and other advantages of the present application can be realized and obtained by the structures specifically pointed out in the written description, claims, and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0059] Figure 1 A schematic diagram of the handle button structure provided in an embodiment of the present application;

[0060] Figure 2A A schematic diagram of the RGB image of the handle collected under visible light is provided for the embodiment of the present application;

[0061] Figure 2B A schematic diagram of an infrared image of a handle collected under infrared light is provided for an embodiment of the present application;

[0062] Figure 3 A schematic diagram of a situation where the number of collected LED light spots is insufficient for the embodiment of the present application is provided;

[0063] Figure 4 This is a schematic diagram of an improved handle according to an embodiment of the present application;

[0064] Figure 5 A schematic plan view of an infrared curved light band on a handle provided in an embodiment of the present application;

[0065] Figure 6 A flowchart of a handle positioning method provided in an embodiment of the present application;

[0066] Figure 7 This is a detection effect diagram of the infrared curve light band provided in the embodiment of the present application;

[0067] Figure 8 A flow chart of a method for extracting a 2D center point in an infrared curve light band provided in an embodiment of the present application;

[0068] Fig. 9 A schematic diagram of a method for extracting a 2D center point in an infrared curve light band provided in an embodiment of the present application;

[0069] Fig.10 A schematic diagram of a 2D center point set in an infrared curve light band provided in an embodiment of the present application;

[0070] Fig.11 A flow chart of a method for optimizing a 6-DOF posture provided in an embodiment of the present application;

[0071] Fig.12 A schematic diagram of matching the 2D projection point of a 3D center point with the 2D center point provided in an embodiment of the present application;

[0072] Fig.13 A rendering of a three-dimensional model of a handle displayed in a virtual screen provided in an embodiment of the present application;

[0073] Fig.14 A structural diagram of a virtual reality device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0074] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the technical solution of the present application, rather than all of the embodiments. Based on the embodiments recorded in the application documents, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the technical solution of the present application.

[0075] The handle is to the VR helmet as the mouse is to the PC, and is an important input device. The handle can be used as a mouse to perform common operations such as clicking and dragging on the virtual screen displayed by the VR device. It can also perform operations that a traditional mouse cannot provide. This is because the mouse only provides 2-degree-of-freedom movement operation information on the plane, while the handle can provide 6-degree-of-freedom movement and rotation operation information in three-dimensional space, such as waving, tapping, turning and other actions. It is an important control device in virtual interactive scenes.

[0076] Accurate and robust 6-DOF pose calculation of the controller is the key to ensuring natural and smooth interaction. Usually, the controller is equipped with multiple infrared LED lights, and the VR device is equipped with multiple cameras, such as Figure 1 As shown, computer vision algorithms can be used to locate and track the handle.

[0077] In order to reduce the power consumption of the handle, multiple infrared LED lights are usually lit in a flashing manner, such as Figure 2A As shown, this is the RGB image of the handle collected under visible light, such as Figure 2B As shown in the figure, the LED light is turned on to collect the infrared image of the handle. Among them, each light spot in the infrared image of the handle has a unique ID, and its 3D coordinates in the handle coordinate system are pre-designed and fixed. Combined with the 2D coordinates of each light spot in the infrared image of the handle, the PnP algorithm can be used to calculate the position and posture of the handle, thereby completing the positioning of the handle.

[0078] However, the handle is constantly moving during the interaction, so the number of light spots in the infrared image of the handle captured by the camera will also change. When the handle is close to the VR device, the LED light spots may not be captured or the number of LED light spots captured may be less than three, such as Figure 3 As shown, (a) is an infrared image of the handle with two LED spots captured, (b) is an infrared image of the handle with one LED spot captured, and (c) is an infrared image of the handle without any LED spot captured.

[0079] At present, when using the PnP algorithm to calculate the 6-DOF posture of the handle, if there are only two LED spots in the collected infrared image of the handle, the positioning effect is poor and the handle is prone to shaking during the interaction. If there is only one LED spot or no LED spot, based on the principle of the PnP algorithm, the 6-DOF posture of the handle cannot be calculated, and therefore interaction is impossible.

[0080] In view of this, the embodiment of the present application improves the structure of the handle and replaces the infrared LED array with an infrared curved light band, thereby effectively expanding the tracking range of the handle and being able to eliminate background interference such as lights (such as point or long strip lights), and based on the improved handle, a handle positioning method is provided. The method uses geometric knowledge to generate an inscribed circle based on multiple contour points on the infrared curved light band detected in the infrared image of the handle collected by the camera, and uses the center of the inscribed circle as the 2D center point set for the 6-degree-of-freedom pose calculation, thereby solving the impact of the insufficient number of light spots on the 6-degree-of-freedom pose calculation, and then combining the preset 3D center point set and the calibrated camera internal parameters to accurately and robustly solve the rotation matrix and translation matrix of the handle relative to the virtual reality device, complete the handle positioning, improve the interaction quality between the handle and the virtual reality device, and thereby enhance the user's immersive experience.

[0081] like Figure 4 As shown, it is a schematic diagram of the improved handle of the embodiment of the present application. A circle of infrared curved light band is installed on the handle ring to replace the array composed of multiple infrared LED lights on the ordinary handle.

[0082] In order to eliminate background interference such as lights (such as point-shaped or long strip lights) during positioning and expand the tracking range of the handle, the infrared curved light band on the handle has at least the following characteristics:

[0083] (1) The shape of the infrared curved light band is wavy;

[0084] (2) The curve distribution of the infrared curve light band has no periodicity;

[0085] (3) The difference between the number of inflection points in the infrared curve light band and the number of infrared LED lights in other handles is less than a preset difference.

[0086] As an example, the effect of unfolding the infrared curved light band on the handle ring onto a plane is as follows: Figure 5 As shown, its mathematical representation is:

[0087] y=0.8sin(6πx 2 sign(x))+0.2cos(6πx 2 ) Formula 1

[0088] It should be noted that Figure 5 This is only an example and is not intended to be a geometric limitation of the infrared curve light band.

[0089] Since the infrared curved light band is continuously wrapped around the handle ring, during the interaction process, when the handle is close to the virtual reality device, the camera can still capture the infrared image of the handle containing the light band, thereby effectively expanding the tracking range of the handle and reducing the interference of point light sources or strip light sources in the environment on the light band, thereby improving the accuracy and robustness of handle positioning.

[0090] Based on the improved handle, the process of the handle positioning method provided in the embodiment of the present application is as follows: Figure 6 As shown in the figure, the process is executed by the virtual reality device and mainly includes the following steps:

[0091] S601: Acquire an infrared image of the handle captured by a camera, and detect infrared curve light bands in the infrared image of the handle.

[0092] In one example, in order to reduce the interference of the environment on the light band and obtain an infrared image of the handle under a dark background, the camera on the virtual reality device can be set to capture the infrared image of the handle in a short exposure mode and capture the environmental image under a bright background in a long exposure mode, wherein the infrared image of the handle is used to calculate the 6-degree-of-freedom posture of the handle, and the environmental image is used to calculate its own 6-degree-of-freedom posture, and the exposure time of the environmental image is greater than the exposure time of the infrared image of the handle.

[0093] The light band detection algorithm in the infrared image of the handle may adopt a traditional image processing algorithm or an artificial intelligence (AI) algorithm.

[0094] Taking AI algorithm as an example, light band detection belongs to target detection task, which can be implemented by using classic YOLO, Single Shot MultiBox Detector (SSD) and other neural networks. The range of the detected light band in the infrared image of the handle can be framed by a rectangle, such as Figure 7 As shown in (a) in the figure, binary conversion and contour detection are then performed within this range to select the largest wavy area as the detection result of the infrared curve light band, as shown in Figure 7 As shown in (b) in .

[0095] S602: Acquire a 2D center point set of the infrared curve light band according to a plurality of contour points on the infrared curve light band.

[0096] The infrared curve light band is continuous and has a certain width. However, the PnP algorithm generally uses discrete points when calculating the 6-DOF posture. Therefore, the detected infrared curve light band needs to be discretized. For the specific process, see Figure 8 , mainly includes the following steps:

[0097] S602 1: Determine a first straight line according to the contour point and the adjacent points of the contour point.

[0098] Take a contour point A as an example, Fig. 9 As shown, a first straight line La can be determined by combining an adjacent point of the contour point A on the contour of the infrared curve light band.

[0099] S6022: Obtain the intersection of a second straight line passing through the contour point and perpendicular to the first straight line with the contour of the infrared curve light band, and determine a third straight line based on the intersection and adjacent points of the intersection.

[0100] like Fig. 9 As shown, a second straight line perpendicular to La is made through the contour point A. Since the infrared curve light band is closed, there will be another intersection point B between the second straight line and the contour of the infrared curve light band. Similarly, a third straight line Lb can be determined by combining the adjacent points of the intersection point B on the contour.

[0101] S6023: Draw a circle passing through the contour point and inscribed in the first straight line and the third straight line respectively, and count the number of contour points contained in the circle.

[0102] like Fig. 9 As shown, a circle is made which passes through the contour point A and is inscribed in the first straight line La and the third straight line Lb respectively, and the number of contour points contained in the circle is counted.

[0103] S6024: Determine whether the number of contour points is less than a preset number threshold, if so, execute S6025, otherwise, obtain the next contour point and return to S6021 for recalculation.

[0104] S6025: Take the center of the circle as a 2D center point.

[0105] When the number of contour points contained in the circle is less than a preset number threshold, the center P of the circle is used as a discrete 2D center point of the infrared curve light band, otherwise, the next contour point is obtained to re-determine the 2D center point.

[0106] S6026: Obtain a 2D center point set based on each 2D center point.

[0107] like Fig.10 , which is a schematic diagram of a determined 2D center point set of an infrared curve light band, including a plurality of tracked discrete 2D points.

[0108] Optionally, when the number of 2D center points in the 2D center point set reaches a preset threshold, or after a sufficient number of contour points are traversed, the above iteration is stopped.

[0109] S603 calculates the target 6-DOF posture of the handle relative to the virtual reality device when a preset convergence condition is met through at least one round of iteration based on the pre-calibrated camera internal parameters, the preset 3D center point set and the 2D center point set of the infrared curve light band.

[0110] In the process of calculating the 6-DOF posture of the handle, positioning can be performed by solving a nonlinear optimization problem so that the projection curve of the infrared curve light band curve in the 3D space on the infrared image of the handle coincides as much as possible with the curve represented by the detected 2D center point set.

[0111] When solving nonlinear optimization problems, some discrete 3D center points can be pre-set for the infrared curve light band on the handle to obtain a 3D center point set. The 3D coordinates of these 3D center points in the handle coordinate system are known and are recorded as m is the number of 3D center points, and the projection point of the 3D center point set on the image collected by the camera is recorded as The 2D coordinates of n discrete 2D center points in the 2D center point set in the image are also known, denoted as The 6-DOF pose of the handle can be represented by a 4*3 matrix W=[R|t], where R represents a 3*3 three-dimensional rotation matrix, t represents a 3*1 translation matrix, and the matrix of the camera internal parameters is expressed as Among them, f x 、f y is the focal length of the camera, c x 、c y is the optical center position of the camera, in pixels. Assuming that the camera is a pinhole camera without distortion, the projection relationship from the 3D point to the 2D point can be expressed as:

[0112]

[0113] Where s is the scalar coefficient.

[0114] Based on the above projection relationship, the 2D projection point of the 3D center point on the infrared curve light band is made to coincide with the 2D center point as much as possible to minimize the reprojection error, thereby obtaining the optimal 6-DOF pose [R|t]. For the specific solution process, see Fig.11 , mainly includes the following steps:

[0115] S6031: Based on the preset initial 6-DOF posture and camera intrinsic parameters, project the preset 3D center point set to obtain a 2D projection point set.

[0116] In one example, given an initial 6-DOF pose W0, combined with the camera intrinsic parameters, and projected using Formula 2, a 2D projection point set can be obtained.

[0117] S6032: Match the 2D projection point set with the 2D center point set to obtain multiple initial matching point pairs.

[0118] The matching process between 2D points is as follows Fig.12 As shown, assuming that the 2D projection point set 2D center point set Point pairs before matching Traverse each 2D center point in the 2D center point set P Find the closest 2D projection point from the 2D projection point set to form a matching point pair. The formula is:

[0119]

[0120] in,

[0121] After traversing all 2D center points, multiple initial matching point pairs can be obtained.

[0122] It should be noted that during the matching process, the 2D projection point set can also be traversed.

[0123] S6033: Calculate an initial reprojection error based on multiple initial matching point pairs.

[0124] In one example, the degree of overlap between the 2D projection point and the 2D center point can be represented by the reprojection error L:

[0125]

[0126] Among them, W is the rotation matrix and translation matrix representing the 6-DOF posture, K is the matching point pair, and are the 2D center point and 2D projection point with the smallest distance, L is the square of the distance between the matching point pair, and is used to characterize the reprojection error L. The smaller L is, the higher the degree of overlap between the 2D projection point and the 2D center point, and the more accurate the pose calculation is.

[0127] S6034: Optimize the initial 6-DOF pose based on the initial reprojection error.

[0128] In one example, the optimization method may use a gradient descent method, which is expressed as:

[0129]

[0130] Where λ is the step size coefficient, λ∈(0,1), W k+1 is the 6-DOF pose after the k-th round of update.

[0131] S6035: Recalculate the reprojection error according to the optimized 6-DOF posture until the reprojection error meets the preset convergence condition to obtain the target 6-DOF posture.

[0132] According to the optimized 6-DOF pose, the 2D projection points of the m 3D center points are recalculated using formula 2, and new matching point pairs are regenerated with the n 2D center points. Based on the new matching point pairs, the reprojection error is recalculated to optimize the 6-DOF pose until the preset convergence condition |W is met. k+1 -W k |<δor k≥k max , and obtain the target 6-DOF pose. Among them, δ is the preset difference, k max The preset maximum number of iterations.

[0133] S604: Displaying a three-dimensional model of the handle in a virtual screen for interaction according to the target 6-DOF posture.

[0134] After obtaining the target 6-DOF posture of the handle relative to the virtual reality device, the virtual reality device can determine the target position of the handle in the virtual screen of the VR application based on the target 6-DOF posture, thereby displaying the 3D model of the handle at the target position to achieve interaction between the handle and the virtual screen.

[0135] like Fig.13 As shown, a rendering of a three-dimensional model of a handle is displayed in a virtual image seen by the left and right eyes, and the user can interact by operating the handle.

[0136] In the handle positioning method provided in the embodiment of the present application, an infrared curved light band is used to replace the multiple infrared LED lights on the handle. Since the infrared curved light band is continuously surrounded by the handle, during the interaction, when the handle is close to the virtual reality device, the camera can still collect the infrared image of the handle containing the light band, and generate a 2D center point set in the image through an inscribed circle based on the multiple contour points on the infrared curved light band, thereby effectively expanding the tracking range of the handle, and can reduce the interference of point light sources or strip light sources in the environment on the light band, thereby improving the accuracy and robustness of handle positioning.

[0137] Based on the same technical concept, an embodiment of the present application provides a virtual reality device, which can implement the steps of the above-mentioned handle positioning method and achieve the same technical effect.

[0138] See also Fig.14 , the virtual reality device includes a processor 1401, a memory 1402, a display 1403, a camera 1404 and a communication interface 1405, and the communication interface 1405, the camera 1404, the display 1403, the memory 1402 and the processor 1401 are connected via a bus 1406;

[0139] The communication interface 1405 is used to interact with the handle equipped with the infrared curved light strip;

[0140] The memory 1402 stores a computer program, and the processor 1401 performs the following operations according to the computer program:

[0141] Acquire the infrared image of the handle captured by the camera 1404, and detect the infrared curve light band in the infrared image of the handle;

[0142] According to a plurality of contour points on the infrared curve light band, a 2D center point set of the infrared curve light band is obtained;

[0143] According to the pre-calibrated camera internal parameters, the preset 3D center point set and the 2D center point set of the infrared curve light band, through at least one round of iteration, the target 6-DOF position of the handle relative to the virtual reality device when the preset convergence condition is met is calculated;

[0144] According to the target 6-DOF posture, a three-dimensional model of the handle is displayed in the virtual screen displayed by the display 1403 for interaction.

[0145] Optionally, the processor obtains a 2D center point set of the infrared curve light band according to multiple contour points on the infrared curve light band, and the specific operation is:

[0146] For each contour point on the infrared curve band, do the following:

[0147] Determine a first straight line according to the contour point and the adjacent points of the contour point;

[0148] Obtain the intersection of a second straight line passing through the contour point and perpendicular to the first straight line with the contour of the infrared curve light band, and determine a third straight line based on the intersection and adjacent points of the intersection;

[0149] Draw a circle passing through the contour point and inscribed in the first and third straight lines respectively, and count the number of contour points contained in the circle;

[0150] If the number of contour points is less than the preset threshold, the center of the circle is taken as a 2D center point.

[0151] Optionally, the processor calculates the target 6-DOF position of the handle relative to the virtual reality device when a preset convergence condition is met through at least one round of iteration according to the pre-calibrated camera internal parameters, the preset 3D center point set and the 2D center point set of the infrared curve light band, and the specific operation is:

[0152] Based on the preset initial 6-DOF pose and camera internal parameters, the preset 3D center point set is projected to obtain a 2D projection point set;

[0153] Match the 2D projection point set with the 2D center point set to obtain multiple initial matching point pairs;

[0154] Calculate the initial reprojection error based on multiple initial matching point pairs;

[0155] According to the initial reprojection error, the initial 6-DOF pose is optimized until the optimized 6-DOF pose meets the preset convergence condition and the target 6-DOF pose is obtained.

[0156] Alternatively, the reprojection error is formulated as:

[0157]

[0158] Among them, W is the rotation matrix and translation matrix, K is the matching point pair, and are the 2D center point and 2D projection point with the smallest distance, and L is the reprojection error represented by the sum of the squared distances of the matching point pairs.

[0159] Optionally, the infrared curved light strip has at least one of the following characteristics:

[0160] The shape of the infrared curved light band is wavy;

[0161] The curve distribution of the infrared curve light band has no periodicity;

[0162] The difference between the number of inflection points in the infrared curve light band and the number of infrared LED lights in other handles is less than a preset difference.

[0163] in, Fig.14 The memory 1402 in the embodiment may be a volatile memory, such as a random-access memory (RAM); the memory 1402 may also be a non-volatile memory, such as a read-only memory, a flash memory, a hard disk drive (HDD) or a solid-state drive (SSD); or the memory 1402 may be any other medium that can be used to carry or store a desired computer program in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory may be a combination of the above memories;

[0164] The processor 1401 may include one or more central processing units (CPU) or a digital processing unit, etc.

[0165] It should be noted that Fig.14This is just an example, and provides the necessary hardware for the virtual reality device to execute the steps of a handle positioning method provided in an embodiment of the present application. If not shown, the virtual reality device may also include conventional hardware such as a speaker, a pickup, left and right eyeglasses, a power supply, and an IMU.

[0166] The handle provided in the embodiment of the present application uses an infrared curved light band to replace multiple infrared LED lights, thereby effectively expanding the tracking range of the handle, and can eliminate background interference such as lights (such as point-shaped or long strip-shaped lights), thereby improving the positioning accuracy of the handle. When the virtual reality device locates the handle containing the infrared curved light band, the 2D center point set of the infrared curved light band is obtained based on the multiple contour points on the detected infrared curved light band for the infrared handle image captured by the camera of the virtual reality device, and the 6-DOF posture of the handle is calculated in combination with the preset 3D center point set camera parameters, thereby avoiding the tracking problem of insufficient or no 2D points collected when the handle is close to the camera, improving the robustness and accuracy of the handle positioning, and then ensuring the accuracy of the interaction between the handle and the virtual reality device, which helps to enhance the user's immersive experience.

[0167] An embodiment of the present application also provides a computer-readable storage medium for storing some instructions. When these instructions are executed, a handle positioning method in the aforementioned embodiment can be completed.

[0168] An embodiment of the present application also provides a computer program product for storing a computer program, wherein the computer program is used to execute a handle positioning method in the aforementioned embodiment.

[0169] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0170] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1A device that provides the functions specified in a block or multiple blocks.

[0171] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0172] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0173] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A handle positioning method, characterized in that: Applied to a virtual reality device, the virtual reality device interacts with a handle, the handle is equipped with an infrared curved light strip, and the method includes: Acquire an infrared image of the handle captured by the camera, and detect an infrared curve light band in the infrared image of the handle; According to a plurality of contour points on the infrared curve light band, a 2D center point set of the infrared curve light band is obtained; According to the pre-calibrated camera internal parameters, the preset 3D center point set of the infrared curved light band and the 2D center point set, through at least one round of iteration, the target 6-DOF position of the handle relative to the virtual reality device when a preset convergence condition is satisfied is calculated; According to the target 6-DOF posture, a three-dimensional model of the handle is displayed in a virtual screen for interaction.

2. The method according to claim 1, characterized in that The step of acquiring a 2D center point set of the infrared curve light band according to a plurality of contour points on the infrared curve light band comprises: For each contour point on the infrared curve light band, perform the following operations: Determine a first straight line according to the contour point and the adjacent points of the contour point; Obtaining the intersection of a second straight line passing through the contour point and perpendicular to the first straight line with the contour of the infrared curve light band, and determining a third straight line according to the intersection and adjacent points of the intersection; Draw a circle passing through the contour point and inscribed in the first straight line and the third straight line respectively, and count the number of contour points contained in the circle; If the number of contour points is less than a preset threshold, the center of the circle is taken as a 2D center point.

3. The method according to claim 1, characterized in that The method of calculating, based on the pre-calibrated camera internal parameters, the preset 3D center point set of the infrared curve light band, and the 2D center point set, a target 6-DOF position of the handle relative to the virtual reality device when a preset convergence condition is satisfied through at least one round of iterations includes: Based on a preset initial 6-DOF posture and the camera intrinsic parameters, projecting the preset 3D center point set to obtain a 2D projection point set; Matching the 2D projection point set with the 2D center point set to obtain a plurality of initial matching point pairs; Calculating an initial reprojection error based on the multiple initial matching point pairs; According to the initial reprojection error, the initial 6-DOF posture is optimized until the optimized 6-DOF posture meets the preset convergence condition, thereby obtaining a target 6-DOF posture.

4. The method according to claim 3, characterized in that The formula of the reprojection error is expressed as: Among them, W is the rotation matrix and translation matrix, K is the matching point pair, and are the 2D center point and 2D projection point with the smallest distance, and L is the reprojection error represented by the sum of the squared distances of the matching point pairs.

5. The method according to any one of claims 1 to 4, characterized in that The infrared curved light band has at least one of the following characteristics: The shape of the infrared curved light band is wavy; The curve distribution of the infrared curve light band has no periodicity; The difference between the number of inflection points in the infrared curve light band and the number of infrared LED lights in other handles is less than a preset difference.

6. A virtual reality device, characterized in that: It includes a processor, a memory, a display, a camera and a communication interface, wherein the communication interface, the camera, the display, the memory and the processor are connected via a bus; The communication interface is used to interact with a handle equipped with an infrared curved light strip; The memory stores a computer program, and the processor performs the following operations according to the computer program: Acquire an infrared image of the handle captured by the camera, and detect an infrared curve light band in the infrared image of the handle; According to a plurality of contour points on the infrared curve light band, a 2D center point set of the infrared curve light band is obtained; According to the pre-calibrated camera internal parameters, the preset 3D center point set of the infrared curved light band and the 2D center point set, through at least one round of iteration, the target 6-DOF position of the handle relative to the virtual reality device when a preset convergence condition is satisfied is calculated; According to the target 6-DOF posture, a three-dimensional model of the handle is displayed in a virtual picture displayed on the display for interaction.

7. The virtual reality device according to claim 6, characterized in that: The processor obtains a 2D center point set of the infrared curve light band according to a plurality of contour points on the infrared curve light band, and the specific operation is: For each contour point on the infrared curve light band, perform the following operations: Determine a first straight line according to the contour point and the adjacent points of the contour point; Obtaining the intersection of a second straight line passing through the contour point and perpendicular to the first straight line with the contour of the infrared curve light band, and determining a third straight line according to the intersection and adjacent points of the intersection; Draw a circle passing through the contour point and inscribed in the first straight line and the third straight line respectively, and count the number of contour points contained in the circle; If the number of contour points is less than a preset threshold, the center of the circle is taken as a 2D center point.

8. The virtual reality device according to claim 6, characterized in that: The processor calculates, through at least one round of iteration, a target 6-DOF position of the handle relative to the virtual reality device when a preset convergence condition is satisfied based on pre-calibrated camera internal parameters, a preset 3D center point set of the infrared curve light band, and the 2D center point set. The specific operation is: Based on a preset initial 6-DOF posture and the camera intrinsic parameters, projecting the preset 3D center point set to obtain a 2D projection point set; Matching the 2D projection point set with the 2D center point set to obtain a plurality of initial matching point pairs; Calculating an initial reprojection error based on the multiple initial matching point pairs; According to the initial reprojection error, the initial 6-DOF posture is optimized until the optimized 6-DOF posture meets the preset convergence condition, thereby obtaining a target 6-DOF posture.

9. The virtual reality device according to claim 8, characterized in that: The formula of the reprojection error is expressed as: Among them, W is the rotation matrix and translation matrix, K is the matching point pair, and are the 2D center point and 2D projection point with the smallest distance, and L is the reprojection error represented by the sum of the squared distances of the matching point pairs.

10. The virtual reality device according to any one of claims 6 to 9, characterized in that: The infrared curved light band has at least one of the following characteristics: The shape of the infrared curved light band is wavy; The curve distribution of the infrared curve light band has no periodicity; The difference between the number of inflection points in the infrared curve light band and the number of infrared LED lights in other handles is less than a preset difference.

Images