Positioning tracking method and device, VR head-mounted device and computer storage medium

Abstract

The application discloses a positioning tracking method and device, a VR head-mounted device and a computer readable storage medium, and is applied to positioning and tracking of a handle by a VR head-mounted device provided with an image acquisition device, the handle being provided with multiple light sources, and the method comprises the following steps: taking a first target image containing multiple light spots by the image acquisition device; determining a target abnormal light spot corresponding to ambient light in the first target image, and removing the target abnormal light spot to obtain a second target image; determining a graphic feature corresponding to each light spot in the second target image, and determining a target camera pose of the image acquisition device based on the graphic feature and the plane coordinates of each light spot; calculating a handle coordinate of the handle according to the target camera pose, and positioning and tracking the handle based on the handle coordinate. The technical effect that the VR head-mounted device can exclude the interference of ambient light when performing optical positioning operation is achieved, so that the accuracy of positioning the handle by the VR head-mounted device is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of virtual reality, in particular to a positioning tracking method and device, a VR head-mounted device and a computer readable storage medium. BACKGROUND

[0002] With the rapid development of virtual reality technology, the accuracy and refresh frequency of VR head-mounted devices for handle positioning tracking are also increasingly high. At present, VR head-mounted devices mainly use the method of fusing optical positioning and IMU (Inertial Measurement Unit) data to improve positioning accuracy and position refresh frequency. When the VR head-mounted device uses the optical positioning method, the camera inside the VR head-mounted device has a shooting frequency of 30fps to 90fps, which causes the problem of device cost expansion if a higher shooting frequency is required. Similarly, when the VR head-mounted device uses IMU data to predict the handle pose, the high refresh frequency of the IMU will continuously accumulate errors, causing the handle to drift over time. Therefore, the VR head-mounted device needs to correct the errors of the IMU data through optical positioning.

[0003] However, the optical positioning method requires multiple infrared light beads to be installed on the handle and an infrared camera to capture pictures of the handle to obtain images, and then complete optical positioning according to the light spots in the images. As a result, the infrared camera is likely to be disturbed by ambient light when capturing pictures, which leads to chaotic light spots in the images captured by the infrared camera and ultimately inaccurate optical positioning. SUMMARY

[0004] The embodiments of the present application provide a positioning tracking method, device, VR head-mounted device and computer readable storage medium, which aims to exclude the interference of ambient light when the VR head-mounted device performs optical positioning operation, thereby improving the accuracy of positioning the handle of the VR head-mounted device.

[0005] The present application provides a positioning tracking method, which is applied to a VR head-mounted device configured with an image acquisition device for positioning tracking of a handle, the handle being configured with multiple light sources, and the positioning tracking method comprising the following steps:

[0006] Taking a first target image containing multiple light spots by the image acquisition device, wherein each light spot is generated by the invisible light and ambient light emitted by each light source on the handle;

[0007] Determining a target abnormal light spot corresponding to the ambient light in the first target image, and removing the target abnormal light spot to obtain a second target image;

[0008] determine a pattern feature corresponding to each of the light spots in the second target image, and determine a target camera pose of the image acquisition device based on the pattern feature and the plane coordinates of each of the light spots;

[0009] calculate a handle coordinate of the handle according to the target camera pose, and track and locate the handle based on the handle coordinate.

[0010] Further, the step of determining the target abnormal light spot corresponding to the ambient light in the first target image comprises:

[0011] determine distance values between a plurality of the light spots in the first target image, and construct a target distance matrix based on the distance values.

[0012] determine a target abnormal light spot from the light spots according to the target distance matrix.

[0013] Further, the step of determining a target abnormal light spot from the light spots according to the target distance matrix comprises:

[0014] determine a minimum non-zero distance value of each row of the target distance matrix, and compare the minimum non-zero distance value with a preset first threshold to obtain a first comparison result;

[0015] when the first comparison result is that the minimum non-zero distance value is less than the first threshold, determine the light spot corresponding to the minimum non-zero distance value as the target abnormal light spot;

[0016] and / or,

[0017] determine a maximum distance value of each row of the target distance matrix, and compare the maximum distance value with a preset second threshold to obtain a second comparison result;

[0018] when the second comparison result is that the maximum distance value is greater than the second threshold, determine the light spot corresponding to the maximum distance value as the target abnormal light spot.

[0019] Further, the step of determining a target abnormal light spot from the light spots according to the target distance matrix further comprises:

[0020] determine a matrix distance mean of the target distance matrix, and a sub-matrix distance mean of each row of the target distance matrix;

[0021] determine a target abnormal light spot based on the matrix distance mean and the sub-matrix distance mean.

[0022] Further, the step of determining a pattern feature corresponding to each of the light spots in the second target image comprises:

[0023] determining slope values between the light spots in the second target image;

[0024] determining a pattern feature corresponding to each of the light spots based on the slope values and the plane coordinates.

[0025] Further, the step of determining the target camera pose of the image acquisition device based on the pattern feature and the plane coordinates of each of the light spots comprises:

[0026] determining light source identification data corresponding to each of the light spots in the second target image based on the pattern feature, and determining spatial coordinates of each of the light sources according to a preset light source arrangement rule and the light source identification data;

[0027] determining each of the to-be-verified camera poses of the image acquisition device according to the plane coordinates and the spatial coordinates, and determining the target camera pose among the to-be-verified camera poses.

[0028] Further, the step of determining the target camera pose among the to-be-verified camera poses comprises:

[0029] determining a rotation degree and a displacement value of each of the to-be-verified camera poses;

[0030] projecting each of the to-be-verified camera poses according to the rotation degree and the displacement value to obtain plane camera coordinates corresponding to each of the to-be-verified camera poses;

[0031] matching the plane camera coordinates to determine target plane camera coordinates, and determining the to-be-verified camera pose corresponding to the target plane camera coordinates as the target camera pose.

[0032] In addition, to achieve the above object, the present application further provides a positioning and tracking device, which is applied to a VR head-mounted device configured with an image acquisition device for positioning and tracking a handle, the handle being configured with a plurality of light sources, and the device comprising:

[0033] a camera module configured to capture a first target image containing a plurality of light spots through the image acquisition device, wherein each of the light spots is generated by invisible light and ambient light emitted by each of the light sources on the handle;

[0034] a noise reduction module configured to determine a target abnormal light spot corresponding to the ambient light in the first target image, and remove the target abnormal light spot to obtain a second target image;

[0035] a calculation module configured to determine a pattern feature corresponding to each of the light spots in the second target image, and determine a target camera pose of the image acquisition device based on the pattern feature and the plane coordinates of each of the light spots;

[0036] a tracking module configured to calculate a handle coordinate of the handle according to the target camera pose, and track the position of the handle based on the handle coordinate.

[0037] In addition, to achieve the above object, the present application also provides a VR head-mounted device, which comprises a memory, a processor, and a position tracking program stored in the memory and executable on the processor, and the position tracking program implements the steps of the position tracking method when executed by the processor.

[0038] In addition, to achieve the above object, the present application also provides a computer readable storage medium, which stores a position tracking program, and the position tracking program implements the steps of the position tracking method when executed by a processor.

[0039] The position tracking method, device, VR head-mounted device and computer readable storage medium provided by the embodiments of the present application are applied to the position tracking of a handle by a VR head-mounted device configured with an image acquisition device, the handle is configured with a plurality of light sources, and a first target image containing a plurality of light spots is taken by the image acquisition device; wherein each of the light spots is generated by the invisible light and the ambient light emitted by each of the light sources on the handle; a target abnormal light spot corresponding to the ambient light is determined in the first target image, and the target abnormal light spot is removed to obtain a second target image; a pattern feature corresponding to each of the light spots in the second target image is determined, and a target camera pose of the image acquisition device is determined based on the pattern feature and the plane coordinates of each of the light spots; a handle coordinate of the handle is calculated according to the target camera pose, and the position of the handle is tracked based on the handle coordinate.

[0040] In this embodiment, when the VR headset is running, first, the built-in image acquisition device is called to capture a first target image of a plurality of light spots generated by the invisible light emitted by the plurality of light sources on the handle and the ambient light, then the VR headset inputs the first target image into the image processing device configured in the VR headset, the image processing device determines each target abnormal light spot generated by the ambient light in the first target image, and removes each target abnormal light spot to obtain a second target image, then the image processing device determines the graphical features corresponding to each light spot in the second target image, and further determines the target camera pose of the VR headset relative to the handle based on the graphical features and the plane coordinates of each light spot in the image, finally, the VR headset determines the positional relationship between the image acquisition device and the handle according to the target camera pose, and calculates the handle coordinates of the handle based on the positional relationship and the target camera pose, and further tracks and locates the handle based on the handle coordinates.

[0041] Thus, the present application adopts the method of acquiring a first target image containing a plurality of light spots by an image acquisition device, determining each target abnormal light spot generated by the ambient light in the first target image, and further removing each target abnormal light spot to obtain a second target image, then determining the graphical features of each light spot in the second target image and determining the camera pose according to the graphical features and the plane coordinates of each light spot, and finally determining the handle coordinates based on the camera pose and tracking and locating the handle, that is, removing the light spots generated by the ambient light in the collected first target image to exclude the interference of the ambient light, then determining the image features corresponding to each light spot in the second target image, and determining the positional relationship between the handle and the image acquisition device according to the image features and the plane coordinates of each light spot, and finally realizing tracking and locating the handle based on the positional relationship, thereby realizing the technical effect that the VR headset can exclude the interference of the ambient light when performing optical positioning operation, and further improving the accuracy of positioning the handle by the VR headset. BRIEF DESCRIPTION OF DRAWINGS

[0042] Figure 1 is a schematic diagram of the hardware running environment of the VR headset involved in the embodiment scheme of the present application;

[0043] Figure 2 is a flowchart of the positioning and tracking method of the first embodiment of the present application;

[0044] Figure 3 is a schematic diagram of the arrangement of infrared lamp beads involved in an embodiment of the positioning and tracking method of the present application;

[0045] Figure 4 is a schematic diagram of the handle structure involved in an embodiment of the positioning and tracking method of the present application;

[0046] Figure 5 Fig. 1 is a schematic diagram of a pose calculation process involved in an embodiment of the positioning and tracking method of the present application;

[0047] Figure 6 Fig. 2 is a schematic diagram of a VR headset structure involved in an embodiment of the positioning and tracking method of the present application;

[0048] Figure 7 Fig. 3 is a schematic diagram of functional modules involved in an embodiment of the positioning and tracking method of the present application.

[0049] The implementation, functional features and advantages of the present application will be further described with reference to the embodiments and the accompanying drawings. DETAILED DESCRIPTION

[0050] It should be understood that the specific embodiments described herein merely exemplify the present application and do not limit the present application.

[0051] Please refer to Figure 1 , Figure 1 Fig. 4 is a schematic diagram of a hardware operating environment of a VR headset involved in an embodiment of the present application.

[0052] The VR headset involved in the embodiments of the present application can be a mobile VR headset or a fixed VR headset configured with an HMD headset and a handle, and the mobile VR headset or the fixed VR headset is composed of the HMD headset and the matching handle.

[0053] As shown in Figure 1 Fig. 4, the VR headset can include a processor 1001 such as a central processing unit (CPU), a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. The communication bus 1002 is used to realize the connection and communication between these components. The user interface 1003 can include a display, an input unit such as a keyboard, and can also include a standard wired interface, a wireless interface. The network interface 1004 can optionally include a standard wired interface, a wireless interface (such as a wireless fidelity (WIreless-FIdelity, WI-FI) interface). The memory 1005 can be a high-speed random access memory (RAM) memory, or a stable non-volatile memory (Non-Volatile Memory, NVM) such as a disk memory. The memory 1005 can also be a storage device independent of the aforementioned processor 1001.

[0054] Those skilled in the art can understand that Figure 1The structure shown in the figure does not constitute a limitation on the VR headset, and can include more or fewer components than shown, or combine certain components, or different component arrangements.

[0055] As shown in Figure 1 The memory 1005 as a storage medium can include an operating system, a data storage module, a network communication module, a user interface module, and a positioning tracking program.

[0056] In the VR headset shown in Figure 1 In the VR headset shown in

[0057] Based on the above VR headset, each embodiment of the positioning tracking method of the present application is provided.

[0058] Please refer to Figure 6 , Figure 6 The VR headset structure diagram related to an embodiment of the positioning tracking method of the present application is shown in

[0059] In this embodiment, the VR headset is composed of an HMD headset and a handle, wherein the HMD headset is configured with an HMD self-tracking module, a camera module, an IMU module, a handle control and transmission module, and a handle tracking module. Similarly, the handle is configured with a plurality of wireless transmission modules, an IMU module, and an infrared LED module, and the infrared LED module should be configured with a plurality of light source devices for emitting invisible light.

[0060] Further, based on the VR headset structure diagram related to the above embodiment of the positioning tracking method of the present application, the first embodiment of the positioning tracking method of the present application is proposed, please refer to Figure 2 , Figure 2 The flowchart of the first embodiment of the positioning tracking method of the present application is shown in

[0061] It should be understood that although the logical order is shown in the flowchart, in some cases, the positioning tracking method of the present application can of course also execute the steps shown or described in an order different from here.

[0062] In this embodiment, the positioning tracking method of the present application can include the following steps:

[0063] Step S10: taking a first target image containing a plurality of light spots by the image acquisition device; wherein the plurality of light spots are generated by the invisible light emitted by each of the light sources on the handle and the ambient light;

[0064] In the embodiment, the image acquisition device is an infrared camera using LED infrared diode as the main material. The setting position of the infrared camera can refer to the setting position of the image acquisition device in other same type of VR head-mounted device. The present application does not limit this. Similarly, the light source is an infrared lamp bead, and the light source should be arranged on the handle matched with the VR head-mounted device. Please refer to Figure 3 , Figure 3 The present application discloses a positioning tracking method. The positioning tracking method is related to the arrangement diagram of the infrared lamp bead. Two rows of infrared lamp beads can be arranged on the handle, and each infrared lamp bead is unevenly distributed with the two ends concentrated and the middle scattered. In addition, each infrared lamp bead in the second row is cross-distributed between two lamp beads in the first row, forming a triangular arrangement to form a light belt. The light belt composed of a plurality of infrared lamp beads can be arranged at the front end of the handle. Please refer to Figure 4 , Figure 4 The present application discloses a positioning tracking method. The positioning tracking method is related to the arrangement diagram of the infrared lamp bead. Two rows of infrared lamp beads can be arranged on the handle, and each infrared lamp bead is unevenly distributed with the two ends concentrated and the middle scattered. In addition, each infrared lamp bead in the second row is cross-distributed between two lamp beads in the first row, forming a triangular arrangement to form a light belt. The light belt composed of a plurality of infrared lamp beads can be arranged at the front end of the handle. Please refer to

[0065] When the VR head-mounted device is running, the image acquisition device arranged in the VR head-mounted device controls the plurality of light sources on the handle to emit light for image acquisition, so as to obtain a first target image containing a plurality of light spots generated by the invisible light emitted by the plurality of light sources and the ambient light.

[0066] For example, when the VR head-mounted device is running, a plurality of infrared lamp beads arranged on the handle matched with the VR head-mounted device emit infrared light respectively. At the same time, the VR head-mounted device controls the built-in infrared camera to take a single frame image containing a plurality of light spots generated by the infrared light emitted by each of the infrared lamp beads and the ambient light in the environment around the VR head-mounted device, and determines the single frame image as the first target image.

[0067] Step S20: determining a target abnormal light spot corresponding to the ambient light in the first target image, and removing the target abnormal light spot to obtain a second target image;

[0068] In the embodiment, the VR head-mounted device extracts each light spot in the first target image and determines the planar coordinates of the center of each light spot through an internally configured image processing device, which further determines target abnormal light spots in each light spot due to the ambient light according to the planar coordinates of each light spot and removes each target abnormal light spot in the first target image to obtain a second target image containing only light spots generated by invisible light emitted by each light source.

[0069] For example, the VR head-mounted device inputs the acquired first target image to the image processing device, which controls the opencv tool installed by the technician to perform grayscale processing on the first target image data. Then, the opencv tool samples the first target image using an image pyramid to remove small bright spots in the first target image. After that, the opencv tool performs binary processing on the first target image by using a preset threshold value and traverses the contour area contained in the first target image to determine whether the contour area meets the geometric characteristics of a circle and an ellipse. Then, the opencv tool performs minimum rotating rectangle on the contour area meeting the geometric characteristics, and further excludes noise points according to whether the length, width, and area meet the threshold value. Meanwhile, the opencv tool extracts the 2D planar coordinates of the remaining light spot centers on the first target image, determines target abnormal light spots in each light spot contained in the first target image according to the 2D planar coordinates, and removes each target abnormal light spot in the first target image to obtain a second target image.

[0070] Further, in a feasible embodiment, the step of "determining target abnormal light spots corresponding to the ambient light in the first target image" in step S20 can specifically include:

[0071] Step S201: determining the distance values between a plurality of light spots in the first target image, and constructing a target distance matrix based on the distance values;

[0072] Step S202: determining target abnormal light spots in each light spot according to the target distance matrix;

[0073] Exemplarily, for example, after the VR head-mounted device controls the above-mentioned opencv tool to obtain the 2D plane coordinates of the centers of the light spots in the above-mentioned first target image, the VR head-mounted device controls the above-mentioned image processing apparatus to further determine the distance values between the light spots in the first target image based on the 2D plane coordinates of the centers of the light spots, and then constructs a target distance matrix according to the distance values. After that, the VR head-mounted device inputs the target distance matrix into a data processing apparatus arranged in the VR head-mounted device, and the data processing apparatus processes the target distance matrix to determine each target abnormal light spot in the light spots, and removes each target abnormal light spot to obtain a second target image.

[0074] It can be understood that in the embodiment, the distance values between the light spots are the distance values between the respective centers of the light spots. Of course, the VR head-mounted device can also control the opencv tool to obtain the 2D plane coordinates of any position on the edges of the light spots, and then take the distance values between any positions on the edges of the light spots as the distance values between the light spots, and the present application does not limit this.

[0075] Further, in a feasible embodiment, the step of “determining target abnormal light spots in the light spots according to the target distance matrix” in the above-mentioned step S202 can specifically include:

[0076] Step S2021: determining the minimum non-0 distance value of each row of the target distance matrix, and comparing the minimum non-0 distance value with a preset first threshold to obtain a first comparison result;

[0077] Step S2022: when the first comparison result is that the minimum non-0 distance value is less than the first threshold, determining the light spot corresponding to the minimum non-0 distance value as a target abnormal light spot;

[0078] Step S2023: determining the maximum distance value of each row of the target distance matrix, and comparing the maximum distance value with a preset second threshold to obtain a second comparison result;

[0079] Step S2024: when the second comparison result is that the maximum distance value is greater than the second threshold, determining the light spot corresponding to the maximum distance value as the target abnormal light spot;

[0080] In the embodiment, the first threshold value and the second threshold value are maximum spot distance threshold value and minimum spot distance threshold value respectively, which are determined by the arrangement positions of the infrared lamp beads according to the spot distribution range after the arrangement positions of the infrared lamp beads are determined by the technician in advance, that is, only the spot between the first threshold value and the second threshold value can be considered as the spot formed by the infrared light emitted by the infrared lamp bead. Of course, there are various ways to obtain the specific numerical value and the first threshold value and the second threshold value, and the present application does not limit this.

[0081] For example, the VR head-mounted device calls the data processing apparatus to traverse the centers of the spots in the first target image to obtain the respective 2D plane coordinates of the spots, and then calculates the distance values between the spots according to the respective 2D plane coordinates. Meanwhile, the data processing apparatus constructs a symmetric matrix with zero diagonal as a target distance matrix based on the distance values. Then, the data processing apparatus extracts the minimum distance value other than 0 in each row of the target distance matrix, and compares the minimum distance value with the first threshold value preset by the developer to obtain a first comparison result. When the first comparison result is that the minimum distance value is less than the first threshold value, the data processing apparatus determines that the spot corresponding to the minimum distance value is a target abnormal spot. Similarly, the data processing apparatus extracts the maximum distance value in each row of the target distance matrix, and compares the maximum distance value with the second threshold value preset by the developer to obtain a second comparison result. When the second comparison result is that the maximum distance value is greater than the second threshold value, the data processing apparatus determines that the spot corresponding to the maximum distance value is a target abnormal spot.

[0082] Further, in a feasible embodiment, the step of determining the target abnormal spot in the step S202 can further include the following steps.

[0083] S2025: determining the matrix distance mean of the target distance matrix and the sub-matrix distance mean of each row of the target distance matrix;

[0084] S2026: determining the target abnormal spot based on the matrix distance mean and the sub-matrix distance mean;

[0085] Exemplarily, for example, after the VR head-mounted device calls the above-mentioned data processing apparatus to construct the above-mentioned target distance matrix, the data processing apparatus calculates the matrix distance mean U of the target distance matrix, at the same time, the data processing apparatus extracts each distance value in the target distance matrix except 0, and then obtains the respective sub-matrix distance mean Ui corresponding to each row, and then the data processing apparatus compares each sub-matrix distance mean Ui with the matrix distance mean U according to the preset calculation formula (Ui-U) / U to determine whether the distance mean of the i-th light spot to other light spots corresponding to the sub-matrix distance mean Ui is higher than the preset third threshold value, if the data processing apparatus determines that the distance mean of the i-th light spot to other light spots is higher than the third threshold value, the VR head-mounted device determines that the i-th light spot is a discrete noise point, and then determines the i-th light spot as the target abnormal light spot.

[0086] Further, in a feasible embodiment, the step of "determining the graphical features corresponding to each of the light spots in the second target image" in the above-mentioned step S20 can specifically include:

[0087] Step S203: determining the slope values between each of the light spots in the second target image;

[0088] Step S204: determining the graphical features corresponding to each of the light spots based on each of the slope values and each of the plane coordinates;

[0089] Exemplarily, for example, the VR head-mounted device selects the center points p1(u1, v1) and p2(u2, v2) of any two light spots in the above-mentioned second target image, and calculates the slope K between the two light spots based on the preset slope calculation formula:

[0090]

[0091] calculates the slope K between the two light spots, and then the VR head-mounted device judges whether the slope K value tends to 0, if the VR head-mounted device judges that the slope K value tends to 0, it is determined that the infrared lamp beads corresponding to the two light spots are the infrared lamp beads on the upper side or the lower side of the lamp strip composed of the infrared lamp beads, similarly, if the VR head-mounted device judges that the slope K value is greater than 0, it is determined that the infrared lamp beads corresponding to the two light spots are arranged in a diagonal direction, similarly, if the VR head-mounted device judges that the slope K value is less than 0, it is determined that the lamp beads corresponding to the two light spots are arranged in a diagonal direction, and then the VR head-mounted device first selects two light spots on the same side and adjacent to each other, and then determines the range of the horizontal coordinates of the two light spots according to the plane coordinates of each of the light spots, and then determines the light spots included in the range, and judges whether the image composed of three light spots is an upper triangle, a lower triangle or other related graphical features such as "V, ∧, W, M".

[0092] It should be noted that in the present embodiment, the graphic feature can be modified by the technician according to the arrangement order of the lamp beads, of course, the technician can also add other graphic features for comparison, and the present application does not limit this.

[0093] Step S30: determining the graphic feature corresponding to each of the light spots in the second target image, and determining the target camera pose of the image acquisition device based on the graphic feature and the plane coordinates of each of the light spots;

[0094] In the present embodiment, the graphic feature is the shape feature corresponding to the graphic formed between the light spots after the light spots in the image are sequentially linked in a certain order, and similarly, the plane coordinate data is the coordinate of the light spot in the second target image as the plane coordinate system, wherein the origin and direction axis of the plane coordinate system can be arbitrarily set, and the present application does not limit this.

[0095] Illustratively, for example, the VR headset inputs the acquired second target image to the above-mentioned image processing device, and the image processing device determines the graphic feature corresponding to each of the light spots contained in the second target image, and the VR headset further inputs the acquired graphic feature to the above-mentioned data processing device, and the data processing device determines the position of the image acquisition device relative to the handle based on the graphic feature and the acquired plane coordinates of each of the light spots, and further determines the target camera pose of the image acquisition device.

[0096] Further, in a feasible embodiment, the step of "determining the target camera pose of the image acquisition device based on the graphic feature and the plane coordinates of each of the light spots" in the above-mentioned step S30 can specifically include:

[0097] Step S301: determining the light source identification data corresponding to each of the light spots in the second target image based on the graphic feature, and determining the spatial coordinates of each of the light sources according to the predetermined light source arrangement rule and the light source identification data;

[0098] Step S302: determining each of the to-be-verified camera poses of the image acquisition device according to the plane coordinates and the spatial coordinates, and determining the target camera pose among the to-be-verified camera poses;

[0099] In the embodiment, the spatial coordinate data is the coordinate data of each infrared lamp bead in a spatial coordinate system constructed with the handle centroid as the origin. Of course, the spatial coordinate system can also be set with other reference objects as the origin, or the skilled person can directly use the world coordinate system as the spatial coordinate system. In addition, the spatial coordinates of each infrared lamp bead are obtained by the skilled person after determining the setting positions of each infrared lamp bead according to the handle structure, and are stored in the VR headset before the VR headset is shipped. Of course, there are many ways to obtain the spatial coordinate data, and the present application does not limit this.

[0100] For example, please refer to Figure 5 , Figure 5 The embodiment of the positioning tracking method of the present application relates to the pose calculation process. The VR headset inputs the graphical features corresponding to each of the above-mentioned light spots and the rotation data of the handle obtained by the built-in IMU sensor of the VR headset to the data processing device. The data processing device determines each target infrared lamp bead that meets the graphical features and the rotation data among the infrared lamp beads based on the graphical features and the rotation data, and determines the respective number identifiers of each target infrared lamp bead. Then, the VR headset reads the storage device to obtain the light source arrangement rule stored by the skilled person in advance. The VR headset inputs the light source arrangement rule to the data processing device. The data processing device determines the respective spatial coordinates of each target infrared lamp bead according to the light source arrangement rule. Then, the data processing device estimates the camera pose by the PnP algorithm set by the skilled person according to the obtained plane coordinates and spatial coordinates to obtain a plurality of to-be-confirmed camera poses. The data processing device further performs brute force matching on each to-be-confirmed camera pose to obtain a target camera pose.

[0101] Further, in a feasible embodiment, the step of "determining a target camera pose among the to-be-verified camera poses" in step S302 can specifically include:

[0102] Step S3021: determining the rotation degree and displacement value of each to-be-verified camera pose;

[0103] Step S3022: projecting each to-be-verified camera pose according to the rotation degree and displacement value to obtain the corresponding plane camera coordinates of each to-be-verified camera pose;

[0104] Step S3023: matching each plane camera coordinate to determine a target plane camera coordinate, and determining the to-be-verified camera pose corresponding to the target plane camera coordinate as a target camera pose;

[0105] Exemplarily, for example, when the VR head-mounted device determines the rotation value R and the displacement value t of each of the above-mentioned to-be-verified poses by the above-mentioned data processing apparatus, the VR head-mounted device first converts each of the to-be-confirmed camera poses from the world coordinate to the camera coordinate relative to the handle by the formula P c = RP w +t, and simultaneously, the VR head-mounted device inputs the intrinsic matrix of the above-mentioned infrared camera device to the data processing apparatus, projects the camera coordinate Pc(Xc, Yc, Zc) of each of the to-be-confirmed camera poses into the plane coordinate system corresponding to the second target image based on the intrinsic matrix by the data processing apparatus, and then determines the target to-be-verified camera pose with the minimum error value in each of the to-be-confirmed camera poses according to the plane coordinates of each of the light spots in the second target image and the plane coordinates of each of the to-be-confirmed camera poses, and determines the target to-be-verified camera pose as the above-mentioned target camera pose.

[0106] Step S40: calculating the handle coordinate of the handle according to the target camera pose, and positioning and tracking the handle based on the handle coordinate;

[0107] Exemplarily, for example, the VR head-mounted device determines the camera coordinate of the image acquisition device relative to the handle according to the acquired target camera pose, then determines the positional relationship between the image acquisition device and the handle according to the camera coordinate, then determines the handle coordinate of the handle relative to the image acquisition device according to the positional relationship, and positions and tracks the handle based on the handle coordinate.

[0108] In the embodiment, when the VR head-mounted device is running, first, the VR head-mounted device controls the image acquisition device arranged in the VR head-mounted device to acquire a first target image containing light spots generated by the invisible light emitted by the multiple light sources and the ambient light when the multiple light sources on the handle emit light, then the VR head-mounted device extracts each light spot in the first target image and determines the plane coordinates of the center of each light spot through the image processing device arranged in the VR head-mounted device, the image processing device further determines the target abnormal light spots formed by the ambient light in each light spot according to the plane coordinates of each light spot, and removes each target abnormal light spot in the first target image to obtain a second target image containing only the light spots generated by the invisible light emitted by each light source, then the VR head-mounted device inputs the acquired second target image into the image processing device to determine the graphical features corresponding to each light spot contained in the second target image, and further inputs the acquired graphical features into the data processing device to determine the position of the image acquisition device relative to the handle based on the graphical features and the plane coordinates of each light spot, and further determine the target camera pose of the image acquisition device, finally, the VR head-mounted device determines the camera coordinates of the image acquisition device relative to the handle based on the acquired target camera pose, and further determines the positional relationship between the image acquisition device and the handle based on the camera coordinates, and further determines the handle coordinates of the handle relative to the image acquisition device based on the positional relationship, and performs positioning tracking on the handle based on the handle coordinates.

[0109] Thus, the present application adopts the manner of acquiring a first target image containing multiple light spots by an image acquisition device, determining each target abnormal light spot generated by the ambient light in the first target image, removing each target abnormal light spot to obtain a second target image, determining the graphical features of each light spot in the second target image, determining the camera pose based on the graphical features and the plane coordinates of each light spot, and finally determining the handle coordinates based on the camera pose and performing positioning tracking on the handle, that is, removing the light spots generated by the ambient light in the acquired first target image to exclude the interference of the ambient light, further determining the image features corresponding to each light spot in the second target image, determining the positional relationship between the handle and the image acquisition device based on the image features and the plane coordinates of each light spot, and finally realizing the positioning tracking on the handle based on the positional relationship, so as to achieve the technical effect of enabling the VR head-mounted device to exclude the interference of the ambient light when performing the optical positioning operation, thereby improving the accuracy of positioning the handle by the VR head-mounted device.

[0110] In addition, the present application further provides a positioning tracking device applied to a VR head-mounted device arranged with an image acquisition device to perform positioning tracking on a handle, the handle is arranged with multiple light sources, please refer to Figure 7 , Figure 7An embodiment of the positioning tracking method of the present application relates to a functional module schematic diagram as shown in the figure Figure 7 The positioning tracking device of the present application comprises:

[0111] The camera module 10 is used to take a first target image containing a plurality of light spots through the image acquisition device; wherein each light spot is generated by the invisible light and the ambient light emitted by each light source on the handle;

[0112] The noise reduction module 20 is used to determine a target abnormal light spot corresponding to the ambient light in the first target image, and remove the target abnormal light spot to obtain a second target image;

[0113] The calculation module 30 is used to determine the graphical features corresponding to each light spot in the second target image, and determine the target camera pose of the image acquisition device based on the graphical features and the plane coordinates of each light spot;

[0114] The tracking module 40 is used to calculate the handle coordinates of the handle according to the target camera pose, and perform positioning tracking on the handle based on the handle coordinates.

[0115] Further, the noise reduction module 20 comprises:

[0116] The matrix construction unit is used to determine the distance values between a plurality of light spots in the first target image, and construct a target distance matrix based on each distance value;

[0117] The light spot screening unit is used to determine a target abnormal light spot in each light spot according to the target distance matrix.

[0118] Further, the light spot screening unit comprises:

[0119] The first comparison subunit is used to determine the minimum non-zero distance value of each row of the target distance matrix, and compare the minimum non-zero distance value with a preset first threshold to obtain a first comparison result;

[0120] The first screening subunit is used to determine the light spot corresponding to the minimum non-zero distance value as a target abnormal light spot when the first comparison result is that the minimum non-zero distance value is less than the first threshold;

[0121] The second comparison subunit is used to determine the maximum distance value of each row of the target distance matrix, and compare the maximum distance value with a preset second threshold to obtain a second comparison result;

[0122] The second screening subunit is configured to determine the spot corresponding to the maximum distance value as the target abnormal spot when the second comparison result is that the maximum distance value is greater than the second threshold value.

[0123] Further, the spot screening unit further comprises:

[0124] The average calculation subunit is configured to determine a matrix distance average of the target distance matrix and a respective submatrix distance average of each row of the target distance matrix.

[0125] The third comparison subunit is configured to determine a target abnormal spot based on the matrix distance average and the respective submatrix distance average.

[0126] Further, the calculation module 30 comprises:

[0127] The slope calculation unit is configured to determine a slope value between each of the spots in the second target image.

[0128] The feature determination unit is configured to determine a graphical feature corresponding to each of the spots based on the respective slope value and the respective plane coordinate.

[0129] Further, the calculation module 30 further comprises:

[0130] The coordinate determination unit is configured to determine light source identification data corresponding to each of the spots in the second target image based on the graphical feature, and determine a spatial coordinate of each of the light sources according to a preset light source arrangement rule and the light source identification data.

[0131] The pose calculation unit is configured to determine a to-be-verified camera pose of the image acquisition device according to each plane coordinate and each spatial coordinate, and determine a target camera pose from the to-be-verified camera poses.

[0132] Further, the pose calculation unit comprises:

[0133] The data extraction subunit is configured to determine a rotation degree and a displacement value of each of the to-be-verified camera poses.

[0134] The coordinate projection subunit is configured to project each of the to-be-verified camera poses according to the respective rotation degree and displacement value to obtain a plane camera coordinate corresponding to each of the to-be-verified camera poses.

[0135] The pose calculation subunit is configured to match each of the plane camera coordinates to determine a target plane camera coordinate, and determine the to-be-verified camera pose corresponding to the target plane camera coordinate as the target camera pose.

[0136] Further, the present application also provides a VR head-mounted device, which has a positioning tracking program capable of running on a processor, and the VR head-mounted device implements the steps of the positioning tracking method according to any one of the above embodiments when executing the positioning tracking program.

[0137] The specific embodiments of the VR head-mounted device of the present application are basically the same as the above-mentioned embodiments of the positioning tracking method, and thus are not described herein.

[0138] Further, the present application also provides a computer readable storage medium, which has a positioning tracking program stored thereon, and the positioning tracking program implements the steps of the positioning tracking method according to any one of the above embodiments when executed by a processor.

[0139] The specific embodiments of the computer readable storage medium of the present application are basically the same as the above-mentioned embodiments of the positioning tracking method, and thus are not described herein.

[0140] It should be noted that, in this document, the terms "comprising", "including", or any other variant thereof are intended to cover non-exclusive inclusion, so that processes, methods, articles, or systems that include a series of elements not only include those elements, but also include other elements that are not explicitly listed, or further include elements inherent in such processes, methods, articles, or systems. Without more limitations, the element defined by the statement "comprising a" does not exclude the presence of additional identical elements in the process, method, article, or system that includes the element.

[0141] The above-mentioned embodiment numbers of the present application are only for description, and do not represent the advantages or disadvantages of the embodiments.

[0142] From the above description of the embodiments, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be realized by means of software and necessary general hardware platforms, of course, they can also be realized by hardware, but in many cases, the former is a better embodiment. Based on such understanding, the technical solutions of the present application can be embodied in the form of a software product, which is stored in a storage medium (such as a ROM / RAM, a magnetic disk, an optical disk) and includes a plurality of instructions for causing a VR head-mounted device (which can be a mobile VR head-mounted device or a fixed VR head-mounted device configured with an HMD head-mounted device and a handle, and the mobile VR head-mounted device or the fixed VR head-mounted device is composed of an HMD head-mounted device and a matching handle, etc.) to execute the methods described in the embodiments of the present application.

[0143] The above merely describes the preferred embodiments of the present application, and is not intended to limit the patent scope of the present application, and any equivalent structure or equivalent process conversion, or direct or indirect application in other related technical fields, which are made by using the content of the present application specification and drawings, are also included in the patent protection scope of the present application.

Claims

1. A method of position tracking, characterized by, The positioning tracking method is applied to a VR head-mounted device configured with an image acquisition device for positioning tracking of a handle, the handle is configured with a plurality of light sources, and the positioning tracking method comprises the following steps: A first target image containing a plurality of light spots is captured by the image acquisition device; wherein the plurality of light spots are generated by the invisible light and ambient light emitted by each of the light sources on the handle; A target abnormal light spot corresponding to the ambient light is determined in the first target image, and the target abnormal light spot is removed to obtain a second target image; The corresponding graphical features of each light spot in the second target image are determined, and the target camera pose of the image acquisition device is determined based on the graphical features and the plane coordinates of each light spot; The handle coordinates of the handle are calculated according to the target camera pose, and the handle is positioned and tracked based on the handle coordinates; The step of determining the target abnormal light spot corresponding to the ambient light in the first target image comprises: The distance values between the plurality of light spots in the first target image are determined, and a target distance matrix is constructed based on the distance values; The target abnormal light spot in each light spot is determined according to the target distance matrix.

2. The positioning tracking method of claim 1, wherein, The step of determining the target abnormal light spot in each light spot according to the target distance matrix comprises: The minimum non-zero distance value of each row of the target distance matrix is determined, and the minimum non-zero distance value is compared with a preset first threshold to obtain a first comparison result; When the first comparison result is that the minimum non-zero distance value is less than the first threshold, the light spot corresponding to the minimum non-zero distance value is determined as the target abnormal light spot; And / or, The maximum distance value of each row of the target distance matrix is determined, and the maximum distance value is compared with a preset second threshold to obtain a second comparison result; When the second comparison result is that the maximum distance value is greater than the second threshold, the light spot corresponding to the maximum distance value is determined as the target abnormal light spot.

3. The positioning tracking method of claim 1, wherein, The step of determining the target abnormal light spot in each light spot according to the target distance matrix further comprises: The matrix distance mean of the target distance matrix and the sub-matrix distance mean of each row of the target distance matrix are determined; The target abnormal light spot is determined based on the matrix distance mean and the sub-matrix distance mean.

4. The position tracking method of claim 1, wherein, The step of determining the corresponding graphical features of each light spot in the second target image comprises: The slope values between each light spot in the second target image are determined; The graphical features corresponding to each light spot are determined based on the slope values and the plane coordinates.

5. The position tracking method of claim 1, wherein, The step of determining the target camera pose of the image acquisition device based on the graphical features and the plane coordinates of each light spot comprises: The light source identification data corresponding to each light spot in the second target image is determined based on the graphical features, and the spatial coordinates of each light source are determined according to the preset light source arrangement rule and the light source identification data. The positioning tracking method is applied to a VR head-mounted device configured with an image acquisition device for positioning tracking of a handle, the handle is configured with a plurality of light sources, and the positioning tracking method comprises the following steps: A first target image containing a plurality of light spots is captured by the image acquisition device; wherein the plurality of light spots are generated by the invisible light and ambient light emitted by each of the light sources on the handle; A target abnormal light spot corresponding to the ambient light is determined in the first target image, and the target abnormal light spot is removed to obtain a second target image; The corresponding graphical features of each light spot in the second target image are determined, and the target camera pose of the image acquisition device is determined based on the graphical features and the plane coordinates of each light spot; The handle coordinates of the handle are calculated according to the target camera pose, and the handle is positioned and tracked based on the handle coordinates; The step of determining the target abnormal light spot corresponding to the ambient light in the first target image comprises: The distance values between the plurality of light spots in the first target image are determined, and a target distance matrix is constructed based on the distance values; The target abnormal light spot in each light spot is determined according to the target distance matrix. The step of determining the target abnormal light spot in each light spot according to the target distance matrix comprises: The minimum non-zero distance value of each row of the target distance matrix is determined, and the minimum non-zero distance value is compared with a preset first threshold to obtain a first comparison result; When the first comparison result is that the minimum non-zero distance value is less than the first threshold, the light spot corresponding to the minimum non-zero distance value is determined as the target abnormal light spot; And / or, The maximum distance value of each row of the target distance matrix is determined, and the maximum distance value is compared with a preset second threshold to obtain a second comparison result; When the second comparison result is that the maximum distance value is greater than the second threshold, the light spot corresponding to the maximum distance value is determined as the target abnormal light spot. The step of determining the target abnormal light spot in each light spot according to the target distance matrix further comprises: The matrix distance mean of the target distance matrix and the sub-matrix distance mean of each row of the target distance matrix are determined; The target abnormal light spot is determined based on the matrix distance mean and the sub-matrix distance mean. The step of determining the corresponding graphical features of each light spot in the second target image comprises: The slope values between each light spot in the second target image are determined; The graphical features corresponding to each light spot are determined based on the slope values and the plane coordinates. The step of determining the target camera pose of the image acquisition device based on the graphical features and the plane coordinates of each light spot comprises: The light source identification data corresponding to each light spot in the second target image is determined based on the graphical features, and the spatial coordinates of each light source are determined according to the preset light source arrangement rule and the light source identification data. Determine each to-be-verified camera pose of the image acquisition device according to each plane coordinate and each space coordinate, and determine a target camera pose in each to-be-verified camera pose.

6. The position tracking method of claim 5, wherein, The step of determining a target camera pose in each to-be-verified camera pose comprises: Determining a rotation degree and a displacement value of each to-be-verified camera pose; Projecting each to-be-verified camera pose according to the rotation degree and the displacement value to obtain a corresponding plane camera coordinate of each to-be-verified camera pose; Matching each plane camera coordinate to determine a target plane camera coordinate, and determining the to-be-verified camera pose corresponding to the target plane camera coordinate as a target camera pose.

7. A position tracking device, characterized by The device is applied to a VR head-mounted device configured with an image acquisition device for positioning tracking of a handle, the handle is configured with a plurality of light sources, and the device comprises: A camera module configured to capture a first target image containing a plurality of light spots through the image acquisition device, wherein each light spot is generated by invisible light and ambient light emitted by each light source on the handle; A noise reduction module configured to determine a target abnormal light spot corresponding to the ambient light in the first target image, and remove the target abnormal light spot to obtain a second target image; A calculation module configured to determine a graphical feature corresponding to each light spot in the second target image, and determine a target camera pose of the image acquisition device based on the graphical feature and a plane coordinate of each light spot; A tracking module configured to calculate a handle coordinate of the handle according to the target camera pose, and perform positioning tracking on the handle based on the handle coordinate; The noise reduction module comprises: A matrix construction unit configured to determine distance values between a plurality of light spots in the first target image, and construct a target distance matrix based on each distance value; A light spot screening unit configured to determine a target abnormal light spot in each light spot according to the target distance matrix.

8. A VR headset, comprising: The VR head-mounted device comprises a memory, a processor, and a positioning tracking program stored on the memory and executable on the processor, and the positioning tracking program, when executed by the processor, implements the steps of the positioning tracking method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a positioning tracking program, and the positioning tracking program, when executed by the processor, implements the steps of the positioning tracking method according to any one of claims 1 to 6.

Images