Hand position tracking method, virtual reality headset, and computer medium

Abstract

The application discloses a handle positioning tracking method, a virtual reality head-mounted device and a computer readable storage medium, and comprises the following steps: taking a target image containing multiple light spots through an image acquisition device, and determining the respective pixel coordinates of each light spot in the target image; determining the respective light spot identifiers of each light spot based on the respective pixel coordinates, and determining the respective light spot vector angle values of each light spot according to the respective pixel coordinates and the respective light spot identifiers; determining the respective light spot opposite side information of each light spot according to the respective pixel coordinates and the respective light spot vector angle values, and combining the respective pixel coordinates, the light spot opposite side information and the light spot vector angle values of each light spot to generate a light spot information structure; generating a light spot information list based on the respective light spot information structures, and determining the respective spatial coordinates of each light spot according to a preset light source information list and the light spot information list, and then positioning and tracking the handle according to the respective spatial coordinates.

Description

Technical Field

[0001] This invention relates to the field of virtual reality technology, and in particular to a method for positioning and tracking a controller, a virtual reality headset, and a computer-readable storage medium. Background Technology

[0002] Currently, virtual reality headsets are considered the hardware entry point for the metaverse industry, and controllers have irreplaceable advantages in terms of accuracy, latency, cost, and physical feedback. Therefore, using controllers for virtual reality interaction has become the main interaction method adopted by virtual reality headsets.

[0003] To enable virtual reality headsets to track and position controllers, engineers primarily employ a method where the headset uses a pre-defined visual algorithm to analyze captured controller images to obtain the controller's 3DOF (Degree of Freedom) displacement. Simultaneously, an IMU (Inertial Measurement Unit) within the controller acquires the controller's three-axis acceleration and angular velocity, outputting the controller's 3DOF rotation. This allows the virtual reality headset to estimate the controller's 6DOF pose. However, a significant challenge in this approach is matching the controller's infrared LEDs with the light spots in the image. Therefore, improving the matching accuracy between infrared LEDs and light spots has become a pressing technical issue that the industry needs to address. Summary of the Invention

[0004] This invention provides a method for positioning and tracking a controller, a virtual reality headset, and a computer-readable storage medium. The aim is to enable the virtual reality headset to determine the spatial coordinates of each light spot in an image, and then determine the controller pose based on the spatial coordinates to complete the positioning and tracking of the controller.

[0005] To achieve the above objectives, the present invention provides a method for positioning and tracking a controller. This method is applied to a virtual reality headset equipped with an image acquisition device for positioning and tracking a controller. The controller is equipped with multiple light sources. The method includes the following steps:

[0006] The image acquisition device captures a target image containing multiple light spots and determines the pixel coordinates of each light spot within the target image.

[0007] Based on the pixel coordinates, determine the spot identifier corresponding to each spot, and determine the angle value of the spot vector corresponding to each spot according to the pixel coordinates and the spot identifier.

[0008] The opposite side information of each light spot is determined according to the pixel coordinates and the angle value of each light spot vector. The pixel coordinates, the opposite side information, and the angle value of each light spot vector are combined to generate a light spot information structure. The opposite side information includes the side on the left side of the handle and the side on the right side of the handle.

[0009] A list of light spot information is generated based on each of the light spot information structures, and the spatial coordinates corresponding to each light spot are determined according to the preset list of light source information and the list of light spot information. Then, the handle is located and tracked according to each of the spatial coordinates.

[0010] Further, the step of determining the pixel coordinates corresponding to each of the light spots within the target image includes:

[0011] Extract the image features of the target image and determine the grayscale value data corresponding to each pixel in the target image;

[0012] Based on the image features and the grayscale data, the pixel coordinates corresponding to each of the light spots in the target image are determined.

[0013] Further, the step of determining the angle value of the light spot vector corresponding to each light spot based on each pixel coordinate and each light spot identifier includes:

[0014] Based on the pixel coordinates and spot identifier corresponding to each spot, a first vector is determined between each spot and its adjacent (i-1)th spot; wherein the endpoint of the first vector is the (i-1)th spot.

[0015] A second vector is determined between each light spot and its adjacent (i+1)th light spot based on the pixel coordinates and the light spot identifier of each light spot; wherein the endpoint of the second vector is the (i+1)th light spot.

[0016] Determine the vector angle value formed between the first vector and the second vector, and set the vector angle value as the vector angle value of the light spot corresponding to the light spot.

[0017] Further, the step of determining the opposite side information of each light spot based on the pixel coordinates and the angle value between the light spot vectors includes:

[0018] The first relative position between the light spot and the (i-1)th light spot is determined based on the pixel coordinates corresponding to each light spot and the angle value between the light spot vectors;

[0019] The second relative position between the light spot and the (i+1)th light spot is determined based on the pixel coordinates corresponding to each light spot and the angle value between the light spot vectors;

[0020] The information of the opposite side of the light spot corresponding to the light spot is determined based on the first relative position and the second relative position.

[0021] Furthermore, before the step of determining the opposite side information of each light spot based on the pixel coordinates and the angle value between the light spot vectors, the method further includes:

[0022] Obtain a preset vector angle range, and compare the angle values ​​of each light spot vector with the vector angle range respectively, and determine the target vector angle value that is not within the vector angle range from the angle values ​​of each light spot vector;

[0023] The spot marker corresponding to the angle value of the target vector is corrected.

[0024] Further, the step of determining the spatial coordinates corresponding to each of the light spots according to the preset light source information list and the light spot information list includes:

[0025] Based on the light spot information list, determine the Euclidean distances corresponding to each light spot, and determine the target Euclidean distance with the largest distance value among all the Euclidean distances;

[0026] Based on the target Euclidean distance, determine the light source identifier corresponding to each of the light sources in the preset light source information list, and determine the spatial coordinates corresponding to each light spot according to each light spot identifier and each light source identifier.

[0027] Further, the step of determining the spatial coordinates corresponding to each of the light spots based on each of the light spot identifiers and each of the light source identifiers includes:

[0028] Determine the target spot identifier corresponding to the target Euclidean distance, and identify the target light source identifier that corresponds to the target spot identifier among the light source identifiers;

[0029] In the spot information list, determine the target spot opposite side information corresponding to the target spot identifier, and in the light source information list, determine the target light source opposite side information corresponding to the target light source identifier;

[0030] Determine whether the information on the opposite side of the target light spot is consistent with the information on the opposite side of the target light source;

[0031] If it is determined that the information on the opposite side of the target light spot is consistent with the information on the opposite side of the target light source, the light source coordinates corresponding to the target light source are determined in the light source information list, and the light source coordinates are determined as the spatial coordinates corresponding to the target light spot.

[0032] Furthermore, the step of determining the spatial coordinates corresponding to each light spot based on each light spot identifier and each light source identifier further includes:

[0033] If it is determined that the information on the opposite side of the target light spot is inconsistent with the information on the opposite side of the target light source, then a second target light source identifier adjacent to the target light source identifier is determined;

[0034] When the target light source opposite information corresponding to the second target light source identifier is consistent with the target light spot opposite information, the light source coordinates corresponding to the second target light source identifier are determined as the spatial coordinates.

[0035] In addition, to achieve the above objectives, the present invention also provides a virtual reality head-mounted device, the virtual reality head-mounted device comprising: a memory, a processor, and a controller positioning and tracking program stored in the memory and executable on the processor, wherein the controller positioning and tracking program, when executed by the processor, implements the steps of the controller positioning and tracking method as described above.

[0036] In addition, to achieve the above objectives, the present invention also provides a computer-readable storage medium storing a controller positioning and tracking program thereon, wherein the controller positioning and tracking program, when executed by a processor, implements the steps of the controller positioning and tracking method as described above.

[0037] The present invention provides a controller positioning and tracking method, a virtual reality headset, and a computer-readable storage medium. This method is applied to a virtual reality headset equipped with an image acquisition device for controller positioning and tracking. The controller is equipped with multiple light sources. The image acquisition device captures a target image containing multiple light spots and determines the pixel coordinates of each light spot within the target image. Based on the pixel coordinates, a light spot identifier is determined for each light spot. The angle between the light spot vectors of each light spot and the pixel coordinates and the light spot identifier are then determined. Opposite side information for each light spot is determined based on the pixel coordinates and the angle between the light spot vectors. The pixel coordinates, opposite side information, and angle between the light spot vectors of each light spot are combined to generate a light spot information structure. The opposite side information includes locations on the left and right sides of the controller. A light spot information list is generated based on the light spot information structures. The spatial coordinates of each light spot are determined based on a preset light source information list and the light spot information list. The controller is then positioned and tracked based on these spatial coordinates.

[0038] In this embodiment, when the virtual reality headset is running, it first calls the image acquisition device configured within the headset to acquire a target image containing multiple light spots formed by invisible light emitted from multiple light sources on the controllers. The target image is then input to the image processing device configured within the headset. The image processing device determines the pixel coordinates corresponding to each light spot based on the image features of the target image, and then uploads these pixel coordinates to the headset. Next, the headset determines the corresponding light spot identifier based on the pixel coordinates of each light spot, and determines the angle value of the light spot vector for each light spot based on the pixel coordinates and the light spot identifier. Finally, the headset determines the angle values ​​of each light spot vector based on the obtained angle values ​​of each light spot vector and the corresponding pixel coordinates. Each light spot corresponds to a light source located on either the left or right side of the controller, thus determining the opposite side information of each light spot. The virtual reality headset then combines the pixel coordinates, opposite side information, and angle values ​​of the light spot vectors to obtain a light spot information structure. Finally, the virtual reality headset sorts the generated light spot information structures to generate a light spot information list. Simultaneously, the virtual reality headset reads the storage device to obtain the light source information list generated by the technician based on the light source arrangement parameters corresponding to each light source, and iterates through the light spot information list and the light source information list to match each light spot with each light source, thereby determining the spatial coordinates corresponding to each light spot. Based on the spatial coordinates, the controller pose is determined, and finally, the controller is positioned and tracked based on the controller pose.

[0039] Thus, this invention employs the following methods: capturing a target image containing light spots formed by invisible light emitted from multiple light sources on the handle; determining the pixel coordinates of each light spot within the target image; determining the light spot identifier for each light spot based on the pixel coordinates; determining the light spot vector angle value for each light spot based on the light spot identifier and pixel coordinates; determining the area of ​​the light source corresponding to each light spot on the handle based on the vector angle value and pixel coordinates; combining the pixel coordinates, opposite side information, and vector angle value of each light spot to generate a light spot information structure and a corresponding light spot information list; and then, based on a preset light source information list and light spot information list, linking the light spots and light sources... The invention employs a method of matching light spots to determine their respective spatial coordinates, and finally using these spatial coordinates to complete the positioning and tracking of the controller. Specifically, this invention compares a list of light spot information containing the pixel coordinates, opposite side information, and vector angle values ​​of each light spot with a list of light source information generated based on the light source arrangement parameters. This method matches each light spot in the image with each light source on the controller, solving the technical problem that current virtual reality headsets cannot accurately match light spots in the image with light sources on the controller. This enables virtual reality headsets to determine the spatial coordinates of each light spot in the image, and then determine the controller pose based on these spatial coordinates to complete the positioning and tracking of the controller. Attached Figure Description

[0040] Figure 1 This is a schematic diagram of the structure of a virtual reality head-mounted device in the hardware operating environment of the embodiment of the present invention;

[0041] Figure 2 This is a flowchart illustrating the first embodiment of the positioning and tracking method for the handle of the present invention;

[0042] Figure 3 This is a schematic diagram of the included angle of the light spot vector in an embodiment of the positioning and tracking method for the handle of the present invention;

[0043] Figure 4 This is a schematic diagram illustrating the correction of light spot markings in an embodiment of the positioning and tracking method for the handle of the present invention;

[0044] Figure 5 This is a flowchart illustrating a preferred embodiment of the positioning and tracking method for the handle of the present invention.

[0045] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0046] It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0047] Reference Figure 1 , Figure 1 This is a schematic diagram of the structure of a virtual reality headset in the hardware operating environment of an embodiment of the present invention.

[0048] The terminal device involved in the embodiments of the present invention may be a mobile VR headset or a fixed VR headset equipped with an HMD head-mounted device and a controller, and the mobile VR headset or the fixed VR headset is composed of an HMD head-mounted device and a matching controller.

[0049] like Figure 1 As shown, the virtual reality headset may include: a processor 1001, such as a central processing unit (CPU), a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. The communication bus 1002 is used to enable communication between these components. The user interface 1003 may include a display screen or an input unit such as a keyboard; optionally, the user interface 1003 may also include a standard wired interface or a wireless interface. The network interface 1004 may optionally include a standard wired interface or a wireless interface (such as a Wi-Fi interface). The memory 1005 may be a high-speed random access memory (RAM) or a stable non-volatile memory (NVM), such as a disk drive. The memory 1005 may also optionally be a storage device independent of the aforementioned processor 1001.

[0050] Those skilled in the art will understand that Figure 1 The structure shown does not constitute a limitation on virtual reality headsets and may include more or fewer components than illustrated, or combine certain components, or have different component arrangements.

[0051] like Figure 1 As shown, the memory 1005, which serves as a storage medium, may include an operating system, a data storage module, a network communication module, a user interface module, and a positioning and tracking program for the controller.

[0052] exist Figure 1In the virtual reality headset shown, the network interface 1004 is mainly used for data communication with other devices; the user interface 1003 is mainly used for data interaction with the user; the processor 1001 and memory 1005 in the virtual reality headset of the present invention can be set in the virtual reality headset. The virtual reality headset calls the controller positioning and tracking program stored in the memory 1005 through the processor 1001 and executes the controller positioning and tracking method provided in the embodiment of the present invention.

[0053] Based on the aforementioned virtual reality headset, various embodiments of the positioning and tracking method for the controller of the present invention are provided.

[0054] Please refer to Figure 2 , Figure 2 This is a flowchart illustrating the first embodiment of the positioning and tracking method for the handle of the present invention.

[0055] It should be understood that although the logical order is shown in the flowchart, in some cases the positioning and tracking method of the handle of the present invention may of course perform the steps shown or described in a different order than that shown here.

[0056] In this embodiment, the positioning and tracking method for the handle of the present invention may include the following steps:

[0057] Step S10: The image acquisition device captures a target image containing multiple light spots and determines the pixel coordinates of each light spot within the target image.

[0058] The image acquisition device is an infrared camera that uses LED infrared light-emitting diodes as the main material or other camera or video recording device that can capture infrared light; in addition, the setting position of the infrared camera can refer to the setting position of the video recording device in other similar virtual reality headsets, and the present invention does not limit it in this regard.

[0059] Similarly, the target handle that is paired with the virtual reality headset should be equipped with multiple light source devices that can emit infrared light. The light source devices should be infrared LEDs, and each infrared LED should be of the same model. In this embodiment, each infrared LED should be arranged on the ring at the front of the handle and divided into two rows, and each infrared LED in the second row should be distributed in a cross pattern between the infrared LEDs in the first row, forming a triangular distribution.

[0060] In this embodiment, when the virtual reality headset is running, it first calls the image acquisition device configured in the virtual reality headset to acquire a target image containing multiple light spots formed by invisible light emitted from multiple light sources on the controller. The target image is then input to the image processing device configured in the virtual reality headset. The image processing device determines the pixel coordinates corresponding to each light spot based on the image features of the target image, and then uploads each pixel coordinate to the virtual reality headset.

[0061] For example, when the VR headset is running, it first calls the infrared camera configured in the VR headset through the control system integrated on the internal main control chip, and then uses the infrared camera to take pictures of the controller that is matched with the VR headset to obtain a target image containing the light spots formed by the infrared light emitted by each infrared LED on the controller. Then, the VR headset inputs the target image to the aforementioned image processing device, which determines the corresponding pixel coordinates of each light spot in the target image, and then uploads the pixel coordinates to the VR headset.

[0062] Furthermore, in a feasible embodiment, the step of "determining the pixel coordinates corresponding to each of the light spots within the target image" in step S10 above may specifically include:

[0063] Step S101: Extract the image features of the target image and determine the grayscale value data corresponding to each pixel in the target image;

[0064] In this embodiment, after acquiring the target image, the virtual reality headset inputs the target image into the image processing device, which extracts the image features contained in the target image. At the same time, the image processing device identifies the grayscale value corresponding to each pixel in the target image.

[0065] Step S102: Determine the pixel coordinates corresponding to each of the light spots in the target image based on the image features and the grayscale data;

[0066] In this embodiment, the image processing device determines the position of each light spot in the target image based on the acquired image features and the gray value corresponding to each pixel, and then determines the pixel coordinates of each light spot in the target image.

[0067] For example, the VR headset inputs the acquired target image to the image processing device. The image processing device first calls the OpenCV tool pre-installed by the technician to identify and extract the image features of the target image in the direction from the bottom to the top of the image and from the right to the left of the image. At the same time, the image processing device uses the OpenCV tool to determine the gray value corresponding to each pixel in the target image. Then, based on the extracted image features and gray value data, it identifies light spots in the target image and determines the pixel coordinates of each light spot in the target image.

[0068] Step S20: Determine the spot identifier corresponding to each spot based on each pixel coordinate, and determine the spot vector angle value corresponding to each spot according to each pixel coordinate and each spot identifier;

[0069] In this embodiment, the virtual reality headset sorts the pixels according to their respective horizontal and vertical coordinates to obtain a sorting result. Then, it determines the spot identifier corresponding to each spot according to the sorting result, and determines the angle value of the spot vector corresponding to each spot based on the pixel coordinates and spot identifier of each spot.

[0070] For example, the VR headset sorts the pixel coordinates according to the value of the vertical axis coordinate v corresponding to each pixel coordinate, and sorts the pixel coordinates according to the value of the horizontal axis coordinate u corresponding to each pixel coordinate to obtain a sorting result. Then, the VR headset adds a spot identifier to each spot according to the sorting result, and then determines the angle value of the spot vector corresponding to each spot according to the pixel coordinates and spot identifiers.

[0071] Furthermore, in a feasible embodiment, the step S20 above, "determining the angle value of the light spot vector corresponding to each light spot based on each pixel coordinate and each light spot identifier," may specifically include:

[0072] Step S201: Determine the first vector formed between the light spot and the adjacent (i-1)th light spot according to the pixel coordinates and light spot identifier corresponding to each light spot; wherein, the vector endpoint of the first vector is the (i-1)th light spot;

[0073] In this embodiment, the virtual reality head-mounted device determines the i-th light spot and the (i-1)-th light spot below the i-th light spot based on the pixel coordinates and light spot identifiers corresponding to each light spot. The virtual reality head-mounted device then uses the i-th light spot as the starting point of the first vector and the (i-1)-th light spot as the ending point of the first vector, and then determines the first vector formed between the i-th light spot and the (i-1)-th light spot based on the starting point and the ending point of the first vector.

[0074] Step S202: Determine the second vector formed between the light spot and the adjacent (i+1)th light spot according to the pixel coordinates corresponding to each light spot and the light spot identifier; wherein the vector endpoint of the second vector is the (i+1)th light spot;

[0075] In this embodiment, the virtual reality headset determines the i-th light spot and the i+1-th light spot above the i-th light spot based on the pixel coordinates and light spot identifiers corresponding to each light spot. The virtual reality headset then uses the i-th light spot as the starting point of the second vector and the i+1-th light spot as the ending point of the second vector, and then determines the second vector formed between the i-th light spot and the i+1-th light spot based on the starting point and the ending point of the second vector.

[0076] Step S203: Determine the vector angle value formed between the first vector and the second vector, and determine the vector angle value as the light spot vector angle value corresponding to the light spot;

[0077] In this embodiment, the virtual reality head-mounted device determines the vector angle formed between the first vector and the second vector, calculates the vector angle value, and then determines the vector angle value as the light spot vector angle value corresponding to the i-th light spot.

[0078] For example, please refer to Figure 3 , Figure 3 This is a schematic diagram of the angle between light spot vectors in an embodiment of the positioning and tracking method for the controller of the present invention. The VR headset first determines the i-th light spot (i.e., ...) based on the light spot identifiers and pixel coordinates. Figure 3 The pixel coordinates corresponding to spot 1) in the image are the (i-1)th spot below and adjacent to spot i (i.e. Figure 3 The pixel coordinates corresponding to the light spot 0 in the image are used to determine the first vector a, with the i-th light spot as the starting point of the first vector and the (i-1)-th light spot as the ending point of the first vector. Simultaneously, the VR headset determines the pixel coordinates corresponding to the i-th light spot based on the light spot identifiers and pixel coordinates. The (i+1)-th light spot above the i-th light spot (i.e.,...) Figure 3 The pixel coordinates corresponding to spot 2) in the image are used to determine the second vector b, with the i-th spot as the starting point and the (i+1)-th spot as the ending point. Then, the VR headset reads the storage device to obtain the vector dot product formula preset by the technician.

[0079]

[0080] The vector angle between the first vector a and the second vector b is calculated according to the vector dot product formula, and this vector angle value is determined as the angle between the light spot vectors corresponding to the i-th light spot.

[0081] Step S30: Determine the opposite side information of each light spot according to the pixel coordinates and the angle value of each light spot vector, and combine the pixel coordinates, the opposite side information and the angle value of each light spot vector to generate a light spot information structure; wherein, the opposite side information includes the left side of the handle and the right side of the handle;

[0082] In this embodiment, the virtual reality headset determines whether the light source corresponding to each light spot is on the left or right side of the controller based on the obtained angle values ​​of each light spot vector and the coordinates of each pixel. Then, it determines the opposite side information of each light spot. The virtual reality headset then combines the pixel coordinates, opposite side information and angle values ​​of each light spot vector to obtain the light spot information structure.

[0083] For example, the VR headset determines the relative positions of the i-th, i-1-th, and i+1-th light spots based on the angle value of the light spot vectors and the pixel coordinates of each light spot. Based on these relative positions, it determines whether the infrared LEDs corresponding to each light spot are placed in the left or right area of ​​the controller, thereby generating the opposite side information of the light spot. Then, the VR headset combines the pixel coordinates, opposite side information, and angle value of the light spot vectors of each light spot to obtain the light spot information structure.

[0084] Furthermore, in a feasible embodiment, the step S30 above, "determining the opposite side information of each light spot based on the pixel coordinates and the angle between each light spot vector," may specifically include:

[0085] Step S301: Determine the first relative position between the light spot and the (i-1)th light spot based on the pixel coordinates corresponding to each light spot and the angle value between the light spot vectors;

[0086] Step S302: Determine the second relative position between the light spot and the (i+1)th light spot based on the pixel coordinates corresponding to each light spot and the angle value between the light spot vectors;

[0087] Step S303: Determine the information of the opposite side of the light spot corresponding to the light spot based on the first relative position and the second relative position;

[0088] For example, the VR headset determines the first relative position between the i-th and i-1-th light spots based on the angle between the light spot vectors corresponding to the i-th light spot and the pixel coordinates, and the angle between the light spot vectors corresponding to the (i-1)-th light spot and the pixel coordinates. Similarly, the VR headset determines the second relative position between the i-th and i+1-th light spots based on the angle between the light spot vectors corresponding to the i-th light spot and the pixel coordinates, and the angle between the light spot vectors corresponding to the (i+1)-th light spot and the pixel coordinates. Then, the VR headset determines that when the (i-1)-th and i+1-th light spots are both on the right side of the target image and the i-th light spot is on the left side of the target image, the VR headset determines that the infrared LED corresponding to the i-th light spot is deployed in the left area of ​​the controller, and then generates light spot countermeasure information corresponding to the i-th light spot.

[0089] Furthermore, in a feasible embodiment, prior to step S30 above, the positioning and tracking method for the handle of the present invention may further include:

[0090] Step A10: Obtain a preset vector angle range, and compare the angle values ​​of each light spot vector with the vector angle range respectively, and determine the target vector angle value that is not within the vector angle range from the angle values ​​of each light spot vector;

[0091] In this embodiment, the virtual reality headset reads from the storage device to obtain the vector angle range preset by the technician. At the same time, the virtual reality headset inputs the obtained vector angle values ​​of each light spot along with the vector angle range to the data processing device configured in the virtual reality headset. The data processing device compares the vector angle values ​​of each light spot with the vector angle range, and then determines the vector angle values ​​of each target light spot that are outside the vector angle range.

[0092] Step A20: Correct the spot marker corresponding to the angle value of the target vector;

[0093] In this embodiment, after determining the angle values ​​of each target spot vector, the data processing device determines that the spot identifier corresponding to each target spot vector angle value is incorrect, and corrects the spot identifier corresponding to each target spot vector angle value.

[0094] For example, please refer to Figure 4 , Figure 4This is a schematic diagram illustrating the correction of light spot markings in an embodiment of the positioning and tracking method for the controller of the present invention. The VR headset reads from a storage device to obtain a vector angle range pre-stored by a technician, and inputs the obtained vector angle range and the vector angle values ​​of each light spot to a data processing device. The data processing device compares the vector angle values ​​of each light spot with the vector angle range to determine the target light spot vector angle values ​​that are not within the vector angle range. Then, when the data processing device determines that a target light spot vector angle value exists, i.e., ... Figure 4 If ∠012 shown is 15° and falls within the range of the vector angle, then the data processing device determines that the vector angle corresponding to the target spot vector angle value is an erroneous vector, and further determines the spot identifier corresponding to the erroneous vector. Subsequently, the data processing device corrects or replaces the spot identifier corresponding to the erroneous vector, that is, it... Figure 4 The light spot 1 and light spot 2 described herein are swapped.

[0095] It should be noted that, in this embodiment, the vector angle range is the range of angle values ​​between the light spots that each infrared lamp bead can form when the handle is in each pose, calculated in advance by the technician based on the deployment position of each infrared lamp bead on the handle. That is, the vector angle range corresponds to the arrangement rule of the infrared lamp beads. Of course, there are many ways to calculate the vector angle range, and this invention does not limit it.

[0096] Step S40: Generate a light spot information list based on each of the light spot information structures, and determine the spatial coordinates corresponding to each light spot according to the preset light source information list and the light spot information list, and then locate and track the handle according to each of the spatial coordinates;

[0097] The light source information list contains the light source coordinates corresponding to each infrared LED and the corresponding opposite side information of each infrared LED. The light source coordinates are the coordinates of the infrared LED in a three-dimensional coordinate system constructed with the center of mass of the handle as the origin. Similarly, the opposite side information is the opposite side information determined according to the deployment position of the infrared LED on the handle, that is, whether the infrared LED is deployed in the left area of ​​the ring at the front of the handle or in the right area of ​​the ring at the front of the handle.

[0098] In this embodiment, the virtual reality headset sorts the generated light spot information structures to generate a light spot information list. At the same time, the virtual reality headset reads the storage device to obtain the light source information list generated by the technician according to the light source arrangement parameters corresponding to each light source, and traverses the light spot information list and the light source information list to match each light spot with each light source to determine the spatial coordinates corresponding to each light spot. Based on each spatial coordinate, the headset determines the hand pose, and finally performs positioning and tracking of the hand based on the hand pose.

[0099] For example, the VR headset first sorts the acquired light spot information structures and integrates them to generate a light spot information list. Simultaneously, the VR headset reads from the storage device to obtain a light source information list containing the coordinates of infrared LEDs and the information of the opposite side of the light source. Then, the VR headset traverses the light spot information list and the light source information list to determine the infrared LEDs corresponding to each light spot in the light spot information list and the corresponding light source information list, thereby determining the spatial coordinates of each light spot. After that, the VR headset calculates the hand pose based on the spatial coordinates of the light spots and then tracks the hand based on the hand pose.

[0100] Furthermore, in a feasible embodiment, the step S40 above, "determining the spatial coordinates corresponding to each of the light spots according to the preset light source information list and the light spot information list," may specifically include:

[0101] Step S401: Determine the Euclidean distances corresponding to each light spot according to the light spot information list, and determine the target Euclidean distance with the largest distance value among the Euclidean distances;

[0102] In this embodiment, the virtual reality headset first traverses the list of light spot information and determines the Euclidean distances between the i-th light spot and the adjacent i+1-th light spot, and between the i-th light spot and the adjacent i-1-th light spot, based on the pixel coordinates contained in the list of light spot information. Then, the virtual reality headset determines the target Euclidean distance with the largest value among the various Euclidean distances.

[0103] Step S402: Based on the target Euclidean distance, determine the light source identifier corresponding to each of the light sources in the preset light source information list, and determine the spatial coordinates corresponding to each light spot according to each light spot identifier and each light source identifier;

[0104] In this embodiment, after determining the target Euclidean distance, the virtual reality headset determines the light source identifier corresponding to each light source in the light source information list based on the target Euclidean distance, and then determines the spatial coordinates corresponding to each light spot based on each light spot identifier and each light source identifier.

[0105] For example, the VR headset first traverses the list of light spot information according to a preset incremental rule, and then calculates the distance according to a preset Euclidean distance formula. The Euclidean distance d between the i-th spot and its adjacent (i+1)-th spot is calculated by examining the pixel coordinates of each pixel in the spot information list. Similarly, the VR headset traverses the spot information list according to a preset decreasing rule and calculates the Euclidean distance using the formula. The Euclidean distance d between the i-th spot and its adjacent (i-1)-th spot is calculated by analyzing the pixel coordinates in the spot information list. Then, the VR headset sorts the obtained Euclidean distances d according to their numerical values, identifying the highest-valued target Euclidean distance and the target spot identifier corresponding to that distance. Next, the VR headset determines the light source identifier for each infrared LED based on the infrared LED arrangement order in the light source information list and the target spot identifier. Finally, the VR headset determines the infrared LED corresponding to each spot based on the spot identifier and the light source identifier, and the light source coordinates of the infrared LED in the light source information list are used as the spatial coordinates of the spot.

[0106] Furthermore, in a feasible embodiment, the step of "determining the spatial coordinates corresponding to each of the light spots according to each of the light spot identifiers and each of the light source identifiers" in step S402 above may further include:

[0107] Step S4021: Determine the target spot identifier corresponding to the target Euclidean distance, and identify the target light source identifier corresponding to the target spot identifier among the light source identifiers;

[0108] In this embodiment, the virtual reality headset first determines the target spot identifier corresponding to the target Euclidean distance, and then determines the target light source identifier corresponding to the target spot identifier in each light source identifier according to the preset light source arrangement order and the target spot identifier.

[0109] Step S4022: Determine the target spot opposite information corresponding to the target spot identifier in the spot information list, and determine the target light source opposite information corresponding to the target light source identifier in the light source information list.

[0110] In this embodiment, the virtual reality headset filters the list of light spot information based on the target light spot identifier to determine the target light spot opposite information corresponding to the target light spot identifier. At the same time, the virtual reality headset filters the list of light source information based on the target light source identifier to determine the target light source opposite information corresponding to the target light source identifier.

[0111] Step S4023: Determine whether the information on the opposite side of the target light spot is consistent with the information on the opposite side of the target light source;

[0112] In this embodiment, the virtual reality headset compares the information on the opposite side of the target light spot with the information on the opposite side of the target light source, and determines whether the information on the opposite side of the target light spot is consistent with the information on the opposite side of the target light source.

[0113] Step S4024: If it is determined that the information on the opposite side of the target light spot is consistent with the information on the opposite side of the target light source, determine the light source coordinates corresponding to the target light source in the light source information list, and determine the light source coordinates as the spatial coordinates corresponding to the target light spot;

[0114] In this embodiment, if the virtual reality headset determines that the information on the opposite side of the target light spot is consistent with the information on the opposite side of the target light source, the virtual reality headset determines that the target light spot matches the target light source, and then in the light source information list, the light source coordinates corresponding to the target light source are added, and the light source coordinates are projected onto the target light spot to determine the spatial coordinates corresponding to the target light spot.

[0115] For example, after determining the target Euclidean distance, the VR headset further determines the target spot identifier of the target spot i corresponding to the target Euclidean distance, and determines the target light source identifier corresponding to the target infrared lamp j in each light source identifier according to the preset lamp arrangement order and the target spot identifier. Then, the VR headset filters the light source information list based on the target light source identifier to determine the target light source opposite side information corresponding to the target light source i. At the same time, the VR headset filters the light source information list based on the target light source identifier to determine the target light source opposite side information corresponding to the target infrared lamp. Then, the VR headset compares the obtained target spot opposite side information with the target light source opposite side information and determines whether the target spot opposite side information and the target light source opposite side information are consistent. When the VR headset determines that the target spot opposite side information and the target light source opposite side information are consistent, it determines that the target spot i matches the target infrared lamp j. Finally, the VR headset determines the light source coordinates corresponding to the target infrared lamp j in the light source information list and projects the light source coordinates onto the target spot i to determine the spatial coordinates corresponding to the target spot i.

[0116] Furthermore, in a feasible embodiment, the step of "determining the spatial coordinates corresponding to each of the light spots according to each of the light spot identifiers and each of the light source identifiers" in step S402 above may further include:

[0117] Step S4025: If it is determined that the information on the opposite side of the target light spot is inconsistent with the information on the opposite side of the target light source, then a second target light source identifier adjacent to the target light source identifier is determined;

[0118] In this embodiment, when the virtual reality headset determines that the information on the opposite side of the target light spot is inconsistent with the information on the opposite side of the target light source, it determines that the target light spot and the target light source do not match, and then determines the second target light spot above and adjacent to the target light source in the light source information list.

[0119] Step S4026: When the target light source opposite information corresponding to the second target light source identifier is consistent with the target light spot opposite information, the light source coordinates corresponding to the second target light source identifier are determined as the spatial coordinates;

[0120] In this embodiment, when the virtual reality headset determines that the light source opposite information of the second target light source is consistent with the light source opposite information of the target light spot, it determines the light source coordinates of the second target light source in the light source information list and sets the light source coordinates as the spatial coordinates corresponding to the target light spot.

[0121] For example, when the VR headset determines that the target light spot opposite information corresponding to target light spot i is inconsistent with the target light source opposite information of target infrared lamp j corresponding to target light spot i, the VR headset determines that the lamp corresponding to target light spot i is located in the left area of ​​the controller, while the target infrared lamp j is located in the right area of ​​the controller. The VR headset then determines that target light spot i and target infrared lamp j do not match. Afterwards, the VR headset determines infrared lamp j+2 located above and adjacent to target infrared lamp j in the light source information list according to the arrangement rules of infrared lamps. When it determines that the light source opposite information of infrared lamp j+2 is consistent with the target light spot opposite information, it determines the light source coordinates of infrared lamp j+2 in the light source information list and determines the light source coordinates as the spatial coordinates corresponding to target light spot i.

[0122] In this embodiment, when the virtual reality headset is running, it first calls the image acquisition device configured within the headset to acquire a target image containing multiple light spots formed by invisible light emitted from multiple light sources on the controllers. The target image is then input to the image processing device configured within the headset. The image processing device determines the pixel coordinates corresponding to each light spot based on the image features of the target image, and then uploads these pixel coordinates to the headset. Next, the headset sorts the pixel coordinates according to their horizontal and vertical coordinates to obtain a sorting result. Based on the sorting result, it determines the light spot identifier corresponding to each light spot. Then, based on the pixel coordinates and the light spot identifier, it determines the angle value of the light spot vector corresponding to each light spot. Finally, the virtual reality headset... The vector angle value and pixel coordinates determine whether the light source corresponding to each light spot is on the left or right side of the controller, and then determine the opposite side information of each light spot. The virtual reality headset then combines the pixel coordinates, opposite side information and vector angle value of each light spot to obtain a light spot information structure. Finally, the virtual reality headset sorts the generated light spot information structures to generate a light spot information list. At the same time, the virtual reality headset reads the storage device to obtain the light source information list generated by the technician according to the light source arrangement parameters corresponding to each light source, and traverses the light spot information list and the light source information list to match each light spot with each light source to determine the spatial coordinates corresponding to each light spot. Based on the spatial coordinates, the controller pose is determined, and finally the controller is positioned and tracked based on the controller pose.

[0123] Thus, this invention employs the following methods: capturing a target image containing light spots formed by invisible light emitted from multiple light sources on the handle; determining the pixel coordinates of each light spot within the target image; determining the light spot identifier for each light spot based on the pixel coordinates; determining the light spot vector angle value for each light spot based on the light spot identifier and pixel coordinates; determining the area of ​​the light source corresponding to each light spot on the handle based on the vector angle value and pixel coordinates; combining the pixel coordinates, opposite side information, and vector angle value of each light spot to generate a light spot information structure and a corresponding light spot information list; and then, based on a preset light source information list and light spot information list, linking the light spots and light sources... The invention employs a method of matching light spots to determine their respective spatial coordinates, and finally using these spatial coordinates to complete the positioning and tracking of the controller. Specifically, this invention compares a list of light spot information containing the pixel coordinates, opposite side information, and vector angle values ​​of each light spot with a list of light source information generated based on the light source arrangement parameters. This method matches each light spot in the image with each light source on the controller, solving the technical problem that current virtual reality headsets cannot accurately match light spots in the image with light sources on the controller. This enables virtual reality headsets to determine the spatial coordinates of each light spot in the image, and then determine the controller pose based on these spatial coordinates to complete the positioning and tracking of the controller.

[0124] Furthermore, based on the first embodiment of the positioning and tracking method for the handle of the present invention described above, a preferred embodiment of the positioning and tracking method for the handle of the present invention is hereby proposed.

[0125] Please refer to Figure 5 , Figure 5 This is a flowchart illustrating a preferred embodiment of the positioning and tracking method for the handle of the present invention.

[0126] In this embodiment, the VR headset first uses an infrared camera to capture an image of the VR controller, including multiple light spots formed by infrared rays emitted from the infrared LEDs on the controller. The image is then processed to determine the centroid coordinates of each light spot. Next, based on the acquired centroid coordinates, the VR headset forms a vector angle between three adjacent light spots and determines the angle value. Based on this angle value, it determines the opposite side information for each light spot. Finally, it combines the centroid coordinates, opposite side information, and the angle to obtain a light spot information structure. A light spot list is then generated based on this structure. Simultaneously, the VR headset... The device reads from the storage device to obtain the infrared LED arrangement rules pre-stored by the technicians, and a three-dimensional coordinate system constructed based on the centroid of the handle as the origin. Then, it determines the 3D light source coordinates corresponding to each infrared LED according to the arrangement rules and the three-dimensional coordinate system. Based on the 3D light source coordinates and opposite side information of each infrared LED, it generates an LED information list. Then, the VR headset traverses the light spot information list and the light source information list to match each light spot with each infrared LED, thereby determining the spatial coordinates corresponding to each light spot. Finally, the VR headset estimates the spatial coordinates corresponding to each light spot and the light source coordinates of each infrared LED based on the camera intrinsic parameter matrix and the preset PnP algorithm to obtain the handle pose, and then performs positioning and tracking of the handle based on the handle pose.

[0127] Furthermore, the present invention also provides a virtual reality headset having a controller positioning and tracking program that can run on a processor. When the virtual reality headset executes the controller positioning and tracking program, it implements the steps of the controller positioning and tracking method as described in any of the above embodiments.

[0128] The specific embodiments of the virtual reality head-mounted device of the present invention are basically the same as the embodiments of the positioning and tracking method of the controller described above, and will not be repeated here.

[0129] Furthermore, the present invention provides a computer-readable storage medium storing a controller positioning and tracking program thereon, which, when executed by a processor, implements the steps of the controller positioning and tracking method as described in any of the above embodiments.

[0130] The specific embodiments of the computer-readable storage medium of this invention are basically the same as the embodiments of the positioning and tracking method of the handle described above, and will not be repeated here.

[0131] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or system. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or system that includes that element.

[0132] The sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0133] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes several instructions to cause a terminal device (which can be a mobile VR headset or a fixed VR headset configured with an HMD head-mounted device and a controller, and the mobile VR headset or the fixed VR headset is composed of an HMD head-mounted device and a matching controller) to execute the methods described in the various embodiments of the present invention.

[0134] The above are merely preferred embodiments of the present invention and do not limit the scope of the patent. Any equivalent structural or procedural transformations made based on the description and drawings of the present invention, or direct or indirect applications in other related technical fields, are similarly included within the scope of patent protection of the present invention.

Claims

1. A method for positioning and tracking a handle, characterized in that, The controller positioning and tracking method is applied to a virtual reality headset equipped with an image acquisition device for controller positioning and tracking. The controller is equipped with multiple light sources, and the controller positioning and tracking method includes the following steps: The image acquisition device captures a target image containing multiple light spots and determines the pixel coordinates of each light spot within the target image. Based on the pixel coordinates, determine the spot identifier corresponding to each spot, and determine the angle value of the spot vector corresponding to each spot according to the pixel coordinates and the spot identifier. The opposite side information of each light spot is determined according to the pixel coordinates and the angle value of each light spot vector. The pixel coordinates, the opposite side information, and the angle value of each light spot vector are combined to generate a light spot information structure. The opposite side information includes the side on the left side of the handle and the side on the right side of the handle. A list of light spot information is generated based on each of the light spot information structures, and the spatial coordinates corresponding to each light spot are determined according to the preset list of light source information and the list of light spot information. Then, the handle is positioned and tracked according to each of the spatial coordinates. The step of determining the angle value of the light spot vector corresponding to each light spot based on each pixel coordinate and each light spot identifier includes: Based on the pixel coordinates and spot identifier corresponding to each spot, a first vector is determined between each spot and its adjacent (i-1)th spot; wherein the endpoint of the first vector is the (i-1)th spot. A second vector is determined between each light spot and its adjacent (i+1)th light spot based on the pixel coordinates and the light spot identifier of each light spot; wherein the endpoint of the second vector is the (i+1)th light spot. Determine the vector angle value formed between the first vector and the second vector, and set the vector angle value as the light spot vector angle value corresponding to the light spot; The step of determining the opposite side information of each light spot based on the pixel coordinates and the angle between the light spot vectors includes: The first relative position between the light spot and the (i-1)th light spot is determined based on the pixel coordinates corresponding to each light spot and the angle value between the light spot vectors; The second relative position between the light spot and the (i+1)th light spot is determined based on the pixel coordinates corresponding to each light spot and the angle value between the light spot vectors; The information of the opposite side of the light spot corresponding to the light spot is determined based on the first relative position and the second relative position.

2. The handle positioning and tracking method as described in claim 1, characterized in that, The step of determining the pixel coordinates corresponding to each of the light spots within the target image includes: Extract the image features of the target image and determine the grayscale value data corresponding to each pixel in the target image; Based on the image features and the grayscale data, the pixel coordinates corresponding to each of the light spots in the target image are determined.

3. The handle positioning and tracking method as described in claim 1, characterized in that, Before the step of determining the spot-opposite information corresponding to each spot based on the pixel coordinates and the angle values ​​between the spot vectors, the method further includes: Obtain a preset vector angle range, and compare the angle values ​​of each light spot vector with the vector angle range respectively, and determine the target vector angle value that is not within the vector angle range from the angle values ​​of each light spot vector; The spot marker corresponding to the angle value of the target vector is corrected.

4. The handle positioning and tracking method as described in claim 1, characterized in that, The step of determining the spatial coordinates corresponding to each light spot based on the preset light source information list and the light spot information list includes: Based on the light spot information list, determine the Euclidean distances corresponding to each light spot, and determine the target Euclidean distance with the largest distance value among the Euclidean distances; Based on the target Euclidean distance, determine the light source identifier corresponding to each of the light sources in the preset light source information list, and determine the spatial coordinates corresponding to each light spot according to each light spot identifier and each light source identifier.

5. The handle positioning and tracking method as described in claim 4, characterized in that, The step of determining the spatial coordinates corresponding to each light spot based on each light spot identifier and each light source identifier includes: Determine the target spot identifier corresponding to the target Euclidean distance, and identify the target light source identifier that corresponds to the target spot identifier among the light source identifiers; In the spot information list, determine the target spot opposite side information corresponding to the target spot identifier, and in the light source information list, determine the target light source opposite side information corresponding to the target light source identifier; Determine whether the information on the opposite side of the target light spot is consistent with the information on the opposite side of the target light source; If it is determined that the information on the opposite side of the target light spot is consistent with the information on the opposite side of the target light source, the light source coordinates corresponding to the target light source are determined in the light source information list, and the light source coordinates are determined as the spatial coordinates corresponding to the target light spot.

6. The handle positioning and tracking method as described in claim 5, characterized in that, The step of determining the spatial coordinates corresponding to each light spot based on each light spot identifier and each light source identifier further includes: If it is determined that the information on the opposite side of the target light spot is inconsistent with the information on the opposite side of the target light source, then a second target light source identifier adjacent to the target light source identifier is determined; When the target light source opposite information corresponding to the second target light source identifier is consistent with the target light spot opposite information, the light source coordinates corresponding to the second target light source identifier are determined as the spatial coordinates.

7. A virtual reality headset, characterized in that, The virtual reality headset includes: a memory, a processor, and a controller positioning and tracking program stored in the memory and executable on the processor, wherein the controller positioning and tracking program, when executed by the processor, implements the steps of the controller positioning and tracking method as described in any one of claims 1 to 6.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a controller positioning and tracking program, which, when executed by a processor, implements the steps of the controller positioning and tracking method as described in any one of claims 1 to 6.

Images