Positioning and tracking method and platform, head-mounted display system, and computer-readable storage medium

Abstract

The present disclosure provides a positioning and tracking method, a positioning and tracking platform, a head-mounted display system, and a computer-readable storage medium, the method including: acquiring odd frame images collected at a preset first exposure time by a tracking camera installed on a head-mounted display device; acquiring even frame images collected at a preset second exposure time, the even frame images including light spots corresponding to each of a plurality of light emitters disposed on at least a steering wheel communicatively connected to the head-mounted display device (S201); determining degree-of-freedom information of the head-mounted display device based on the odd frame images and attitude information of the head-mounted display device (S202); and determining degree-of-freedom information of the steering wheel based on the even frame images, attitude information of the steering wheel, and attitude information of the head-mounted display device (S203).

Description

CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims priority to a Chinese patent application filed with the State Intellectual Property Office of the People's Republic of China on September 16, 2020, bearing application number 202010974857.9 and entitled "Positioning and Tracking Method and Platform, and Head-Mounted Display System," the entire contents of which are incorporated herein by reference. [Technical field]

[0002] FIELD Embodiments of the present disclosure relate to the field of video processing technology, and more particularly, to a positioning and tracking method, a positioning and tracking platform, a head-mounted display system, and a computer-readable storage medium. [Background technology]

[0003] Positioning technology is widely applied in fields such as virtual reality, augmented reality, and mixed reality, and is an important component of man-machine interaction.

[0004] Currently, when tracking a steering wheel, it is usually necessary to provide an electromagnetic sensor or an ultrasonic sensor on the steering wheel to track the steering wheel position. When tracking a bare hand, it is necessary to add an infrared camera or a depth camera to the head-mounted display device. This means that the same camera cannot be used to track the position of the head-mounted display device, the steering wheel, and the bare hand, and there are problems such as high power consumption, poor stability, and high cost during use.

[0005] Therefore, it is necessary to provide a positioning and tracking scheme that allows for synchronous tracking of the head mounted display device and the handle. Summary of the Invention

[0006] An embodiment of the present disclosure aims to provide a technical solution for synchronously tracking a head-mounted display device and a steering wheel.

[0007] According to a first aspect of the embodiment of the present disclosure, acquiring odd frame images collected at a preset first exposure time and even frame images collected at a preset second exposure time by a tracking camera disposed on a head mounted display device, wherein the even frame images include light spots corresponding to each of a plurality of light emitters disposed on a handle communicatively connected to at least the head mounted display device; determining degree of freedom information of the head mounted display device based on the odd frame images and posture information of the head mounted display device; determining degree of freedom information of the steering wheel based on the even frame images, pose information of the steering wheel, and degree of freedom information of the head mounted display device.

[0008] In at least one example embodiment, the second exposure time is shorter than the first exposure time.

[0009] In at least one exemplary embodiment, the first exposure time is adaptively adjusted according to an intensity of external ambient light.

[0010] In at least one exemplary embodiment, at least two tracking cameras are positioned on the head mounted display device, each tracking camera having the same center point of exposure time.

[0011] In at least one example embodiment, acquiring odd frame images collected at a preset first exposure time and even frame images collected at a preset second exposure time by a tracking camera disposed on the head mounted display device includes: The method includes collecting, by the at least two tracking cameras, a plurality of odd frame images each corresponding to each tracking camera at a preset first exposure time, and acquiring, by the at least two tracking cameras, a plurality of even frame images each corresponding to each tracking camera at a preset second predetermined exposure time.

[0012] In at least one exemplary embodiment, the method further includes controlling a light emitter on the handle to light up for a preset lighting time when collecting even frame images at a preset second exposure time by a tracking camera disposed on a head mounted display device, wherein a midpoint time of the second exposure time corresponds to a midpoint time of the lighting time.

[0013] In at least one example embodiment, the illumination time is greater than or equal to the second exposure time.

[0014] In at least one exemplary embodiment, determining degree of freedom information of the head mounted display device based on the odd frame image and the attitude information of the head mounted display device includes: According to the spatial structure information in the odd frame image, a simultaneous localization and mapping SLAM algorithm is used to perform calculations to obtain the movement freedom of the head mounted display device; Obtaining a rotational degree of freedom of the head mounted display device based on the attitude information of the head mounted display device collected by a first inertial measurement unit integrated in the head mounted display device; and determining the degree of freedom information for the head mounted display device using the translational degree of freedom and the rotational degree of freedom.

[0015] Determining degree of freedom information of the head mounted display device based on the odd frame images and the attitude information of the head mounted display device, and then The method includes pre-integrating the degree-of-freedom information of the head mounted display device in combination with the posture information of the head mounted display device to obtain high frequency degree-of-freedom information of the head mounted display device.

[0016] In at least one exemplary embodiment, determining the degree of freedom information of the steering wheel based on the even frame image, the attitude information of the steering wheel, and the degree of freedom information of the head mounted display device includes: performing light spot detection on the even-numbered frame images to determine position information of all light spots in the even-numbered frame images; determining three-dimensional coordinates of light-emitting bodies on the handle corresponding to the light spots in the even-numbered frame images based on position information of all the light spots in the even-numbered frame images; The method includes determining degree of freedom information of the handle based on three-dimensional coordinates of light-emitting bodies on the handle corresponding to light spots in the even frame images, posture information of the handle, and degree of freedom information of the head-mounted display device.

[0017] In at least one exemplary embodiment, performing light spot detection on the even frame image and determining position information of all light spots in the even frame image includes: Performing light spot detection on each even frame image collected by the tracking camera to determine position information of all light spots in each even frame image, including position information of at least four light spots; determining a correspondence between the light spots in the even-numbered frame images and the light emitters on the handle based on a distribution rule of each light spot in the even-numbered frame images; and using the two-dimensional coordinates of the light spots corresponding to the light emitters on the handle to determine the three-dimensional coordinates of the corresponding light emitters on the handle.

[0018] In at least one exemplary embodiment, determining degree of freedom information of the steering wheel based on three-dimensional coordinates of light-emitting bodies on the steering wheel corresponding to light spots in the even frame images, attitude information of the steering wheel, and degree of freedom information of the head-mounted display device includes: determining a degree of freedom of movement of the handle based on three-dimensional coordinates of a light-emitting body on the handle corresponding to a light spot in the even-numbered frame image; determining a rotational degree of freedom of the steering wheel based on the attitude information of the steering wheel; determining degree of freedom information of the handle based on the translational degrees of freedom and the rotational degrees of freedom; The method includes performing coordinate transformation on the degree of freedom information of the handle based on the degree of freedom information of the head mounted display device.

[0019] In at least one exemplary embodiment, after determining the degree of freedom information of the handle based on the three-dimensional coordinates of the light-emitting body on the handle corresponding to the light spot in the even frame image, the attitude information of the handle, and the degree of freedom information of the head mounted display device, further: The method includes performing pre-integration on the degree of freedom information of the steering wheel in combination with the attitude information of the steering wheel to obtain high frequency degree of freedom information of the steering wheel.

[0020] In at least one example embodiment, the odd frame image includes at least a hand, and the method further comprises: The method further includes determining hand degree of freedom information based on the odd frame images.

[0021] In at least one exemplary embodiment, determining hand degree of freedom information based on the odd frame images includes: inputting the odd frame image into a predetermined convolutional neural network model to obtain multiple key point positions of the hand; and determining degrees of freedom information of the hand based on the positions of a plurality of key points of the hand.

[0022] In at least one exemplary embodiment, the method further comprises training a predetermined convolutional neural network model; The step of training the predetermined convolutional neural network model includes: Collecting a plurality of scene images including a hand; marking the locations of a number of key points of a hand in the scene images, and forming the marked images into an image training sample set; and training the convolutional neural network model based on the image training sample set.

[0023] According to a second aspect of the embodiment of the present disclosure, an image capture module arranged to capture, by a tracking camera arranged on a head mounted display device, odd frame images collected at a first preset exposure time and even frame images collected at a second preset exposure time, the even frame images including light spots corresponding to each of a plurality of light emitters arranged on at least a handle communicatively connected to the head mounted display device; a positioning and tracking module arranged to determine degree of freedom information of the head mounted display device based on the odd frame images and pose information of the head mounted display device, and to determine degree of freedom information of the handle based on the even frame images, pose information of the handle, and degree of freedom information of the head mounted display device; or A method for implementing a method according to any one of the first aspects of the embodiments of the present disclosure, comprising: a processor; and a memory storing computer instructions that, when executed by the processor, cause the method to perform a method according to any one of the first aspects of the embodiments of the present disclosure; Provides a positioning and tracking platform.

[0024] According to a third aspect of the embodiment of the present disclosure, A head mounted display system is provided, the head mounted display system including a head mounted display device having at least two tracking cameras arranged thereon, a handle connected to the head mounted display device having a plurality of light emitters arranged thereon, and a positioning and tracking platform.

[0025] In at least one exemplary embodiment, the center point of the exposure time for each tracking camera disposed on the head mounted display device is the same.

[0026] In at least one exemplary embodiment, four tracking cameras are disposed on the head mounted display device, and the four tracking cameras are positioned within the head mounted display device according to a predetermined first position constraint rule to ensure that each tracking camera meets a field of view requirement of the head mounted display device.

[0027] In at least one exemplary embodiment, the multiple light-emitting elements disposed on the handle are positioned according to a predetermined second positional constraint rule, which satisfies the condition that no overlap or adhesion of local pattern information occurs in images collected by the tracking camera within a preset distance range from the handle, and corresponding light spots of at least four light-emitting elements appear in all images collected by the tracking camera at any angle.

[0028] According to a fourth aspect of an embodiment of the present disclosure, there is provided a computer-readable storage medium having stored thereon a computer program which, when executed by a processor, causes the method to be implemented as described in any one of the method embodiments and exemplary embodiments.

[0029] According to an embodiment of the present disclosure, odd frame images collected at a preset first exposure time and even frame images collected at a preset second exposure time are acquired by a tracking camera installed on a head mounted display device, and 6DoF information for the head mounted display device is determined based on the odd frame images and posture information of the head mounted display device, and 6DoF information for the handlebars is determined based on the even frame images and posture information of the handlebars. This eliminates the need for additional hardware and enables synchronous tracking of the head mounted display and handlebars using the same hardware, thereby reducing power consumption of the head mounted display device and improving the user experience.

[0030] Other features and advantages of the present disclosure will become more apparent from the following detailed description of exemplary embodiments of the present disclosure, taken in conjunction with the accompanying drawings. [Brief description of the drawings]

[0031] In order to more clearly describe the technical solutions of the embodiments of the present disclosure, the following drawings are briefly introduced which are required to be used in the embodiments. Note that the following drawings only show some embodiments of the present disclosure, and should not be regarded as limiting the scope. Those skilled in the art can obtain other related drawings based on these drawings without any creative effort. [Figure 1] FIG. 1 is a hardware configuration diagram of a head mounted display system used to realize an embodiment of the present disclosure. [Diagram 2] FIG. 2 is a flow chart of a positioning and tracking method according to an embodiment of the present disclosure. [Diagram 3] FIG. 3 is a schematic diagram of a handle according to an embodiment of the present disclosure. [Figure 4] FIG. 4 is a structural block diagram of a positioning and tracking platform according to an embodiment of the present disclosure. [Diagram 5] FIG. 5 is a structural block diagram of a positioning and tracking platform according to an embodiment of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0032] Each of the exemplary embodiments of the present disclosure will be described in detail below with reference to the drawings. However, unless otherwise specified, the relative arrangement of parts and steps, numerical expressions, and values ​​described in these embodiments do not limit the scope of the present disclosure.

[0033] The following description of at least one illustrative embodiment is merely exemplary in nature and is in no way intended to limit the present disclosure and its application or uses.

[0034] Techniques, methods and equipment known to those skilled in the art may not be described in detail, but where appropriate, said techniques, methods and equipment are considered part of this specification.

[0035] In all examples shown and described herein, any specific values ​​should be construed as merely exemplary and not limiting, and therefore other examples of the example embodiment may have different values.

[0036] It should be noted that in the following drawings, like reference numbers and letters represent like items, so that once an item is defined in one drawing, no further explanation is required for it in subsequent drawings.

[0037] <Hardware configuration> FIG. 1 is a block diagram showing a hardware configuration of a head mounted display system 100 according to an embodiment of the present disclosure.

[0038] 1, the head mounted display device 100 includes a head mounted display device 1000, a handle 2000, and a positioning and tracking platform 3000. The positioning and tracking platform 3000 is communicatively connected to the head mounted display device 1000 and the handle 2000 to enable positioning tracking for the head mounted display device 1000 and the handle 2000, respectively.

[0039] The head mounted display device 1000 may be, for example, a VR (Virtual Reality) facility, an AR (Augmented Reality) facility, an MR (Mixed Reality) facility, or the like.

[0040] In one embodiment, the head mounted display device 1000 may include a processor 1100, a memory 1200, an interface device 1300, a communication device 1400, a display device 1500, an input device 1600, an audio device 1700, an inertial measurement unit 1800, a tracking camera 1900, etc., as shown in FIG.

[0041] Here, the processor 1100 may be, for example, a central processor CPU, a microprocessor MCU, or the like. The memory 1200 may include, for example, a non-volatile memory such as a ROM (read only memory), a RAM (random access memory), or a hard disk. The interface device 1300 may include, for example, a serial bus interface (including a USB interface), a parallel bus interface, a high definition multimedia interface HDMI (registered trademark) interface, or the like. The communication device 1400 may be capable of, for example, wired or wireless communication. The display device 1500 may be, for example, a liquid crystal display, an LED display, a touch panel display, or the like. The input device 1600 may include, for example, a touch panel, a body sensory input device, or the like. The audio device 1700 may be arranged to input and output audio information. The inertial measurement unit 1800 may be arranged to measure a change in posture of the head mounted display device 1000. The tracking camera 1900 may be arranged to acquire image information. At least two tracking cameras 1900 may be arranged.

[0042] In one embodiment, the tracking camera 1900 may be a large-angle tracking camera, for example, the horizontal field of view (FOV) of the tracking camera 1900 is 100°-160°, the vertical field of view is 80°-120°, the depth field of view is 130°-160°, the frame rate of the tracking camera is greater than 60Hz, the resolution of the tracking camera is VGA (Video Graphics Array) or 720P, the tracking camera adopts a Global Shutter exposure method, and the tracking camera can pass the visible light wavelength band and the infrared light wavelength band. For example, the model number of the tracking camera is OV7251 or OV9282.

[0043] Although multiple devices are illustrated in FIG. 1 for head mounted display device 100, the present disclosure may relate to only some of those devices.

[0044] In one embodiment, the handle 2000 may include a processor 2100, a memory 2200, an interface device 2300, a communication device 2400, an inertial measurement unit 2500, a light emitter 2600, etc., as shown in FIG.

[0045] Here, the processor 2100 may be, for example, a central processor CPU, a microprocessor MCU, or the like. The memory 2200 includes, for example, a non-volatile memory such as a ROM (read only memory), a RAM (random access memory), and a hard disk. The interface device 2300 includes, for example, a serial bus interface (including a USB interface), a parallel bus interface, a high definition multimedia interface HDMI (registered trademark) interface, or the like. The communication device 2400 may, for example, be capable of wired or wireless communication. The inertial measurement unit 2500 may be arranged to measure a change in posture of the head mounted display device 1000. A plurality of light emitters 2600 may be arranged to identify the handle. The light emitters 2600 may be, for example, a visible light source such as an LED lamp or an infrared light source. In FIG. 1, a plurality of devices are shown for the handle 2000, but the present disclosure may relate to only a portion of those devices.

[0046] The positioning and tracking platform 3000 may be, for example, a mobile terminal. In one embodiment, the positioning and tracking platform 3000 includes a processor 3100, a memory 3200, an interface device 3300, a communication device 3400, a display device 3500, and an input device 3600, as shown in FIG. 1. The processor 3100 may be a desktop processor, a mobile processor, or the like that meets performance requirements, and is not limited here. The memory 3200 includes, for example, a non-volatile memory such as a ROM (read-only memory), a RAM (random access memory), and a hard disk. The interface device 3300 includes, for example, various bus interfaces such as a serial bus interface (including a USB interface), a parallel bus interface, and a high-definition multimedia interface (HDMI) interface. The communication device 3400 is capable of, for example, wired or wireless communication. The display device 3500 is, for example, a liquid crystal display, an LED display, a touch panel display, or the like. The input device 3600 may include, for example, a touch panel, a keyboard, or the like. In other embodiments, the positioning and tracking platform 3000 may further include a speaker, a microphone, etc., without being limited thereto. Although multiple devices are shown in FIG. 1 for the positioning and tracking platform 3000, the present disclosure may relate to only some of those devices, for example, the positioning and tracking platform 3000 may relate to only the memory 3200 and the processor 3100.

[0047] In one embodiment, the memory 3200 of the positioning and tracking platform 3000 is arranged to store commands that provide associated support for the positioning and tracking method of this embodiment by controlling the processor 3100 to perform corresponding steps.

[0048] Note that while FIG. 1 shows only one head-mounted display device 1000, one handle 2000, and one positioning and tracking platform 3000, this is not intended to limit the number of each, and the head-mounted display system 100 may include multiple head-mounted display devices 1000, multiple handles 2000, and multiple positioning and tracking platforms 3000.

[0049] With the above description, those skilled in the art can design commands according to the solutions provided by the present disclosure. How commands control and operate a processor is well known in the art, so it will not be described in detail here.

[0050] The head mounted display system shown in FIG. 1 is exemplary only and is not intended to limit the present disclosure and its applications or uses.

[0051] Example of the method A positioning and tracking method according to an embodiment of the present disclosure will be described with reference to Fig. 2. The method is applied to a positioning and tracking platform connected to a head-mounted display device and a handle respectively. The positioning and tracking method includes the following steps S201 to S203.

[0052] In S201, an odd frame image collected at a preset first exposure time and an even frame image collected at a preset second exposure time are acquired by a tracking camera disposed on the head mounted display device, where the even frame image includes light spots each corresponding to a plurality of light emitters disposed on a handle communicatively connected to at least the head mounted display device.

[0053] In an embodiment of the present disclosure, at least two tracking cameras are disposed on the head mounted display device for collecting images. In one embodiment, the tracking cameras may collect images with different exposure times, for example, collect images of odd frames with a first preset exposure time and collect images of even frames with a second preset exposure time, provided that the at least two tracking cameras are collected at the same time, i.e., the center point of the exposure time of each tracking camera is the same.

[0054] In a more specific example, four tracking cameras are arranged on a head mounted display device, and the four tracking cameras are arranged in the head mounted display device according to a predetermined first position constraint rule, so as to ensure that each tracking camera meets the viewing angle requirement of the head mounted display device. According to an embodiment of the present disclosure, positioning tracking is performed on the head mounted display device and the handle based on the images collected by the four tracking cameras, thereby improving robustness and improving the accuracy of positioning tracking.

[0055] As shown in FIG. 3, a plurality of light emitters 2600 are arranged on the handle 2000, and the plurality of light emitters are arranged according to a predetermined second position constraint rule, so that the overlap or adhesion of local pattern information in the image collected when the tracking camera is close to the handle can be avoided, and the light spots corresponding to at least four light emitters can be guaranteed to appear in any image collected by the tracking camera at any angle, so that the handle can be positioned based on the collected image. In a specific example, the plurality of light emitters are not located on the same plane. In a more specific example, 17 or 24 light emitters are arranged. The light emitters may be, for example, a visible light source or an infrared light source, such as an LED lamp.

[0056] According to the head-mounted display device and handle described above, the positioning tracking method provided by the embodiments of the present disclosure can also determine degree-of-freedom information of the head-mounted display device based on images collected by at least two tracking cameras and posture information of the head-mounted display device, and can also determine degree-of-freedom information of the handle based on images collected by at least two tracking cameras and posture information of the handle.

[0057] In an embodiment of the present disclosure, the at least two tracking cameras alternately collect images at a first preset exposure time and a second preset exposure time, i.e., collect a plurality of odd frame images corresponding to each tracking camera at a first preset exposure time by the at least two tracking cameras, and collect a plurality of even frame images corresponding to each tracking camera at a second preset exposure time by the at least two tracking cameras. The odd frame images are used to determine the degree of freedom information of the head mounted display device. The even frame images are used to determine the degree of freedom information of the handle.

[0058] The longer the exposure time of the tracking camera, the more information of the external environment light is acquired when the image is collected. In one embodiment, the first exposure time is adaptable to the external environment light, i.e., the first exposure time is adaptively adjusted according to the intensity of the external environment light. According to an embodiment of the present disclosure, by determining the degree of freedom information of the head mounted display device based on the odd frame image collected at the preset first exposure time, more external environment features can be acquired, and the accuracy of tracking the head mounted display device can be improved.

[0059] In one embodiment, the second exposure time is shorter than the first exposure time. The even frame image is collected at a preset second exposure time and includes at least light spots corresponding to a plurality of light emitters arranged on the handle. The second exposure time may be preset. According to an embodiment of the present disclosure, by collecting the even frame image at a shorter second exposure time, it is possible to present light spots corresponding to a plurality of light emitters arranged on the handle while avoiding obtaining excessive external ambient light information, thereby reducing the interference of the external ambient light, improving the quality of the even frame image, and improving the accuracy of tracking the handle.

[0060] In a more specific example, the positioning tracking method further includes controlling the light-emitting element on the handle to be lit for a preset lighting time when collecting even frame images at a preset second exposure time by a tracking camera disposed on the head-mounted display device, wherein a midpoint time of the second exposure time corresponds to a midpoint time of the lighting time.

[0061] The intermediate time of the second exposure time is synchronized with the intermediate time of the lighting time, i.e., by controlling the light-emitting elements on the handle to be turned on within the exposure time period during which the tracking camera collects even frame images, it is possible to ensure that light spots corresponding to the multiple light-emitting elements arranged on the handle are included in the even frame images.

[0062] The lighting time of the light-emitting body of the handle may be set according to a preset second exposure time. According to the present disclosure, by setting the lighting time of the light-emitting body of the handle according to a preset short second exposure time, the lighting time of the light-emitting body of the handle can be shortened and the power consumption of the device can be reduced.

[0063] In one embodiment, the lighting time may be the same as the second exposure time, and the start time and end time of the second exposure time may correspond to the start time and end time of the lighting time. For example, the lighting time may be 100 μs. In another embodiment, the lighting time may be longer than the second exposure time, that is, the lighting time of the light-emitting body may be a certain time longer than the time before and after the exposure time. For example, the lighting time of the light-emitting body is 30 μs longer than the time before and after the exposure time. According to the embodiment of the present disclosure, by making the lighting time longer than the second exposure time, it is possible to avoid the accuracy error that exists when the collection of the even frame image by the tracking camera and the lighting of the light-emitting body of the handle are synchronously controlled by wireless communication, and it is ensured that the light spot generated by the light-emitting body can be captured when the even frame image is collected by the tracking camera.

[0064] After acquiring odd frame images collected at a preset first exposure time and even frame images collected at a preset second exposure time by a tracking camera disposed on the head-mounted display device, the process proceeds to S202.

[0065] In S202, information on the degrees of freedom of the head mounted display device is determined based on the odd frame images and the attitude information of the head mounted display device.

[0066] The head mounted display device incorporates a first inertial measurement unit (IMU) positioned to collect attitude information of the head mounted display device.

[0067] The degree of freedom information of the head mounted display device is six degrees of freedom information (hereinafter, abbreviated as 6DoF information) of the head mounted display device, which includes a translation degree of freedom and a rotation degree of freedom.

[0068] In one embodiment, based on the spatial structure information in the odd frame images, a simultaneous localization and mapping (SLAM) algorithm can be used to perform calculations to obtain the translational degrees of freedom (i.e., three-dimensional coordinates) of the head mounted display device; based on the attitude information (e.g., quaternion information) of the head mounted display device collected by the first inertial measurement unit, the rotational degrees of freedom (i.e., yaw angle, pitch angle, and roll angle) of the head mounted display device can be obtained; and 6DoF information of the head mounted display device can be determined based on the translational degrees of freedom and the rotational degrees of freedom.

[0069] In one embodiment, the positioning and tracking method may further include, after determining the 6DoF information of the head mounted display device, combining it with posture information of the head mounted display device to perform pre-integration on the 6DoF information of the head mounted display device to obtain high frequency 6DoF information of the head mounted display device.

[0070] The output frequency of the 6DoF information of the head mounted display device depends on the frequency of collecting the attitude information of the head mounted display device by the first inertial measurement unit and the frequency of collecting odd frame images by the tracking camera. The frequency of collecting the attitude information of the head mounted display device by the first inertial measurement unit is higher, typically 200Hz. According to an embodiment of the present disclosure, by pre-integrating the 6DoF information of the head mounted display device with the attitude information of the head mounted display device collected by the first inertial measurement unit, the high frequency 6DoF information of the head mounted display device can be obtained, the delay can be reduced, and the user experience can be improved.

[0071] After determining the degree of freedom information of the head mounted display device, the process proceeds to S203.

[0072] In S203, the degree of freedom information of the steering wheel is determined based on the even-numbered frame images, the attitude information of the steering wheel, and the degree of freedom information of the head mounted display device.

[0073] A second inertial measurement unit is incorporated within the steering wheel and positioned to obtain attitude information of the steering wheel.

[0074] The information on the degrees of freedom of the handle is information on six degrees of freedom of the handle (hereinafter, abbreviated as 6DoF information), and includes the degree of freedom of movement and the degree of freedom of rotation.

[0075] In one embodiment, determining the degree of freedom information of the steering wheel based on the even frame images, the attitude information of the steering wheel, and the degree of freedom information of the head mounted display device may further include S301 to S303.

[0076] In S301, light spot detection is performed on the even-numbered frame images, and position information of all light spots in the even-numbered frame images is determined.

[0077] In S302, based on the position information of all the light spots in the even-numbered frame images, the three-dimensional coordinates of the light emitters on the handle corresponding to the light spots in the even-numbered frame images are determined.

[0078] Light spot detection is performed on each even frame image collected by the tracking camera to determine position information (two-dimensional coordinates) of all light spots in each even frame image, including position information of at least four light spots. Based on the distribution rule of each light spot in the even frame image, the correspondence relationship between the light spots in the even frame image and the light-emitting body on the steering wheel is determined. Using the two-dimensional coordinates of the light spots corresponding to the light-emitting body on the steering wheel, the three-dimensional coordinates of the corresponding light-emitting body on the steering wheel are determined.

[0079] In S303, degree of freedom information of the steering wheel is determined based on the three-dimensional coordinates of the light-emitting bodies on the steering wheel corresponding to the light spots in the even-numbered frame images, the attitude information of the steering wheel, and the degree of freedom information of the head mounted display device.

[0080] The degree of freedom of movement of the handle can be determined based on the three-dimensional coordinates of the light emitters on the handle corresponding to the light spots in the even-numbered frame images. The degree of freedom of rotation of the handle can be determined based on the attitude information of the handle. The 6DoF information of the handle can be determined based on the degree of freedom of movement and the degree of freedom of rotation. Then, a coordinate transformation is performed on the 6DoF information of the handle according to the degree of freedom information of the head-mounted display device.

[0081] In one embodiment, once the three-dimensional coordinates of each light emitter are determined, they are combined with handle attitude information collected by a second inertial measurement unit to determine 6DoF information for the handle using a PNP algorithm or other prior art correlation algorithm, thereby enabling position tracking of the handle.

[0082] According to an embodiment of the present disclosure, by using at least two tracking cameras to obtain multiple even frame images and calculating 6DoF information of the handle based on the multiple even frame images, robustness can be improved and tracking accuracy can be improved.

[0083] In one embodiment, the positioning and tracking method may further include, after determining the 6DoF information of the handle, combining it with the attitude information of the handle and performing pre-integration on the 6DoF information of the handle to obtain high-frequency 6DoF information of the handle.

[0084] The output frequency of the 6DoF information of the steering wheel depends on the frequency at which the steering wheel collects attitude information by the second inertial measurement unit and the frequency at which the tracking camera collects images of even frames. The frequency at which the second inertial measurement unit collects attitude information of the steering wheel is large, typically 200Hz. According to an embodiment of the present disclosure, by combining the second inertial measurement unit with collecting attitude information of the steering wheel and performing pre-integration on the 6DoF information of the steering wheel, high-frequency 6DoF information of the steering wheel can be obtained, delay can be reduced, and user experience can be improved.

[0085] According to an embodiment of the present disclosure, a tracking camera installed on a head mounted display device acquires odd frame images collected at a preset first exposure time and even frame images collected at a preset second exposure time, and determines 6DoF information for the head mounted display device based on the odd frame images and posture information of the head mounted display device, and determines 6DoF information for the handlebars based on the even frame images and posture information of the handlebars. This eliminates the need for additional hardware and enables synchronous tracking of the head mounted display and handlebars using the same hardware, thereby reducing power consumption of the head mounted display device and improving the user experience.

[0086] In some scenes, the user may want to use his / her bare hands to select some menus through the main interface and / or interact with some objects in the virtual scene. Thus, in the embodiment of the present disclosure, bare hand tracking can be enabled. This will be described in detail below.

[0087] In this embodiment, the odd frame image includes at least a hand. The positioning and tracking method may further include determining freedom information of the hand based on the odd frame image. According to the embodiment of the present disclosure, tracking of the user's hand can be realized when the user is not using the steering wheel.

[0088] In one embodiment, the step of determining the hand freedom information based on the odd frame image may further include S401 to S402.

[0089] In S401, odd frame images are input into a predetermined convolutional neural network model to obtain the positions of multiple key points of the hand.

[0090] In S402, the degree of freedom information of the hand is determined based on the positions of multiple key points of the hand.

[0091] The hand degree of freedom information is twenty-six degrees of freedom (26DoF) information of the hand.

[0092] In one embodiment, the positioning and tracking method may further include training a predetermined convolutional neural network model. The training of the predetermined convolutional neural network model may further include S501 to S503.

[0093] In S501, a plurality of scene images including hands are acquired.

[0094] For example, we collect image data of different sizes including hands collected by four tracking cameras for 200 people in different lighting environments and different usage scene environments. For example, we collect 1.25 million scene images including hands.

[0095] S502 marks the positions of multiple key points of the hand in the scene image, and the marked images are used as an image training sample set.

[0096] In S503, a convolutional neural network model is trained based on the image training sample set.

[0097] For example, the positions of 24 keypoints of a hand in multiple scene images are marked, and the marked images are a set of image training samples. The positions of multiple estimated keypoints in multiple scene images are estimated based on a convolutional neural network model. An optimization residual equation is constructed and combined with a PNP optimization algorithm to iteratively estimate a convolutional neural network model, so that the distance error between the estimated keypoint positions on the image corresponding to the convolutional neural network model and the marked keypoint positions on the image is minimized to obtain a convolutional neural network model after training.

[0098] According to an embodiment of the present disclosure, 26DoF information of the hand can be rapidly determined based on a convolutional neural network model.

[0099] According to an embodiment of the present disclosure, an odd frame image collected at a preset first exposure time and an even frame image collected at a preset second exposure time are obtained by a tracking camera disposed on a head mounted display device. Then, when a user uses a handle, 6DoF information of the head mounted display device is determined based on the odd frame image and the attitude information of the head mounted display device, and 6DoF information of the handle is determined based on the even frame image and the attitude information of the handle. When a user does not use a handle, 6DoF information of the head mounted display device is determined based on the odd frame image and the attitude information of the head mounted display device, and 26DoF information of the user's hand is determined based on the odd frame image. The positioning and tracking method provided by the embodiment of the present disclosure realizes synchronous tracking of the head mounted display device, the handle, and the bare hand by the same hardware, thereby reducing the power consumption of the head mounted display device and improving the user experience.

[0100] <Equipment Example 1> 4, an embodiment of the present disclosure provides a positioning and tracking platform 40, which may be the positioning and tracking platform 3000 as shown in FIG. 1. For example, the positioning and tracking platform may be a mobile terminal. The positioning and tracking platform 40 includes an image acquisition module 41 and a positioning and tracking module 42.

[0101] The image acquisition module 41 is arranged to acquire, by a tracking camera disposed on the head mounted display device, odd frame images collected at a first preset exposure time and even frame images collected at a second preset exposure time, where the even frame images include light spots corresponding to each of a plurality of light emitters disposed on a handle communicatively connected with at least the head mounted display device.

[0102] In one embodiment, the second exposure time is shorter than the first exposure time.

[0103] In one embodiment, when even frame images are collected with a preset second exposure time by a tracking camera disposed on the head-mounted display device, the light emitter on the handle is controlled to be illuminated for a preset lighting time, where the midpoint of the second exposure time corresponds to the midpoint of the lighting time.

[0104] In a more specific example, the lighting time is equal to or longer than the second exposure time.

[0105] The positioning and tracking module 42 is arranged to determine degree of freedom information of the head mounted display device based on the odd frame images and pose information of the head mounted display device.

[0106] The positioning and tracking module 42 is further arranged to determine degree of freedom information of the handle based on the even frame images, pose information of the handle, and degree of freedom information of the head mounted display device.

[0107] In one embodiment, the positioning and tracking module 42 is configured to perform light spot detection on the even frame images and determine the position information of all light spots in the even frame images.

[0108] The positioning and tracking module 42 is further arranged to determine, based on the position information of all light spots in the even frame images, three-dimensional coordinates of light emitters on the handle corresponding to the light spots in the even frame images.

[0109] The positioning tracking module 42 is further configured to determine degree of freedom information of the handle based on three-dimensional coordinates of light-emitting bodies on the handle corresponding to light spots in the even frame images, posture information of the handle, and degree of freedom information of the head-mounted display device.

[0110] In one embodiment, the odd frame images include at least a hand. The positioning and tracking module 42 is further arranged to determine hand degree of freedom information based on the odd frame images.

[0111] In a more specific example, the positioning and tracking module 42 is arranged to input the odd frame images into a predefined convolutional neural network model to obtain the positions of multiple key points of the hand.

[0112] The positioning and tracking module 42 is also arranged to determine degrees of freedom information of the hand based on the positions of a number of key points of the hand.

[0113] In one embodiment, the positioning and tracking platform 40 further includes a model training module configured to train a predetermined convolutional neural network model.

[0114] In one more specific example, the model training module is arranged to collect a number of scene images including a hand.

[0115] The model training module is further configured to mark locations of a plurality of key points of a hand in the scene images, and the marked images are an image training sample set.

[0116] The model training module is further configured to train the convolutional neural network model based on the image training sample set.

[0117] 5, an embodiment of the present disclosure provides a positioning and tracking platform 50 including a processor 51 and a memory 52. ​​The memory 52 is arranged to store a computer program which, when executed by the processor 51, realizes the positioning and tracking method disclosed in any of the above embodiments.

[0118] <Apparatus Example 2> An embodiment of the present disclosure further provides a head mounted display system, which includes a head mounted display device and a handle and a positioning and tracking platform connected to the head mounted display device, and may be, for example, the head mounted display system 100 shown in FIG.

[0119] In one embodiment, at least two tracking cameras and a first inertial measurement unit are disposed on the head mounted display device.

[0120] The at least two tracking cameras are positioned to collect images, specifically, the at least two tracking cameras collect a plurality of images of odd frames with a first predetermined exposure time and collect a plurality of images of even frames with a second predetermined exposure time.

[0121] The first inertial measurement unit is positioned to detect attitude information of the head mounted display device.

[0122] In a more specific example, four tracking cameras are arranged on the head mounted display device, and the four tracking cameras are arranged in the head mounted display device according to a predetermined first position constraint rule to ensure that each tracking camera meets the viewing angle requirement of the head mounted display device. According to an embodiment of the present disclosure, positioning tracking is performed on the head mounted display device and the handle based on the images collected by the four tracking cameras, thereby improving robustness and improving the accuracy of positioning tracking.

[0123] In a more specific example, a large-angle tracking camera may be selected as the tracking camera 1900, for example, the horizontal field angle of view (FOV) of the tracking camera 1900 is 100°-160°, the vertical field angle of view is 80°-120°, the depth field angle is 130°-160°, the frame rate of the tracking camera is greater than 60Hz, the resolution of the tracking camera is VGA (Video Graphics Array) or 720P, the tracking camera adopts a Global Shutter exposure method, and the tracking camera can pass visible light wavelength band and infrared light wavelength band. For example, the model number of the tracking camera is OV7251 or OV9282.

[0124] A second inertial measurement unit and a number of light emitters are disposed on the handle.

[0125] The multiple light emitters are positioned according to a predetermined second position constraint rule, thereby avoiding overlap or adhesion of local pattern information in images collected when the tracking camera is close to the handle, and ensuring that corresponding light spots of at least four light emitters appear in any of the images collected by the tracking camera at any angle, and allowing the handle to be positioned based on the collected images.

[0126] In a more specific example, the number of light emitters is 17 or 24. The light emitters may be, for example, visible light sources or infrared light sources, such as LED lamps.

[0127] The second inertial measurement unit is arranged to detect attitude information of the steering wheel.

[0128] The positioning and tracking platform is connected to the head mounted display and the handle and is arranged to acquire odd frame images, even frame images, attitude information of the head mounted display device, and attitude information of the handle.

[0129] The positioning and tracking platform is further arranged to determine degree-of-freedom information of the head mounted display device based on the odd frame images and the pose information of the head mounted display device, and realize positioning and tracking for the head mounted display device.

[0130] In one embodiment, the positioning and tracking platform is further configured to determine degree of freedom information of the handle based on the even frame image, the handle posture information, and the degree of freedom information of the head-mounted display device when a user uses the handle, and realize positioning tracking for the handle.

[0131] In one embodiment, the positioning and tracking platform is further arranged to determine hand degree-of-freedom information based on the odd frame images when the user does not use the handle, and realize positioning tracking for the user's hand.

[0132] In one embodiment, the positioning and tracking platform may be the positioning and tracking platform 3000 shown in FIG.

[0133] In one embodiment, the positioning and tracking platform may be, for example, a mobile terminal.

[0134] According to an embodiment of the present disclosure, a tracking camera installed on a head mounted display device acquires odd frame images collected at a preset first exposure time and even frame images collected at a preset second exposure time, and determines 6DoF information for the head mounted display device based on the odd frame images and posture information of the head mounted display device, and determines 6DoF information for the handlebars based on the even frame images and posture information of the handlebars. This eliminates the need for additional hardware and enables synchronous tracking of the head mounted display and handlebars using the same hardware, thereby reducing power consumption of the head mounted display device and improving the user experience.

[0135] According to an embodiment of the present disclosure, an odd frame image collected at a preset first exposure time and an even frame image collected at a preset second exposure time are obtained by a tracking camera disposed on a head mounted display device. Then, when a user uses a handle, 6DoF information of the head mounted display device is determined based on the odd frame image and the attitude information of the head mounted display device, and 6DoF information of the handle is determined based on the even frame image and the attitude information of the handle. When a user does not use a handle, 6DoF information of the head mounted display device is determined based on the odd frame image and the attitude information of the head mounted display device, and 26DoF information of the user's hand is determined based on the odd frame image. The positioning and tracking method provided by the embodiment of the present disclosure realizes synchronous tracking of the head mounted display device, the handle, and the bare hand by the same hardware, thereby reducing the power consumption of the head mounted display device and improving the user experience.

[0136] Each embodiment in this specification is described in a progressive manner, and similar or similar parts between each embodiment may be mutually referenced. Each embodiment is described focusing on the differences from other embodiments, but those skilled in the art will understand that the above-mentioned embodiments may be used alone or in combination with each other as necessary. It should be noted that the device embodiments correspond to the method embodiments, and therefore the description is relatively simple, and in this respect, the corresponding parts of the method embodiments may be referred to. The above-mentioned system embodiments are merely examples, and the modules illustrated as separate components may or may not be physically separated.

[0137] The present disclosure may be a system, a method, and / or a computer program product. The computer program product may include a computer-readable storage medium having computer-readable program instructions stored thereon for causing a processor to implement aspects of the present disclosure.

[0138] A computer readable storage medium may be a tangible device capable of holding and storing commands used by a command execution device. A computer readable storage medium may be, for example, but is not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the above. More specific examples of computer readable storage media (not an exhaustive list) include portable computer disks, hard disks, random access memories (RAMs), read-only memories (ROMs), erasable programmable read-only memories (EPROMs or flash memories), static random access memories (SRAMs), portable compact disk read-only memories (CD-ROMs), digital versatile disks (DVDs), memory sticks, floppy disks, mechanical coding devices such as punch cards or raised structures in grooves on which commands are stored, and any suitable combination of the above. A computer readable storage medium as used herein is not to be construed as a transitory signal per se, such as electric waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a wave guide or other transmission medium (e.g., light pulses through a fiber optic cable), or electrical signals transmitted through wires.

[0139] The computer readable program commands described herein may be downloaded from computer readable storage media into various computers / processing devices, or may be downloaded to an external computer or storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network may include copper transmission cables, fiber optic transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer readable program commands from the network and forwards the computer readable program commands for storage in a computer readable storage medium in the respective computing / processing device.

[0140] The computer program commands for carrying out the operations of the present disclosure may be assembly commands, command set architecture (ISA) commands, machine language commands, machine-related commands, microcode, firmware commands, state setting data, or source or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, and conventional procedural programming languages ​​such as the "for example" programming language or similar programming languages. The computer readable program commands may be executed entirely on the user's computer, partially on the user's computer, partially as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or wide area network (WAN) network, or may be connected to an external computer (e.g., via the Internet using an Internet Service Provider). In some examples, electronic circuitry, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), can be personalized with state information of computer readable program commands such that the electronic circuitry can execute the computer readable program commands to implement various aspects of the present disclosure.

[0141] Aspects of the present disclosure are described herein with reference to flowchart and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present disclosure. It should be noted that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer readable program commands.

[0142] These computer readable program commands are provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to create a machine that, when executed by the processor of the computer or other programmable data processing apparatus, generates means for implementing the functions / operations specified in one or more blocks of the flowcharts and / or block diagrams. These computer readable program commands may be stored on a computer readable storage medium, and these comments cause the computer, programmable data processing apparatus, and / or other apparatus to function in a particular manner, such that the computer readable medium having the commands stored thereon includes an article of manufacture containing commands for implementing each aspect of the functions / operations specified in one or more blocks of the flowcharts and / or block diagrams.

[0143] The computer readable program commands are loaded into a computer, other programmable data processing apparatus, or other device to implement the functions / operations specified in one or more blocks of the flowcharts and / or block diagrams, and cause the computer, other programmable data processing apparatus, or other device to execute a series of operations to generate a computer-implemented process.

[0144] The flowcharts and block diagrams in the drawings illustrate possible architectures, functions, and operations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowcharts or block diagrams may represent a module, segment, or part of a command, which includes one or more executable commands for implementing a specified logical function. In some alternative implementations, the functions described in the blocks may occur in a different order from the order described in the drawings. For example, two consecutive blocks may in fact be executed substantially in parallel, or may be executed in reverse order depending on the functions involved. It should be noted that each block of the block diagrams and / or flowchart diagrams, and combinations of blocks in the block diagrams and / or flowchart diagrams, may be realized in a dedicated system by hardware that performs a specific function or operation, or may be realized in a combination of dedicated hardware and computer commands. It is well known to those skilled in the art that hardware implementation, software implementation, and a combination of software and hardware implementation are all equivalent.

[0145] Although the embodiments of the present disclosure have been described above, the above description is illustrative and not exhaustive, and is not limited to the disclosed embodiments. It is obvious to those skilled in the art that many modifications and variations can be made without departing from the scope and spirit of the described embodiments. The selection of terms used in this specification is intended to best explain the principles, practical applications, or technical improvements in the market of various embodiments, or to allow those skilled in the art to understand the various embodiments disclosed in this specification. The scope of the present disclosure is defined by the appended claims.

Claims

1. 1. A method of positioning and tracking, comprising: acquiring odd frame images collected at a preset first exposure time and even frame images collected at a preset second exposure time by at least two tracking cameras disposed on a head mounted display device, wherein the even frame images include light spots corresponding to each of a plurality of light emitters disposed on a handle communicatively connected to at least the head mounted display device; determining degree of freedom information of the head mounted display device based on the odd frame images and posture information of the head mounted display device; determining degree of freedom information of the steering wheel based on the even-numbered frame image, posture information of the steering wheel, and degree of freedom information of the head mounted display device; the at least two tracking cameras are positioned within the head mounted display device according to a predetermined first position constraint rule to ensure that each tracking camera satisfies a viewing angle requirement of the head mounted display device; The method for positioning and tracking further includes controlling a tracking camera disposed on the head mounted display device to light up a light-emitting element on the handle for a preset lighting time when collecting even frame images at the preset second exposure time, wherein a mid-time of the second exposure time corresponds to a mid-time of the lighting time, and the lighting time is equal to or greater than the second exposure time.

2. The method of claim 1 , wherein the second exposure time is less than the first exposure time.

3. The method of claim 1 , wherein the first exposure time is adaptively adjusted according to an intensity of external ambient light.

4. The method of claim 1 , wherein four tracking cameras are positioned on the head mounted display device and / or each tracking camera has the same center point of exposure time.

5. Acquiring odd frame images collected at the preset first exposure time and even frame images collected at the preset second exposure time by a tracking camera disposed on the head mounted display device includes:

2. The method of claim 1, comprising: collecting, by the at least two tracking cameras, a plurality of odd frame images corresponding to each tracking camera at the preset first exposure time; and acquiring, by the at least two tracking cameras, a plurality of even frame images corresponding to each tracking camera at the preset second predetermined exposure time.

6. Determining degree of freedom information of the head mounted display device based on the odd frame images and posture information of the head mounted display device includes: Performing calculations using a simultaneous localization and mapping SLAM algorithm based on spatial structure information in the odd frame images to obtain the movement freedom of the head mounted display device; Obtaining a rotational degree of freedom of the head mounted display device based on the attitude information of the head mounted display device collected by a first inertial measurement unit integrated in the head mounted display device; and determining the degree of freedom information for the head mounted display device using the translational degree of freedom and the rotational degree of freedom.

7. Determining degree of freedom information of the head mounted display device based on the odd frame images and the attitude information of the head mounted display device, and then The method of claim 1 or claim 6, comprising pre-integrating the degree of freedom information of the head mounted display device in combination with the pose information of the head mounted display device to obtain high frequency degree of freedom information of the head mounted display device.

8. Determining the degree of freedom information of the steering wheel based on the even-numbered frame image, the attitude information of the steering wheel, and the degree of freedom information of the head mounted display device, performing light spot detection on the even-numbered frame images to determine position information of all light spots in the even-numbered frame images; determining three-dimensional coordinates of light-emitting bodies on the handle corresponding to the light spots in the even-numbered frame images based on position information of all the light spots in the even-numbered frame images; The method of claim 1 , further comprising determining degree of freedom information of the handle based on three-dimensional coordinates of light-emitting bodies on the handle corresponding to light spots in the even frame images, posture information of the handle, and degree of freedom information of the head-mounted display device.

9. performing light spot detection on the even-numbered frame images and determining position information of all light spots in the even-numbered frame images; Performing light spot detection on each even frame image collected by the at least two tracking cameras to determine position information of all light spots in each even frame image, including position information of at least four light spots; determining a correspondence between the light spots in the even-numbered frame images and the light emitters on the handle based on a distribution rule of each light spot in the even-numbered frame images; and using the two-dimensional coordinates of the light spots corresponding to the light emitters on the handle to determine the three-dimensional coordinates of corresponding light emitters on the handle.

10. determining degree of freedom information of the steering wheel based on three-dimensional coordinates of light-emitting bodies on the steering wheel corresponding to light spots in the even-numbered frame images, attitude information of the steering wheel, and degree of freedom information of the head mounted display device; determining a degree of freedom of movement of the handle based on three-dimensional coordinates of a light-emitting body on the handle corresponding to a light spot in the even-numbered frame image; determining a rotational degree of freedom of the steering wheel based on the attitude information of the steering wheel; determining degree of freedom information of the handle based on the translational degrees of freedom and the rotational degrees of freedom; The method of claim 8 , further comprising: performing a coordinate transformation on the degree of freedom information of the handle based on the degree of freedom information of the head mounted display device.

11. determining degree of freedom information of the steering wheel based on three-dimensional coordinates of light-emitting bodies on the steering wheel corresponding to light spots in the even-numbered frame images, attitude information of the steering wheel, and degree of freedom information of the head mounted display device, The method according to claim 8 or claim 10, comprising performing pre-integration on the degree of freedom information of the steering wheel in combination with the attitude information of the steering wheel to obtain high frequency degree of freedom information of the steering wheel.

12. the odd frame image includes at least a hand; The method of claim 1 , further comprising determining hand degree of freedom information based on the odd frame images.

13. Determining hand freedom information based on the odd frame image includes: inputting the odd frame image into a predetermined convolutional neural network model to obtain multiple key point positions of the hand; and determining hand degree of freedom information based on positions of a plurality of key points of the hand.

14. training the predetermined convolutional neural network model; The step of training the predetermined convolutional neural network model includes: Collecting a plurality of scene images including a hand; marking the locations of a number of key points of a hand in the scene images, and forming the marked images into an image training sample set; and training the convolutional neural network model based on the image training sample set.

15. 1. A positioning and tracking platform, comprising: an image capture module arranged to capture odd frame images collected at a preset first exposure time by at least two tracking cameras arranged on a head mounted display device, and even frame images collected at a preset second exposure time, the even frame images including light spots corresponding to each of a plurality of light emitters arranged on at least a handle communicatively connected to the head mounted display device; The positioning and tracking platform further comprises: a positioning and tracking module arranged to determine degree of freedom information of the head mounted display device based on the odd frame images and pose information of the head mounted display device, and to determine degree of freedom information of the handle based on the even frame images, pose information of the handle, and degree of freedom information of the head mounted display device; or or A processor and a memory having stored thereon computer instructions which, when executed by the processor, cause the device to carry out the method of any one of claims 1 to 8, the at least two tracking cameras are positioned within the head mounted display device according to a predetermined first position constraint rule to ensure that each tracking camera satisfies a viewing angle requirement of the head mounted display device; The positioning and tracking platform, wherein the image acquisition module is further configured to control the light-emitting body on the handle to be lit for a preset lighting time when collecting even frame images at the preset second exposure time by a tracking camera disposed on the head-mounted display device, wherein a mid-time of the second exposure time corresponds to a mid-time of the lighting time, and the lighting time is equal to or greater than the second exposure time.

16. A head mounted display system, A head mounted display device having at least two tracking cameras arranged thereon, a handle connected to the head mounted display device and having a plurality of light emitters arranged thereon, and a positioning and tracking platform; the at least two tracking cameras are positioned within the head mounted display device according to a predetermined first position constraint rule to ensure that each tracking camera satisfies a viewing angle requirement of the head mounted display device; A head-mounted display system, in which when collecting even frame images at the preset second exposure time using a tracking camera arranged on the head-mounted display device, a light-emitting element on the handle is controlled to be lit for a preset lighting time, the midpoint of the second exposure time corresponds to the midpoint of the lighting time, and the lighting time is equal to or longer than the second exposure time.

17. The system of claim 16 , wherein the center point of the exposure time for each tracking camera disposed on the head mounted display device is the same.

18. The system of claim 16 , wherein four tracking cameras are positioned on the head mounted display device.

19. The system of claim 16, wherein the multiple light-emitting objects arranged on the handle are arranged according to a predetermined second positional constraint rule, and the predetermined second positional constraint rule satisfies the following conditions: no overlap or adhesion of local pattern information occurs in images collected by the tracking camera within a preset distance range from the handle, and corresponding light spots of at least four light-emitting objects appear in all images collected by the tracking camera at any angle.

20. A computer readable storage medium having stored thereon a computer program which, when executed by a processor, implements the method according to any one of claims 1 to 14.

Images