Entity controller positioning method and device, equipment and storage medium

Abstract

The embodiment of the invention provides an entity controller positioning method and device, equipment and a storage medium. The method comprises: determining a relative pose relationship between an entity controller and an associated part of the entity controller, the relative pose relationship being determined by at least one entity controller pose and at least one associated part pose, the entity controller pose being determined according to a first type of image, the pose of the associated part is determined according to a second type of images, and the first type of images and the second type of images adopt different exposure strategies; determining a current associated part pose, wherein the current associated part pose is determined according to a current second-class image closest to the current; and determining an estimated pose of the entity controller according to the relative pose relationship and the current associated part pose. According to the invention, accurate positioning of the entity controller can be realized, a wider positioning range is provided for the entity controller, and a positioning blind area when the entity controller is shielded is reduced.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of data processing technology, and in particular to a method, device, equipment and storage medium for locating a physical controller. Background Art

[0002] At present, the application scenarios of Extended Reality (XR) technology are becoming more and more extensive, including Virtual Reality (VR), Augmented Reality (AR) and Mixed Reality (MR). In order to ensure the user's real motion experience in the virtual scene, XR devices are usually equipped with corresponding physical controllers, such as handles, rings, body trackers, etc., and the head-mounted display and each physical controller in the XR device are positioned and tracked to simulate the user's motion behavior in the virtual scene.

[0003] Generally, corresponding optical devices can be set in the head-mounted display and each physical controller to analyze the position information of each physical controller by optical positioning tracking. However, when the user uses the physical controller to perform some larger actions, the physical controller may be blocked by the body part holding the physical controller, and it cannot be optically positioned and tracked, which makes the positioning of the physical controller have certain limitations. Summary of the invention

[0004] The embodiments of the present application provide a positioning method, device, equipment and storage medium for a physical controller, which can achieve accurate positioning of the physical controller, provide a wider positioning range for the physical controller, reduce the positioning blind area when the physical controller is blocked, and improve the positioning stability of the physical controller.

[0005] In a first aspect, an embodiment of the present application provides a method for positioning a physical controller, the method comprising:

[0006] Determine a relative posture relationship between a physical controller and a part associated with the physical controller, wherein the relative posture relationship is determined by at least one physical controller posture and at least one associated part posture, wherein the physical controller posture is determined based on a first type of image, and the associated part posture is determined based on a second type of image, and the first type of image and the second type of image use different exposure strategies;

[0007] Determine a current associated part posture, wherein the current associated part posture is determined based on a current second-category image that is closest to the current one;

[0008] The estimated posture of the entity controller is determined according to the relative posture relationship and the current associated part posture.

[0009] In a second aspect, an embodiment of the present application provides a positioning device for a physical controller, the device comprising:

[0010] A relative posture determination module, used to determine a relative posture relationship between a physical controller and an associated part of the physical controller, wherein the relative posture relationship is determined by at least one physical controller posture and at least one associated part posture, wherein the physical controller posture is determined based on a first type of image, and the associated part posture is determined based on a second type of image, and the first type of image and the second type of image use different exposure strategies;

[0011] A related part positioning module, used to determine the current related part position and posture, wherein the current related part position and posture is determined according to the current second-category image closest to the current one;

[0012] The entity positioning module is used to determine the estimated posture of the entity controller according to the relative posture relationship and the current associated part posture.

[0013] In a third aspect, an embodiment of the present application provides an electronic device, the electronic device comprising:

[0014] A processor and a memory, the memory being used to store a computer program, and the processor being used to call and run the computer program stored in the memory to execute the method for positioning the entity controller provided in the first aspect of the present application.

[0015] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium for storing a computer program, wherein the computer program enables a computer to execute the positioning method of the entity controller provided in the first aspect of the present application.

[0016] In a fifth aspect, an embodiment of the present application provides a computer program product, including a computer program / instructions, which enable a computer to execute the positioning method of the entity controller provided in the first aspect of the present application.

[0017] Through the technical solution of the present application, different exposure strategies are first used to determine the first type of image and the second type of image, and the relative posture relationship between the physical controller and the associated part of the physical controller is determined by at least one physical controller posture determined according to the first type of image and at least one associated part posture determined according to the second type of image. Then, the current associated part posture is determined according to the current second type of image closest to the current one, so as to determine the estimated posture of the physical controller according to the relative posture relationship and the current associated part posture, and realize the accurate positioning of the physical controller, avoid the situation that the physical controller cannot be positioned when it is blocked, and combine the positioning of the associated part of the physical controller to provide a wider positioning range for the physical controller, reduce the positioning blind area when the physical controller is blocked, and improve the stability and comprehensiveness of the positioning of the physical controller. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0019] Figure 1 A flowchart of a method for positioning a physical controller provided in an embodiment of the present application;

[0020] Figure 2 A schematic diagram of the overall positioning process of the physical controller provided in an embodiment of the present application;

[0021] Figure 3 A schematic diagram of the principle of the positioning process when the physical controller provided in the embodiment of the present application is blocked;

[0022] Figure 4 A principle block diagram of a positioning device of a physical controller provided in an embodiment of the present application;

[0023] Figure 5 A schematic block diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0024] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0025] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or server that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0026] In the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or scheme described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or schemes. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

[0027] In order to solve the problem that the physical controller configured by the XR device cannot be optically positioned when it is blocked during some large actions, the inventive concept of the present application is: different exposure strategies are used to determine the first type of image and the second type of image, and the relative posture relationship between the physical controller and the associated part of the physical controller is determined by at least one physical controller posture determined according to the first type of image and at least one associated part posture determined according to the second type of image. Then, the current associated part posture is determined according to the current second type of image closest to the current one, so as to determine the estimated posture of the physical controller according to the relative posture relationship and the current associated part posture, so as to achieve accurate positioning of the physical controller, avoid the situation that the physical controller cannot be positioned when it is blocked, and combine the associated part positioning of the physical controller to provide a wider positioning range for the physical controller, reduce the positioning blind area when the physical controller is blocked, and improve the stability and comprehensiveness of the positioning of the physical controller.

[0028] Figure 1 A flowchart of a method for positioning a physical controller provided in an embodiment of the present application, which method can be applied to XR devices, but is not limited thereto. The method can be executed by a positioning device for a physical controller provided in the present application, wherein the positioning device for the physical controller can be implemented in any software and / or hardware manner. Exemplarily, the positioning device for the physical controller can be configured in an electronic device such as AR / VR / MR that can simulate a virtual scene, and the present application does not impose any restrictions on the specific type of electronic device.

[0029] Specifically, Figure 1As shown, the method may include the following steps:

[0030] S110, determining a relative position and posture relationship between the physical controller and associated parts of the physical controller.

[0031] In order to ensure the convenient movement of users in virtual scenes, XR devices are usually equipped with corresponding physical controllers, such as handles, rings, body trackers, etc. Users wear or hold various physical controllers on the corresponding body parts, such as hands, fingers or torsos, to drive each physical controller to perform various motion control operations, so as to locate and track each physical controller to realize various interactions of users in virtual scenes.

[0032] Normally, when positioning and tracking each physical controller, the head-mounted display of the XR device will use the Simultaneous Localization And Mapping (SLAM) technology to determine the corresponding head-mounted posture in real time to achieve the positioning of the head-mounted display. When positioning and tracking each physical controller, the posture of the physical controller and the posture of the associated parts of the physical controller involved in this application can be the relative posture of the physical controller and the associated parts relative to the head-mounted display, that is, the posture in the head-mounted coordinate system, or it can be the relative posture of the physical controller and the associated parts relative to the head-mounted display after calculating the relative posture of the physical controller and the associated parts relative to the head-mounted display, and then further using the posture of the head-mounted display to convert the relative posture of the physical controller and the associated parts relative to the head-mounted display to obtain the real posture of the physical controller and the associated parts in the world coordinate system.

[0033] In order to realize the optical positioning tracking of the physical controller by the XR device, multiple light-emitting elements, such as infrared light sources, can be set on the physical controller. In addition, a corresponding camera, such as an infrared camera, can be set on the head-mounted display of the XR device to continuously shoot the physical controller in motion so as to perform optical positioning tracking on it.

[0034] Since the physical controller may be blocked when the user drives it through the corresponding body part to move, it is impossible to optically locate it. Moreover, the physical controller is usually fixedly worn or held on the corresponding body part of the user, and it is usually the body part that blocks the physical controller during movement. Usually, there is a relatively fixed association relationship between each physical controller and the body part that wears or holds the physical controller in the spatial structure.

[0035] Therefore, in order to solve the problem that the physical controller cannot be optically located when it is blocked, for each physical controller configured by the XR device, the present application can use the body part that wears or holds the physical controller as the associated part of the physical controller. Then, by positioning and tracking the associated parts of each physical controller and combining the relatively fixed association relationship between the two, the positioning and tracking of the physical controller is achieved.

[0036] For example, assuming that the physical controller is a handle, the associated part of the physical controller may be a hand. The position tracking of the handle is achieved by analyzing the hand posture and combining the relative holding posture of the hand when holding the handle.

[0037] Then, in order to achieve the positioning and tracking of the physical controller, considering that the physical controller will not be always blocked during the movement, the present application can capture the first type of image of the physical controller in real time, and use the optical positioning algorithm to complete the successful optical positioning of the physical controller in the first type of image at at least one moment, thereby determining the position and posture of at least one physical controller.

[0038] Wherein, the physical controller is provided with at least one light-emitting element, then for the first type of image captured by the physical controller, the first type of image may contain light spots formed by the various light-emitting elements provided on the physical controller. Furthermore, the present application may adopt an optical positioning algorithm to analyze the light spot information in the first type of image at least at a moment, so as to achieve successful optical positioning of the physical controller. It can be seen that the position and posture of at least one physical controller in the present application can be determined based on the light spots in at least one first type of image. It can be understood that in order to be able to capture the light spots formed by the various light-emitting elements provided on the physical controller in the first type of image, the light-emitting time of each light-emitting element provided on the physical controller can be controlled to match the acquisition time of the first type of image.

[0039] Moreover, for each physical controller, the present application can also photograph the associated parts of the physical controller in real time to obtain a second type of image. Then, the associated parts of the physical controller in each second type of image can be located and tracked using an image positioning algorithm to complete the successful positioning of the associated parts in the second type of image at at least one moment, thereby determining at least one associated part posture.

[0040] In the present application, among at least one physical controller posture and at least one associated part posture, the physical controller posture and the associated part posture at the same time can be found. Then, by analyzing the conversion relationship between the physical controller posture and the associated part posture at the same time, the relative posture relationship between the physical controller and the associated part of the physical controller can be determined.

[0041] It can be understood that since there is a relatively fixed association relationship between the physical controller and the associated parts in the spatial structure, in order to ensure accurate identification of the physical controller in the first type of image and the associated parts in the second type of image, the present application can adopt different exposure strategies to shoot the physical controller separately.

[0042] Among them, for the first type of images captured by a certain exposure strategy, when the physical controller is not blocked, the first type of images can clearly show the light spot formed by the light-emitting element arranged on the physical controller, so as to realize the optical positioning of the physical controller. For the second type of images captured by another exposure strategy, the second type of images can clearly show the associated parts of the physical controller, so as to realize the image positioning of the associated parts.

[0043] It can be understood that by positioning the physical controller through the light spot in the first type of image and combining it with the positioning of the associated parts in the second type of image, the relative posture relationship between the physical controller and the associated parts can be calculated more accurately with less computing overhead.

[0044] In some possible implementations, for the first type of image, in order to ensure that the light spots emitted by each light-emitting element on the physical controller are accurately captured, the present application can set a first exposure time for the camera on the head-mounted display, and the first exposure time can be a smaller determined value. Then, if the camera on the head-mounted display uses the first exposure time to shoot the physical controller, a first type of image with low exposure can be obtained. When the physical controller is not blocked, the first type of image can weaken the background image of the physical controller and clearly capture the light spots formed by each light-emitting element on the physical controller, so as to ensure the optical positioning accuracy of the physical controller.

[0045] Moreover, for the second type of images, in order to ensure that the associated parts of the physical controller are accurately captured, the present application can also set a second exposure time for the camera on the head-mounted display, and the second exposure time is a larger value that changes dynamically according to the ambient light brightness, so that the second exposure time can be larger than the first exposure time as a whole. Then, if the camera on the head-mounted display uses the second exposure time to shoot the physical controller, a second type of image with a higher exposure can be obtained. Regardless of whether the physical controller is blocked, the second type of image can clearly capture the background picture of the physical controller, so that the associated parts of the physical controller can be clearly displayed in the second type of image, so that the associated parts of the physical controller can be identified and located later, to assist in the positioning and tracking of the physical controller when it is blocked.

[0046] It is understandable that since the second type of image can clearly show the actual objects in the scene, the head-mounted display of the XR device can use the second type of image as the corresponding environment image, and use SLAM technology to analyze the second type of image, and thus combine it with the inertial data continuously obtained by the head-mounted display to determine its own head-mounted posture in real time and realize the positioning of the head-mounted display.

[0047] S120, determining a current associated part posture, where the current associated part posture is determined based on a current second-category image that is closest to the current one.

[0048] In order to ensure the accurate positioning of the physical controller when it is blocked, considering that the associated parts of the physical controller will be clearly displayed in the second-class image, the present application can obtain the current second-class image closest to the current one according to the acquisition time of each second-class image that has been taken. Then, the image positioning algorithm is used to locate and track the associated parts of the physical controller in the current second-class image, and the corresponding current associated part posture can be obtained.

[0049] Specifically, for each current second-category image acquired, the present application can perform corresponding feature analysis on the associated parts of the entity controller in the current second-category image to extract the key point features of the associated parts of the entity controller. Then, through the image positioning algorithm, the key point features of the associated parts can be subjected to corresponding positioning analysis to determine the posture information of the associated parts in the current second-category image as the current posture of the associated parts in the present application.

[0050] Exemplarily, in order to ensure the accuracy of positioning the associated parts in the second category image, the present application can pre-train a neural network model, and the neural network model can perform key point feature analysis on the target object in any image, so as to achieve positioning and tracking of the target object. Therefore, the present application can input the current second category image into the trained neural network model, and perform key point feature analysis on the associated parts in the current second category image through the neural network model, thereby outputting the posture information of the associated parts in the current second category image, and obtaining the current posture of the associated parts in the present application.

[0051] S130, determining an estimated posture of the physical controller according to the relative posture relationship and the current posture of the associated parts.

[0052] For the associated parts of each physical controller, after determining the current posture of the associated parts, in order to achieve accurate positioning of the physical controller when it is obscured, the present application can use the predetermined relative posture relationship between the physical controller and the associated parts to perform corresponding posture transformation on the current posture of the associated parts, and the posture information of the physical controller at the same time as the current posture of the associated parts can be obtained as the estimated posture of the physical controller in the present application, thereby achieving accurate positioning of the physical controller when it is obscured.

[0053] The technical solution provided by the embodiment of the present application first adopts different exposure strategies to determine the first type of image and the second type of image, and determines the relative posture relationship between the physical controller and the associated part of the physical controller by at least one physical controller posture determined according to the first type of image and at least one associated part posture determined according to the second type of image. Then, the current associated part posture is determined according to the current second type of image closest to the current one, so as to determine the estimated posture of the physical controller according to the relative posture relationship and the current associated part posture, realize the accurate positioning of the physical controller, avoid the situation that the physical controller cannot be positioned when it is blocked, and combine the positioning of the associated part of the physical controller to provide a wider positioning range for the physical controller, reduce the positioning blind area when the physical controller is blocked, and improve the stability and comprehensiveness of the positioning of the physical controller.

[0054] This application can fully realize the accurate positioning of the physical controller by combining the optical positioning algorithm of the physical controller with the positioning of the associated parts of the physical controller. The first type of image does not detect the light spots formed by the various light-emitting elements on the physical controller, indicating that the physical controller is blocked. Then, this application needs to use the posture information of the associated parts of the physical controller and the relative posture relationship between the two to locate and track the physical controller.

[0055] In order to ensure the positioning accuracy of the physical controller, this application usually sets an inertial measurement unit (IMU) on the physical controller. The IMU unit can use a higher frequency to detect the three-axis angular velocity and acceleration of the physical controller in real time during movement, so as to obtain the IMU data of the physical controller, which is the inertial data in this application.

[0056] Since the detection frequency of the IMU unit is higher than the acquisition frequency of the physical controller and related parts by the camera on the head mounted display, the IMU unit can usually detect multiple inertial data of the physical controller in the time interval between any two images in the first and second categories of images, thereby indicating the change in the position and posture of the physical controller in this time interval.

[0057] It can be seen from this that the estimated posture of the physical controller in this application can be obtained not only by observing the posture of the associated parts and combining the relative posture relationship between the physical controller and the associated parts, but also by inertial data within a certain time interval.

[0058] Therefore, in order to ensure the accurate positioning of the physical controller, the estimated posture of the physical controller in the present application can be determined by the following steps: determine the estimated posture of the physical controller based on the relative posture relationship, the current posture of the associated parts and the first confidence, as well as the estimated posture of the previous physical controller, inertial data and the second confidence.

[0059] In other words, considering that the inertial data obtained by the physical controller through the IMU unit can only guarantee the short-term positioning accuracy, but cannot guarantee the long-term positioning accuracy of the physical controller. Figure 2 As shown, when the present application analyzes the posture of the physical controller by observing the posture of the associated parts and combining the relative posture relationship between the physical controller and the associated parts, it can also combine the inertial data of the physical controller detected by the IMU unit within a certain time interval to optimize the estimated posture of the physical controller, thereby improving the positioning accuracy of the physical controller.

[0060] On the one hand, by observing the current associated part posture in the current second-category image closest to the current one, and combining the relative posture relationship between the entity controller and the associated part, the first posture of the entity controller at the time of acquisition of the current second-category image can be determined. Moreover, by analyzing the positioning accuracy of the image positioning algorithm used for the current associated part posture, the confidence of the image positioning algorithm for the current associated part posture can be determined as the first confidence in this application.

[0061] On the other hand, since the estimated position and posture of the physical controller is determined in real time during the movement of the physical controller, the previously determined estimated position and posture of the previous physical controller can be obtained at the time of collecting the current second type of image.

[0062] In one case, if no light spot is detected in the previous first category image before the current second category image that is closest to the current one, then the estimated posture of the previous entity controller can be determined by the posture of the associated parts in the previous second category image before the current second category image that is closest to the current one and the relative posture relationship between the two.

[0063] In another case, if a light spot has been detected in a previous first-category image before the current second-category image that is closest to the current one, then the estimated position of the previous entity controller can be determined by the optical positioning result of the entity controller in the previous first-category image before the current second-category image that is closest to the current one.

[0064] Then, based on the estimated posture of the previous entity controller, the inertial data of the entity controller from the moment when the previous entity controller estimated the posture to the moment when the current second type of image is collected can be obtained from the IMU unit. Then, according to the integral of the estimated posture of the previous entity controller and the inertial data, the second posture of the entity controller at the moment of collecting the current second type of image can be determined. Moreover, by analyzing the positioning accuracy of the inertial data of the IMU unit, the confidence of the IMU unit in positioning the entity controller can be determined as the second confidence in this application.

[0065] Then, the first confidence level and the second confidence level are used to perform weighted fusion on the first pose and the second pose of the physical controller at the current acquisition moment of the second type of image, respectively, to obtain the corresponding estimated pose of the physical controller.

[0066] As an exemplary solution in the present application, in order to ensure the positioning accuracy of the physical controller, the present application can introduce an extended Kalman filter (Extended Kalman Filter, referred to as EKF) algorithm, or an error state Kalman filter (Error State Kalman Filter, referred to as ESKF) algorithm. Among them, the EKF algorithm and the ESKF algorithm are both provided with corresponding weights for the poses of the physical controller obtained in different ways, that is, the first confidence and the second confidence mentioned above. In other words, the poses of the physical controller obtained in different ways can be fused using the EKF algorithm and the ESKF algorithm to obtain the final estimated pose of the physical controller, thereby ensuring the accuracy of the position of the physical controller.

[0067] Then, the present application can directly input the relative posture relationship between the entity controller and the associated parts and the current posture of the associated parts, as well as the estimated posture of the previous entity controller and the inertial data continuously acquired from the collection time corresponding to the estimated posture of the previous entity controller to the collection time corresponding to the current posture of the associated parts, into the EKF algorithm or the ESKF algorithm.

[0068] Then, in the EKF algorithm or the ESKF algorithm, the relative posture relationship and the current posture of the associated parts can be used to determine the posture of the physical controller under one calculation method. The posture of the physical controller under another calculation method is determined by using the estimated posture of the previous physical controller and the continuously acquired inertial data. Then, the postures of the physical controller determined by the above two methods are fused through the various weights pre-set in the EKF algorithm or the ESKF algorithm to obtain the estimated posture of the physical controller.

[0069] It can be seen from the above content that the estimated posture of the entity controller determined each time in the present application can be the posture information of the entity controller at the acquisition moment of the current second-type image that is closest to the current one.

[0070] As an optional implementation scheme in the present application, the present application can adopt different exposure strategies to respectively shoot the first type of image and the second type of image. When the first type of image contains a light spot formed by a light-emitting element, an optical positioning algorithm can be used to determine the corresponding physical controller posture.

[0071] Therefore, when locating the associated parts of the physical controller through the second type of image taken each time, the present application can also determine the current physical controller posture, which is determined based on the current first type of image that is closest to the current one.

[0072] That is to say, according to the acquisition time of each first-category image that has been taken, the current first-category image closest to the current one is obtained. Then, when the current first-category image contains a light spot formed by a light-emitting element, an optical positioning algorithm can be used to locate and track the physical controller in the current first-category image to obtain the corresponding current physical controller posture.

[0073] Then, considering that the associated parts in the current second-category image may not be clearly photographed, the current associated parts posture is inaccurate, resulting in inaccurate estimated posture of the entity controller. Alternatively, the light spot formed by the light-emitting element of the entity controller in the current first-category image is not clear, which makes the current entity controller posture inaccurate. Therefore, in order to ensure the accurate positioning of the entity controller, the present application can also determine the estimated posture of the entity controller based on the relative posture relationship and the current associated parts posture, as well as the current entity controller posture and the inertial data from the acquisition time of the current first-category image to the acquisition time of the current second-category image.

[0074] That is to say, through the relative posture relationship between the entity controller and the associated parts, the posture of the current associated parts determined by the current second-type image closest to the current one is transformed, and the posture information of the entity controller at the time of acquisition of the current second-type image can be determined.

[0075] Moreover, after determining the current entity controller posture based on the current first-category image that is closest to the current one, the current entity controller posture can be the posture information of the entity controller at the time of acquisition of the current first-category image. Then, the present application can determine another posture information of the entity controller at the time of acquisition of the current second-category image based on the current entity controller posture determined by the current first-category image that is closest to the current one and the integral of the inertial data obtained from the time of acquisition of the current first-category image to the time of acquisition of the current second-category image. Furthermore, the present application can fuse the two posture information determined by different methods at the time of acquisition of the current second-category image by the entity controller to obtain the corresponding estimated posture of the entity controller.

[0076] It should be noted that, for the estimated pose of the physical controller after fusion, the estimated pose of the physical controller can be further updated by using the fused estimated pose of the physical controller and the first confidence, as well as the estimated pose of the previous physical controller, inertial data and the second confidence, so as to ensure the positioning accuracy of the physical controller.

[0077] According to one or more embodiments of the present application, considering that within the time interval between any two images in the first category and the second category, the IMU unit can usually detect multiple inertial data of the entity controller. Therefore, after determining the estimated posture of the entity controller each time, that is, the posture information of the entity controller at the acquisition time of the current second category image closest to the current one, the present application can also determine the real-time posture of the entity controller based on the estimated posture of the entity controller and the inertial data from the acquisition time of the current second category image to the current time.

[0078] That is to say, after capturing the current second-class image, the IMU unit will also detect multiple inertial data of the physical controller in real time. Then, the inertial data detected by the IMU unit on the physical controller from the acquisition time of the current second-class image to the current moment is obtained. Then, based on the posture information of the physical controller at the acquisition time of the current second-class image closest to the current moment, combined with the integral processing of the inertial data from the acquisition time of the current second-class image to the current moment, the real-time posture of the physical controller at the current moment can be obtained.

[0079] As an optional implementation scheme in the present application, considering that within the time interval between any two images in the first type of image and the second type of image, the IMU unit can usually detect multiple inertial data of the physical controller. The relative posture relationship between the physical controller and the associated parts can be determined by at least one physical controller posture determined according to the first type of image and at least one associated part posture determined according to the second type of image.

[0080] In the present application, in the continuously captured first-type images and second-type images, the physical controller posture can be continuously determined based on the light spots in each first-type image, and the associated part posture can be continuously determined based on each second-type image. Then, the relative posture relationship between the physical controller and the associated part can be determined by at least one physical controller posture, at least one associated part posture, and the inertial data between the acquisition time of the first-type image corresponding to the physical controller posture and the acquisition time of the second-type image corresponding to the associated part posture.

[0081] That is to say, for at least one physical controller posture and at least one associated part posture, the present application can directly obtain the inertial data detected by the IMU unit for the physical controller between the acquisition time of the first type of image corresponding to the physical controller posture and the acquisition time of the second type of image corresponding to the associated part posture.

[0082] Then, according to the before-and-after acquisition relationship between at least one physical controller posture and at least one associated part posture, the physical controller posture and the associated part posture at the same moment can be obtained based on one of the physical controller posture and the associated part posture in combination with the integral processing of the inertial data, so as to determine the relative posture relationship between the two.

[0083] In some possible implementations, for at least one associated part posture determined based on the second category image, the associated part posture can be determined based on the entity region of interest within the second category image, and the entity region of interest within the second category image can be determined based on the entity controller posture corresponding to the previous first category image of the second category image and the inertial data from the acquisition time of the previous first category image to the acquisition time of the second category image.

[0084] That is to say, since the physical controller is worn or held on the associated part, the spatial posture of the two is not much different. The IMU unit on the physical controller will detect the inertial data of the physical controller in real time, and obtain the real-time inertial posture of the physical controller with higher precision. Therefore, for each captured second-class image, the present application can use an optical positioning algorithm to process the spot information in the previous first-class image of the second-class image, and obtain the posture information of the physical controller at the acquisition time of the previous first-class image. Then, the inertial data from the acquisition time of the previous first-class image to the acquisition time of the second-class image is obtained, and according to the posture information of the physical controller at the acquisition time of the previous first-class image and the integral of the inertial data from the acquisition time of the previous first-class image to the acquisition time of the second-class image, the posture information of the physical controller at the acquisition time of the second-class image can be obtained.

[0085] Then, the present application can analyze the posture information of the entity controller at the time of acquisition of the second type of image to determine the approximate area where the entity controller is located in the second type of image, thereby determining the entity area of ​​interest in the second type of image.

[0086] Furthermore, since the physical controller is worn or held on the associated part, the spatial posture between the two is not much different. Considering that the performance cost of the gesture recognition algorithm itself is relatively large, if the gesture recognition algorithm is executed in the entire area of ​​the second type of image, it will cause greater performance cost. Therefore, after determining the entity region of interest in the second type of image, it can be known that the entity region of interest may contain the associated part of the physical controller. Then, the present application can perform feature analysis of the associated part on the entity region of interest to extract the key point features of the associated part, without performing feature analysis of the associated part on the entire second type of image. Executing the gesture recognition algorithm in the entity region of interest can greatly save performance cost.

[0087] Then, through the image positioning algorithm, the key point features of the associated parts can be subjected to corresponding positioning analysis to determine the posture information of the associated parts in the second type of image, and obtain at least one associated part posture in the present application to ensure the stability of the associated part positioning and improve the positioning stability when the physical controller is obscured.

[0088] The technical solution provided in the embodiment of the present application locates and tracks the associated parts of the physical controller in at least one second type of image to assist in the positioning of the physical controller when it is obscured, reduce the positioning blind spot when the physical controller is obscured, and ensure the positioning accuracy and stability of the physical controller by improving the positioning accuracy and stability of the associated parts.

[0089] According to one or more embodiments of the present application, the relative posture relationship between the physical controller and the associated parts usually needs to be determined by analyzing the posture information of the two at the same time. However, the camera on the head-mounted display cannot use two different exposure strategies to simultaneously capture the first type of image and the second type of image at the same time. Then, in order to avoid the problem of inaccurate relative posture relationship between the physical controller and the associated parts of the physical controller when the posture information of the physical controller and the associated parts of the physical controller at different times is transformed to the same time due to the large posture transformation error, such as Figure 3 As shown, the first type of images and the second type of images in the present application can be acquired alternately at a preset frequency.

[0090] Then, the relative posture relationship between the entity controller and the associated parts in the present application can be determined by at least one entity controller posture, at least one associated part posture, and the inertial data between the acquisition time of the first type of image corresponding to the entity controller posture and the acquisition time of the second type of image corresponding to the associated part posture.

[0091] The acquisition frequency of the inertial data of the physical controller by the IMU unit may be higher than the acquisition frequency of the first type of images and / or the second type of images.

[0092] Moreover, in order to avoid errors in the relative posture relationship between the physical controller and the associated parts caused by the posture changes of the physical controller and the associated parts at different times, the present application, for at least one physical controller posture and at least one associated part posture adopted for the relative posture relationship, the acquisition time of the first type of image corresponding to the physical controller posture and the acquisition time of the second type of image corresponding to the associated part posture are adjacent.

[0093] That is to say, for every two adjacent first-type images and second-type images, the first-type image will contain a light spot formed by a light-emitting element. Then, the present application can use an optical positioning algorithm to perform positioning analysis on the light spot in the first-type image to determine the corresponding physical controller posture. Moreover, the present application can also use an image positioning algorithm to perform positioning analysis on the associated parts of the physical controller in the second-type image to determine the corresponding associated parts posture.

[0094] Then, by acquiring a plurality of inertial data detected by the IMU unit between two adjacent acquisition moments of the first type of image and the second type of image. According to the sequential acquisition relationship of the adjacent first type of image and the second type of image, the physical controller posture and the associated part posture at the same moment can be obtained based on the first posture information of the physical controller posture and the associated part posture respectively determined by the adjacent first type of image and the second type of image and the integration of the plurality of inertial data detected between two adjacent acquisition moments of the first type of image and the second type of image, thereby determining the corresponding relative posture relationship.

[0095] It is understandable that, since the present application will alternately collect the first type of images and the second type of images, so as to continuously determine the pose of the entity controller in the first type of images and the pose of the associated parts in the second type of images. Therefore, the present application can continuously update the relative pose relationship between the entity controller and the associated parts of the entity controller by continuously analyzing the pose of the entity controller and the pose of the associated parts at adjacent moments, as well as the multiple inertial data detected by the IMU unit in this adjacent time period, to ensure the accuracy of the relative pose relationship, so as to further improve the accurate positioning of the entity controller.

[0096] For the positioning process of the entity controller, such as Figure 3As shown, the present application can be that the acquisition times of the current first-category image and the current second-category image that are closest to the current one are adjacent. Then, the current entity controller posture can be determined according to the current first-category image, and the current associated part posture can be determined according to the current second-category image. By integrating the current entity controller posture and the inertial data from the acquisition time of the current first-category image to the acquisition time of the current second-category image, the posture information of the entity controller at the acquisition time of the current second-category image can be obtained. Then, based on the posture information of the entity controller at the acquisition time of the current second-category image and the posture of the current associated part, the latest relative posture relationship between the entity controller and the associated part can be determined.

[0097] Then, according to the latest relative posture relationship, the posture of the current associated part is transformed to determine the estimated posture of the physical controller. Then, according to the estimated posture of the physical controller and the integral of the inertial data from the acquisition time of the current second type of image to the current time, the real-time posture of the physical controller can be determined, so as to achieve accurate positioning of the physical controller.

[0098] The technical solution provided in the embodiment of the present application improves the positioning accuracy of the physical controller when it is obscured by continuously updating the relative posture relationship between the associated parts of the physical controller and the physical controller during the alternating acquisition process of the first type of images and the second type of images.

[0099] Figure 4 This is a principle block diagram of a positioning device of a physical controller provided in an embodiment of the present application. Figure 4 As shown, the positioning device 400 of the physical controller may include:

[0100] A relative posture determination module 410 is used to determine a relative posture relationship between a physical controller and an associated part of the physical controller, wherein the relative posture relationship is determined by at least one physical controller posture and at least one associated part posture, wherein the physical controller posture is determined based on a first type of image, and the associated part posture is determined based on a second type of image, and the first type of image and the second type of image use different exposure strategies;

[0101] The associated part positioning module 420 is used to determine the current associated part position and posture, wherein the current associated part position and posture is determined according to the current second-category image closest to the current one;

[0102] The entity positioning module 430 is used to determine the estimated posture of the entity controller according to the relative posture relationship and the current associated part posture.

[0103] In some possible implementations, the physical controller is provided with a light-emitting element, the first type of image includes a light spot formed by the light-emitting element, and the position and posture of the physical controller is determined according to the light spot in the first type of image.

[0104] In some implementations, the entity positioning module 430 may be specifically configured to:

[0105] The estimated pose of the entity controller is determined according to the relative pose relationship, the current associated part pose and the first confidence, and the previous entity controller estimated pose, inertial data and the second confidence.

[0106] In some implementations, the entity positioning module 430 may also be used to:

[0107] The current entity controller posture is determined, wherein the current entity controller posture is determined according to a current first-category image that is closest to the current one.

[0108] In some implementations, the entity positioning module 430 may also be specifically configured to:

[0109] The estimated posture of the entity controller is determined according to the relative posture relationship and the current posture of the associated part, as well as the current posture of the entity controller and the inertial data from the acquisition time of the current first type of image to the acquisition time of the current second type of image.

[0110] In some implementations, the positioning device 400 of the physical controller may further include:

[0111] The entity real-time positioning module is used to determine the real-time position and posture of the entity controller according to the estimated position and posture of the entity controller and the inertial data from the acquisition time of the current second type of image to the current time.

[0112] In some possible implementations, the associated part posture is determined based on the entity region of interest within the second category image, and the entity region of interest within the second category image is determined based on the entity controller posture corresponding to the previous first category image of the second category image and the inertial data from the acquisition time of the previous first category image to the acquisition time of the second category image.

[0113] In some possible implementations, the relative posture relationship is determined by at least one physical controller posture, at least one associated part posture, and inertial data between the capture time of a first type of image corresponding to the physical controller posture and the capture time of a second type of image corresponding to the associated part posture.

[0114] In some possible implementations, the first type of images and the second type of images are acquired alternately at a preset frequency.

[0115] In some possible implementations, the relative posture relationship is determined by at least one physical controller posture, at least one associated part posture, and inertial data between a capture time of a first type of image corresponding to the physical controller posture and a capture time of a second type of image corresponding to the associated part posture, and the capture time of the first type of image corresponding to the physical controller posture and the capture time of the second type of image corresponding to the associated part posture are adjacent.

[0116] In some possible implementations, the inertial data is inertial data of the physical controller, and a collection frequency of the inertial data is higher than a collection frequency of the first category of images and / or the second category of images.

[0117] In the embodiment of the present application, different exposure strategies are first used to determine the first type of image and the second type of image, and the relative posture relationship between the physical controller and the associated part of the physical controller is determined by at least one physical controller posture determined according to the first type of image and at least one associated part posture determined according to the second type of image. Then, the current associated part posture is determined according to the current second type of image closest to the current one, so as to determine the estimated posture of the physical controller according to the relative posture relationship and the current associated part posture, and realize the accurate positioning of the physical controller, avoid the situation that the physical controller cannot be positioned when it is blocked, and combine the positioning of the associated part of the physical controller to provide a wider positioning range for the physical controller, reduce the positioning blind area when the physical controller is blocked, and improve the stability and comprehensiveness of the positioning of the physical controller.

[0118] It should be understood that the device embodiment and the method embodiment in the present application may correspond to each other, and similar descriptions may refer to the method embodiment in the present application. To avoid repetition, they will not be described here.

[0119] Specifically, Figure 4 The device 400 shown can execute any method embodiment provided in this application, and Figure 4 The aforementioned and other operations and / or functions of each module in the illustrated device 400 are respectively for implementing the corresponding processes of the above method embodiment, and for the sake of brevity, they are not described again here.

[0120] The above method embodiment of the embodiment of the present application is described from the perspective of the functional module in conjunction with the accompanying drawings. It should be understood that the functional module can be implemented in hardware form, can be implemented by instructions in software form, and can also be implemented by a combination of hardware and software modules. Specifically, the steps of the method embodiment in the embodiment of the present application can be completed by the hardware integrated logic circuit and / or software form instructions in the processor, and the steps of the method disclosed in the embodiment of the present application can be directly embodied as a hardware decoding processor to perform, or a combination of hardware and software modules in the decoding processor to perform. Optionally, the software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, an electrically erasable programmable memory, a register, etc. The storage medium is located in a memory, and the processor reads the information in the memory, and completes the steps in the above method embodiment in conjunction with its hardware.

[0121] Figure 5 A schematic block diagram of an electronic device provided in an embodiment of the present application.

[0122] like Figure 5 As shown, the electronic device 500 may include:

[0123] The memory 510 and the processor 520, the memory 510 is used to store the computer program and transmit the program code to the processor 520. In other words, the processor 520 can call and run the computer program from the memory 510 to implement the method in the embodiment of the present application.

[0124] For example, the processor 520 may be configured to execute the above method embodiments according to instructions in the computer program.

[0125] In some embodiments of the present application, the processor 520 may include but is not limited to:

[0126] General-purpose processor, digital signal processor (DSP), application-specific integrated circuit (ASIC), field programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic device, discrete hardware components, etc.

[0127] In some embodiments of the present application, the memory 510 includes but is not limited to:

[0128] Volatile memory and / or non-volatile memory. Among them, the non-volatile memory can be read-only memory (ROM), programmable ROM (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM) or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link DRAM (SLDRAM), and direct RAM bus random access memory (DR RAM).

[0129] In some embodiments of the present application, the computer program may be divided into one or more modules, which are stored in the memory 510 and executed by the processor 520 to complete the method provided by the present application. The one or more modules may be a series of computer program instruction segments capable of completing specific functions, and the instruction segments are used to describe the execution process of the computer program in the electronic device 500.

[0130] like Figure 5 As shown, the electronic device may also include:

[0131] The transceiver 530 may be connected to the processor 520 or the memory 510 .

[0132] The processor 520 may control the transceiver 530 to communicate with other devices, specifically, to send information or data to other devices, or to receive information or data sent by other devices. The transceiver 530 may include a transmitter and a receiver. The transceiver 530 may further include an antenna, and the number of antennas may be one or more.

[0133] It should be understood that the various components in the electronic device 500 are connected via a bus system, wherein the bus system includes not only a data bus but also a power bus, a control bus and a status signal bus.

[0134] The present application also provides a computer storage medium on which a computer program is stored. When the computer program is executed by a computer, the computer is enabled to execute the method of the above method embodiment.

[0135] The embodiment of the present application also provides a computer program product including a computer program / instruction, which, when executed by a computer, enables the computer to execute the method of the above method embodiment.

[0136] When software is used for implementation, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function according to the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (digital subscriber line, DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integration. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a digital video disc (digital video disc, DVD)), or a semiconductor medium (e.g., a solid state drive (solid state disk, SSD)), etc.

[0137] The above are only specific implementations of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A method for positioning a physical controller, characterized in that: include: Determine a relative posture relationship between a physical controller and a part associated with the physical controller, wherein the relative posture relationship is determined by at least one physical controller posture and at least one associated part posture, wherein the physical controller posture is determined based on a first type of image, and the associated part posture is determined based on a second type of image, and the first type of image and the second type of image use different exposure strategies; Determine a current associated part posture, wherein the current associated part posture is determined based on a current second-category image that is closest to the current one; The estimated posture of the entity controller is determined according to the relative posture relationship and the current associated part posture.

2. The method according to claim 1, characterized in that The physical controller is provided with a light-emitting element, the first type of image includes a light spot formed by the light-emitting element, and the position and posture of the physical controller is determined according to the light spot in the first type of image.

3. The method according to claim 1, characterized in that The step of determining the estimated pose of the entity controller according to the relative pose relationship and the current pose of the associated part includes: The estimated pose of the entity controller is determined according to the relative pose relationship, the current associated part pose and the first confidence, and the previous entity controller estimated pose, inertial data and the second confidence.

4. The method according to claim 1, characterized in that: Also includes: Determine a current entity controller posture, wherein the current entity controller posture is determined according to a current first-category image that is closest to the current one; The step of determining the estimated pose of the entity controller according to the relative pose relationship and the current pose of the associated part includes: The estimated posture of the entity controller is determined according to the relative posture relationship and the current posture of the associated part, as well as the current posture of the entity controller and the inertial data from the acquisition time of the current first type of image to the acquisition time of the current second type of image.

5. The method according to claim 1, characterized in that Also includes: The real-time posture of the entity controller is determined according to the estimated posture of the entity controller and the inertial data from the acquisition time of the current second-type image to the current time.

6. The method according to claim 1, characterized in that The associated part posture is determined according to the entity interest region in the second type of image, and the entity interest region in the second type of image is determined according to the entity controller posture corresponding to the previous first type of image of the second type of image and the inertial data from the acquisition time of the previous first type of image to the acquisition time of the second type of image.

7. The method according to claim 1, characterized in that The relative posture relationship is determined by at least one physical controller posture, at least one associated part posture, and inertial data between the acquisition time of the first type of image corresponding to the physical controller posture and the acquisition time of the second type of image corresponding to the associated part posture.

8. The method according to any one of claims 1 to 7, characterized in that: The first type of images and the second type of images are acquired alternately at a preset frequency.

9. The method according to claim 8, characterized in that The relative posture relationship is determined by at least one entity controller posture, at least one associated part posture, and inertial data between the acquisition time of the first type of image corresponding to the entity controller posture and the acquisition time of the second type of image corresponding to the associated part posture. The acquisition time of the first type of image corresponding to the entity controller posture and the acquisition time of the second type of image corresponding to the associated part posture are adjacent.

10. The method according to claim 9, characterized in that The inertial data is inertial data of the physical controller, and a collection frequency of the inertial data is higher than a collection frequency of the first type of images and / or the second type of images.

11. A positioning device for a physical controller, characterized in that: include: A relative posture determination module, used to determine a relative posture relationship between a physical controller and an associated part of the physical controller, wherein the relative posture relationship is determined by at least one physical controller posture and at least one associated part posture, wherein the physical controller posture is determined based on a first type of image, and the associated part posture is determined based on a second type of image, and the first type of image and the second type of image use different exposure strategies; A related part positioning module, used to determine the current related part position and posture, wherein the current related part position and posture is determined according to the current second-category image closest to the current one; The entity positioning module is used to determine the estimated posture of the entity controller according to the relative posture relationship and the current associated part posture.

12. An electronic device, characterized in that: include: processor; as well as A memory, configured to store executable instructions of the processor; Wherein, the processor is configured to execute the positioning method of the entity controller described in any one of claims 1-10 by executing the executable instructions.

13. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for positioning the entity controller according to any one of claims 1 to 10 is implemented.

14. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions contained in the computer program product are executed on an electronic device, the electronic device executes the method for locating the entity controller as described in any one of claims 1 to 10.

Images