Handle positioning and tracking method and device, terminal equipment and computer storage medium

Abstract

The invention discloses a positioning and tracking method and device of a handle, terminal equipment and a computer storage medium, and relates to the technical field of VR, and the positioning and tracking method of the handle determines the offset of the to-be-selected pose of the current frame of the handle according to the pose of the previous frame for positioning and tracking the handle, the to-be-selected pose of the current frame is a first solution or a plurality of second solutions obtained by performing positioning tracking calculation on the handle; and when the offset is within a preset effective offset range, taking the first solution as the current frame pose of the handle, or selecting an optimal solution from the plurality of second solutions as the current frame pose of the handle. By adopting the method and the device, the stability and the accuracy of positioning and tracking the handle can be ensured, so that the use experience of a user for the VR all-in-one machine is improved.

Description

Technical Field

[0001] The present application relates to the field of VR (Virtual Reality) technology, and in particular to a method, device, terminal equipment and computer storage medium for positioning and tracking a handle. Background Art

[0002] With the rapid development of VR (Virtual Reality) technology, accurate and stable positioning of the handles in all-in-one VR devices has become increasingly important, because the accuracy and stability of positioning and tracking of the handles will directly affect the user experience.

[0003] However, the optical positioning technology used for positioning and tracking the handle often causes the handle's posture to have multiple solutions due to the influence of the handle's LED (Light Emitting Diode) matching points and the handle's free movement. The existing solution selection method will cause the handle to have posture jumps, resulting in poor accuracy in positioning and tracking the handle. Therefore, how to ensure the stability and accuracy of positioning and tracking the handle, so as to improve the user experience of the VR all-in-one machine, is a technical problem that needs to be solved urgently in the industry. Summary of the invention

[0004] The main purpose of this application is to provide a handle positioning and tracking method, device, terminal device and computer storage medium, aiming to ensure the stability and accuracy of positioning and tracking of the handle, thereby improving the user experience of the VR all-in-one machine.

[0005] To achieve the above object, the present application provides a method for positioning and tracking a handle, and the method for positioning and tracking a handle comprises the following steps:

[0006] Determine an offset of a candidate pose of a current frame of the handle according to a previous frame pose of the handle for positioning and tracking, wherein the candidate pose of the current frame is a first solution or multiple second solutions obtained by positioning and tracking the handle;

[0007] When the offset is within a preset valid offset range, the first solution is used as the current frame pose of the handle, or an optimal solution is selected from multiple second solutions as the current frame pose of the handle.

[0008] Optionally, before the step of determining the offset of the current frame to-be-selected pose of the handle according to the previous frame pose for positioning and tracking the handle, the method further comprises:

[0009] Acquire the positioning LED spot coordinates of the handle from the positioning tracking image of the handle;

[0010] According to the number of the positioned LED spot coordinates, a corresponding PnP algorithm is selected to calculate and obtain the current frame candidate posture of the handle, wherein the LED spot coordinates include: 2D point coordinates and / or 3D point coordinates of the center of mass of the positioned LED spot.

[0011] Optionally, the offset includes an x-axis offset, a y-axis offset and a z-axis offset;

[0012] The step of determining the offset of the current frame to be selected posture of the handle includes:

[0013] Calculating an x-axis offset, a y-axis offset, and a z-axis offset of the first solution relative to the previous frame posture;

[0014] Alternatively, the x-axial offset, y-axial offset and z-axial offset of each of the second solutions relative to the previous frame posture are calculated, wherein the offset is within a preset effective offset range means that the x-axial offset, y-axial offset and z-axial offset are all within the effective offset range.

[0015] Optionally, the step of selecting an optimal solution from a plurality of the second solutions as the current frame pose of the handle includes:

[0016] Calculating a reprojection error for each of the second solutions;

[0017] An optimal solution with the smallest reprojection error and the smallest offset is selected from each of the second solutions, and the optimal solution is used as the current frame pose of the handle.

[0018] Optionally, the method further comprises:

[0019] When the offset exceeds the effective offset range, calling a preset Kalman filter to perform posture prediction on the handle;

[0020] The predicted value obtained by the Kalman filter through predicting the posture of the handle is used as the current frame posture of the handle.

[0021] Optionally, the preset Kalman filter is a Kalman filter that is corrected and updated based on the previous frame posture;

[0022] The step of calling a preset Kalman filter to predict the position and posture of the handle includes:

[0023] Determine the target position state of the handle according to the three-dimensional coordinates of the handle in the world coordinate system;

[0024] Determine a state transfer matrix of the handle according to the target position state;

[0025] The modified and updated Kalman filter is called to predict the position and posture of the handle using the state transfer matrix.

[0026] Optionally, the method further comprises:

[0027] The Kalman filter is corrected and updated using the current frame posture as an observation value, so as to continuously predict the posture of the handle based on the corrected and updated Kalman filter.

[0028] In addition, to achieve the above-mentioned purpose, the present application also provides another handle positioning tracking device, the handle positioning tracking device comprising:

[0029] An offset determination module, used to determine an offset of a candidate pose of a current frame of the handle according to a pose of a previous frame in which positioning and tracking is performed on the handle, wherein the candidate pose of the current frame is a first solution or multiple second solutions obtained by positioning and tracking calculations for the handle;

[0030] A handle posture determination module is used to use the first solution as the current frame posture of the handle when the offset is within a preset effective offset range, or to select the optimal solution from multiple second solutions as the current frame posture of the handle.

[0031] Optionally, each functional module of the handle positioning tracking device implements the steps of the positioning tracking method of each handle as described above during operation.

[0032] In addition, to achieve the above-mentioned purpose, the present application also provides a terminal device, which includes: a memory, a processor, and a handle positioning tracking program stored in the memory and executable on the processor, and the handle positioning tracking program, when executed by the processor, implements the steps of the handle positioning tracking method as described above.

[0033] In addition, to achieve the above-mentioned purpose, the present application also provides a computer storage medium, on which a handle positioning tracking program is stored. When the handle positioning tracking program is executed by a processor, the steps of the handle positioning tracking method as described above are implemented.

[0034] The embodiments of the present application provide a positioning and tracking method, apparatus, terminal device, and computer storage medium for a handle. The positioning and tracking method for the handle determines an offset of a candidate pose of a current frame of the handle based on a pose of a previous frame in which positioning and tracking is performed on the handle, wherein the candidate pose of the current frame is a first solution or multiple second solutions obtained by positioning and tracking the handle. When the offset is within a preset effective offset range, the first solution is used as the current frame pose of the handle, or the optimal solution is selected from multiple second solutions as the current frame pose of the handle.

[0035] In an embodiment of the present application, a positioning and tracking method for a handle provided by the present application, during the process of positioning and tracking the handle, when it is calculated that the current frame candidate pose of the handle includes a first solution or multiple second solutions, based on the previous frame pose determined for positioning and tracking the handle, the offset of one or more current frame candidate poses of the handle is first calculated and determined, and then, when it is determined that the offset is within a preset valid offset range, the first solution is directly used as the current frame pose for tracking and positioning the handle, or the optimal solution is selected from the multiple second solutions as the current frame pose of the handle.

[0036] In this way, the embodiment of the present application is based on determining the offset of one or more current frame candidate poses of the handle, so that only when the offset is within the valid offset range, the current frame pose of the handle is determined from the one or more current frame candidate poses, thereby ensuring the stability and accuracy of positioning and tracking the handle. Moreover, in view of the problem of multiple pose solutions in the process of positioning and tracking the handle, the embodiment of the present application selects the optimal solution from multiple solutions as the current frame pose of the handle, thereby avoiding the situation of handle pose jumping, so as to ensure the accuracy of positioning and tracking the handle. That is, the embodiment of the present application can ensure the stability and accuracy of positioning and tracking the handle, thereby improving the user experience of the VR all-in-one machine. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 It is a schematic diagram of the structure of a terminal device in the hardware operating environment involved in the embodiment of the present application;

[0038] Figure 2 This is a schematic diagram of the steps of an embodiment of a method for positioning and tracking a handle of the present application;

[0039] Figure 3 This is a schematic diagram of an application flow involved in an embodiment of a method for positioning and tracking a handle of the present application;

[0040] Figure 4 This is a schematic diagram of the functional module structure involved in an embodiment of the positioning and tracking device of the handle of the present application.

[0041] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0042] It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0043] Reference Figure 1 , Figure 1 This is a schematic diagram of the terminal device structure of the hardware operating environment involved in the embodiment of the present application.

[0044] It should be noted that Figure 1 That is, it is a structural diagram of the hardware operating environment of the terminal device. The terminal device of the embodiment of the present application can be a VR all-in-one machine that executes the positioning and tracking method of the handle of the present application, a processing terminal or module configured in the VR all-in-one machine, or a terminal device connected to the VR all-in-one machine or the processing terminal or module, wherein the terminal device can specifically be a data storage control terminal, a PC, or a portable computer, etc. that executes the positioning and tracking method of the handle of the present application.

[0045] like Figure 1 As shown, the terminal device may include: a processor 1001, such as a central processing unit (CPU), a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. Among them, the communication bus 1002 is used to realize the connection and communication between these components. The user interface 1003 may include a display screen (Display), an input unit such as a keyboard (Keyboard), and the optional user interface 1003 may also include a standard wired interface and a wireless interface. The network interface 1004 may optionally include a standard wired interface and a wireless interface (such as a wireless fidelity (WIreless-FIdelity, WI-FI) interface). The memory 1005 may be a high-speed random access memory (Random Access Memory, RAM) memory, or a stable non-volatile memory (Non-Volatile Memory, NVM), such as a disk memory. The memory 1005 may also be a storage device independent of the aforementioned processor 1001.

[0046] Those skilled in the art will understand that Figure 1 The structure shown in the figure does not constitute a limitation on the terminal device, and may include more or less components than shown in the figure, or combine certain components, or arrange the components differently.

[0047] like Figure 1As shown, the memory 1005 as a storage medium may include an operating system, a data storage module, a network communication module, a user interface module, and a positioning tracking program of the handle.

[0048] exist Figure 1 In the terminal device shown, the network interface 1004 is mainly used for data communication with other devices; the user interface 1003 is mainly used for data interaction with the user; in the embodiment of the present application, the processor 1001 and the memory 1005 can be set in the terminal device. Based on this, when the terminal device is the above-mentioned mobile terminal, the positioning tracking program of the handle stored in the memory 1005 is called by the processor 1001, and the following operations are performed:

[0049] Determine an offset of a candidate pose of a current frame of the handle according to a previous frame pose of the handle for positioning and tracking, wherein the candidate pose of the current frame is a first solution or multiple second solutions obtained by positioning and tracking the handle;

[0050] When the offset is within a preset valid offset range, the first solution is used as the current frame pose of the handle, or an optimal solution is selected from multiple second solutions as the current frame pose of the handle.

[0051] Furthermore, the processor 1001 calls the positioning and tracking program of the handle stored in the memory 1005, and before executing the step of determining the offset of the current frame candidate posture of the handle according to the previous frame posture of the positioning and tracking for the handle, further performs the following operations:

[0052] Acquire the positioning LED spot coordinates of the handle from the positioning tracking image of the handle;

[0053] According to the number of the positioned LED spot coordinates, a corresponding PnP algorithm is selected to calculate and obtain the current frame candidate posture of the handle, wherein the LED spot coordinates include: 2D point coordinates and / or 3D point coordinates of the center of mass of the positioned LED spot.

[0054] Further, the offset includes an x-axis offset, a y-axis offset, and a z-axis offset; the processor 1001 calls the handle positioning tracking program stored in the memory 1005, and also performs the following operations:

[0055] Calculating an x-axis offset, a y-axis offset, and a z-axis offset of the first solution relative to the previous frame posture;

[0056] Alternatively, the x-axial offset, y-axial offset and z-axial offset of each of the second solutions relative to the previous frame posture are calculated, wherein the offset is within a preset effective offset range means that the x-axial offset, y-axial offset and z-axial offset are all within the effective offset range.

[0057] Furthermore, the processor 1001 calls the handle positioning tracking program stored in the memory 1005, and also performs the following operations:

[0058] Calculating a reprojection error for each of the second solutions;

[0059] An optimal solution with the smallest reprojection error and the smallest offset is selected from each of the second solutions, and the optimal solution is used as the current frame pose of the handle.

[0060] Furthermore, the processor 1001 calls the handle positioning tracking program stored in the memory 1005, and performs the following operations:

[0061] When the offset exceeds the effective offset range, calling a preset Kalman filter to perform posture prediction on the handle;

[0062] The predicted value obtained by the Kalman filter through predicting the posture of the handle is used as the current frame posture of the handle.

[0063] Furthermore, the preset Kalman filter is a Kalman filter that is corrected and updated based on the previous frame posture; the processor 1001 calls the positioning and tracking program of the handle stored in the memory 1005, and also performs the following operations:

[0064] Determine the target position state of the handle according to the three-dimensional coordinates of the handle in the world coordinate system;

[0065] Determine a state transfer matrix of the handle according to the target position state;

[0066] The modified and updated Kalman filter is called to predict the position and posture of the handle using the state transfer matrix.

[0067] Furthermore, the processor 1001 calls the handle positioning tracking program stored in the memory 1005, and performs the following operations:

[0068] The Kalman filter is corrected and updated using the current frame posture as an observation value, so as to continuously predict the posture of the handle based on the corrected and updated Kalman filter.

[0069] Based on the above-mentioned terminal device, various embodiments of the positioning and tracking method of the handle of the present application are provided.

[0070] Please refer to Figure 2 , Figure 2 This is a flow chart of the first embodiment of the positioning and tracking method of a handle of the present application.

[0071] It should be understood that although the logical order is shown in the flowchart, in some cases, the positioning and tracking method of the handle of the present application can of course perform the steps shown or described in a different order than here. In addition, the positioning and tracking method of the handle of the present application is applied to the above-mentioned terminal device. In order to clearly illustrate the positioning and tracking method of the handle of the present application, the following text uses one of the terminal devices, the VR all-in-one machine, as the execution subject to explain the positioning and tracking method of the handle of the present application.

[0072] like Figure 2 As shown, in the first embodiment of the positioning tracking method of the handle of the present application, the positioning tracking method of the handle of the present application may include the following steps:

[0073] Step S10: determining an offset of a candidate pose of the handle in the current frame according to a pose of the previous frame in which positioning and tracking is performed on the handle, wherein the candidate pose of the current frame is a first solution or multiple second solutions obtained by positioning and tracking calculation for the handle;

[0074] In this embodiment, when the VR all-in-one machine is performing positioning and tracking on a handle, when it is calculated that the current frame candidate pose of the handle includes a first solution or multiple second solutions, based on the previous frame pose determined by positioning and tracking the handle, the offset of one or more current frame candidate poses of the handle (i.e., a first solution or multiple second solutions) is first calculated and determined.

[0075] It should be noted that in this embodiment and other feasible embodiments described below, the VR all-in-one machine can specifically call the PnP algorithm to calculate and determine the above one or more current frame candidate poses of the handle. That is, the VR all-in-one machine uses optical positioning technology to estimate the 6DOF pose of the handle using the PnP algorithm based on the positioning LED feature points on the handle and their projections in the current frame image. Among them, the PnP algorithm specifically solves the current frame candidate pose of the handle based on the points in the 3D space of the N positioning LEDs of the handle and their corresponding 2D point lists.

[0076] Based on this, in some feasible embodiments, before the above step S10: determining the offset of the current frame candidate posture of the handle according to the previous frame posture for positioning and tracking the handle, the positioning and tracking method of the handle of the present application may also include:

[0077] Step A: acquiring the positioning LED spot coordinates of the handle from the positioning tracking image of the handle;

[0078] Step B: Selecting a corresponding PnP algorithm according to the number of the positioned LED spot coordinates to calculate and obtain the current frame candidate posture of the handle, wherein the LED spot coordinates include: 2D point coordinates and / or 3D point coordinates of the center of mass of the positioned LED spot.

[0079] In this embodiment, the VR all-in-one machine continuously collects the positioning tracking image of the handle through the camera, and then obtains the positioning LED spot coordinates of the handle from the positioning tracking image.

[0080] It should be noted that in this embodiment and other feasible embodiments described below, the positioning LED spot coordinates of the handle obtained by the VR all-in-one machine from the positioning tracking image of the handle include: the 2D point coordinates of the center of mass of the positioning LED spot - the 2D point coordinates of the center of mass of the IR LEDs of the handle corresponding to the handle in the image, and / or the 3D point coordinates - the 3D point coordinates of the LEDs on the handle in the image. Among them, the coordinate system of the LEDs 3D point coordinates is a coordinate system constructed with the center of mass of the handle as the origin; and the 2D point coordinates of the center of mass of the spot can be obtained by the VR all-in-one machine through OpenCV to detect the spot operation, thereby using the spot geometric features.

[0081] In this embodiment, after the VR all-in-one machine obtains the positioning LED spot coordinates of the above-mentioned handle from the positioning tracking image of the handle, that is, based on the different numbers N of the positioning LED spot coordinates (including the 2D point coordinates and / or 3D point coordinates of the center of mass of the positioning LED spot), different PnP algorithms are selected to solve one or more current frame candidate poses of the handle.

[0082] It should be noted that, in this embodiment and other feasible embodiments described below, a current frame candidate pose of the handle obtained by the VR all-in-one machine through the PnP algorithm is the above-mentioned first solution, and multiple current frame candidate poses of the handle obtained by the VR all-in-one machine through the PnP algorithm are the above-mentioned multiple second solutions.

[0083] Step S20: when the offset is within a preset valid offset range, using the first solution as the current frame pose of the handle, or selecting the best solution from a plurality of the second solutions as the current frame pose of the handle;

[0084] In this embodiment, after calculating and determining the offset of one or more current frame candidate poses of the handle, the VR all-in-one machine further uses the first solution as the current frame pose for tracking and positioning the handle only when it is determined that the offset is within a preset valid offset range, or selects the optimal solution from the multiple second solutions as the current frame pose of the handle.

[0085] In this embodiment, the positioning and tracking method of the handle of the present application uses a VR all-in-one machine to position and track the handle. When it is calculated that the current frame candidate pose of the handle includes a first solution or multiple second solutions, the offset of one or more current frame candidate poses of the handle is first calculated and determined according to the previous frame pose determined for positioning and tracking the handle. Then, when it is determined that the offset is within a preset valid offset range, the first solution is directly used as the current frame pose for tracking and positioning the handle, or the optimal solution is selected from the multiple second solutions as the current frame pose of the handle.

[0086] In this way, the embodiment of the present application is based on determining the offset of one or more current frame candidate poses of the handle, so that only when the offset is within the valid offset range, the current frame pose of the handle is determined from the one or more current frame candidate poses, thereby ensuring the stability and accuracy of positioning and tracking the handle. Moreover, in view of the problem of multiple pose solutions in the process of positioning and tracking the handle, the embodiment of the present application selects the optimal solution from multiple solutions as the current frame pose of the handle, thereby avoiding the situation of handle pose jumping, so as to ensure the accuracy of positioning and tracking the handle. That is, the embodiment of the present application can ensure the stability and accuracy of positioning and tracking the handle, thereby improving the user experience of the VR all-in-one machine.

[0087] It should be noted that, in this embodiment and other feasible embodiments described below, the offset of one or more current frame candidate poses of the handle determined by the VR all-in-one machine in calculation includes: x-axis offset, y-axis offset and z-axis offset.

[0088] Based on this, in the above step S10, the step of "determining the offset of the current frame candidate posture of the handle" may include:

[0089] Step S101, calculating the x-axis offset, y-axis offset and z-axis offset of the first solution relative to the previous frame posture;

[0090] In this embodiment, when the VR all-in-one machine selects the current frame candidate posture of the handle obtained by solving the PnP algorithm as the above-mentioned first solution, the VR all-in-one machine directly calculates the x-axis offset of the first solution in the x-axis direction relative to the previous frame posture of the handle, the y-axis offset of the first solution in the y-axis direction relative to the previous frame posture, and the z-axis offset of the first solution in the z-axis direction relative to the previous frame posture.

[0091] Step S102, alternatively, calculate the x-axis offset, y-axis offset and z-axis offset of each of the second solutions relative to the previous frame posture, wherein the offset is within a preset effective offset range means that the x-axis offset, y-axis offset and z-axis offset are all within the effective offset range.

[0092] In this embodiment, when the VR all-in-one machine selects the current frame candidate posture of the handle obtained by the PnP algorithm as the above-mentioned multiple second solutions, the VR all-in-one machine will sequentially calculate the x-axis offset of each second solution in the x-axis direction relative to the previous frame posture of the handle, the y-axis offset of each of the multiple second solutions in the y-axis direction relative to the previous frame posture, and the z-axis offset of each of the multiple second solutions in the z-axis direction relative to the previous frame posture.

[0093] It should be noted that, in this embodiment and other feasible embodiments described below, since the offset of one or more current frame candidate positions of the handle determined by the VR all-in-one machine in calculation includes: x-axial offset, y-axial offset and z-axial offset, when the VR all-in-one machine determines whether the offset of one or more current frame candidate positions of the handle is within the preset valid offset range, it is necessary to determine whether the x-axial offset, y-axial offset and z-axial offset of the current frame candidate position (a first solution or any one of multiple second solutions) are all within the valid offset range, and only when the x-axial offset, y-axial offset and z-axial offset are each within the valid offset range, the VR all-in-one machine determines that the offset of the current frame candidate position is within the preset valid offset range.

[0094] In addition, the preset effective offset range can be the offset threshold values ​​of the first solution or the second solution relative to the previous frame posture in the x-axis, y-axis and z-axis directions, which can be set by the R&D designers of the VR all-in-one machine based on actual needs and experience. It should be understood that based on different design needs of actual applications, in different feasible embodiments, the specific size of the effective offset range can of course be set to different values, and the positioning and tracking method of the handle of this application does not limit the specific value size of the effective offset range.

[0095] Optionally, in some feasible embodiments, in the above step S20, the step of “selecting the best solution from a plurality of the second solutions as the current frame pose of the handle” may include:

[0096] Step S201, calculating the reprojection error of each of the second solutions;

[0097] Step S202: Select an optimal solution with the smallest reprojection error and the smallest offset from each of the second solutions, and use the optimal solution as the current frame pose of the handle.

[0098] In this embodiment, when the VR all-in-one machine calculates and determines that the current frame candidate posture of the handle is a plurality of second solutions, if it is further determined that the offsets of the plurality of second solutions are within the above-mentioned effective offset range, the VR all-in-one machine can calculate the reprojection error of each second solution, and thereby screen out the optimal solution with the smallest reprojection error and offset from each second solution based on the comparison of the reprojection error and the offset of each second solution, and thus use the optimal solution as the current frame posture for positioning and tracking the handle.

[0099] For example, Figure 3 As shown, after the VR all-in-one machine uses the PnP algorithm to solve one or more current frame candidate poses R / t of the handle, it first calculates the offsets of each solution and the previous frame pose of the handle in the x / y / z three-axis directions - the x-axis offset, the y-axis offset and the z-axis offset.

[0100] If the VR all-in-one determines through comparison that the x-axis offset, the y-axis offset, and the z-axis offset are all within the effective offset range, then when the VR all-in-one obtains multiple solutions (second solutions) for the current frame candidate pose R / t of the handle using the PnP algorithm, the VR all-in-one will reproject the 3D coordinates Pw of the feature point and the R / t of the solution for each solution, calculate the projection position Puv of the solution using the formula sPuv=KPc=K(RPw+t), and calculate its reprojection error e=||Puv-Pi|| based on the feature point Pi identified in the image. Thus, the VR all-in-one can select the solution with the smallest reprojection error e from among the multiple solutions as the optimal solution of PnP, and use the optimal solution as the current frame pose for positioning and tracking the handle in the current frame.

[0101] In this embodiment, the positioning and tracking method of the handle of the present application, when the VR all-in-one machine calculates and determines that the current frame candidate pose of the handle is a plurality of second solutions, the VR all-in-one machine calculates the reprojection error of each second solution, and thus selects the optimal solution with the smallest error and offset as the handle pose of the current frame. In this way, the purpose of stable tracking of the handle can be achieved by selecting the PnP optimal solution.

[0102] Based on the first embodiment of the positioning tracking method of the handle of the present application, a second embodiment of the positioning tracking method of the handle of the present application is proposed.

[0103] In the second embodiment of the positioning tracking method of the handle of the present application, the positioning tracking method of the handle of the present application may further include the following steps:

[0104] Step S30: When the offset exceeds the effective offset range, calling a preset Kalman filter to perform posture prediction on the handle;

[0105] Step S40: using the predicted value obtained by the Kalman filter through posture prediction of the handle as the current frame posture of the handle.

[0106] In this embodiment, the VR all-in-one machine, in the process of calculating and solving one or more current frame candidate poses of the handle based on the above-mentioned PnP algorithm, also synchronously or asynchronously calls a preset Kalman filter to perform pose prediction operations for the handle. Therefore, after the VR all-in-one machine calculates and determines the offset of one or more current frame candidate poses of the handle, if it is determined that the offset exceeds the above-mentioned preset effective offset range, the VR all-in-one machine will call the Kalman filter to predict the pose of the handle and obtain the predicted value as the current frame pose for positioning and tracking the handle in the current frame.

[0107] It should be noted that in this embodiment and other feasible embodiments described below, due to the presence of noise points in the positioning and tracking image of the handle collected by the VR all-in-one machine, errors may occur in the feature point matching of the handle, or the handle rotates too fast, causing large changes in the feature points of the handle's infrared LED in the positioning and tracking image. This will cause the offset between the current candidate pose calculated by the VR all-in-one machine based on the PnP algorithm and the previous frame pose to be large (i.e., exceeding the above-mentioned effective offset range). In this case, if the current estimated current frame candidate pose is directly used as the current frame pose of the handle when the offset is large, the handle position will jump and affect the user experience. Therefore, if Figure 3 As shown, the VR all-in-one machine needs to use Kalman filtering to predict the position and posture of the handle based on the previous movement of the handle when the above-mentioned offset exceeds the effective offset range.

[0108] In addition, in some feasible embodiments, when the handle moves out of the camera's field of view in a short period of time, and the positioning tracking image of the handle cannot be used to locate the handle, the VR all-in-one machine is also required to use Kalman filtering to predict the position of the handle based on its previous movement.

[0109] Optionally, in some feasible embodiments, the VR all-in-one machine can use a Kalman filter to predict the position of the handle in the next time interval, that is, the coordinates of the center of mass of the handle in 3D space, based on the position of the handle calculated by the positioning tracking image of the handle taken by the camera. Based on this, the above-mentioned preset Kalman filter is a Kalman filter that is corrected and updated based on the position of the handle in the previous frame (the correction and update of the Kalman filter based on the position of the handle will be explained later).

[0110] Optionally, the positioning and tracking method of the handle of the present application may further include the following steps:

[0111] Step C: determining the target position state of the handle according to the three-dimensional coordinates of the handle in the world coordinate system;

[0112] Step D: determining the state transfer matrix of the handle according to the target position state;

[0113] Step E: Call the corrected and updated Kalman filter to predict the position and posture of the handle using the state transfer matrix.

[0114] In this embodiment, when the VR all-in-one machine calls the above-mentioned Kalman filter to predict the position of the handle, it first determines the target position state of the handle based on the three-dimensional coordinates of the handle in the world coordinate system estimated before the current frame or also predicted. After that, the VR all-in-one machine can further determine the state transfer matrix of the handle based on the target position state, and call the Kalman filter that has been corrected and updated using the previous frame position of the handle to use the state transfer matrix to predict the position of the handle in the current frame to obtain a predicted value.

[0115] For example, the VR all-in-one machine can specifically set the target position state of the handle to x=[Xw, Yw, Zw, v x , v y , v z , a x , a y , a z ], where Xw, Yw, and Zw represent the coordinates of the handle in the world coordinate system, v x 、v y and v z Indicates the speed at which the handle coordinates change in the x / y / z directions. x 、a y and a z Indicates the acceleration of the handle coordinates in the x / y / z axis directions.

[0116] Based on the target position state x of the handle, the VR all-in-one machine can determine the state transfer matrix A corresponding to the handle as:

[0117] Since the 9-dimensional target position state and state transfer matrix A comprehensively utilize the speed and acceleration of the handle's previous movement, the VR all-in-one machine can use the formula X k+1 =AX k +Bu k To predict the position and posture of the handle and get the predicted value.

[0118] It should be noted that, in this embodiment, the VR all-in-one machine predicts the position and posture of the handle to obtain the predicted value, which is X k In addition, since the prediction value refers to both the overall motion state of the handle and its current motion state, the VR all-in-one machine can obtain the optimal prediction of the handle position at the next moment by calling the Kalman filter to predict the handle posture.

[0119] Optionally, in some feasible embodiments, when the VR all-in-one machine predicts the posture of the handle, the first three dimensions of the initial value of the target state of the Kalman filter are not [0, 0, 0], but the previous frame posture of the handle obtained by the VR all-in-one machine based on the PnP algorithm. This also helps the predicted value of the Kalman filter to quickly become consistent with the actual movement of the handle, and gives the optimal prediction result for the posture prediction of the handle.

[0120] Optionally, in some feasible embodiments, the positioning tracking method of the handle of the present application may further include the following steps:

[0121] Step F: Using the current frame posture as an observation value to correct and update the Kalman filter, so as to continuously predict the posture of the handle based on the corrected and updated Kalman filter.

[0122] In this embodiment, after each time the VR all-in-one machine locates and tracks the handle to determine the current frame pose of the handle, the VR all-in-one machine further uses the current frame pose as an observation value to correct and update the Kalman filter. Therefore, in the subsequent process of positioning and tracking the handle, the VR all-in-one machine can continue to predict the pose of the handle through the corrected and updated Kalman filter.

[0123] For example, Figure 3 As shown, after the VR all-in-one machine determines the current frame posture of the handle, it corrects the Klaman filter by taking the current frame posture as the observation value, that is, updates the Kalman gain, posterior probability and covariance matrix, so that the corrected and updated Kalman filter can make subsequent posture predictions for the handle based on the current motion state of the handle, thereby giving the optimal prediction of the change in the posture of the handle.

[0124] It should be noted that, in this embodiment, each time the VR all-in-one machine locates and tracks the handle to determine the current frame pose of the handle, the VR all-in-one machine can determine the pose by selecting the optimal solution based on the PnP algorithm, or the VR all-in-one machine can determine the pose of the handle in the current frame by calling the Kalman filter that has been corrected and updated based on the pose of the previous frame. That is, the positioning and tracking method of the handle of the present application can achieve the purpose of stable tracking of the handle by selecting the PnP optimal solution and using the Kalaman filter to predict the movement of the handle.

[0125] Furthermore, to achieve the above objectives, the present application also provides a positioning and tracking device for a handle.

[0126] Please refer to Figure 4 , the positioning tracking device of the handle of the present application includes:

[0127] An offset determination module, used to determine an offset of a candidate pose of a current frame of the handle according to a pose of a previous frame in which positioning and tracking is performed on the handle, wherein the candidate pose of the current frame is a first solution or multiple second solutions obtained by positioning and tracking calculations for the handle;

[0128] A handle posture determination module is used to use the first solution as the current frame posture of the handle when the offset is within a preset effective offset range, or to select the optimal solution from multiple second solutions as the current frame posture of the handle.

[0129] Optionally, the positioning tracking device of the handle of the present application further includes:

[0130] A light spot coordinate acquisition module, used to acquire the positioning LED light spot coordinates of the handle from the positioning tracking image of the handle;

[0131] The PnP posture calculation module is used to select the corresponding PnP algorithm according to the number of the positioning LED spot coordinates to calculate the current frame candidate posture of the handle, wherein the LED spot coordinates include: 2D point coordinates and / or 3D point coordinates of the center of mass of the positioning LED spot.

[0132] Optionally, the offset includes an x-axis offset, a y-axis offset and a z-axis offset; and the offset determination module includes:

[0133] A first offset calculation module, used to calculate the x-axis offset, y-axis offset and z-axis offset of the first solution relative to the previous frame posture;

[0134] The second offset calculation module is used to calculate the x-axial offset, y-axial offset and z-axial offset of each of the second solutions relative to the previous frame posture, wherein the offset is within a preset effective offset range, that is, the x-axial offset, y-axial offset and z-axial offset are all within the effective offset range.

[0135] Optionally, the handle posture determination module includes:

[0136] A reprojection error calculation module, used for calculating the reprojection error of each of the second solutions;

[0137] A posture determination unit is used to select an optimal solution with the smallest reprojection error and the smallest offset from each of the second solutions, and use the optimal solution as the current frame posture of the handle.

[0138] Optionally, the handle posture determination module of the positioning and tracking device of the handle of the present application is also used to use the predicted value obtained by using a preset Kalman filter to predict the posture of the handle as the current frame posture of the handle when the offset exceeds the effective offset range.

[0139] Optionally, the preset Kalman filter is a Kalman filter that is corrected and updated based on the previous frame posture; the positioning and tracking device of the handle of the present application also includes:

[0140] The Kalman prediction module is used to determine the target position state of the handle according to the three-dimensional coordinates of the handle in the world coordinate system; determine the state transfer matrix of the handle according to the target position state; and call the corrected and updated Kalman filter to use the state transfer matrix to predict the posture of the handle.

[0141] Optionally, the positioning tracking device of the handle of the present application further includes:

[0142] The Kalman filter update module is used to correct and update the Kalman filter by taking the current frame posture as an observation value, so as to continuously predict the posture of the handle based on the corrected and updated Kalman filter.

[0143] The specific embodiments of the operation of the various functional modules of the handle positioning and tracking device provided in the present application are basically the same as the embodiments of the handle positioning and tracking method described above, and will not be described in detail here.

[0144] In addition, the present application also provides a terminal device, which includes: a memory, a processor, and a handle positioning tracking program stored in the memory and executable on the processor, and when the processor of the terminal device executes the handle positioning tracking program, it implements the steps of the handle positioning tracking method described in any of the above embodiments.

[0145] The specific embodiments of the terminal device of the present application are basically the same as the embodiments of the positioning and tracking method of the above-mentioned handle, and will not be described in detail here.

[0146] In addition, the present application also provides a computer storage medium, on which a handle positioning tracking program is stored. When the handle positioning tracking program is executed by a processor, the steps of the handle positioning tracking method described in any of the above embodiments are implemented.

[0147] The specific embodiments of the computer storage medium of the present invention are basically the same as the embodiments of the positioning and tracking method of the handle described above, and will not be described in detail here.

[0148] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or system including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or system. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or system including the element.

[0149] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0150] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus the necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product is stored in a storage medium (such as ROM / RAM, disk, CD) as described above, including a number of instructions for enabling a terminal device (which can be a device for executing the positioning and tracking method of the handle of the present application, and the terminal device can specifically be a mobile terminal, a data storage control terminal, a PC or a portable computer terminal, etc.) to execute the methods described in each embodiment of the present application.

[0151] The above are only preferred embodiments of the present application, and are not intended to limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A method for positioning and tracking a handle, It is characterized in that The positioning tracking method of the handle comprises the following steps: Determine an offset of a candidate pose of a current frame of the handle according to a previous frame pose of the handle for positioning and tracking, wherein the candidate pose of the current frame is a first solution or multiple second solutions obtained by positioning and tracking the handle; When the offset is within a preset valid offset range, the first solution is used as the current frame pose of the handle, or an optimal solution is selected from multiple second solutions as the current frame pose of the handle.

2. The method for positioning and tracking a handle according to claim 1, It is characterized in that Before the step of determining the offset of the current frame candidate pose of the handle according to the previous frame pose for positioning and tracking the handle, the method further includes: Acquire the positioning LED spot coordinates of the handle from the positioning tracking image of the handle; According to the number of the positioned LED spot coordinates, a corresponding PnP algorithm is selected to calculate and obtain the current frame candidate posture of the handle, wherein the LED spot coordinates include: 2D point coordinates and / or 3D point coordinates of the centroid of the positioned LED spot.

3. The method for positioning and tracking a handle according to claim 1, It is characterized in that The offset includes an x-axis offset, a y-axis offset and a z-axis offset; The step of determining the offset of the current frame to be selected posture of the handle includes: Calculating an x-axis offset, a y-axis offset, and a z-axis offset of the first solution relative to the previous frame posture; Alternatively, the x-axial offset, y-axial offset and z-axial offset of each of the second solutions relative to the previous frame posture are calculated, wherein the offset is within a preset effective offset range means that the x-axial offset, y-axial offset and z-axial offset are all within the effective offset range.

4. The method for positioning and tracking a handle according to claim 1, It is characterized in that The step of selecting the optimal solution from the plurality of second solutions as the current frame pose of the handle comprises: Calculating a reprojection error for each of the second solutions; An optimal solution with the smallest reprojection error and the smallest offset is selected from each of the second solutions, and the optimal solution is used as the current frame pose of the handle.

5. The method for positioning and tracking a handle according to claim 1, It is characterized in that The method further comprises: When the offset exceeds the effective offset range, a predicted value obtained by performing posture prediction on the handle by a preset Kalman filter is used as the current frame posture of the handle.

6. The method for positioning and tracking a handle as claimed in claim 5, It is characterized in that The preset Kalman filter is a Kalman filter that is corrected and updated based on the previous frame posture; the method also includes: Determine the target position state of the handle according to the three-dimensional coordinates of the handle in the world coordinate system; Determine a state transfer matrix of the handle according to the target position state; The modified and updated Kalman filter is called to predict the position and posture of the handle using the state transfer matrix.

7. The method for positioning and tracking a handle according to any one of claims 1 to 6, It is characterized in that The method further comprises: The Kalman filter is corrected and updated using the current frame posture as an observation value, so as to continuously predict the posture of the handle based on the corrected and updated Kalman filter.

8. A positioning tracking device for a handle, It is characterized in that The positioning tracking device of the handle includes: An offset determination module, used to determine an offset of a candidate pose of a current frame of the handle according to a pose of a previous frame in which positioning and tracking is performed on the handle, wherein the candidate pose of the current frame is a first solution or multiple second solutions obtained by positioning and tracking calculations for the handle; A handle posture determination module is used to use the first solution as the current frame posture of the handle when the offset is within a preset effective offset range, or to select the optimal solution from multiple second solutions as the current frame posture of the handle.

9. A terminal device, It is characterized in that The terminal device includes: a memory, a processor, and a handle positioning tracking program stored in the memory and executable on the processor. When the handle positioning tracking program is executed by the processor, the steps of the handle positioning tracking method as described in any one of claims 1 to 7 are implemented.

10. A computer storage medium, It is characterized in that The computer storage medium stores a handle positioning tracking program, and when the handle positioning tracking program is executed by the processor, the steps of the handle positioning tracking method according to any one of claims 1 to 7 are implemented.

Images