Method for estimating pose of handle and virtual display equipment

Abstract

The invention relates to the technical field of virtual reality interaction, provides a method for estimating the pose of a handle and virtual display equipment, and realizes joint optimization of the pose by vision and inertial navigation by utilizing an IMU (Inertial Measurement Unit) and a plurality of light emitters on the handle. Before the pose estimation, the 3D space structure of each illuminator is optimized according to the marking result of each illuminator on the handle in multiple frames of initial handle images collected at different position angles so as to improve the accuracy of the relative pose calculation; in the pose estimation process, based on an optimized 3D space structure, a target light spot set extracted from a target handle image collected by a camera and observation data of an IMU, the relative pose between the handle and the virtual display device is initialized, and due to the fact that interference of environmental factors is eliminated from the target light spot set, the accuracy of relative pose calculation is improved, and the accuracy of pose estimation is improved. When the relative pose between the handle and the virtual display device is predicted and optimized subsequently, the stable and accurate target relative pose can be obtained.

Description

A method for estimating handle posture and virtual display device

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This disclosure claims priority to Chinese patent applications filed with the Intellectual Property Office of the People's Republic of China on September 27, 2022, with application number 202211183832.2, titled "A method for estimating the position of a handle and a virtual display device", filed with the Intellectual Property Office of the People's Republic of China on September 21, 2022, with application number 202211149262.5, titled "A method for detecting light emitters on a handle and a virtual display device", and filed with the Intellectual Property Office of the People's Republic of China on November 7, 2022, with application number 202211390797.1, titled "Method and electronic device for detecting light spot marks in handle images", the entire contents of which are incorporated by reference into this disclosure. Technical Field

[0003] The present disclosure relates to the technical field of virtual reality interaction, and provides a method for estimating a handle posture and a virtual display device. Background Art

[0004] For virtual display devices such as virtual reality (VR) and augmented reality (AR), controllers are usually used to achieve conventional interactions, just like the control relationship between a personal computer (PC) and a mouse.

[0005] However, interacting with the virtual world through a controller requires a 6DOF positional relationship between the controller and the virtual display device, enabling the controller to control the display based on this 6DOF positional relationship. Therefore, the controller's position relative to the virtual display device determines the accuracy of its control over the virtual display device, affecting the user's immersive experience. Therefore, this research is of great value.

[0006] Summary of the Invention

[0007] The present disclosure provides a method for estimating the position and posture of a handle and a virtual display device, which are used to improve the accuracy of the relative position and posture estimation between the handle and the virtual display device.

[0008] In one aspect, the present disclosure provides a method for estimating the position and pose of a handle, which is applied to a virtual display device. The virtual display device interacts with a handle, and the handle is used to control the image displayed by the virtual display device. The handle is equipped with an IMU and multiple light emitters, and the virtual display device is equipped with a multi-camera that matches the type of the light emitters. The method includes:

[0009] For the first frame of the target handle image captured by each of the multi-cameras, a target light spot set of each light emitter is obtained based on the target handle image, and the relative position between the handle and the virtual display device is initialized based on the target light spot set, the observation data synchronously collected by the IMU, and the optimized 3D spatial structure of each light emitter on the handle; wherein the 3D spatial structure is optimized based on the labeling results of each light emitter in multiple frames of initial handle images captured at different positions and angles;

[0010] For the non-first frame target handle images captured by each of the multi-cameras, the current relative pose between the handle and the virtual display device is predicted based on the relative pose corresponding to the historical target handle images, and the current target relative pose between the handle and the virtual display device is determined in combination with the observation data continuously collected by the IMU.

[0011] On the other hand, the present disclosure provides a virtual display device, including a processor, a memory, a display screen, a communication interface, and a multi-eye camera, wherein the display screen is used to display an image, the virtual display device communicates with a handle via the communication interface, the handle is used to control the image displayed on the display screen, and the type of the multi-eye camera matches the light-emitting type of multiple light emitters on the handle;

[0012] The communication interface, the multi-camera, the display screen, the memory, and the processor are connected via a bus. The memory stores a computer program. The processor performs the following operations according to the computer program:

[0013] For the first frame of the target handle image captured by the multi-camera, a target light spot set of each light emitter is obtained based on the target handle image, and the relative position between the handle and the virtual display device is initialized based on the target light spot set, the observation data synchronously collected by the IMU, and the optimized 3D spatial structure of each light emitter on the handle; wherein the 3D spatial structure is optimized based on the labeling results of each light emitter in multiple frames of initial handle images captured at different positions and angles;

[0014] For the non-first frame target handle images captured by each of the multi-cameras, the current relative pose between the handle and the virtual display device is predicted based on the relative pose corresponding to the historical target handle images, and the current target relative pose between the handle and the virtual display device is determined in combination with the observation data continuously collected by the IMU.

[0015] In some embodiments of the present disclosure, the processor optimizes the 3D spatial structure of each light emitter on the handle in the following manner:

[0016] Obtaining the 3D coordinates and first identification of each light emitter according to the 3D spatial structure of each light emitter before optimization;

[0017] According to the first identification of each light emitter, for the light emitters pre-annotated on multiple frames of initial handle images collected at different positions and angles, the 2D coordinates and second identification of the light spot formed by each light emitter on the corresponding initial handle image are obtained;

[0018] For each frame of the initial handle image, determine the relative position between the handle and the acquisition camera based on the 3D coordinates of the light source and the 2D coordinates of the light spot with the same first identifier and second identifier, and the observation data of the IMU corresponding to the corresponding frame;

[0019] A reprojection error equation is constructed, and the relative postures and 3D coordinates of each light emitter are simultaneously optimized according to the reprojection error equation to obtain the 3D spatial structure after the first optimization.

[0020] In some embodiments of the present disclosure, the processor optimizes the 3D spatial structure of each light emitter on the handle and further performs:

[0021] After obtaining the first optimized 3D spatial structure, determining a conversion pose between the first 3D point cloud and the second 3D point cloud before and after optimization based on a first 3D point cloud composed of the light emitters on the handle corresponding to the optimized 3D spatial structure and a second 3D point cloud composed of the light emitters on the handle corresponding to the 3D spatial structure before optimization;

[0022] According to the transformed posture, the 3D coordinates of each light emitter on the handle are re-determined to obtain the 3D spatial structure after the second optimization.

[0023] In some embodiments of the present disclosure, the reprojection error equation is:

[0024] Among them, K n represents the projection parameters of the nth camera, Respectively represent the rotation matrix and translation vector between the handle and camera No. 0, Respectively represent the rotation matrix and translation vector between the nth camera and the 0th camera, represents the 3D coordinates of the first light emitter identified as m on the handle, p m,n represents the 2D coordinates of the second light spot marked as m.

[0025] In some embodiments of the present disclosure, the processor obtains the target light spot set of each light emitter according to the target handle image, and the specific operation is:

[0026] Obtaining the current ambient brightness, determining a binarization threshold for each of at least two binarization methods based on the current ambient brightness, and performing binarization processing on the target handle image based on each binarization threshold to obtain a binarized handle image;

[0027] Performing contour detection on the global scope of the binary handle image to obtain a set of candidate contours of each light emitter, wherein each contour represents a light spot;

[0028] Abnormal contours in the candidate contour set are eliminated according to the prior contour shape information and the contour contrast information, so as to obtain the target light spot set of each light emitter.

[0029] In some embodiments of the present disclosure, the processor determines, based on the current ambient brightness, respective binarization thresholds of at least two binarization methods, and binarizes the visible light handle image according to the respective binarization thresholds to obtain a binarized handle image. Specifically, the processor performs the following operations:

[0030] Eliminating pixels whose grayscale values ​​are lower than a preset grayscale threshold in a grayscale handle image obtained by grayscaling the visible light handle image, and determining the respective binarization thresholds of the at least two binarization methods based on a new histogram of the grayscale handle image after pixel elimination;

[0031] Comparing the current ambient brightness with a preset brightness threshold, and determining weights corresponding to the at least two binarization thresholds respectively according to the comparison result;

[0032] According to each binarization threshold and the corresponding weight, the target binarization threshold is obtained by weighting;

[0033] The grayscale handle image is binarized according to the target binarization threshold to obtain a binarized handle image.

[0034] In some embodiments of the present disclosure, the processor determines the weights corresponding to the at least two binarization thresholds respectively according to the comparison result, and the specific operation is:

[0035] When the current ambient brightness is greater than the preset brightness threshold, setting a first weight corresponding to a first binarization threshold calculated by the first binarization method to be greater than a second weight corresponding to a second binarization threshold calculated by the second binarization method;

[0036] When the current ambient brightness is less than or equal to the preset brightness threshold, setting a first weight corresponding to a first binarization threshold calculated by the first binarization method to be less than a second weight corresponding to a second binarization threshold calculated by the second binarization method;

[0037] The first binarization method is used to solve a histogram distribution containing a single peak, and the second binarization method is used to solve a histogram distribution containing two peaks.

[0038] In some embodiments of the present disclosure, the processor performs light spot detection in the global range of the binary handle image to obtain a target light spot set of each light emitter, and the specific operations are:

[0039] Performing contour detection on the binary handle image to obtain a set of candidate contours of each light emitter, wherein each contour represents a light spot;

[0040] Abnormal contours in the candidate contour set are eliminated according to the prior contour shape information and contour contrast information, to obtain the target light spot set of each light emitter.

[0041] In some embodiments of the present disclosure, the processor eliminates abnormal contours from the candidate contour set according to the contour contrast information in one or more of the following ways:

[0042] For each two candidate contours in the candidate contour set, determining the Euclidean distance between the center points of the circumscribed rectangles of the two candidate contours and the minimum Manhattan distance between the edges of the two candidate contours, and eliminating abnormal contours based on the Euclidean distance and the minimum Manhattan distance;

[0043] Sort all candidate contours in the candidate contour set according to their areas, and eliminate abnormal contours based on the relationship between the number of pixels in the candidate contour with the largest area and the candidate contour with the second largest area;

[0044] For each candidate contour in the candidate contour set, calculating the distance between the candidate contour and the nearest neighbor candidate contour, and eliminating outlier contours based on the distance;

[0045] The brightness mean of each candidate contour in the candidate contour set is calculated, and abnormal contours are eliminated based on the brightness mean values.

[0046] In some embodiments of the present disclosure, the processor removes abnormal contours according to the Euclidean distance and the minimum Manhattan distance, and the specific operation is:

[0047] When at least one of the Euclidean distance and the minimum Manhattan distance is less than a preset distance threshold, respectively calculating the areas of the two candidate contours;

[0048] If the areas of both candidate contours are smaller than the preset area threshold, both candidate contours are eliminated at the same time;

[0049] If at least one of the areas of the two candidate contours is not less than the preset area threshold, the brightness mean values ​​of the two candidate contours are calculated respectively, and the candidate contour corresponding to the smaller brightness mean value is eliminated.

[0050] In some embodiments of the present disclosure, the processor removes abnormal contours based on the quantitative relationship between pixels within the candidate contour with the largest area and the candidate contour with the second largest area. The specific operation is:

[0051] If the number of pixels in the candidate contour with the largest area and the candidate contour with the second largest area both exceeds a preset pixel number threshold, then calculate the multiple of the number of pixels in the candidate contour with the largest area and the candidate contour with the second largest area;

[0052] If the multiple is greater than a preset multiple threshold, the candidate contour with the largest area is eliminated.

[0053] In some embodiments of the present disclosure, the processor removes outlier abnormal contours based on the distance, and the specific operation is:

[0054] Determining an adaptive outlier distance according to the side length of the candidate contour and the median of the side lengths of all candidate contours;

[0055] If the number of all candidate contours is greater than a preset number threshold, and the distance is greater than the adaptive outlier distance, the candidate contour is eliminated.

[0056] In some embodiments of the present disclosure, the processor eliminates abnormal contours from the candidate contour set according to the prior contour shape information in one or more of the following ways:

[0057] Eliminate candidate contours whose aspect ratio exceeds a first preset ratio threshold according to a relationship between the area of ​​the candidate contour and the aspect ratio of the circumscribed rectangle of the candidate contour;

[0058] Eliminate candidate contours whose area ratio to the circumscribed rectangle of the candidate contour is less than a preset ratio threshold;

[0059] Calculating the distances between the grayscale centroid of the candidate contour and the center point of the circumscribed rectangle of the candidate contour on the horizontal axis and the vertical axis, respectively, and calculating the ratio of each distance to the side length of the candidate contour, and if at least one of the two ratios exceeds a second preset ratio threshold, eliminating the candidate contour;

[0060] Determining the roundness of the candidate contour according to the total number of pixels included in the candidate contour and the side length of the candidate contour, and eliminating the candidate contour if the roundness is lower than a preset roundness threshold;

[0061] Calculating the average brightness of the candidate contours, and if the average brightness is less than a preset brightness threshold, eliminating the candidate contours;

[0062] Determine the brightness mean of a preset peripheral area of ​​the circumscribed rectangle of the candidate area and the brightness mean of the candidate contour, and if the brightness difference between the two brightness mean values ​​is less than a preset difference, eliminate the candidate contour.

[0063] In some embodiments of the present disclosure, the processor initializes the relative position between the handle and the virtual display device based on the target light spot set, the observation data synchronously collected by the IMU, and the optimized 3D spatial structure of each light emitter on the handle, specifically by:

[0064] Matching each light emitter on the optimized 3D spatial structure with a target light spot in the target light spot set to establish a corresponding relationship between the 3D light emitter and the 2D light spot;

[0065] The relative position between the handle and the virtual display device is initialized according to the 3D coordinates of the light emitter and the 2D coordinates of the light spot, which have a corresponding relationship, and the observation data synchronously collected by the IMU.

[0066] In some embodiments of the present disclosure, the processor matches each light emitter on the optimized 3D spatial structure with a target light spot in the target light spot set to establish a correspondence between the 3D light emitter and the 2D light spot. The specific operations are:

[0067] For any target light spot in the target light spot set, screen out a first specified number of candidate light spots adjacent to the target light spot from the target light spot set, and connect the target light spot with the first specified number of candidate light spots to obtain a plane figure;

[0068] According to the set of light emitters that are actually adjacent in the optimized 3D spatial structure, each light spot in the plane graphic is matched with each light emitter in the set of light emitters that are actually adjacent to each other, to obtain adjacent light spot matching pairs, wherein each adjacent light spot matching pair includes an image light spot index of the light spot and a first identifier of a light emitter that matches the light spot;

[0069] For any set of adjacent light spot matching pairs, determine multiple predicted positions of the handle corresponding to the adjacent light spot matching pairs according to the 2D coordinates of each light spot in the adjacent light spot matching pairs and the 3D coordinates of each light emitter;

[0070] For any predicted pose, projecting each light emitter into a specified image according to the predicted pose to obtain each projection light spot, and matching other light spots in the specified image except the light spots included in the plane figure with each light emitter on the handle based on the projection light spots to obtain other light spot matching pairs, wherein each other light spot matching pair includes an image spot index of the other light spot and a first identifier of the light emitter corresponding to the projection light spot matching the other light spot;

[0071] Each spot matching pair is screened according to the number of each other spot matching pair, and each target spot matching pair is obtained according to the number of each screened spot matching pair, and the first identifier of the light emitter in the target spot matching pair is determined as the second identifier of the target spot corresponding to the image spot index, wherein the spot matching pair includes the adjacent spot matching pair and the other spot matching pairs, and each matching pair represents the correspondence between the 3D light emitter and the 2D spot.

[0072] In some embodiments of the present disclosure, the processor selects a first specified number of candidate light spots adjacent to the target light spot from the target light spot set by:

[0073] Obtaining a distance between the target light spot and the other light spots according to the 2D coordinates of the target light spot and the 2D coordinates of the other light spots in the target light spot set;

[0074] According to the order of the distances between the target light spot and the other light spots from small to large, the other light spots corresponding to the first specified number of distances are selected as the candidate light spots.

[0075] In some embodiments of the present disclosure, the processor matches each light spot in the plane figure with each light emitter in the set of light emitters that are actually adjacent in the optimized 3D spatial structure, to obtain matching pairs of adjacent light spots. The specific operations are:

[0076] Arrange the light spots in the plane graph in ascending order of image light spot index to obtain a light spot list;

[0077] Traversing each light emitter in the set of actually adjacent light emitters in a specified order, taking the light emitter currently traversed as the initial position, and sorting other light emitters actually adjacent to the light emitter in the specified order to obtain a sorted list;

[0078] For any light emitter in the sorted list, add the first identifier of the light emitter and the image spot index of the spot in the light spot list having the same position as the light emitter in the sorted list to the same adjacent spot matching pair;

[0079] Determining whether there is any light emitter that has not been traversed in the set of actually adjacent light emitters;

[0080] If so, return to the step of traversing each light emitter in the set of actually adjacent light emitters in a specified order until there is no untraversed light emitter in the set of actually adjacent light emitters.

[0081] In some embodiments of the present disclosure, before projecting each light emitter into a specified image according to the predicted pose, the processor further executes:

[0082] For a plurality of predicted postures of the handle corresponding to any set of adjacent light spot matching pairs, respectively obtain predicted gravity direction vectors of the handle corresponding to the adjacent light spot matching pairs;

[0083] Obtaining an actual gravity direction vector of the handle according to a current position of the virtual display device when the designated image is captured;

[0084] Adjacent light spot matching pairs that need to be deleted are determined by using the predicted gravity direction vectors corresponding to each adjacent light spot matching pair and the actual direction vector, and the adjacent light spot matching pairs that need to be deleted are deleted.

[0085] In some embodiments of the present disclosure, the processor determines adjacent light spot matching pairs to be deleted by using the predicted gravity direction vectors corresponding to each adjacent light spot matching pair and the actual direction vector, and the specific operation is:

[0086] For any set of adjacent light spot matching pairs, obtaining the angle between the gravity direction vectors according to the predicted gravity direction vectors corresponding to the adjacent light spot matching pairs and the actual direction vector;

[0087] If the angle between the gravity direction vectors is greater than a specified angle, the adjacent light spot matching pair is determined to be the adjacent light spot matching pair that needs to be deleted.

[0088] In some embodiments of the present disclosure, the processor matches, based on the projection light spots, light spots other than the light spots included in the plane figure in the designated image with the light emitters on the handle to obtain other light spot matching pairs, specifically by performing the following operations:

[0089] For any other light spot in the designated image, obtaining the distances between the other light spot and each of the projection light spots according to the 2D coordinates of the other light spot and the 2D coordinates of each of the projection light spots;

[0090] If the shortest distance among the distances is less than the specified distance, the image spot index of the other spot and the first identifier of the light emitter corresponding to the projection spot corresponding to the shortest distance are added to the same spot matching pair, and the spot matching pair is determined as the other spot matching pair.

[0091] In some embodiments of the present disclosure, the processor screens the spot matching pairs according to the number of the other spot matching pairs, and obtains the target spot matching pairs according to the number of the screened spot matching pairs. The specific operations are:

[0092] For any predicted posture, if the number of other spot matching pairs corresponding to the predicted posture is less than a second specified number, deleting the predicted posture and other spot matching pairs corresponding to the predicted posture;

[0093] For any adjacent spot matching pair, if multiple predicted poses corresponding to the adjacent spot matching pair have been deleted, then the adjacent spot matching pair is deleted;

[0094] Count the number of matching pairs of each spot remaining after elimination;

[0095] For each spot matching pair having the same image spot index, the spot matching pair with the largest number among the spot matching pairs is determined as the target spot matching pair corresponding to the image spot index.

[0096] In some embodiments of the present disclosure, the processor determines the current target relative pose between the handle and the virtual display device based on the predicted current relative pose between the handle and the virtual display device and the observation data continuously collected by the IMU, specifically by performing the following operations:

[0097] Determining the local range of each light emitter in the current target handle image based on the 3D coordinates of each light emitter on the handle in the optimized 3D spatial structure and the predicted current relative position between the handle and the virtual display device;

[0098] Extracting the current light spots of each current light emitter within the local range of the current target handle image, and determining the light emitter corresponding to each current light spot based on nearest neighbor matching;

[0099] Establishing a reprojection constraint equation based on the corresponding 2D coordinates of the current light spot and the 3D coordinates of the 3D light emitter, and the poses of the IMU and the camera when the observation data and the current target handle image are synchronized;

[0100] Establishing a pre-integration constraint equation based on the IMU posture and the movement speed of the handle corresponding to two consecutive frames of observation data;

[0101] Combining the pre-integration constraint equation and the reprojection constraint equation, solving the pose of the IMU, the pose of the camera, and the relative pose of the IMU and the handle corresponding to the current target handle image;

[0102] According to the relative posture of the IMU and the handle, the posture of the IMU and the posture of the camera, the current target relative posture between the handle and the virtual display device is obtained.

[0103] In some embodiments of the present disclosure, the pre-integration constraint equation is:

[0104] The reprojection constraint equation is:

[0105] in, They respectively represent the rotation matrix and translation vector of the IMU in the world coordinate system corresponding to the j-th frame observation data collected by the IMU, represents the translation vector of the IMU in the world coordinate system corresponding to the j+1th frame observation data collected by the IMU, They represent the motion speed of the IMU in the world coordinate system corresponding to the j-th frame and the j+1-th frame observation data, respectively, W represents the acceleration of gravity, Δt represents the time interval between the j-th frame and the j+1-th frame of observation data collected by the IMU, LOG(·) represents the logarithmic function on the Lie group SO3 corresponding to the quaternion array, Respectively represent the pre-integrated variables of the translation vector, the motion velocity and the rotation matrix of the IMU, They respectively represent the rotation matrix and translation vector of the camera on the virtual display device in the world coordinate system corresponding to the j-th frame observation data collected by the IMU, They represent the rotation matrix and translation vector of the IMU in the handle coordinate system respectively. represents the 3D coordinates of the first light emitter on the handle, which is marked as m, and p m represents the 2D coordinates of the current light spot with the second marker m on the handle, and pro j(·) represents the projection equation of the camera.

[0106] In some embodiments of the present disclosure, the result of combining the pre-integration constraint equation and the reprojection constraint equation is:

[0107] in, They represent the rotation matrix and translation vector of the IMU in the world coordinate system corresponding to the j-th frame observation data collected by the IMU, j represents the number of frames of observation data collected by the IMU, and f j represents the pre-integration constraint equation, g j represents the reprojection constraint equation.

[0108] On the other hand, the present disclosure provides a computer-readable storage medium storing computer-executable instructions for causing a computer device to execute the method for estimating a handle pose according to some embodiments.

[0109] In the method for estimating the position of a handle and the virtual display device provided by the present invention, an IMU and multiple light emitters are installed on the handle, and multiple cameras are installed on the virtual display device, and the type of the camera matches the type of the light emitter. By estimating the relative position between the handle and the virtual display device, the handle can control the image displayed by the virtual display device, thereby completing the interaction with the virtual world. Before estimating the relative pose between the handle and the virtual display device, multiple frames of initial handle images are collected from different positions and angles to ensure that the complete number of light emitters on the handle are obtained. The 3D spatial structure of the light emitters is optimized based on the light emitters in the multiple frames of initial handle images, thereby improving the accuracy of subsequent relative pose calculations. During the pose estimation process, the relative pose between the handle and the virtual display device is initialized based on the optimized 3D spatial structure, the target light spot set extracted from the first frame target handle image collected by each camera, and the IMU observation data. Since the interference of environmental factors is eliminated when extracting the target light spot set, it helps to improve the accuracy of the relative pose calculation. After initialization is completed, for the non-first frame target handle images collected by the camera, the relative pose between the handle corresponding to the current target handle image and the virtual display device is predicted based on the relative pose between the handle and the virtual display device corresponding to the historical target handle images. Combined with the IMU observation data, the relative pose is jointly optimized by the visual inertial navigation system, thereby obtaining a smooth and accurate target relative pose between the current handle and the virtual display device. BRIEF DESCRIPTION OF THE DRAWINGS

[0110] FIG1 is a schematic diagram of an application scenario of a VR device and a controller according to some embodiments;

[0111] FIG2A is a schematic diagram of a virtual display device including multiple cameras according to some embodiments;

[0112] FIG2B is a schematic diagram of a 6DOF controller including multiple LED white lights according to some embodiments;

[0113] FIG2C is a schematic diagram of a 6DOF controller including multiple LED infrared lights according to some embodiments;

[0114] FIG3A is a schematic diagram of light emitter abnormality detection according to some embodiments;

[0115] FIG3B is a schematic diagram of light emitter abnormality detection according to some embodiments;

[0116] FIG4 is an overall architecture diagram of a method for estimating a handle pose according to some embodiments;

[0117] FIG5 is a flow chart of a method for optimizing the 3D spatial structure of light emitters on a handle according to some embodiments;

[0118] FIG6A is an image of a handle captured by a binocular infrared camera before annotation according to some embodiments;

[0119] FIG6B is an image of the handle captured by a binocular infrared camera after annotation according to some embodiments;

[0120] FIG7 is a schematic diagram of a PnP principle according to some embodiments;

[0121] FIG8 is a diagram illustrating an architecture for visual-inertial navigation combined optimization to estimate handle pose according to some embodiments;

[0122] FIG9 is a flow chart of a method for jointly estimating handle pose using visual inertial navigation according to some embodiments;

[0123] FIG10 is a flow chart of a light spot detection method according to some embodiments;

[0124] FIG11 is a flow chart of a method for image binarization according to some embodiments;

[0125] FIG12 is a flow chart of a method for eliminating abnormal contours by using the Euclidean distance and the minimum Manhattan distance between every two candidate contours according to some embodiments;

[0126] FIG13 is a flow chart of a method for eliminating abnormal contours by utilizing the quantitative relationship between pixels within two selected candidate contours according to some embodiments;

[0127] FIG14 is a flow chart of a method for eliminating outlier abnormal contours by using the distance between a candidate contour and its nearest neighbor candidate contour according to some embodiments;

[0128] FIG15 is a flow chart of a method for matching a 2D light spot with a 3D light emitter according to some embodiments;

[0129] FIG16 is a schematic diagram of a planar pattern composed of adjacent light spots according to some embodiments;

[0130] FIG17 is a flow chart of a method for quickly matching each light spot in a planar pattern with an actually adjacent set of light emitters according to some embodiments;

[0131] FIG18 is a flow chart of a method for screening matching pairs of adjacent light spots according to some embodiments;

[0132] FIG19 is a flow chart of a method for determining other spot matching pairs according to some embodiments;

[0133] FIG20 is a flow chart of a method for estimating the relative posture between a handle and a virtual display device in real time according to some embodiments;

[0134] FIG. 21 is a block diagram of a virtual display device according to some embodiments. DETAILED DESCRIPTION

[0135] To make the purposes and advantages of the embodiments of the present disclosure more clear, the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings of the embodiments of the present disclosure. Obviously, the described embodiments are part of the embodiments of the present disclosure, but not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments described in this document without making any creative efforts shall fall within the scope of protection of this disclosure.

[0136] Virtual display devices such as AR and VR generally refer to head-mounted display devices (referred to as HMDs or helmets, such as VR glasses and AR glasses) with independent processors, which provide independent computing, input, and output capabilities. Virtual display devices can be connected to external controllers, which users use to control the virtual images displayed by the virtual display device, enabling conventional interaction.

[0137] Taking a gaming scenario as an example, see Figure 1, which illustrates a virtual display device and controller application scenario according to some embodiments. In the gaming scenario shown in Figure 1, players interact with the virtual world using the controller, controlling the game screen on the virtual display device by adjusting the relative position of the controller and the virtual display device, and physically reacting to changes in the game scene. This creates an immersive experience, making the game more engaging. In particular, leveraging the large screen of a television to project the virtual game screen from the virtual display device onto the television offers enhanced entertainment value.

[0138] Generally, depending on the output posture, commonly used controllers include 3DOF controllers and 6DOF controllers. Among them, 3DOF outputs 3D rotation posture, and 6DOF controllers output 3D translation position and 3D rotation posture. Compared with 3DOF controllers, 6DOF controllers can perform more complex game actions and are more interesting.

[0139] Currently, commonly used 6DOF handles are equipped with multiple light emitters (such as LED lights), where the light emitters can emit different types of light (such as infrared light, white light, etc.), and the type of the multi-eye camera (circled in Figure 2A) on the virtual display device should be compatible with the light type.

[0140] For example, see Figure 2B, which illustrates a 6DOF controller according to some embodiments. As shown in Figure 2B, the LEDs on the controller emit white light, with the white holes representing the locations of each LED. To estimate the pose between the controller and the virtual display device based on the positions of the LEDs on the controller, the multi-camera on the virtual display device should be an RGB camera.

[0141] For another example, see Figure 2C, which is a schematic diagram of another 6DOF controller according to some embodiments. As shown in Figure 2C, the LED lights on this 6DOF controller emit infrared light (invisible to the human eye). In this case, in order to estimate the position between the controller and the virtual display device based on the position of the LED lights on the controller, the multi-camera on the virtual display device should be an infrared camera.

[0142] In practical applications, using a handle to interact with the virtual world requires obtaining the position of the handle in the virtual world, so that the handle can control the display screen of the virtual display device according to the 6DOF position.

[0143] At present, in most products on the market, the method for locating the handle posture is mainly: using the infrared camera on the virtual display device to capture the infrared image of the light emitter on the handle, through image recognition and image tracking of these infrared light emitters, and combining the 3D spatial structure of the light emitter on the handle to perform light emitter matching, 3D coordinate calculation and other operations, and finally the relative posture between the handle and the virtual display device can be obtained.

[0144] However, in the above method, since the 3D spatial structure of the light emitter on the handle is measured based on the design drawings of the handle, the accuracy is low, resulting in a large error in the pose estimation. At the same time, the pose of the handle in the current frame can be calculated through the 3D spatial structure of the light emitter on the handle and the 2D light spot in the image. However, on the one hand, the number of light emitters in a single-frame image captured by the camera is limited, resulting in low pose estimation accuracy. On the other hand, the observations of the light emitters between the consecutive multi-frame images captured by the camera are not correlated with each other, resulting in poor pose smoothness during the interaction process, affecting the visual experience.

[0145] Generally, as shown in FIG2B and FIG2C , an inertial measurement unit (IMU) is installed inside the handle to measure the movement speed of the handle, including acceleration and angular velocity. The movement speed of the handle will also affect the relative position between the handle and the virtual display device.

[0146] In view of this, the embodiments of the present disclosure provide a method for estimating the posture of a handle and a virtual display device. Based on the labeling results of the light emitters in the handle images captured by the multi-camera of the virtual display device at different positions and angles, the 3D spatial structure of the light emitters on the handle is optimized, thereby improving the accuracy of the handle posture estimation; and, by utilizing the observation data collected by the IMU on the handle and the handle images captured by the camera on the virtual display device, a posture estimation method jointly optimized by visual inertial navigation is adopted to obtain a smoother and more accurate handle posture.

[0147] At the same time, considering that the light emitter on the handle is detected through visual images, it will be affected by environmental factors to a certain extent. For example, some lights in the corridor and room shown in Figure 3A (a) may be detected incorrectly, as shown in Figure 3A (b); for another example, the text on the LED display screen shown in Figure 3B (a) may be detected incorrectly, as shown in Figure 3B (b). If the light emitter on the handle in the image is not detected accurately, it will cause a large error in the relative posture between the handle and the virtual display device, reducing the control accuracy and seriously affecting the user experience. Therefore, when estimating the posture, the embodiment of the present disclosure performs a series of processing operations on the image captured by the camera, and performs abnormal elimination on the 2D light spot of the detected light emitter in the image to improve the accuracy and robustness of the light emitter detection.

[0148] When calculating the relative pose between the controller and the virtual display device, it is necessary to match the 3D points of the light emitter on the controller with the 2D points of the light spot in the image captured by the camera. Traditional brute-force matching is time-consuming and reduces positioning efficiency. Therefore, the disclosed embodiment improves matching efficiency and accuracy by splicing adjacent light spots into a planar shape.

[0149] See Figure 4, which is an overall architecture diagram of the method for estimating the handle pose according to some embodiments, which mainly includes two parts: preprocessing and relative pose estimation. Among them, the preprocessing part mainly uses the annotation results of each light emitter in the multiple frames of initial handle images captured by the multi-camera on the virtual display device at different positions and angles to optimize the 3D spatial structure of the light emitter on the handle, obtain more accurate 3D coordinates of the light emitter, and thus improve the accuracy of the handle pose estimation. The relative pose estimation part mainly uses the target handle image captured by the camera and the observation data collected by the IMU, and adopts the visual inertial navigation joint optimization method to estimate the relative pose between the handle and the virtual display device in real time.

[0150] In the relative pose estimation part, spot detection is performed on the target handle image captured by the camera to obtain the target spot set of each light emitter on the handle in the image. The relative pose estimation is performed by combining the optimized 3D spatial structure of each light emitter and the observation data collected by the IMU. During the pose estimation process, it is necessary to match the 3D points of each light emitter on the handle with the 2D points of the light spot formed by each light emitter in the image. Usually, the first identification of each light emitter on the handle is set on the design drawing. Therefore, the matching process can be regarded as the second identification process of determining the matching light spot of each light emitter.

[0151] Before the controller leaves the factory, the 3D spatial structure of each light emitter can be obtained based on the design drawings of the controller, including the position of each light emitter (represented by 3D coordinates) and the first identification (represented by a digitally encoded ID). However, due to differences in production processes, the actual 3D spatial structure of each light emitter may differ from the design drawings. If the 3D spatial structure of each light emitter on the controller is directly used for pose estimation, it may cause estimation errors and affect the user's immersive experience.

[0152] Therefore, before estimating the relative pose between the controller and the virtual display device, the disclosed embodiments optimize the 3D spatial structure of each light emitter based on multiple frames of different initial controller images. This optimization process can utilize controller images captured by at least two pre-calibrated cameras on the virtual display device, or by multiple pre-calibrated independent cameras. Regardless of the camera type used, the camera type must be compatible with the light-emitting type of the controller.

[0153] In some embodiments of the present disclosure, the specific optimization process of the 3D spatial structure of each light emitter on the handle is shown in FIG5 , and mainly includes the following steps:

[0154] S501: Obtaining the 3D coordinates and the first identifier of each light emitter according to the 3D spatial structure of each light emitter before optimization.

[0155] The 3D spatial structure of each light emitter before optimization is determined by the design drawing of the handle. By measuring the design drawing of the handle, the 3D coordinates of each light emitter on the handle in the 3D spatial structure before optimization and the first identifier of each light emitter can be obtained.

[0156] S502: According to the light emitters pre-marked on the multiple frames of initial handle images collected at different positions and angles, the 2D coordinates and the second identifier of the light spot formed by each light emitter on the corresponding initial handle image are obtained.

[0157] In the embodiment of the present disclosure, when each light emitter on the handle is lit, a multi-lens camera that matches the light emission type of the light emitter is used to capture multiple frames of initial handle images from different positions and angles to ensure that all light emitters on the handle are captured. After obtaining multiple frames of initial handle images, the position of the center point of each light emitter in each frame of the initial handle image (expressed as 2D coordinates) and the second identification of each light emitter (expressed as a digitally encoded ID) are manually pre-marked. The second identification of each light emitter is consistent with the 3D spatial structure of each light emitter.

[0158] For example, if the light emitter on the handle is an LED infrared light and the acquisition camera is a binocular infrared camera on a virtual display device, the initial handle image is an infrared handle image. Figure 6A shows the infrared handle image captured by the binocular infrared camera before annotation. After manual annotation, the binocular infrared handle image is shown in Figure 6B.

[0159] Because the binocular infrared cameras have different positions and angles relative to the same handle, the positions and number of the handle's light emitters differ in the synchronously captured single-frame infrared handle images. For example, as shown in Figures 6A and 6B, the infrared handle image captured by one infrared camera contains five LED infrared light spots with first identifiers 2, 3, 4, 5, and 7, while the infrared handle image captured by the other infrared camera contains eight LED infrared light spots with first identifiers 2, 3, 4, 5, 6, 7, 8, and 9.

[0160] After labeling each frame of the initial handle image captured by the multi-eye camera at different positions and angles, the 2D coordinates and second identifier of the light spot formed on the initial handle image corresponding to each light emitter can be obtained based on the labeling results of each frame of the initial handle image.

[0161] Furthermore, based on the 2D coordinates and second identifier of each light spot in each frame of the initial handle image, the 3D coordinates of each light emitter are optimized using the Structure from Motion (SFM) idea to obtain the optimized 3D spatial structure of each light emitter, see S503-S506 for details.

[0162] S503: For each frame of the initial handle image, determine the relative position between the handle and the acquisition camera based on the 2D coordinates and 3D coordinates of the light emitter with the same first identifier and second identifier, and the observation data of the IMU corresponding to the corresponding frame.

[0163] For each frame of the initial handle image, the following operations are performed: based on the 2D coordinates of the light spot with the same second marker in the 2D image and the first marker in the 3D space and the 3D coordinates of the light emitter, the PnP (Perspective-n-Points) algorithm is used to determine the first relative pose between the handle and the acquisition camera corresponding to the frame, and the second relative pose between the handle and the acquisition camera is obtained by integrating the observation data of the IMU corresponding to the frame. The relative pose between the handle and the acquisition camera corresponding to the frame is obtained by fusing the first relative pose and the second relative pose.

[0164] The PnP algorithm solves the problem of object motion positioning based on 3D and 2D point pairs. Its principle is shown in Figure 7, where O represents the optical center of the camera. Several 3D points (such as A, B, C, and D) of the object in 3D space are projected onto the image plane by the camera, resulting in corresponding 2D points (such as a, b, c, and d). Given the coordinates of the 3D points and the projected relationship between the 3D points and the 2D points, the pose between the camera and the object can be estimated. In the disclosed embodiment, the projected relationship between the 3D points and the 2D points can be reflected by the first and second identifiers of the light emitter.

[0165] S504: Construct a reprojection error equation, and simultaneously optimize each relative pose and 3D coordinate according to the reprojection error equation to obtain a 3D spatial structure after the first optimization.

[0166] Since each camera is calibrated before use, the projection parameters (also called intrinsic parameters) of each camera and the relative pose between the cameras are known. Therefore, in S504, a reprojection error equation is constructed based on the projection parameters of each camera, the relative pose between the cameras, the 3D coordinates of each light emitter on the handle, and the 2D coordinates of the light spot formed by each light emitter in the initial handle image captured by each camera. By minimizing the reprojection error, the relative pose between the handle and the acquisition camera corresponding to each frame of the initial handle image, as well as the 3D coordinates of each light emitter on the handle, are simultaneously optimized to obtain the 3D spatial structure after the first optimization.

[0167] Among them, the reprojection error equation formula is expressed as follows:

[0168] In formula 1, K n represents the projection parameters of the nth camera, They represent the rotation matrix and translation vector between the handle and camera No. 0 respectively. Represent the rotation matrix and translation vector between camera No. n and camera No. 0 respectively, represents the 3D coordinates of the first light emitter identified as m on the handle, p m,nIndicates the 2D coordinates of the light spot formed by the second light emitter with the identifier m on the initial handle image captured by the nth camera.

[0169] in, Indicates the relative position between the handle and camera No. 0, Represents the relative pose between camera No. n and camera No. 0.

[0170] In some embodiments of the present disclosure, camera 0 may be the camera that captures the most light spots, also known as the primary camera. For example, in Figure 6B , the number of light spots captured by the right infrared camera is greater than that captured by the left infrared camera. In this case, the right infrared camera is camera 0 (primary camera).

[0171] After the first 3D spatial structure optimization, relatively accurate 3D coordinates of each light emitter can be obtained, but there will be a certain drift between the origin of the optimized 3D spatial structure and the origin of the 3D spatial structure before optimization. In some embodiments, to further improve the accuracy of the 3D coordinates of each light emitter, the similarity transformation (SIM3) method of three pairs of points is used to align the handle coordinate systems before and after optimization, thereby achieving a secondary optimization of the 3D spatial structure of each light emitter. The second optimization process includes:

[0172] S505: Determine the conversion pose between the first 3D point cloud and the second 3D point cloud before and after optimization based on the first 3D point cloud composed of the light emitters on the handle corresponding to the optimized 3D spatial structure and the second 3D point cloud composed of the light emitters on the handle corresponding to the 3D spatial structure before optimization.

[0173] After the first optimization of the 3D spatial structure of each light emitter on the handle, the 3D points of each light emitter form a first 3D point cloud. Before the first optimization of the 3D spatial structure of each light emitter on the handle, the 3D points of each light emitter form a second 3D point cloud. In the first 3D point cloud and the second 3D point cloud, the 3D point coordinates of each light emitter before and after optimization are known. By minimizing the drift error between the 3D coordinates of each light emitter before and after optimization, the transformation pose between the first 3D point cloud and the second 3D point cloud is obtained. The calculation formula for the transformation pose is as follows:

[0174] in, Indicates the 3D coordinates of the emitter marked as m in the handle coordinate system after the first optimization. represents the 3D coordinates of the light emitter marked as m in the handle coordinate system before the first optimization, s represents the scale transformation coefficient of the first 3D point cloud and the second 3D point cloud, (R, t) represents the conversion pose between the first 3D point cloud and the second 3D point cloud, where R represents the rotation matrix between the handle coordinate systems before and after optimization, and t represents the translation vector between the handle coordinate systems before and after optimization.

[0175] S506: Re-determine the 3D coordinates of each light emitter on the handle according to the transformed posture to obtain the 3D spatial structure after the second optimization.

[0176] According to the quasi-transposed poses between the first 3D point cloud and the second 3D point cloud of each light emitter before and after the first optimization of the 3D spatial structure, the final 3D coordinates of each light emitter on the handle are calculated and recorded as The calculation formula is as follows:

[0177] Based on the final 3D coordinates of each emitter, a second optimized 3D spatial structure can be obtained. By optimizing the 3D spatial structure of each emitter on the handle, more accurate 3D coordinates can be obtained for each emitter. This, in turn, improves the accuracy of pose estimation when estimating the relative pose between the handle and the virtual display device in real time based on the optimized 3D coordinates of each emitter.

[0178] It should be noted that the handles of the same batch are produced based on the same design drawings. Therefore, only one optimization is required for the handles of the same batch.

[0179] It should be noted that the above method for optimizing the 3D spatial structure of each light emitter on the handle can be executed by a virtual display device, and can also be executed by other devices, such as a laptop computer, a desktop computer, etc.

[0180] After optimizing the 3D spatial structure of each light emitter on the handle, the handle can be imaged using the multi-camera on the virtual display device. Combined with the observation data collected by the IMU inside the handle, the joint optimization of the posture by vision and inertial navigation can be achieved.

[0181] See FIG8 , which is a diagram illustrating an architecture for visual inertial navigation combined optimization and estimation of handle pose according to some embodiments. In FIG8 , They represent the relative pose between the IMU coordinate system on the handle and the world coordinate system, the relative pose between the handle coordinate system and the world coordinate system, and the relative pose between the camera (i.e., virtual display device) coordinate system and the world coordinate system corresponding to the jth (j=1,2,…n) frame, respectively. Indicates the relative pose between the controller coordinate system and the IMU coordinate system.

[0182] As shown in Figure 8, the joint optimization of the relative pose between the handle and the virtual display device by visual inertial navigation is achieved through the pre-integration constraints between multiple frames of observation data continuously collected by the IMU, and the reprojection constraints between the same frame of data collected by the IMU and the camera (that is, the observation data and the target handle image have the same timestamp).

[0183] FIG9 is a flowchart of a method for estimating handle pose by combining visual inertial navigation with control, according to some embodiments. The method mainly includes the following steps:

[0184] S901: Determine whether the relative posture between the handle and the virtual display device has been initialized. If not, execute S902; if so, execute S903.

[0185] In the process of real-time estimation of the relative posture between the handle and the virtual display device, the relative posture between the handle and the virtual display device can be predicted. The prediction process requires the initial value of the relative posture between the handle and the virtual display device to be given. Therefore, in the posture estimation process, it is first determined whether the relative posture between the handle and the virtual display device has been initialized. If not, the relative posture between the handle and the virtual display device is initialized. If it has been initialized, the relative posture between the handle and the virtual display device is predicted and optimized.

[0186] S902: For the first frame of the target handle image captured by the camera, obtain the target light spot set of each light emitter based on the target handle image, and initialize the relative position between the handle and the virtual display device based on the target light spot set, the observation data synchronously collected by the IMU, and the optimized 3D spatial structure of each light emitter on the handle.

[0187] In actual applications, VR experiences experience significant differences in ambient brightness, and the ambient light source can affect the detection of the light emitters on the controller. To address this issue, the disclosed embodiments provide a method that can accurately detect the 2D light spots of each light emitter in an image in both bright and dim environments.

[0188] 10 is a flow chart of a light spot detection method according to an embodiment of the present disclosure, which mainly includes the following steps:

[0189] S9021: Obtain the current ambient brightness, determine the binarization thresholds of at least two binarization methods according to the current ambient brightness, and perform binarization processing on the target handle image according to the binarization thresholds to obtain a binarized handle image.

[0190] Typically, illumination features can be extracted from images captured by a camera, and the current ambient brightness can be obtained through the illumination features.

[0191] In some disclosed embodiments, the image captured by the camera can be grayscaled to obtain a grayscale image, including but not limited to floating-point method, integer method, shift method, averaging method, etc. Further, the current ambient brightness can be determined based on the histogram of the grayscale image.

[0192] For example, when the peak of the histogram is located on the dark side with a grayscale value less than 100, it indicates that there is no bright light in the current environment. At this time, the current environment brightness is determined to be dim; when the peak of the histogram is located on the bright side with a grayscale value greater than or equal to 100, it indicates that there is bright light in the current environment. At this time, the current environment brightness is determined to be bright.

[0193] According to the current ambient brightness, the target handle image can be binarized using a target binarization threshold that matches the current ambient brightness to improve the accuracy and robustness of light emitter detection in different environments. There are two main methods for binarizing a target handle image containing multiple light emitters:

[0194] Maximum inter-class variance method: also known as Otsu's method, is a binary threshold solution method proposed in 1979. This method is based on maximizing the inter-class variance of foreground and background images. It is suitable for solving the binary threshold when the histogram distribution tends to be bimodal.

[0195] Triangulation method: It is a binary threshold solution algorithm, which is more suitable for solving the binary threshold when the histogram distribution tends to be unimodal. This method constructs a straight line from the highest peak of the histogram to the histogram on the farther side, and then solves the vertical distance from each histogram to the straight line. The histogram position corresponding to the maximum vertical distance is taken as the binarization threshold.

[0196] Due to the complex virtual game experience and the large differences in ambient brightness, neither of the two methods above can achieve a satisfactory binarization effect. To accommodate a wider range of usage scenarios, the disclosed embodiment combines the Otsu method with the triangulation method based on these two main binarization adaptive thresholding algorithms to obtain a target binarization threshold that can adapt to both bright and dim environments and achieve a more reasonable image binarization.

[0197] FIG11 is a flowchart of a method for image binarization processing in an embodiment of the present disclosure, which mainly includes the following steps:

[0198] S9021_1: Eliminate pixels whose grayscale values ​​are lower than a preset grayscale threshold in the grayscale handle image after grayscale processing of the target handle image, and determine the binarization thresholds of at least two binarization methods respectively based on the new histogram of the grayscale handle image after pixel elimination.

[0199] The brightness of each light emitter on the handle is generally stable under different environments. When calculating the binarization threshold using a binarization method, dim backgrounds with extremely low brightness should be excluded. Therefore, pixels with grayscale values ​​below a preset grayscale threshold are removed from the grayscale handle image after grayscale processing of the target handle image. A new histogram of the current image is calculated based on the remaining pixels in the grayscale handle image. Based on this new histogram, the binarization thresholds of at least two binarization methods are determined.

[0200] In certain embodiments of the present disclosure, due to the complex and diverse environments in which the controller operates, a minimum guaranteed threshold can be pre-set for each binarization method to prevent unexpected situations. If the binarization threshold calculated based on the new histogram falls below the preset minimum guaranteed threshold, the calculated binarization threshold is forcibly set to the preset minimum guaranteed threshold, thereby enhancing the algorithm's stability in special circumstances.

[0201] For example, according to the new histogram, when the binarization threshold calculated by the Otsu method is lower than the preset minimum guarantee threshold, the preset minimum guarantee threshold is set to the binarization threshold corresponding to the Otsu method; when the binarization threshold calculated by the trigonometric method is lower than the preset minimum guarantee threshold, the preset minimum guarantee threshold is set to the binarization threshold corresponding to the trigonometric method.

[0202] It should be noted that, in addition to the binarization thresholds of the Otsu method and the triangular method described above determined based on the new histogram, the binarization thresholds of other binarization methods can also be determined.

[0203] S9021_2: Compare the current environment brightness with a preset brightness threshold, and determine weights corresponding to at least two binarization thresholds according to the comparison result.

[0204] By comparing the current environment brightness with the preset brightness threshold, the degree of adaptation of the current environment brightness to the binarization threshold solved by each binarization method can be determined, and the degree of adaptation can be reflected by the weight.

[0205] Take the process of weighting the binarization thresholds solved by two binarization methods to obtain the target binarization threshold as an example, where the first binarization method is used to solve the histogram distribution containing a single peak, and the second binarization method is used to solve the histogram distribution containing two peaks. For example, the first binarization method is the triangulation method, and the second binarization method is the Otsu method. First, determine whether the current ambient brightness is greater than a preset brightness threshold. If so, it indicates that the handle is in a bright environment. At this time, the first binarization threshold calculated by the first binarization method is more adapted to the current ambient brightness, that is, the first binarization threshold calculated by the first binarization method is more accurate. Therefore, the first weight corresponding to the first binarization threshold calculated by the first binarization method is set to be greater than the second weight corresponding to the second binarization threshold calculated by the second binarization method; if not, it indicates that the handle is in a dim environment. At this time, the second binarization threshold calculated by the second binarization method is more adapted to the current ambient brightness, that is, the second binarization threshold calculated by the second binarization method is more accurate. Therefore, the first weight corresponding to the first binarization threshold calculated by the first binarization method is set to be less than the second weight corresponding to the second binarization threshold calculated by the second binarization method.

[0206] S9021_3: According to each binarization threshold and the corresponding weight, a target binarization threshold is obtained by weighting.

[0207] After obtaining the weights corresponding to each binarization method, the target binarization threshold is obtained by weighting.

[0208] Taking the first binarization method as the triangulation method and the second binarization method as the Otsu method as an example, assuming that the first binarization threshold is recorded as S1, the corresponding first weight is α, the second binarization threshold is recorded as S2, and the corresponding second weight is β. At this time, the calculation formula of the target binarization threshold S is: S=α*S1+β*S2 Formula 4

[0209] In some embodiments of the present disclosure, when the current ambient brightness is greater than a preset brightness threshold, α=0.7, β=0.3; when the current ambient brightness is greater than a preset brightness threshold, α=0.3, β=0.7.

[0210] S9021_4: Binarize the grayscale handle image according to the target binarization threshold to obtain a binarized handle image.

[0211] After obtaining a target binarization threshold that matches the current ambient brightness, the grayscale handle image is binarized according to the target binarization threshold to produce a binarized handle image. Because the target binarization threshold is obtained by weighting the binarization thresholds of different binarization methods based on the current ambient brightness, the target binarization threshold is more reasonably set and can adapt to the current ambient brightness, thereby reducing interference from ambient light and improving the accuracy of light emitter detection.

[0212] S9022: Perform contour detection on the global scale of the binary handle image to obtain a candidate contour set for each light emitter.

[0213] Before initialization, the relative position between the controller and the virtual display device is unknown. The positions of the light spots projected by each light emitter on the controller in the target controller image captured by the camera on the virtual display device are also unknown. Therefore, it is necessary to detect each light emitter globally within the binarized controller image, using the detected light spots as 2D points in the image representing each light emitter in 3D space.

[0214] In certain embodiments of the present disclosure, contour extraction algorithms used in image processing (such as HOG and Canny) can be used to detect light emitters. A contour is formed by the outermost pixels in a disconnected binary region after the image is binarized. Each disconnected binary region has one and only one outermost contour. The contour area can be obtained by calculating the sum of the areas of all pixels in the region surrounded by the outermost pixel. Each contour represents a light spot.

[0215] It should be noted that the embodiments of the present disclosure do not impose any restrictive requirements on the detection method of the light emitter. For example, a deep learning model (such as CNN, YOLO, etc.) can also be used to detect the light emitter.

[0216] S9023: Eliminate abnormal contours in the candidate contour set based on the prior contour shape information and contour contrast information, and obtain a target light spot set of each light emitter.

[0217] Since in addition to the multiple light emitters on the handle, other light-emitting devices in the surrounding environment can also emit light, the candidate contour set for contour detection may include the contours of the light emitters, and may also include the contours of other light-emitting devices that interfere with the light emitters. Therefore, the candidate contour set needs to be screened.

[0218] In some embodiments of the present disclosure, at least one of the following elimination operations is performed based on the prior contour shape information:

[0219] Elimination operation 1: Eliminate candidate contours whose length-to-width ratio exceeds a first preset ratio threshold according to the relationship between the area of ​​the candidate contour and the length-to-width ratio of the circumscribed rectangle of the candidate contour. The first preset ratio threshold is equal to the area of ​​the candidate contour.

[0220] When the area of ​​a candidate contour expands, the length and width of the circumscribed rectangle of the candidate contour are required to be closer. Therefore, in the first elimination operation, in order to improve the accuracy of contour detection, the embodiment of the present disclosure adopts a stepped ratio threshold to eliminate abnormal contours, that is, the first preset ratio threshold and the area of ​​the candidate contour are in a stepped state. The larger the area of ​​the candidate contour, the smaller the first preset ratio threshold. When the length-to-width ratio of the circumscribed rectangle of the candidate contour exceeds the first preset ratio threshold, it is considered a false detection and the candidate contour is eliminated.

[0221] Elimination operation 2: Eliminate candidate contours whose area ratio to the circumscribed rectangle of the candidate contour is less than a preset ratio threshold.

[0222] Elimination operation three: calculate the distance between the grayscale centroid of the candidate contour and the center point of the circumscribed rectangle of the candidate contour on the horizontal axis and the vertical axis respectively, and calculate the ratio of each distance to the side length of the candidate contour respectively. If at least one of the two ratios exceeds the second preset ratio threshold, the candidate contour is eliminated.

[0223] Elimination operation 4: Determine the roundness of the candidate contour based on the total number of pixels contained in the candidate contour and the side length of the candidate contour. If the roundness is lower than a preset roundness threshold, the candidate contour is eliminated.

[0224] Assuming that the total number of pixels contained in the candidate contour (including pixels inside the candidate contour and pixels on the contour boundary) is P, and the perimeter of the candidate contour is C, the calculation formula for roundness R is: R = (4*π*P) / C 2 Formula 5

[0225] Elimination operation 5: Calculate the brightness mean of the candidate contours. If the brightness mean is less than the preset brightness threshold, the candidate contour is eliminated.

[0226] Elimination operation six: determine the brightness mean of the preset peripheral area of ​​the circumscribed rectangle of the candidate area and the brightness mean of the candidate contour. If the brightness difference between the two brightness means is less than the preset brightness difference, the candidate contour is eliminated.

[0227] When removing abnormal contours from the candidate contour set based on prior contour shape information, the removal is performed on a single candidate contour without considering the relationship between the candidate contours. Therefore, abnormal contours from the candidate contour set can be further removed based on contour comparison information.

[0228] In some embodiments of the present disclosure, methods for eliminating abnormal contours in the candidate contour set based on contour contrast information include one or more of the following:

[0229] Elimination operation seven: For every two candidate contours in the candidate contour set, determine the Euclidean distance between the center points of the circumscribed rectangles of the two candidate contours and the minimum Manhattan distance between the edges of the two candidate contours, and eliminate abnormal contours based on the Euclidean distance and the minimum Manhattan distance.

[0230] The specific process of eliminating abnormal contours based on the Euclidean distance and the minimum Manhattan distance between each two candidate contours is shown in Figure 12, which mainly includes the following steps:

[0231] S9023_11: Determine whether at least one of the Euclidean distance and the minimum Manhattan distance between two candidate contours is less than a preset distance threshold. If so, execute S9023_12; otherwise, execute S9023_16.

[0232] The degree of similarity between two candidate contours can be determined based on the Euclidean distance and minimum Manhattan distance between them. If at least one of the Euclidean distance and the minimum Manhattan distance between the two candidate contours is less than the preset distance threshold, the two candidate contours are highly similar and require further abnormality determination, executing S9023_12. If both the Euclidean distance and the minimum Manhattan distance between the two candidate contours are greater than the preset distance threshold, the two candidate contours are less similar and executing S9023_16.

[0233] S9023_12: Calculate the areas of the two candidate contours respectively.

[0234] S9023_13: Determine whether the areas of the two candidate contours are both smaller than a preset area threshold. If so, execute S9023_14; otherwise, execute S9023_15.

[0235] The abnormality judgment is further performed by comparing the calculated areas of the two candidate contours with the preset area threshold.

[0236] S9023_14: Eliminate two candidate contours at the same time.

[0237] When the areas of two candidate contours are both smaller than the preset area threshold, it indicates that the two candidate contours may be noise points and should be removed at the same time.

[0238] S9023_15: Calculate the brightness mean of the two candidate contours respectively, and eliminate the candidate contour corresponding to the smaller brightness mean.

[0239] When at least one of the two candidate contours has an area no less than a preset area threshold, outliers can be eliminated using the mean brightness value. In practice, the mean brightness values ​​of the two candidate contours are calculated and compared, and the contour with the smaller mean brightness value is removed from the candidate contour set.

[0240] S9023_16: Keep two candidate contours at the same time.

[0241] When the Euclidean distance and the minimum Manhattan distance between two candidate contours are both greater than the preset distance threshold, it indicates that the two candidate contours are less similar, and the two candidate contours in the candidate contour set can be retained at the same time.

[0242] Elimination operation eight: sort all candidate contours in the candidate contour set according to their areas, and eliminate abnormal contours based on the quantitative relationship between the pixels in the candidate contour with the largest area and the candidate contour with the second largest area.

[0243] After sorting the candidate contours by area, the candidate contour with the largest area and the candidate contour with the second largest area can be selected from the candidate contour set. The specific process of eliminating abnormal contours based on the quantitative relationship between the pixels in these two selected candidate contours is shown in Figure 13. It mainly includes the following steps:

[0244] S9023_21: Determine whether the number of pixels in the candidate contour with the largest area and the candidate contour with the second largest area exceeds a preset pixel number threshold. If so, execute S9023_22; otherwise, execute S9023_25.

[0245] The number of pixels in two candidate contours can reflect the degree of similarity between the two candidate contours. Therefore, whether the two candidate contours are similar in shape can be determined by comparing the number of pixels in the candidate contour with the largest area and the candidate contour with the second largest area with a preset pixel number threshold.

[0246] S9023_22: Calculate the multiple of the number of pixels in the candidate contour with the largest area and the number of pixels in the candidate contour with the second largest area.

[0247] S9023_23: Determine whether the multiple is greater than a preset multiple threshold. If so, execute S9023_24; otherwise, execute S9023_25.

[0248] Further abnormality judgment is performed based on the multiple of the number of pixels in the candidate contour with the largest area and the second largest area.

[0249] S9023_24: Eliminate the candidate contour with the largest area.

[0250] When the multiple of the number of pixels in the candidate contour with the largest area and the number of pixels in the candidate contour with the second largest area is greater than the preset multiple threshold, the candidate contour with the largest area may be an interference object with a shape similar to the light emitter on the handle and should be removed from the candidate contour set.

[0251] S9023_25: retain the candidate contour with the largest area and the candidate contour with the second largest area.

[0252] When the number of pixels in one of the candidate contour with the largest area and the candidate contour with the second largest area does not exceed the preset pixel number threshold, or the multiple of the number of pixels in the candidate contour with the largest area and the candidate contour with the second largest area is not greater than the preset multiple threshold, the candidate contour with the largest area and the candidate contour with the second largest area are retained.

[0253] Elimination operation 9: For each candidate contour in the candidate contour set, calculate the distance between the candidate contour and the nearest neighbor candidate contour, and eliminate outlier contours based on the distance.

[0254] The process of eliminating outlier contours based on the distance between the candidate contour and the nearest neighbor candidate contour is shown in Figure 14 and mainly includes the following steps:

[0255] S9023_31: Determine the adaptive outlier distance based on the side length of the candidate contour and the median of the side lengths of all candidate contours.

[0256] All candidate contours in the candidate contour set are sorted according to their side lengths to obtain the median of the side lengths. The distance between the median of the side lengths and the current candidate contour is used as the adaptive outlier distance.

[0257] S9023_32: Determine whether the distance between the candidate contour and the nearest neighbor candidate contour is greater than the adaptive outlier distance. If so, execute S9023_33; otherwise, execute S9023_36.

[0258] S9023_33: Determine whether the number of all candidate contours is greater than a preset number threshold. If so, execute S9023_34; otherwise, execute S9023_35.

[0259] S9023_34: Eliminate candidate contours.

[0260] When the distance between a candidate contour and its nearest neighbor candidate contour is greater than the adaptive outlier distance, and the number of all candidate contours is greater than a preset threshold, it indicates that the candidate contour is an abnormal outlier contour and should be removed.

[0261] S9023_35: Keep candidate contours.

[0262] S9023_36: Outlier removal is completed.

[0263] When the total number of candidate contours is small, it may not be able to represent a group. In this case, it may not be possible to eliminate abnormal contours by outliers, and other methods must be used to eliminate abnormalities.

[0264] Elimination operation 10: Calculate the brightness mean of each candidate contour in the candidate contour set, and eliminate abnormal contours based on the brightness mean.

[0265] In the elimination operation ten, the brightness mean values ​​of the candidate contours in the candidate contour set are sorted from large to small, the first N (N is an integer greater than or equal to 1) candidate contours are retained, and the remaining candidate contours are eliminated.

[0266] It should be noted that there is no strict execution order in the above-mentioned elimination operations one to ten. Abnormal contours can be eliminated first according to the prior contour shape information, and then according to the contour comparison information; abnormal contours can be eliminated first according to the contour comparison information, and then according to the prior contour shape information; the two types of abnormal elimination methods, contour comparison information and prior contour shape information, can also be interspersed.

[0267] In the disclosed embodiment of obtaining a target light spot set, to further adapt to different ambient lighting scenarios and enable the algorithm to operate stably and robustly in complex environments, the binarization thresholds of different binarization methods are weighted according to the current ambient brightness to obtain a target binarization threshold for binarizing the target handle image. This ensures the accuracy of light emitter detection on the handle under different brightness levels and significantly reduces development difficulty and cost. At the same time, to improve the detection speed of light emitters on the handle, image processing technology is used to eliminate abnormalities in the detected contours, which improves the operation speed while reducing memory resource usage, making it easier to deploy on portable wearable devices. On the one hand, compared with light emitter detection methods based on AI neural networks, the disclosed embodiment does not require a high-configuration processor for network training or large amounts of data annotation, reducing the development hardware resource requirements, development cost, and workload. Compared with general image processing light emitter detection methods, the disclosed embodiment can adaptively adjust the binarization threshold according to the current ambient brightness. By weighting the binarization thresholds of at least two binarization methods, the algorithm improves its robustness in complex scenarios and expands its scope of application. On the other hand, the embodiment of the present disclosure eliminates the light spot of the light emitter that interferes with the positioning of the handle based on the contour features of the light emitter, further improving the performance of the algorithm and the accuracy of detection.

[0268] After obtaining the accurately detected target light spot set, it is unknown which light emitter in the optimized 3D spatial structure each target light spot in the target light spot set is the projection of, that is, the correspondence between the 2D light spot and the 3D light emitter is unknown. Therefore, it is necessary to match each target light spot in the target light spot set with each light emitter after the 3D spatial structure is optimized, and establish a one-to-one correspondence between the 2D light spot and the 3D light emitter. Therefore, based on the corresponding 3D light emitters and 2D light spots, the PNP algorithm is used to align the coordinate system between the handle and the virtual display device, and the observation data collected by the IMU on the handle after alignment (including but not limited to the acceleration and angular velocity of the handle) is pre-integrated to obtain the relative 6DOF posture between the handle and the virtual display device, completing the initialization process of the relative posture between the handle and the virtual display device.

[0269] Generally, the acquisition frequencies of the IMU and the camera may be different. The pose estimation process needs to ensure that the observation data collected by the IMU is synchronized with the target handle image collected by the camera. The synchronization relationship between the observation data and the target handle image can be determined based on the timestamp.

[0270] In some embodiments of the present disclosure, the one-to-one correspondence between the 2D light spot and the 3D light emitter can be represented by the first identifier of the 3D light emitter and the image spot index of the 2D light spot. Therefore, the process of matching the 2D light spot with the 3D light emitter can be regarded as the process of determining the second identifier of the light spot corresponding to a certain image spot index in the target handle image.

[0271] Currently, brute-force matching is the most common method for matching 2D light spots with 3D light emitters. This method involves randomly selecting three target light spots from a set of target light spots, guessing their IDs based on their 3D spatial structure, and then using a P3P algorithm to calculate their relative poses. Each P3P algorithm has four solutions. Based on the solved relative poses, all light emitters are reprojected into the image, and the number and error of matching point pairs are calculated. All combinations are then sorted, prioritizing the result with the largest number of matches. If the number of matches is the same, the result with the smallest error is selected.

[0272] Usually, the number of combinations of brute force matching is huge, the overall time consumption is large, and the positioning efficiency is reduced. Assuming that there are m light spots in the target light spot set and n light emitters in the handle, the formula for calculating the number of combinations is:

[0273] To solve this problem, the embodiments of the present disclosure provide an efficient matching method, which splices adjacent light spots into a plane figure for matching. Experimental measurements show that, taking a plane triangle as an example, the number of combinations of adjacent light spots is usually less than 500, which is less than the number of combinations of brute force matching, and can effectively improve the efficiency and accuracy of matching.

[0274] 15 is a flow chart of a method for matching a 2D light spot with a 3D light emitter in an embodiment of the present disclosure, which mainly includes the following steps:

[0275] S9024: For any target light spot in the target light spot set, select a first specified number of candidate light spots adjacent to the target light spot from the target light spot set, and connect the target light spot with the first specified number of candidate light spots to obtain a plane figure.

[0276] Taking any target light spot in the target light spot set as an example, the process of determining the candidate light spots includes: obtaining the distances between the target light spot and the other light spots according to the 2D coordinates of the target light spot and the 2D coordinates of the other light spots in the target light spot set, sorting the distances between the target light spot and the other light spots in ascending order, and determining the other light spots corresponding to the first specified number of distances as candidate light spots. The distance between the target light spot and any other light spot can be obtained by Formula 6:

[0277] Where d is the distance between the target light spot and any other light spot, x1 is the horizontal coordinate of the target light spot in the image, y1 is the vertical coordinate of the target light spot in the image, x2 is the horizontal coordinate of the other light spot in the image, and y2 is the vertical coordinate of the other light spot in the image.

[0278] In some embodiments of the present disclosure, the first specified number is 2, but the first specified number in the embodiments of the present disclosure is not limited and can be set according to actual conditions.

[0279] The first specified number corresponds to the plane figure. If the plane figure is a triangle, the first specified number is 2; if the plane figure is a tetrahedron, the first specified number is 3.

[0280] Taking a plane figure as a triangle as an example, as shown in FIG16 , a plurality of triangles are obtained by connecting adjacent light spots in the target light spot set.

[0281] S9025: According to the actually adjacent light emitter sets in the optimized 3D spatial structure, each light spot in the plane figure is matched with each light emitter in the actually adjacent light emitter set to obtain each adjacent light spot matching pair.

[0282] Using the handle's design drawings, we can obtain the first identifiers of each light emitter on the handle and the actual adjacent relationships between them, thereby obtaining a set of actually adjacent light emitters. Therefore, we can quickly match each light spot in the plane graphic with the set of actually adjacent light emitters, obtaining matching pairs of adjacent light spots. Each matching pair includes the image spot index of the light spot and the first identifier of the light emitter that matches the light spot.

[0283] As shown in Figure 17, the process of quickly matching each light spot in the plane figure with the actual adjacent light emitter set mainly includes the following steps:

[0284] S9025_1: Arrange the light spots in the plane figure in ascending order of the image light spot index to obtain a light spot list.

[0285] S9025_2: Traverse each light emitter in the set of actually adjacent light emitters in a specified order, use the light emitter as the initial position for the currently traversed light emitter, and sort the other light emitters that are actually adjacent to the light emitter in a specified order to obtain a sorted list.

[0286] Among them, the designated order in this embodiment includes a clockwise order and a counterclockwise order, but the designated order in this embodiment is not limited. The designated order in this embodiment can be set according to actual conditions.

[0287] For example, taking a plane figure as a triangle, in a set of actually adjacent light emitters, a group of light emitters includes light emitter 1, light emitter 2, and light emitter 3. If the specified order in this implementation is counterclockwise, then the traversal order for each light emitter in the set of actually adjacent light emitters is light emitter 3, light emitter 2, and light emitter 1. When light emitter 3 is traversed, the corresponding sorted list is: light emitter 3, light emitter 2, light emitter 1; when light emitter 2 is traversed, the corresponding sorted list is: light emitter 2, light emitter 1, light emitter 3; when light emitter 1 is traversed, the corresponding sorted list is: light emitter 1, light emitter 3, light emitter 2.

[0288] S9025_3: For any light emitter in the sorted list, add the first identifier of the light emitter and the image spot index of the spot in the spot list that has the same position as the light emitter in the sorted list to the same adjacent spot matching pair.

[0289] For example, the order in the light spot list is: light spot A, light spot B, light spot C. Taking the sorted list as: light emitter 3, light emitter 2, light emitter 1 as an example, the obtained adjacent light spot matching degrees are: light spot A-light emitter 3, light spot B-light emitter 2, light spot C-light emitter 1.

[0290] S9025_4: Determine whether there is any light emitter that has not been traversed in the actual adjacent light emitter set. If so, return to S9025_2; if not, end.

[0291] By traversing the light emitters in the actual adjacent light emitter set, it can be ensured that each light emitter has a corresponding image light spot index, and a matching result based on each adjacent light spot can be obtained.

[0292] To further improve the matching efficiency, in some embodiments of the present disclosure, after obtaining each adjacent spot matching pair, the adjacent spot matching pairs may be screened. FIG18 is a schematic diagram of a process for screening adjacent spot matching pairs, which includes the following steps:

[0293] S9025_5: For a plurality of predicted poses of the handle corresponding to any set of adjacent light spot matching pairs, respectively obtain the predicted gravity direction vector of the handle corresponding to the adjacent light spot matching pairs.

[0294] In practical applications, based on multiple predicted postures of the handle corresponding to any set of adjacent light spot matching pairs, the predicted gravity direction vector of the handle corresponding to the adjacent light spot matching pair can be solved through a preset IMU integration algorithm.

[0295] S9025_6: Obtain the actual gravity direction vector of the handle according to the current position of the virtual display device when the specified image is captured.

[0296] In practical applications, the actual gravity direction vector of the handle can be obtained according to the 6Dof posture of the virtual display device when capturing a specified image.

[0297] S9025_7: Determine the adjacent light spot matching pairs that need to be deleted through the predicted gravity direction vector and the actual direction vector corresponding to each adjacent light spot matching pair, and delete the adjacent light spot matching pairs that need to be deleted.

[0298] In practical applications, for any set of adjacent spot matching pairs, the gravity direction vector angle is obtained based on the predicted gravity direction vector and the actual gravity direction vector corresponding to the adjacent spot matching pair; if the gravity direction vector angle is greater than the specified angle, the adjacent spot matching pair is determined to be an adjacent spot matching pair that needs to be deleted. The gravity direction vector angle can be obtained by formula 7:

[0299] Where θ is the angle between the gravity direction vector and To predict the gravity direction vector, is the actual gravity direction vector.

[0300] For example, if the specified angle is 10°, if the angle of the gravity direction vector corresponding to the first adjacent spot matching pair is 4°, it is determined that the first adjacent spot matching pair does not need to be deleted, and if the angle of the gravity direction vector corresponding to the second adjacent spot matching pair is 12°, it is determined that the second adjacent spot matching pair needs to be deleted.

[0301] It should be noted that the designated angle in this embodiment can be set according to actual conditions, and this embodiment does not limit the specific value of the designated angle.

[0302] S9026: For any set of adjacent light spot matching pairs, determine multiple predicted positions of the handle corresponding to the adjacent light spot matching pairs based on the 2D coordinates of each light spot in the adjacent light spot matching pairs and the 3D coordinates of each light emitter.

[0303] Taking a triangle as an example, each set of adjacent light spot matching pairs contains the matching results of three light spots. The 2D coordinates of each light spot in this set of adjacent light spot matching pairs and the 3D coordinates of each light emitter are input into the p3p algorithm. Multiple predicted poses of the handle corresponding to this set of adjacent light spot matching pairs can be obtained, including rotation matrices and translation vectors.

[0304] Among them, the p3p algorithm can output four results, so a set of adjacent spot matching pairs corresponds to four predicted poses.

[0305] S9027: For any predicted posture, project each light emitter into the specified image according to the predicted posture to obtain each projection light spot, and match the other light spots in the specified image except the light spots contained in the plane figure with the light emitters on the handle based on each projection light spot to obtain other light spot matching pairs.

[0306] At the same time, the multi-camera on the virtual display device can synchronously capture multiple handle images, where the designated image is at least one of the target handle images acquired at the current moment. The designated image can be one or more. The number of designated images and the specific image to be used can be set according to actual conditions.

[0307] When a virtual display device leaves the factory, the intrinsic parameters of the multi-camera are already calibrated. Alternatively, before positioning, it can be pre-calibrated using the checkerboard calibration method. Combined with the predicted pose, each light emitter in 3D space can be projected onto a specified 2D image to obtain each projected light spot. Since the light emitters matching the light spots contained in the planar pattern have already been determined, the only thing left to do is to determine the light emitters matching the light spots in the specified image other than those contained in the planar pattern.

[0308] FIG19 is a flow chart showing the process of determining other spot matching pairs, including the following steps:

[0309] S9027_1: For any other light spot in the specified image, obtain the distances between the other light spot and each projection light spot according to the 2D coordinates of the other light spot and the 2D coordinates of each projection light spot.

[0310] The distance between the other light spots and the projection light spot can be determined by the distance formula in Formula 6, which will not be described in detail in this embodiment.

[0311] S9027_2: Determine whether the shortest distance among the distances is less than the specified distance. If so, execute S9027_3; if not, end.

[0312] S9027_3: Add the image spot index of other spots and the first identifier of the illuminator corresponding to the projection spot corresponding to the shortest distance to the same spot matching pair, and determine the spot matching pair as the other spot matching pair.

[0313] Among them, each other spot matching pair includes the image spot index of other spots and the first identifier of the illuminator corresponding to the projection spot matching with other spots.

[0314] For example, the specified image includes other spot C and other spot D. If the distance between other spot C and the first projection spot is m, and the distance between it and the second projection spot is n, the first projection spot is the projection spot of illuminator 1, and the second projection spot is the projection spot of illuminator 2. If m > n, then determine that n is the shortest distance. If n is less than the specified distance, then determine an other spot matching pair as (C, 2). If the distance between other spot D and the first projection spot is p, and the distance between it and the second projection spot is q, if p < q, then determine that p is the shortest distance. If p is greater than the specified distance, then determine that other spot D has no corresponding illuminator.

[0315] It should be noted that: the specified distance in this embodiment can be set according to the actual situation, and this embodiment does not limit the specified distance here.

[0316] S9028: Screen each spot matching pair according to the number of each other spot matching pair, and obtain each target spot matching pair according to the number of the screened spot matching pairs, and determine the first identifier of the illuminator in the target spot matching pair as the second identifier of the target spot corresponding to the image spot index.

[0317] Among them, the spot matching pair includes an adjacent spot matching pair and an other spot matching pair, and each matching pair represents the corresponding relationship between the 3D illuminator and the 2D spot.

[0318] In some embodiments of the present disclosure, for the predicted pose of any handle, if the number of other spot matching pairs corresponding to the predicted pose is less than the second specified number, then delete the predicted pose and the other spot matching pairs corresponding to the predicted pose.

[0319] In some embodiments of the present disclosure, for any adjacent spot matching pair, if multiple predicted poses corresponding to the adjacent spot matching have been deleted, then delete the adjacent spot matching pair.

[0320] For example, each adjacent spot matching pair has corresponding four predicted poses. If the four predicted poses corresponding to any adjacent spot matching pair have been deleted, the adjacent spot matching pair will be deleted.

[0321] It should be noted that the second specified number in this embodiment can be set according to actual conditions, and this embodiment does not limit the specific value of the second specified number.

[0322] After screening each spot matching pair, the number of each spot matching pair after elimination is counted. For each spot matching pair with the same image spot index, the spot matching pair with the largest number among the spot matching pairs is determined as the target spot matching pair corresponding to the image spot index, and the first identifier of the light emitter in the target spot matching pair is determined as the second identifier of the target spot corresponding to the image spot index.

[0323] For example, after elimination, the spot matching pairs are: (A, 1), (A, 2), (A, 2), (A, 2), (A, 1), (B, 3), (B, 1), (B, 3), (B, 3), (B, 1). From the spot matching pairs after elimination, we can obtain that the number of spot matching pairs (A, 1) is 2, the number of spot matching pairs (A, 2) is 3, the number of spot matching pairs (B, 1) is 2, and the number of spot matching pairs (B, 3) is 3. Then, the target spot matching pair with the image spot index A is determined to be (A, 2). At this time, the second identifier of the target spot with the image spot index A is 2. The target spot matching pair with the image spot index B is determined to be (B, 3). At this time, the second identifier of the spot with the image spot index B is 3.

[0324] In the matching process of the 3D light emitter and the 2D light spot in the embodiment of the present invention, adjacent light spots are connected into a plane figure, and then each light spot is quickly matched and the posture is predicted with the actual adjacent light emitter set in units of the plane figure to obtain each light spot matching pair, which effectively reduces the number of combinations in the matching process, and improves the matching accuracy by screening each light spot matching pair, thereby improving the positioning efficiency and accuracy.

[0325] S9029: Initialize the relative position between the controller and the virtual display device based on the 3D coordinates of the light emitter and the 2D coordinates of the target light spot, combined with the observation data collected by the IMU.

[0326] After obtaining the matching pairs of each target light spot, the correspondence between the 3D light emitter and the 2D light spot is obtained. Therefore, the 3D coordinates of the light emitter and the 2D coordinates of the target light spot in each target light spot matching pair can be used to align the coordinate system between the handle and the virtual display device using the PNP algorithm, and the 6DOF posture between the handle and the virtual display device based on visual calculation is obtained. The observation data collected by the IMU on the handle after alignment is pre-integrated to use the inertial navigation positioning results to optimize the relative 6DOF posture between the handle and the virtual display device, completing the initialization process of the relative posture between the handle and the virtual display device.

[0327] S903: For the non-first frame target handle image captured by the camera, predict the current relative pose between the handle and the virtual display device based on the relative pose between the handle and the virtual display device corresponding to the historical target handle image, and combine the observation data continuously collected by the IMU to determine the target relative pose between the current handle and the virtual display device.

[0328] In the process of real-time estimation of the relative pose between the handle and the virtual display device, when the relative pose between the handle and the virtual display device has been initialized, the relative pose between the current handle and the virtual display device is predicted based on the initialization result for the non-first frame target handle image captured by the camera.

[0329] In some embodiments of the present disclosure, the relative posture between the handle corresponding to the target handle image of the first frame and the virtual display device is predicted based on the relative posture between the handle corresponding to the target handle image of the second frame and the virtual display device, and then the relative posture between the handle corresponding to the target handle image of the first frame and the second frame and the virtual display device is predicted based on the relative posture between the handle corresponding to the target handle image of the third frame and the virtual display device, and so on.

[0330] In the disclosed embodiment, during the posture estimation process, the relative posture between the handle corresponding to the historical target handle image and the virtual display device is predicted, thereby ensuring the smoothness of the relative posture between multiple consecutive frames of target handle images. In this way, during the actual interaction process, when the handle is used to control the picture displayed by the virtual display device, the smoothness of the virtual display picture is guaranteed, thereby enhancing the user's immersive experience.

[0331] To further improve the accuracy of the relative pose, the observation data continuously collected by the IMU can be used to optimize the predicted current relative pose, so as to obtain the accurate target relative pose between the current handle and the virtual display device in real time.

[0332] The process of determining the relative pose of the target is shown in Figure 20, which mainly includes the following steps:

[0333] S9031: Determine the local range of each light emitter in the current target handle image based on the 3D coordinates of each light emitter on the handle in the optimized 3D spatial structure and the predicted current relative position between the handle and the virtual display device.

[0334] During the pose estimation process, the current relative pose between the handle and the virtual display device is obtained through prediction. Based on the current relative pose, the approximate position of the light spot projected by each light emitter on the handle to the current target handle image in 3D space can be determined, thereby reducing the image range of the light emitter detection and improving detection efficiency.

[0335] S9032: Extract the current light spots of each current light emitter within the local range of the current target handle image, and determine the light emitter corresponding to each current light spot based on nearest neighbor matching.

[0336] Because the current relative pose between the controller and the virtual display device is known, the approximate position of the current light spot projected by each light emitter on the controller into the current target controller image after 3D spatial structure optimization can be predicted. Therefore, during the pose estimation process, a nearest neighbor matching method can be used for each light emitter, with the light spot closest to the projected light spot among the current light spots extracted in the current target controller image being used as the current light spot matched to that light emitter.

[0337] S9033: Establish a reprojection constraint equation based on the corresponding 2D coordinates of the current light spot and the 3D coordinates of the 3D light emitter, as well as the poses of the IMU and the camera when the observation data and the current target handle image are synchronized.

[0338] The reprojection constraint equation is as follows:

[0339] In formula 8, They represent the rotation matrix and translation vector of the IMU in the world coordinate system corresponding to the j-th frame observation data collected by the IMU, They represent the rotation matrix and translation vector of the camera on the virtual display device in the world coordinate system corresponding to the j-th frame observation data collected by the IMU, They represent the rotation matrix and translation vector of the IMU in the handle coordinate system respectively. Indicates the 3D coordinates of the first light emitter on the handle, which is marked as m, p m represents the 2D coordinates of the current light spot with the second identifier m on the current target handle image, and pro j(·) represents the projection equation of the camera. is the pose of the IMU in the world coordinate system when the IMU is synchronized with the camera, is the camera's pose in the world coordinate system when the IMU is synchronized with the camera. It is the relative pose between the IMU and the controller when the IMU is synchronized with the camera.

[0340] S9034: Establish a pre-integration constraint equation based on the IMU pose and handle motion speed corresponding to two consecutive frames of observation data.

[0341] The pre-integration constraint equation is as follows:

[0342] In formula 9, Indicates the translation vector of the IMU in the world coordinate system corresponding to the j+1 frame observation data collected by the IMU, They represent the motion speed of the IMU corresponding to the observation data of the jth frame and the j+1th frame in the world coordinate system, which can be obtained by integrating the acceleration in the observation data of the jth frame and the j+1th frame respectively. W represents the acceleration of gravity, Δt represents the time interval between the j-th frame and the j+1-th frame of observation data collected by the IMU, LOG(·) represents the logarithmic function on the Lie group (Special Orthometri, SO3) corresponding to the quaternion array, Represent the pre-integrated variables of the IMU translation vector, motion velocity, and rotation matrix respectively.

[0343] S9035: Combine the pre-integration constraint equation and the reprojection constraint equation to solve the IMU pose, camera pose, and relative pose between the IMU and the controller corresponding to the current target controller image.

[0344] The combined formula of the pre-integration constraint equation and the reprojection constraint equation is as follows:

[0345] In formula 10, j represents the number of frames of observation data collected by IMU, f j represents the pre-integration constraint equation, g j Represents the reprojection constraint equation.

[0346] By solving formula 10, we can get the pose of the IMU corresponding to the current target handle image in the world coordinate system: The position of the camera (i.e., virtual display device) in the world coordinate system And the relative position of IMU and handle

[0347] S9036: Based on the relative pose of the IMU and the handle, as well as the current pose of the IMU and the pose of the camera, obtain the target relative pose between the current handle and the virtual display device.

[0348] The formula for the handle's position in the world coordinate system after visual inertial navigation joint optimization is as follows:

[0349] In formula 8, Indicates the current position of the handle in the world coordinate system. Indicates the relative pose of the IMU and the controller.

[0350] because and In the same world coordinate system, the target relative position between the current handle and the virtual display device can be obtained, so that the image displayed by the virtual display device can be controlled by operating the handle.

[0351] It should be noted that since the camera is located on the virtual display device, the camera's pose can represent the pose of the virtual display device. While a virtual display device typically has multiple cameras, each capturing data synchronously, in this disclosed embodiment, pose estimation can be performed using an image of the target handle captured by a single camera.

[0352] In the method for estimating the pose of a handle according to some embodiments, the multiple light emitters of the IMU on the handle and the multi-camera on the virtual display device are used to realize the joint optimization of the relative pose between the handle and the virtual display device by visual inertial navigation. Before the pose estimation, the light emitters are annotated for multiple frames of initial handle images captured at different positions and angles, so that the 3D spatial structure of the light emitters is optimized according to the annotated results of each light emitter, thereby improving the accuracy of subsequent relative pose calculations. During the pose estimation process, the relative pose between the handle and the virtual display device is initialized based on the optimized 3D spatial structure and the first frame target handle image captured by the camera. After the initialization is completed, for the non-first frame target handle image captured by the camera, the relative pose between the current handle and the virtual display device is predicted based on the relative pose between the handle and the virtual display device corresponding to the historical target handle image. Then, combined with the observation data of the IMU, the joint optimization of the relative pose by visual inertial navigation is realized, thereby obtaining a smooth and accurate target relative pose between the current handle and the virtual display device.

[0353] Based on the same technical concept, an embodiment of the present disclosure provides a virtual display device, which can execute the above-mentioned method for detecting the light emitter on the handle and achieve the same technical effect.

[0354] 21 , the virtual display device includes a processor 2101, a memory 2102, a display screen 2103, a communication interface 2104, and a multi-eye camera 2105. The display screen 2103 is used to display images. The virtual display device communicates with a handle via the communication interface 2104. The handle is used to control the images displayed on the display screen 2103. The type of the multi-eye camera 2105 matches the light-emitting type of the multiple light emitters on the handle.

[0355] The communication interface 2104, the multi-camera 2105, the display screen 2103, the memory 2102, and the processor 2101 are connected via a bus 2106. The memory 2102 stores a computer program. The processor 2101 performs the following operations according to the computer program:

[0356] For the first frame of the target handle image captured by the multi-camera 2105, a target light spot set of each light emitter is obtained based on the target handle image, and the relative position between the handle and the virtual display device is initialized based on the target light spot set, the observation data synchronously collected by the IMU, and the optimized 3D spatial structure of each light emitter on the handle; wherein the 3D spatial structure is optimized based on the annotation results of each light emitter in multiple frames of initial handle images captured at different positions and angles;

[0357] For the non-first frame target handle image captured by the multi-eye camera 2105, the current relative pose between the handle and the virtual display device is predicted based on the relative pose corresponding to the historical target handle image, and the current target relative pose between the handle and the virtual display device is determined in combination with the observation data continuously collected by the IMU.

[0358] In some embodiments of the present disclosure, the processor 2101 optimizes the 3D spatial structure of each light emitter on the handle by:

[0359] Obtaining the 3D coordinates and first identification of each light emitter according to the 3D spatial structure of each light emitter before optimization;

[0360] According to the first identification of each light emitter, for the light emitters pre-annotated on multiple frames of initial handle images collected at different positions and angles, the 2D coordinates and second identification of the light spot formed by each light emitter on the corresponding initial handle image are obtained;

[0361] For each frame of the initial handle image, determine the relative position between the handle and the acquisition camera based on the 3D coordinates of the light source and the 2D coordinates of the light spot with the same first identifier and second identifier, and the observation data of the IMU corresponding to the corresponding frame;

[0362] A reprojection error equation is constructed, and the relative postures and 3D coordinates of each light emitter are simultaneously optimized according to the reprojection error equation to obtain the 3D spatial structure after the first optimization.

[0363] In some embodiments of the present disclosure, after obtaining the first optimized 3D spatial structure, the processor 2101 further executes:

[0364] Determining, based on a first 3D point cloud composed of light emitters on the handle corresponding to the optimized 3D spatial structure, and a second 3D point cloud composed of light emitters on the handle corresponding to the 3D spatial structure before optimization, a conversion pose between the first 3D point cloud and the second 3D point cloud before and after optimization;

[0365] According to the transformed posture, the 3D coordinates of each light emitter on the handle are re-determined to obtain the 3D spatial structure after the second optimization.

[0366] In some embodiments of the present disclosure, the reprojection error equation is:

[0367] Among them, K n represents the projection parameters of the nth camera, Respectively represent the rotation matrix and translation vector between the handle and camera No. 0, Respectively represent the rotation matrix and translation vector between the nth camera and the 0th camera, represents the 3D coordinates of the first light emitter identified as m on the handle, p m,n represents the 2D coordinates of the second light spot marked as m.

[0368] In some embodiments of the present disclosure, the processor 2101 obtains the target light spot set of each light emitter according to the target handle image, and the specific operation is:

[0369] Obtaining the current ambient brightness, determining a binarization threshold for each of at least two binarization methods based on the current ambient brightness, and performing binarization processing on the target handle image based on each binarization threshold to obtain a binarized handle image;

[0370] Performing contour detection on the global scope of the binary handle image to obtain a set of candidate contours of each light emitter, wherein each contour represents a light spot;

[0371] Abnormal contours in the candidate contour set are eliminated according to the prior contour shape information and contour contrast information, to obtain the target light spot set of each light emitter.

[0372] In some embodiments of the present disclosure, the processor 2101 determines, based on the current ambient brightness, respective binarization thresholds of at least two binarization methods, and binarizes the visible light handle image based on the respective binarization thresholds to obtain a binarized handle image. Specifically, the operations are as follows:

[0373] Eliminating pixels whose grayscale values ​​are lower than a preset grayscale threshold in a grayscale handle image obtained by grayscaling the visible light handle image, and determining the respective binarization thresholds of the at least two binarization methods based on a new histogram of the grayscale handle image after pixel elimination;

[0374] Comparing the current ambient brightness with a preset brightness threshold, and determining weights corresponding to the at least two binarization thresholds respectively according to the comparison result;

[0375] According to each binarization threshold and the corresponding weight, the target binarization threshold is obtained by weighting;

[0376] The grayscale handle image is binarized according to the target binarization threshold to obtain a binarized handle image.

[0377] In some embodiments of the present disclosure, the processor 2101 determines the weights corresponding to the at least two binarization thresholds respectively according to the comparison result, and the specific operation is:

[0378] When the current ambient brightness is greater than the preset brightness threshold, setting a first weight corresponding to a first binarization threshold calculated by the first binarization method to be greater than a second weight corresponding to a second binarization threshold calculated by the second binarization method;

[0379] When the current ambient brightness is less than or equal to the preset brightness threshold, setting a first weight corresponding to a first binarization threshold calculated by the first binarization method to be less than a second weight corresponding to a second binarization threshold calculated by the second binarization method;

[0380] The first binarization method is used to solve a histogram distribution containing a single peak, and the second binarization method is used to solve a histogram distribution containing two peaks.

[0381] In some embodiments of the present disclosure, the processor 2101 performs light spot detection in the global range of the binary handle image to obtain a target light spot set of each light emitter, specifically by:

[0382] Performing contour detection on the binary handle image to obtain a set of candidate contours of each light emitter, wherein each contour represents a light spot;

[0383] Abnormal contours in the candidate contour set are eliminated according to the prior contour shape information and the contour contrast information, so as to obtain the target light spot set of each light emitter.

[0384] In some embodiments of the present disclosure, the processor 2101 eliminates abnormal contours from the candidate contour set according to the contour comparison information in one or more of the following ways:

[0385] For each two candidate contours in the candidate contour set, determining the Euclidean distance between the center points of the circumscribed rectangles of the two candidate contours and the minimum Manhattan distance between the edges of the two candidate contours, and eliminating abnormal contours based on the Euclidean distance and the minimum Manhattan distance;

[0386] Sort all candidate contours in the candidate contour set according to their areas, and eliminate abnormal contours based on the relationship between the number of pixels in the candidate contour with the largest area and the candidate contour with the second largest area;

[0387] For each candidate contour in the candidate contour set, calculating the distance between the candidate contour and the nearest neighbor candidate contour, and eliminating outlier contours based on the distance;

[0388] The brightness mean of each candidate contour in the candidate contour set is calculated, and abnormal contours are eliminated based on the brightness mean values.

[0389] In some embodiments of the present disclosure, the processor 2101 removes abnormal contours according to the Euclidean distance and the minimum Manhattan distance, and the specific operations are:

[0390] When at least one of the Euclidean distance and the minimum Manhattan distance is less than a preset distance threshold, respectively calculating the areas of the two candidate contours;

[0391] If the areas of both candidate contours are smaller than the preset area threshold, both candidate contours are eliminated at the same time;

[0392] If at least one of the areas of the two candidate contours is not less than the preset area threshold, the brightness mean values ​​of the two candidate contours are calculated respectively, and the candidate contour corresponding to the smaller brightness mean value is eliminated.

[0393] In some embodiments of the present disclosure, the processor 2101 removes abnormal contours based on the quantitative relationship between the pixels in the candidate contour with the largest area and the candidate contour with the second largest area. The specific operation is:

[0394] If the number of pixels in the candidate contour with the largest area and the candidate contour with the second largest area both exceeds a preset pixel number threshold, then calculate the multiple of the number of pixels in the candidate contour with the largest area and the candidate contour with the second largest area;

[0395] If the multiple is greater than a preset multiple threshold, the candidate contour with the largest area is eliminated.

[0396] In some embodiments of the present disclosure, the processor 2101 removes outlier abnormal contours based on the distance, specifically by performing the following operations:

[0397] Determining an adaptive outlier distance according to the side length of the candidate contour and the median of the side lengths of all candidate contours;

[0398] If the number of all candidate contours is greater than a preset number threshold, and the distance is greater than the adaptive outlier distance, the candidate contour is eliminated.

[0399] In some embodiments of the present disclosure, the processor 2101 eliminates abnormal contours from the candidate contour set according to the prior contour shape information in one or more of the following ways:

[0400] Eliminate candidate contours whose aspect ratio exceeds a first preset ratio threshold according to a relationship between the area of ​​the candidate contour and the aspect ratio of the circumscribed rectangle of the candidate contour;

[0401] Eliminate candidate contours whose area ratio to the circumscribed rectangle of the candidate contour is less than a preset ratio threshold;

[0402] Calculating the distances between the grayscale centroid of the candidate contour and the center point of the circumscribed rectangle of the candidate contour on the horizontal axis and the vertical axis, respectively, and calculating the ratio of each distance to the side length of the candidate contour, respectively, if at least one of the two ratios exceeds a second preset ratio threshold, eliminating the candidate contour;

[0403] Determining the roundness of the candidate contour according to the total number of pixels included in the candidate contour and the side length of the candidate contour, and eliminating the candidate contour if the roundness is lower than a preset roundness threshold;

[0404] Calculating the brightness mean of the candidate contours, and if the brightness mean is less than a preset brightness threshold, eliminating the candidate contours;

[0405] Determine the brightness mean of a preset peripheral area of ​​the circumscribed rectangle of the candidate area and the brightness mean of the candidate contour, and if the brightness difference between the two brightness mean values ​​is less than a preset difference, eliminate the candidate contour.

[0406] In some embodiments of the present disclosure, the processor 2101 initializes the relative position between the handle and the virtual display device based on the target light spot set, the observation data synchronously collected by the IMU, and the optimized 3D spatial structure of each light emitter on the handle. The specific operations are:

[0407] Matching each light emitter on the optimized 3D spatial structure with a target light spot in the target light spot set to establish a corresponding relationship between the 3D light emitter and the 2D light spot;

[0408] The relative position between the handle and the virtual display device is initialized according to the 3D coordinates of the light emitter and the 2D coordinates of the light spot, which have a corresponding relationship, and the observation data synchronously collected by the IMU.

[0409] In some embodiments of the present disclosure, the processor 2101 matches each light emitter on the optimized 3D spatial structure with a target light spot in the target light spot set to establish a correspondence between the 3D light emitter and the 2D light spot. The specific operations are as follows:

[0410] For any target light spot in the target light spot set, screen out a first specified number of candidate light spots adjacent to the target light spot from the target light spot set, and connect the target light spot with the first specified number of candidate light spots to obtain a plane figure;

[0411] According to the set of light emitters that are actually adjacent in the optimized 3D spatial structure, each light spot in the plane graphic is matched with each light emitter in the set of light emitters that are actually adjacent to each other, to obtain adjacent light spot matching pairs, wherein each adjacent light spot matching pair includes an image light spot index of the light spot and a first identifier of a light emitter that matches the light spot;

[0412] For any set of adjacent light spot matching pairs, determine multiple predicted positions of the handle corresponding to the adjacent light spot matching pairs according to the 2D coordinates of each light spot in the adjacent light spot matching pairs and the 3D coordinates of each light emitter;

[0413] For any predicted pose, projecting each light emitter into a specified image according to the predicted pose to obtain each projection light spot, and matching other light spots in the specified image except the light spots included in the plane figure with each light emitter on the handle based on the projection light spots to obtain other light spot matching pairs, wherein each other light spot matching pair includes an image spot index of the other light spot and a first identifier of the light emitter corresponding to the projection light spot matching the other light spot;

[0414] Each spot matching pair is screened according to the number of each other spot matching pair, and each target spot matching pair is obtained according to the number of each screened spot matching pair, and the first identifier of the light emitter in the target spot matching pair is determined as the second identifier of the target spot corresponding to the image spot index, wherein the spot matching pair includes the adjacent spot matching pair and the other spot matching pairs, and each matching pair represents the correspondence between the 3D light emitter and the 2D spot.

[0415] In some embodiments of the present disclosure, the processor 2101 selects a first specified number of candidate light spots adjacent to the target light spot from the target light spot set by:

[0416] Obtaining a distance between the target light spot and the other light spots according to the 2D coordinates of the target light spot and the 2D coordinates of the other light spots in the target light spot set;

[0417] According to the order of the distances between the target light spot and the other light spots from small to large, the other light spots corresponding to the first specified number of distances are selected as the candidate light spots.

[0418] In some embodiments of the present disclosure, the processor 2101 matches each light spot in the plane figure with each light emitter in the set of light emitters that are actually adjacent in the optimized 3D spatial structure, to obtain each matching pair of adjacent light spots. The specific operations are:

[0419] Arrange the light spots in the plane graph in ascending order of image light spot index to obtain a light spot list;

[0420] Traversing each light emitter in the set of actually adjacent light emitters in a specified order, taking the light emitter currently traversed as the initial position, and sorting other light emitters actually adjacent to the light emitter in the specified order to obtain a sorted list;

[0421] For any light emitter in the sorted list, add the first identifier of the light emitter and the image spot index of the spot in the light spot list having the same position as the light emitter in the sorted list to the same adjacent spot matching pair;

[0422] Determining whether there is any light emitter that has not been traversed in the set of actually adjacent light emitters;

[0423] If so, return to the step of traversing each light emitter in the set of actually adjacent light emitters in a specified order until there is no untraversed light emitter in the set of actually adjacent light emitters.

[0424] In some embodiments of the present disclosure, before projecting each light emitter into a specified image according to the predicted pose, the processor 2101 further executes:

[0425] For a plurality of predicted postures of the handle corresponding to any set of adjacent light spot matching pairs, respectively obtain predicted gravity direction vectors of the handle corresponding to the adjacent light spot matching pairs;

[0426] Obtaining an actual gravity direction vector of the handle according to a current position of the virtual display device when the designated image is captured;

[0427] Adjacent light spot matching pairs that need to be deleted are determined by using the predicted gravity direction vectors corresponding to each adjacent light spot matching pair and the actual direction vector, and the adjacent light spot matching pairs that need to be deleted are deleted.

[0428] In some embodiments of the present disclosure, the processor 2101 determines adjacent light spot matching pairs to be deleted by using the predicted gravity direction vectors and the actual direction vectors corresponding to the adjacent light spot matching pairs. Specifically, the processor 2101 performs the following operations:

[0429] For any set of adjacent light spot matching pairs, obtaining the angle between the gravity direction vectors according to the predicted gravity direction vectors corresponding to the adjacent light spot matching pairs and the actual direction vector;

[0430] If the angle between the gravity direction vectors is greater than a specified angle, the adjacent light spot matching pair is determined to be the adjacent light spot matching pair that needs to be deleted.

[0431] In some embodiments of the present disclosure, the processor 2101 matches the light spots in the designated image other than the light spots included in the plane figure with the light emitters on the handle based on the projection light spots to obtain other light spot matching pairs. The specific operations are as follows:

[0432] For any other light spot in the designated image, obtaining the distances between the other light spot and each of the projection light spots according to the 2D coordinates of the other light spot and the 2D coordinates of each of the projection light spots;

[0433] If the shortest distance among the distances is less than the specified distance, the image spot index of the other spot and the first identifier of the light emitter corresponding to the projection spot corresponding to the shortest distance are added to the same spot matching pair, and the spot matching pair is determined as the other spot matching pair.

[0434] In some embodiments of the present disclosure, the processor 2101 screens the spot matching pairs according to the number of the other spot matching pairs, and obtains the target spot matching pairs according to the number of the screened spot matching pairs. The specific operations are:

[0435] For any predicted posture, if the number of other spot matching pairs corresponding to the predicted posture is less than a second specified number, deleting the predicted posture and other spot matching pairs corresponding to the predicted posture;

[0436] For any adjacent spot matching pair, if multiple predicted poses corresponding to the adjacent spot matching pair have been deleted, then the adjacent spot matching pair is deleted;

[0437] Count the number of matching pairs of each spot remaining after elimination;

[0438] For each spot matching pair having the same image spot index, the spot matching pair with the largest number among the spot matching pairs is determined as the target spot matching pair corresponding to the image spot index.

[0439] In some embodiments of the present disclosure, the processor 2101 determines the current target relative pose between the handle and the virtual display device based on the predicted current relative pose between the handle and the virtual display device and the observation data continuously collected by the IMU, specifically by performing the following operations:

[0440] Determining the local range of each light emitter in the current target handle image based on the 3D coordinates of each light emitter on the handle in the optimized 3D spatial structure and the predicted current relative position between the handle and the virtual display device;

[0441] Extracting the current light spots of each current light emitter within the local range of the current target handle image, and determining the light emitter corresponding to each current light spot based on nearest neighbor matching;

[0442] Establishing a reprojection constraint equation based on the corresponding 2D coordinates of the current light spot and the 3D coordinates of the 3D light emitter, and the poses of the IMU and the camera when the observation data and the current target handle image are synchronized;

[0443] Establishing a pre-integration constraint equation based on the IMU posture and the movement speed of the handle corresponding to two consecutive frames of observation data;

[0444] Combining the pre-integration constraint equation and the reprojection constraint equation, solving the pose of the IMU, the pose of the camera, and the relative pose of the IMU and the handle corresponding to the current target handle image;

[0445] According to the relative posture of the IMU and the handle, the posture of the IMU and the posture of the camera, the current target relative posture between the handle and the virtual display device is obtained.

[0446] In some embodiments of the present disclosure, the pre-integration constraint equation is:

[0447] The reprojection constraint equation is:

[0448] in, They respectively represent the rotation matrix and translation vector of the IMU in the world coordinate system corresponding to the j-th frame observation data collected by the IMU, represents the translation vector of the IMU in the world coordinate system corresponding to the j+1th frame observation data collected by the IMU, They represent the motion speed of the IMU in the world coordinate system corresponding to the j-th frame and the j+1-th frame observation data, respectively, W represents the acceleration of gravity, Δt represents the time interval between the j-th frame and the j+1-th frame of observation data collected by the IMU, LOG(·) represents the logarithmic function on the Lie group SO3 corresponding to the quaternion array, Respectively represent the pre-integrated variables of the translation vector, the motion velocity and the rotation matrix of the IMU, They respectively represent the rotation matrix and translation vector of the camera on the virtual display device in the world coordinate system corresponding to the j-th frame observation data collected by the IMU, They represent the rotation matrix and translation vector of the IMU in the handle coordinate system respectively. represents the 3D coordinates of the first light emitter on the handle, which is marked as m, and p m represents the 2D coordinates of the current light spot with the second marker m on the handle, and pro j(·) represents the projection equation of the camera.

[0449] In some embodiments of the present disclosure, the result of combining the pre-integration constraint equation and the reprojection constraint equation is:

[0450] in, They represent the rotation matrix and translation vector of the IMU in the world coordinate system corresponding to the j-th frame observation data collected by the IMU, j represents the number of frames of observation data collected by the IMU, and f j represents the pre-integration constraint equation, g j represents the reprojection constraint equation.

[0451] It should be noted that Figure 21 is only an example, showing the hardware necessary for the virtual display device to implement the method steps for estimating the handle posture provided by the present disclosure. Not shown, the virtual display device also includes conventional hardware such as speakers, receivers, lenses, and power interfaces.

[0452] The processor involved in Figure 21 of the embodiment of the present disclosure can be a central processing unit (CPU), a general-purpose processor, a graphics processing unit (GPU), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof.

[0453] The embodiment of the present disclosure further provides a computer-readable storage medium for storing some instructions. When these instructions are executed, the method for estimating the handle posture in the above embodiment can be completed.

[0454] The embodiments of the present disclosure further provide a computer program product for storing a computer program for executing the method for estimating the handle posture in the aforementioned embodiments.

[0455] Those skilled in the art will appreciate that the embodiments of the present disclosure may be provided as methods, apparatus, or computer program products. Therefore, the present disclosure may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present disclosure may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0456] The present disclosure is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the present disclosure. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device produce a device for implementing the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0457] These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a product including an instruction device that implements the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0458] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0459] Obviously, those skilled in the art may make various changes and modifications to the present disclosure without departing from the spirit and scope of the present disclosure. Thus, if these modifications and variations of the present disclosure fall within the scope of the claims of the present disclosure and their equivalents, the present disclosure is intended to include these modifications and variations.

Claims

1. A method for estimating the position and posture of a handle, applied to a virtual display device, wherein the virtual display device interacts with a handle, the handle being used to control the image displayed by the virtual display device, the handle being mounted with an IMU and multiple light emitters, and the virtual display device being mounted with a multi-camera matching the light emitter types, the method comprising: For the first frame of the target handle image captured by each of the multi-cameras, a target light spot set of each light emitter is obtained based on the target handle image, and the relative position between the handle and the virtual display device is initialized based on the target light spot set, the observation data synchronously collected by the IMU, and the optimized 3D spatial structure of each light emitter on the handle; wherein the 3D spatial structure is optimized based on the labeling results of each light emitter in multiple frames of initial handle images captured at different positions and angles; For the non-first frame target handle images captured by each of the multi-cameras, the current relative pose between the handle and the virtual display device is predicted based on the relative pose corresponding to the historical target handle images, and the current target relative pose between the handle and the virtual display device is determined in combination with the observation data continuously collected by the IMU.

2. The method according to claim 1, wherein the 3D spatial structure of each light emitter on the handle is optimized by: Obtaining the 3D coordinates and first identification of each light emitter according to the 3D spatial structure of each light emitter before optimization; According to the first identification of each light emitter, for the light emitters pre-annotated on multiple frames of initial handle images collected at different positions and angles, the 2D coordinates and second identification of the light spot formed by each light emitter on the corresponding initial handle image are obtained; For each frame of the initial handle image, determine the relative position between the handle and the acquisition camera based on the 3D coordinates of the light source and the 2D coordinates of the light spot with the same first identifier and second identifier, and the observation data of the IMU corresponding to the corresponding frame; A reprojection error equation is constructed, and the relative postures and 3D coordinates of each light emitter are simultaneously optimized according to the reprojection error equation to obtain the 3D spatial structure after the first optimization.

3. The method according to claim 2, wherein optimizing the 3D spatial structure of each light emitter on the handle further comprises: After obtaining the first optimized 3D spatial structure, determining a conversion pose between the first 3D point cloud and the second 3D point cloud before and after optimization based on a first 3D point cloud composed of the light emitters on the handle corresponding to the optimized 3D spatial structure and a second 3D point cloud composed of the light emitters on the handle corresponding to the 3D spatial structure before optimization; According to the transformed posture, the 3D coordinates of each light emitter on the handle are re-determined to obtain the 3D spatial structure after the second optimization.

4. The method according to claim 2 or 3, wherein the reprojection error equation is: in, K n represents the projection parameters of the nth camera, Respectively represent the rotation matrix and translation vector between the handle and camera No. 0, Respectively represent the rotation matrix and the square matrix between the nth camera and the 0th camera Shift vector, represents the 3D coordinates of the first light emitter identified as m on the handle, p m,n represents the 2D coordinates of the second light spot marked as m.

5. The method according to claim 1, wherein obtaining the target light spot set of each light emitter according to the target handle image comprises: Obtaining the current ambient brightness, determining a binarization threshold for each of at least two binarization methods based on the current ambient brightness, and performing binarization processing on the target handle image based on each binarization threshold to obtain a binarized handle image; Performing contour detection on the global scope of the binary handle image to obtain a set of candidate contours of each light emitter, wherein each contour represents a light spot; Abnormal contours in the candidate contour set are eliminated according to the prior contour shape information and contour contrast information, to obtain the target light spot set of each light emitter.

6. The method of claim 5, wherein determining a binarization threshold for each of at least two binarization methods based on the current ambient brightness, and binarizing the target handle image based on each binarization threshold to obtain a binarized handle image comprises: Eliminating pixels whose grayscale values ​​are lower than a preset grayscale threshold in a grayscale handle image obtained by grayscaling the target handle image, and determining the respective binarization thresholds of the at least two binarization methods based on a new histogram of the grayscale handle image after the pixel elimination; Comparing the current ambient brightness with a preset brightness threshold, and determining weights corresponding to the at least two binarization thresholds respectively according to the comparison result; According to each binarization threshold and the corresponding weight, the target binarization threshold is obtained by weighting; The grayscale handle image is binarized according to the target binarization threshold to obtain a binarized handle image.

7. The method according to claim 6, wherein determining the weights corresponding to the at least two binarization thresholds respectively according to the comparison results comprises: When the current ambient brightness is greater than the preset brightness threshold, setting a first weight corresponding to a first binarization threshold calculated by the first binarization method to be greater than a second weight corresponding to a second binarization threshold calculated by the second binarization method; When the current ambient brightness is less than or equal to the preset brightness threshold, setting a first weight corresponding to a first binarization threshold calculated by the first binarization method to be less than a second weight corresponding to a second binarization threshold calculated by the second binarization method; The first binarization method is used to solve a histogram distribution containing a single peak, and the second binarization method is used to solve a histogram distribution containing two peaks.

8. The method according to claim 5, wherein the method of eliminating abnormal contours in the candidate contour set according to the contour contrast information comprises one or more of the following: For each two candidate contours in the candidate contour set, the Euclidean distance between the center points of the circumscribed rectangles of the two candidate contours and the minimum Manhattan distance between the edges of the two candidate contours are determined respectively, and the Euclidean distance and the minimum Manhattan distance between the edges of the two candidate contours are determined respectively. Describe the minimum Manhattan distance and remove abnormal contours; Sort all candidate contours in the candidate contour set according to their areas, and eliminate abnormal contours based on the relationship between the number of pixels in the candidate contour with the largest area and the candidate contour with the second largest area; For each candidate contour in the candidate contour set, calculating the distance between the candidate contour and the nearest neighbor candidate contour, and eliminating outlier contours based on the distance; The brightness mean of each candidate contour in the candidate contour set is calculated, and abnormal contours are eliminated based on the brightness mean values.

9. The method according to claim 8, wherein the step of eliminating abnormal contours according to the Euclidean distance and the minimum Manhattan distance comprises: When at least one of the Euclidean distance and the minimum Manhattan distance is less than a preset distance threshold, respectively calculating the areas of the two candidate contours; If the areas of both candidate contours are smaller than the preset area threshold, both candidate contours are eliminated at the same time; If at least one of the areas of the two candidate contours is not less than the preset area threshold, the brightness mean values ​​of the two candidate contours are calculated respectively, and the candidate contour corresponding to the smaller brightness mean value is eliminated.

10. The method according to claim 8, wherein the step of eliminating abnormal contours based on the quantitative relationship between pixels within the largest candidate contour and the second largest candidate contour comprises: If the number of pixels in the candidate contour with the largest area and the candidate contour with the second largest area both exceeds a preset pixel number threshold, then calculate the multiple of the number of pixels in the candidate contour with the largest area and the candidate contour with the second largest area; If the multiple is greater than a preset multiple threshold, the candidate contour with the largest area is eliminated.

11. The method according to claim 8, wherein the step of removing outlier abnormal contours according to the distance comprises: Determining an adaptive outlier distance according to the side length of the candidate contour and the median of the side lengths of all candidate contours; If the number of all candidate contours is greater than a preset number threshold, and the distance is greater than the adaptive outlier distance, the candidate contour is eliminated.

12. The method according to claim 5, wherein the method of eliminating abnormal contours in the candidate contour set according to the prior contour shape information comprises one or more of the following: Eliminate candidate contours whose aspect ratio exceeds a first preset ratio threshold according to a relationship between the area of ​​the candidate contour and the aspect ratio of the circumscribed rectangle of the candidate contour; Eliminate candidate contours whose area ratio to the circumscribed rectangle of the candidate contour is less than a preset ratio threshold; Calculating the distances between the grayscale centroid of the candidate contour and the center point of the circumscribed rectangle of the candidate contour on the horizontal axis and the vertical axis, respectively, and calculating the ratio of each distance to the side length of the candidate contour, and if at least one of the two ratios exceeds a second preset ratio threshold, eliminating the candidate contour; Determining the roundness of the candidate contour according to the total number of pixels included in the candidate contour and the side length of the candidate contour, and eliminating the candidate contour if the roundness is lower than a preset roundness threshold; Calculating the brightness mean of the candidate contours, and if the brightness mean is less than a preset brightness threshold, eliminating the candidate contours; Determine the brightness mean of a preset peripheral area of ​​the circumscribed rectangle of the candidate area and the brightness mean of the candidate contour, and if the brightness difference between the two brightness mean values ​​is less than a preset difference, eliminate the candidate contour.

13. The method of claim 1, wherein initializing the relative position between the handle and the virtual display device based on the target light spot set, the observation data synchronously collected by the IMU, and the optimized 3D spatial structure of each light emitter on the handle comprises: Matching each light emitter on the optimized 3D spatial structure with a target light spot in the target light spot set to establish a corresponding relationship between the 3D light emitter and the 2D light spot; The relative position between the handle and the virtual display device is initialized according to the 3D coordinates of the light emitter and the 2D coordinates of the light spot, which have a corresponding relationship, and the observation data synchronously collected by the IMU.

14. The method of claim 13, wherein matching each light emitter in the optimized 3D spatial structure with a target light spot in the target light spot set to establish a correspondence between the 3D light emitter and the 2D light spot comprises: For any target light spot in the target light spot set, screen out a first specified number of candidate light spots adjacent to the target light spot from the target light spot set, and connect the target light spot with the first specified number of candidate light spots to obtain a plane figure; According to the set of light emitters that are actually adjacent in the optimized 3D spatial structure, each light spot in the plane graphic is matched with each light emitter in the set of light emitters that are actually adjacent to each other, to obtain adjacent light spot matching pairs, wherein each adjacent light spot matching pair includes an image light spot index of the light spot and a first identifier of a light emitter that matches the light spot; For any set of adjacent light spot matching pairs, determine multiple predicted positions of the handle corresponding to the adjacent light spot matching pairs according to the 2D coordinates of each light spot in the adjacent light spot matching pairs and the 3D coordinates of each light emitter; For any predicted pose, projecting each light emitter into a specified image according to the predicted pose to obtain each projection light spot, and matching other light spots in the specified image except the light spots included in the plane figure with each light emitter on the handle based on the projection light spots to obtain other light spot matching pairs, wherein each other light spot matching pair includes an image spot index of the other light spot and a first identifier of the light emitter corresponding to the projection light spot matching the other light spot; Each spot matching pair is screened according to the number of each other spot matching pair, and each target spot matching pair is obtained according to the number of each screened spot matching pair, and the first identifier of the light emitter in the target spot matching pair is determined as the second identifier of the target spot corresponding to the image spot index, wherein the spot matching pair includes the adjacent spot matching pair and the other spot matching pairs, and each matching pair represents the correspondence between the 3D light emitter and the 2D spot.

15. The method according to claim 14, wherein screening out a first specified number of candidate light spots adjacent to the target light spot from the target light spot set comprises: Obtaining a distance between the target light spot and the other light spots according to the 2D coordinates of the target light spot and the 2D coordinates of the other light spots in the target light spot set; According to the order of the distances between the target light spot and the other light spots from small to large, the other light spots corresponding to the first specified number of distances are selected as the candidate light spots.

16. The method of claim 14, wherein the step of matching each light spot in the planar graphic with each light emitter in the set of light emitters that are actually adjacent in the optimized 3D spatial structure to obtain matching pairs of adjacent light spots comprises: Arrange the light spots in the plane graph in ascending order of image light spot index to obtain a light spot list; Traversing each light emitter in the set of actually adjacent light emitters in a specified order, taking the light emitter currently traversed as the initial position, and sorting other light emitters actually adjacent to the light emitter in the specified order to obtain a sorted list; For any light emitter in the sorted list, add the first identifier of the light emitter and the image spot index of the spot in the light spot list having the same position as the light emitter in the sorted list to the same adjacent spot matching pair; Determining whether there is any light emitter that has not been traversed in the set of actually adjacent light emitters; If so, return to the step of traversing each light emitter in the set of actually adjacent light emitters in a specified order until there is no untraversed light emitter in the set of actually adjacent light emitters.

17. The method according to claim 14, further comprising: before projecting each light emitter into a designated image according to the predicted position; For a plurality of predicted postures of the handle corresponding to any set of adjacent light spot matching pairs, respectively obtain predicted gravity direction vectors of the handle corresponding to the adjacent light spot matching pairs; Obtaining an actual gravity direction vector of the handle according to a current position of the virtual display device when the designated image is captured; Adjacent light spot matching pairs that need to be deleted are determined by using the predicted gravity direction vectors corresponding to each adjacent light spot matching pair and the actual gravity direction vector, and the adjacent light spot matching pairs that need to be deleted are deleted.

18. The method according to claim 17, wherein determining the adjacent light spot matching pairs to be deleted by using the predicted gravity direction vectors corresponding to the adjacent light spot matching pairs and the actual gravity direction vectors comprises: For any set of adjacent light spot matching pairs, obtaining the angle between the gravity direction vectors according to the predicted gravity direction vectors corresponding to the adjacent light spot matching pairs and the actual gravity direction vector; If the angle between the gravity direction vectors is greater than a specified angle, the adjacent light spot matching pair is determined to be the adjacent light spot matching pair that needs to be deleted.

19. The method of claim 14, wherein the step of matching light spots other than the light spots included in the planar pattern in the designated image with the light emitters on the handle based on the projected light spots to obtain other light spot matching pairs comprises: For any other light spot in the designated image, obtaining the distances between the other light spot and each of the projection light spots according to the 2D coordinates of the other light spot and the 2D coordinates of each of the projection light spots; If the shortest distance among the distances is less than the specified distance, the image spot index of the other spot and the first identifier of the light emitter corresponding to the projection spot corresponding to the shortest distance are added to the same spot matching pair, and the spot matching pair is determined as the other spot matching pair.

20. The method of claim 14, wherein screening the spot matching pairs according to the number of the other spot matching pairs, and obtaining the target spot matching pairs according to the number of the screened spot matching pairs, comprises: For any predicted posture, if the number of other spot matching pairs corresponding to the predicted posture is less than a second specified number, deleting the predicted posture and other spot matching pairs corresponding to the predicted posture; For any adjacent spot matching pair, if multiple predicted poses corresponding to the adjacent spot matching pair have been deleted, then the adjacent spot matching pair is deleted; Count the number of matching pairs of each spot remaining after elimination; For each spot matching pair having the same image spot index, the spot matching pair with the largest number among the spot matching pairs is determined as the target spot matching pair corresponding to the image spot index.

21. The method of claim 1, further comprising determining a current target relative pose between the handle and the virtual display device based on the predicted current relative pose between the handle and the virtual display device and observation data continuously collected by the IMU, comprising: Determining the local range of each light emitter in the current target handle image based on the 3D coordinates of each light emitter on the handle in the optimized 3D spatial structure and the predicted current relative position between the handle and the virtual display device; Extracting the current light spots of each current light emitter within the local range of the current target handle image, and determining the light emitter corresponding to each current light spot based on nearest neighbor matching; Establishing a reprojection constraint equation based on the corresponding 2D coordinates of the current light spot and the 3D coordinates of the 3D light emitter, and the poses of the IMU and the camera when the observation data and the current target handle image are synchronized; Establishing a pre-integration constraint equation based on the IMU posture and the movement speed of the handle corresponding to two consecutive frames of observation data; Combining the pre-integration constraint equation and the reprojection constraint equation, solving the pose of the IMU, the pose of the camera, and the relative pose of the IMU and the handle corresponding to the current target handle image; According to the relative posture of the IMU and the handle, the posture of the IMU and the posture of the camera, the current target relative posture between the handle and the virtual display device is obtained.

22. The method according to claim 21, wherein the pre-integration constraint equation is: The reprojection constraint equation is: in, They respectively represent the rotation matrix and translation vector of the IMU in the world coordinate system corresponding to the j-th frame observation data collected by the IMU, represents the translation vector of the IMU in the world coordinate system corresponding to the j+1th frame observation data collected by the IMU, They represent the motion speed of the IMU in the world coordinate system corresponding to the j-th frame and the j+1-th frame observation data, respectively, W represents the acceleration of gravity, Δt represents the time interval between the j-th frame and the j+1-th frame of observation data collected by the IMU, LOG(·) represents the logarithmic function on the Lie group SO3 corresponding to the quaternion array, Respectively represent the pre-integrated variables of the translation vector, the motion velocity and the rotation matrix of the IMU, They respectively represent the rotation matrix and translation vector of the camera on the virtual display device in the world coordinate system corresponding to the j-th frame observation data collected by the IMU, They represent the rotation matrix and translation vector of the IMU in the handle coordinate system respectively. represents the 3D coordinates of the first light emitter marked as m on the handle, p m represents the 2D coordinates of the current spot with the second mark m on the handle, and pro j(·) represents the projection equation of the camera.

23. The method of claim 22, wherein a result of combining the pre-integration constraint equation and the reprojection constraint equation is: in, They represent the rotation matrix and translation vector of the IMU in the world coordinate system corresponding to the j-th frame observation data collected by the IMU, j represents the number of frames of observation data collected by the IMU, and f j represents the pre-integration constraint equation, g j represents the reprojection constraint equation.

24. A virtual display device comprising a processor, a memory, a display screen, a communication interface, and a multi-camera, wherein the display screen is used to display images, the virtual display device communicates with a handle via the communication interface, the handle is used to control the images displayed on the display screen, and the type of the multi-camera matches the light-emitting type of multiple light emitters on the handle; The communication interface, the multi-camera, the display screen, the memory, and the processor are connected via a bus. The memory stores a computer program. The processor performs the following operations according to the computer program: For the first frame target handle image captured by each of the multi-cameras, the target light spot set of each light emitter is obtained according to the target handle image, and the relative position between the handle and the virtual display device is initialized according to the target light spot set, the observation data synchronously collected by the IMU and the optimized 3D spatial structure of each light emitter on the handle. posture; among them, The 3D spatial structure is optimized based on the labeling results of each light emitter in multiple frames of initial handle images collected at different positions and angles; For the non-first frame target handle images captured by each of the multi-cameras, the current relative pose between the handle and the virtual display device is predicted based on the relative pose between the handle and the virtual display device corresponding to the historical target handle images. Combined with the observation data continuously collected by the IMU, the target relative pose between the current handle and the virtual display device is determined.

Images