Left and right handle positioning methods, devices, electronic devices and readable storage media

Abstract

This application discloses a method, apparatus, electronic device, and readable storage medium for distinguishing and locating left and right handheld controllers. The method includes: acquiring infrared images of the left and right handheld controllers; performing image processing on the infrared images to obtain the centers of each light spot; clustering the light spot centers to classify them into different categories; determining the light spot centers and positioning information corresponding to the left and right handheld controllers respectively based on the correspondence and coordinates between the light spots on the left and right handheld controllers and the light spot centers of each category; and predicting the positioning information of the left and right handheld controllers in the current frame based on the positioning information of the left and right handheld controllers in the previous two frames after obtaining the positioning information of the left and right handheld controllers in the previous two frames. This application solves the technical problem of poor environmental adaptability of infrared light-based methods for distinguishing and locating left and right handheld controllers.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of VR (Virtual Reality), in particular to a left and right handle distinguishing and positioning method and device, electronic equipment and readable storage medium. BACKGROUND

[0002] In the existing VR device, the VR handle is still an indispensable component, and the key to the successful use of the handle is whether the positioning of the VR handle in the VR space is accurate. Since the handle needs to be used in pairs in practice, how to correctly and quickly distinguish the left handle and the right handle is the top priority of handle positioning.

[0003] The main means for distinguishing and positioning the left and right handles of the VR device at present include structure distinction, visible light distinction and infrared light distinction, etc., so as to finally separate the left handle and the right handle in the VR space. The advantage of structure distinction is that the effect is obvious, but it needs to distinguish the double handles from the design and needs different molds to make, so the cost is high. The advantage of visible light distinction is that the algorithm difficulty is low, the calculation amount is low, and the precision is high, but it has high requirements for the indoor environment, and cannot work in very dark or very bright conditions, so the environmental adaptability is poor. The advantage of infrared light distinction is that it can ensure high precision and low cost, and has strong anti-interference ability, but the algorithm involved is complex, and when encountering point-like strong light source interference and handle partial overlap, the precision of distinguishing and positioning the double handles will decrease, so the environmental adaptability of the left and right handle distinguishing and positioning method based on infrared light is poor. SUMMARY

[0004] The main purpose of the present application is to provide a left and right handle distinguishing and positioning method, device, electronic equipment and readable storage medium, which aims to solve the technical problem of poor environmental adaptability of the left and right handle distinguishing and positioning method based on infrared light.

[0005] To achieve the above purpose, the present application provides a left and right handle distinguishing and positioning method, which comprises:

[0006] An infrared image of the left and right handle is obtained, and the infrared image is processed to obtain the center of each light point;

[0007] Each light point center is clustered to divide each light point center into different categories;

[0008] According to the correspondence relationship and coordinates of each light point on the left and right handle and the center of each category of light points, the corresponding light point center and positioning information of the left and right handle are determined;

[0009] After obtaining the positioning information of the left and right controllers in the first two frames, the positioning information of the left and right controllers in the current frame is predicted based on the positioning information of the left and right controllers in the first two frames.

[0010] Optionally, the step of performing image processing on the infrared image to obtain the center of each light spot includes:

[0011] Contour extraction is performed on each light spot in the infrared image to obtain the contour corresponding to each light spot;

[0012] The contours of each light point are fitted to obtain the shape of each target contour.

[0013] Extract the geometric center of each of the contour shapes to obtain the center of each of the light spots.

[0014] Optionally, the step of performing light spot clustering on each of the light spot centers to divide each of the light spot centers into different categories includes:

[0015] Select the first cluster center from the centers of each light spot, and determine the cluster interval based on the ratio of the light spacing to the hand length on the handle model and the camera focal length;

[0016] The center of the light spot that is farthest from the first cluster center and at a distance greater than the cluster interval is set as another cluster center;

[0017] Repeat the following steps: If the distance between the center of the light spot and the farthest cluster center is greater than the clustering interval, then set the center of the light spot as the cluster center, until the distance between all light spot centers and the nearest cluster center is less than the clustering interval;

[0018] Each of the light spot centers is assigned to the category of the nearest cluster center.

[0019] Optionally, the step of selecting a first cluster center from the centers of each of the light spots and determining the cluster interval based on the ratio of the light spacing to the hand length on the handle model and the camera focal length includes:

[0020] The center of the light spot in the upper left corner of the infrared image is selected as the first cluster center;

[0021] Calculate the ratio of the lamp spacing to the hand length on the handle model, and multiply the ratio by the camera focal length to obtain the clustering interval.

[0022] Optionally, the step of determining the center of the light spot and the positioning information corresponding to the left and right handles respectively based on the correspondence and coordinates between the light spots on the left and right handles and the centers of light spots of each category includes:

[0023] Based on the correspondence and coordinates between the center of each type of light spot and the light spots on the left and right handles, predict the handle pose corresponding to the center of each type of light spot respectively;

[0024] Based on the handle pose, camera intrinsic parameters, two-dimensional coordinates of each light spot center, and three-dimensional coordinates of the light spots in the three-dimensional models of the left and right handles, the average reprojection error of each handle pose relative to the left and right handles is calculated.

[0025] Set the centers of the two categories of light spots with the smallest average reprojection errors corresponding to the left and right handles as the centers of the light spots corresponding to the left and right handles, respectively.

[0026] The handle poses corresponding to the centers of the light spots on the left and right handles are respectively set as the positioning information of the left and right handles.

[0027] Optionally, before the step of predicting the handle pose corresponding to the center of each type of light spot based on the correspondence and coordinates between the center of each type of light spot and the light spots on the left and right handles, the method further includes:

[0028] Sequentially determine whether the number of light spot centers of each category is less than a first preset threshold;

[0029] If the number is less than the first preset threshold, then the center of the light spot of the category is determined to be an interference light spot;

[0030] If it is not less than the first preset threshold, then the following step is performed: predict the handle pose corresponding to the center of the light spot of the category and the coordinates of each light spot on the left and right handles.

[0031] Optionally, the step of predicting the positioning information of the left and right handles in the current frame based on the positioning information of the left and right handles in the previous two frames includes:

[0032] Predict the current hand poses of the left and right hand controllers based on the hand controller poses in the positioning information of the first two frames.

[0033] The two-dimensional coordinates of the light spots on the left and right handles are determined based on the current handle pose and the two-dimensional coordinates of the light spot centers of the left and right handles in the current frame infrared image.

[0034] The two-dimensional coordinates of the light spots on the left and right handles are matched with the two-dimensional coordinates of the centers of the light spots on the left and right handles in the current frame of the infrared image to obtain the number of successful matches.

[0035] When the number of successful matches is not less than the second preset threshold, the updated handle poses of the left and right handles are predicted according to the correspondence and coordinates between each light spot on the left and right handles and the center of the light spot in the current frame infrared image.

[0036] If the average reprojection error of the updated handle pose is less than a third preset threshold, then the updated handle pose is set to the positioning information of the left and right handles in the current frame.

[0037] Optionally, after the step of matching the two-dimensional coordinates of the light spots corresponding to the left and right handles with the two-dimensional coordinates of the centers of the light spots of the left and right handles in the current frame infrared image to obtain the number of successful matches, the method further includes:

[0038] When the number of successful matches is less than the second preset threshold, the prediction is determined to be unsuccessful, and the execution step is returned: perform light spot clustering on each light spot center to divide each light spot center into different categories;

[0039] After predicting the updated handle poses corresponding to the left and right handles based on the correspondence and coordinates between the light spots on the left and right handles and the centers of the light spots in the current frame infrared image, the method further includes:

[0040] If the average reprojection error of the updated handle pose is not less than the third preset threshold, the prediction is determined to be unsuccessful, and the execution step is returned: perform light spot clustering on each of the light spot centers to divide each of the light spot centers into different categories.

[0041] This application also provides a left and right handle differentiation and positioning device, which is applied to a left and right handle differentiation and positioning device, and the left and right handle differentiation and positioning device includes:

[0042] The light spot extraction module is used to acquire infrared images of the left and right handles, perform image processing on the infrared images, and obtain the center of each light spot.

[0043] The light spot clustering module is used to cluster the light spot centers to divide the light spot centers into different categories;

[0044] The differentiation and positioning module is used to determine the center of the light spot and the positioning information corresponding to the left and right handles respectively, based on the correspondence and coordinates between each light spot on the left and right handles and the center of each type of light spot;

[0045] The prediction and positioning module is used to predict the positioning information of the left and right controllers in the current frame based on the positioning information of the left and right controllers in the previous two frames.

[0046] This application also provides an electronic device, which is a physical device, comprising: a memory, a processor, and a program for the left and right handle differentiation and positioning method stored in the memory and executable on the processor. When the program for the left and right handle differentiation and positioning method is executed by the processor, it can implement the steps of the left and right handle differentiation and positioning method as described above.

[0047] This application also provides a computer-readable storage medium storing a program for implementing a left and right hand controller differentiation and positioning method. When the program for the left and right hand controller differentiation and positioning method is executed by a processor, it implements the steps of the left and right hand controller differentiation and positioning method as described above.

[0048] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the left and right handle differentiation and positioning method described above.

[0049] This application provides a method, device, electronic device, and readable storage medium for distinguishing and positioning left and right hand controllers. First, infrared images of the left and right hand controllers are acquired. The infrared images are then processed to obtain the center of each light spot. Next, the light spot centers are clustered to divide them into different categories. Then, based on the correspondence and coordinates between each light spot on the left and right hand controllers and the center of each category, the light spot center and positioning information corresponding to the left and right hand controllers are determined. After obtaining the positioning information of the left and right hand controllers in the previous two frames, the positioning information of the left and right hand controllers in the current frame is predicted based on the positioning information of the left and right hand controllers in the previous two frames. The technical solution of this application extracts the center of each light spot and then divides the center of each light spot into different categories using a clustering method. Based on the correspondence between each light spot on the left and right handles and the center of each category, calculations are performed to obtain a more accurate representation of the center of the light spot on the left and right handles and the corresponding positioning information. This technical solution only requires a single frame image to complete the division of the center of the light spot on the left and right handles and the prediction of the handle pose, obtaining positioning information. It has few dependencies, requires no input parameters, has a simple algorithm, and strong anti-interference capabilities. After completing the pose prediction of the first two frames using the above method, the prediction of subsequent frames can be directly performed based on the positioning information of the first two frames. After obtaining the positioning information of the left and right handles in the first two frames, it is not necessary to further cluster and distinguish the center of each light spot in the infrared image. Therefore, even when encountering interference from point-like strong light sources or partial overlap of the handles, the differentiation and positioning of the left and right handles can continue, improving the environmental adaptability of the infrared light-based left and right handle differentiation and positioning method. Attached Figure Description

[0050] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0051] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0052] Figure 1 This is a flowchart illustrating the first embodiment of the left and right handle differentiation and positioning method of this application;

[0053] Figure 2 The infrared images of the left and right handles in the first embodiment of the left and right handle differentiation and positioning method of this application are shown below.

[0054] Figure 3 Images of light spots that have been classified in the first embodiment of the left and right handle differentiation and positioning method of this application;

[0055] Figure 4 Images of light points for the left and right handles that have been matched in the first embodiment of the left and right handles differentiation and positioning method of this application;

[0056] Figure 5 A schematic diagram illustrating the overall inventive concept of the left and right handle differentiation and positioning method of this application;

[0057] Figure 6 This is a schematic diagram of the structure of the left and right handle differentiation and positioning device of this application;

[0058] Figure 7 This is a schematic diagram of the device structure of the hardware operating environment involved in the left and right handle differentiation and positioning method in the embodiments of this application.

[0059] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0060] To make the above-mentioned objectives, features, and advantages of this application more apparent and understandable, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of this application, and not all embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0061] Example 1

[0062] In today's rapidly evolving VR technology, VR controllers remain an indispensable component of existing VR devices. The key to successful controller use lies in their accurate positioning within the VR space. Since controllers are typically used in pairs, accurately and quickly distinguishing between the left and right controllers is paramount for accurate positioning. Existing technologies primarily utilize infrared light for controller differentiation, offering advantages such as high accuracy, low cost, and strong anti-interference capabilities. However, the algorithms involved are complex, and issues like strong point light sources or partial controller overlap can negatively impact positioning accuracy, even preventing it from being established. Therefore, existing infrared-based left and right controller differentiation methods have poor environmental adaptability.

[0063] This application provides a method for distinguishing and positioning left and right handles. In the first embodiment of this method, refer to... Figure 1 The method for distinguishing and positioning the left and right handles includes:

[0064] Step S10: Acquire infrared images of the left and right handles, perform image processing on the infrared images, and obtain the center of each light spot;

[0065] Step S20: Perform light spot clustering on each of the light spot centers to divide each of the light spot centers into different categories;

[0066] Step S30: Based on the correspondence and coordinates between each light spot on the left and right handles and the center of each type of light spot, determine the center of the light spot and the positioning information corresponding to the left and right handles respectively;

[0067] Step S40: After obtaining the positioning information of the left and right controllers in the previous two frames, predict the positioning information of the left and right controllers in the current frame based on the positioning information of the left and right controllers in the previous two frames.

[0068] In the embodiments of this application, it should be noted that, referring to Figure 2The infrared images corresponding to the left and right controllers include VR controllers with infrared LEDs (light-emitting diodes). The target contour can be an ellipse, a circle, or other polygons. The image processing process is the extraction of the center of the light spot of each infrared LED, including image binarization, Gaussian blur, image erosion, light spot contour extraction, and ellipse fitting. Each light spot on the left and right controllers is an LED light spot. The coordinate point of the light spot in the image coordinate system can be understood as a 2D light spot, while the coordinate point of the light spot in the actual environment can be understood as a 3D light spot. The coordinate axis of the 3D light spot is preset, so the coordinates of the 3D light spot are known and can be directly obtained (since the coordinate axis is on the controller, its coordinate value does not change no matter how the controller is moved). In this embodiment, the 2D light spots are first clustered to classify the 2D light spots on the same controller into the same category, which facilitates the subsequent differentiation of which 2D light spots belong to which controller. Furthermore, in the process of determining the light point center and positioning information corresponding to the left and right handles respectively based on the correspondence and coordinates between each light point on the left and right handles and the center of each type of light point, the PNP (Perspective-n-Point, 3D to 2D point correspondence) algorithm can be used to solve the problem. Specifically, this includes calculating the current pose of the handle by using the correspondence between each 2D point and 3D point, the two-dimensional coordinates of the 2D point and the three-dimensional coordinates of the 3D point in the three-dimensional model of the handle. The handle pose is the positioning information.

[0069] This application embodiment first distinguishes the light spots of the two controllers and their corresponding predicted poses through an adaptive clustering process. When the predicted poses of the first two frames have been determined, that is, when the positioning distinction in the early stage is stable, the controller poses of the next frame can be directly predicted based on the previous frame to obtain the corresponding positioning information. This solves the problem of overlapping of the two controllers and sticking of interfering light sources affecting the positioning accuracy during the use of VR controllers.

[0070] As an example, steps S10 to S40 include: acquiring an infrared image corresponding to a VR controller containing infrared LEDs; sequentially performing image binarization, Gaussian blurring, and image erosion on the infrared image to obtain a first infrared image; extracting the contours of the light spots in the first infrared image to obtain the contours corresponding to each light spot; extracting the center of the light spot of the target ellipse corresponding to each light spot by fitting each contour to an ellipse; calculating the clustering interval based on the LED spacing on the left and right controller models, the user's arm length, and the camera focal length, selecting a cluster center in the first infrared image, and taking the light spot center that is farthest from the clustering interval as another cluster center; repeatedly selecting multiple cluster centers, wherein the mutual distance between multiple cluster centers is greater than the clustering interval, and the distance from each light spot center to its nearest cluster center is less than the clustering interval; classifying each light spot center into the category of the nearest cluster center to complete the classification of each light spot center; and classifying each light spot on the left and right controllers according to the category of each light spot. The correspondence between the light spot centers, the three-dimensional coordinates of each light spot in the three-dimensional model, and the two-dimensional coordinates of each light spot center in the infrared image are solved using the PNP algorithm to obtain the handle poses corresponding to each category of light spot centers. The average reprojection error relative to the left and right handles is calculated based on each handle pose. The handle pose with the smallest average reprojection error is selected as the handle poses corresponding to the left and right handles respectively, and the two handle poses with the smallest average reprojection errors are used as the positioning information of the left and right handles in the current frame. After completing the positioning of the left and right handles in two consecutive frames, the handle pose of the current frame can be predicted by using the positioning of the left and right handles in the previous two frames. The two-dimensional coordinates of the light spot corresponding to the handle pose of the current frame are matched with the light spot centers obtained by performing step S10 in the current frame. When the number of successful matches exceeds a preset number, the pose is solved again using the PNP algorithm to obtain the updated pose. When the average reprojection error of the updated pose is lower than a preset threshold, the prediction is successful, and the obtained updated pose is used as the positioning information of the left and right handles in the current frame.

[0071] The step of performing image processing on the infrared image to obtain the center of each light spot includes:

[0072] Step S11: Extract the contours of each light spot in the infrared image to obtain the contours corresponding to each light spot;

[0073] Step S12: Fit the contour of each light point to obtain the shape of each target contour;

[0074] Step S13: Extract the geometric center of each contour shape to obtain the center of each light spot.

[0075] In this embodiment, it should be noted that the contour corresponding to each light point includes the contour area and the roundness of the contour, which is used to filter the contours to eliminate interference from external light. As an example, the contour shape can be fitted as an ellipse, which is closer to the shape of an actual LED light.

[0076] As an example, steps S11 to S13 include: performing image binarization, Gaussian blurring, and image erosion sequentially on the infrared image to obtain a first infrared image; extracting the contours of the light spots in the first infrared image to obtain the contour lines and contour areas corresponding to each light spot; performing ellipse fitting on each contour based on the contour lines corresponding to each light spot to obtain each target contour ellipse; and extracting the ellipse center of each target contour ellipse to obtain the center of each light spot.

[0077] The step of performing light spot clustering on each of the light spot centers to divide each of the light spot centers into different categories includes:

[0078] Step S21: Select the first cluster center from the centers of each light spot, and determine the cluster interval based on the ratio of the light spacing to the hand length on the handle model and the camera focal length;

[0079] Step S22: The center of the light spot that is farthest from the first cluster center and whose distance is greater than the cluster interval is set as another cluster center;

[0080] Step S23, repeat the following steps: if the distance between the center of the light spot and the farthest cluster center is greater than the clustering interval, then set the center of the light spot as the cluster center, until the distance between all light spot centers and the nearest cluster center is less than the clustering interval;

[0081] Step S24: Assign each of the light spot centers to the category of the nearest cluster center.

[0082] In this embodiment, it should be noted that a method for clustering the centers of light spots is provided to divide the light spots into different categories. Centers of light spots within the same category can be considered to belong to the same handle or be interference points. After classifying the light spot centers, PNP can be performed according to the set of light spot centers for each category to obtain the predicted poses corresponding to the light spot centers of each category for further filtering. Specifically, the handle model is a 3D model of the left and right handles, where the spacing between each LED is the maximum spacing between each LED, and the hand length is the default adult arm length, used to characterize the distance between the handle and the camera. As an example, the hand length can be 0.7m. If there is a previous frame with a successfully predicted handle pose, the hand length can be replaced with the actual distance between the camera and the handle in the previous frame. In this embodiment, the clustering interval is calculated to classify the light spot centers, thereby obtaining multiple sets of light spot centers corresponding to each category for subsequent differentiation of the left and right handles.

[0083] As an example, steps S21 to S24 include: selecting the center of the light spot located at the top left corner in the infrared image as the first cluster center; obtaining the clustering interval based on the product of the ratio of the lamp spacing and hand length on the handle model and the camera focal length; calculating the distance of each light spot center from the first cluster center based on the infrared image, and selecting the light spot center with the farthest distance to set another cluster center, wherein the distance between the light spot center and the first cluster center is greater than the clustering interval; randomly selecting another cluster center, wherein the cluster center is the farthest from the existing cluster center and is greater than the clustering interval; repeating the above steps until the distance of all light spot centers from the nearest cluster center is less than the clustering interval, thereby obtaining multiple categories of cluster centers and the light spot centers corresponding to each cluster center, wherein each light spot center belongs to the category corresponding to the nearest cluster center.

[0084] As an example, such as Figure 3 As shown, this includes the centers of each light spot that have already been categorized, where the light spot is located... Figure 3 The center points of the upper light spots, 0, 1, 2, 3, and 4, belong to one of the categories and are located in... Figure 3 The center points of light in the middle, 0, 1, 2, 3, 4, and 5, belong to another category.

[0085] Further, the step of selecting a first cluster center from the centers of each of the light spots and determining the cluster interval based on the ratio of the light spacing on the handle model to the hand length and the camera focal length includes:

[0086] Step S211: Select the center of the light spot in the upper left corner of the infrared image as the first cluster center;

[0087] Step S212: Calculate the ratio of the lamp spacing to the hand length on the handle model, and multiply the ratio by the camera focal length to obtain the clustering interval.

[0088] In this embodiment, a method for selecting cluster centers and calculating cluster intervals is provided. Specifically, the center of the light spot in the upper left corner of the infrared image is selected as the first cluster center, maximizing the distance between the cluster centers so that the light spot centers assigned to each category more closely resemble the categories of the left and right handles in reality. The cluster interval takes into account the lamp spacing and hand length, and can better reflect the lower limit of the distance between the light spot centers of different categories under normal circumstances.

[0089] As an example, steps S211 to S212 include: selecting the center of the light spot located at the top left corner of the infrared image as the first cluster center; first calculating the ratio of the lamp spacing to the hand length on the handle model; specifically, if there is a previously successfully predicted handle pose, then obtaining the actual distance between the camera and the handle in the successfully predicted frame to replace the hand length; calculating the product of the ratio and the camera focal length to obtain the clustering interval.

[0090] As a feasible implementation, the formula for calculating the clustering interval is as follows:

[0091]

[0092] Among them, D tho The clustering interval is d, where d is the maximum spacing between LED lights on the model, in meters. h The distance between the camera and the handle is set to the length of an adult's arm by default, which can be 0.7m. f is the camera's focal length.

[0093] Furthermore, in this embodiment of the application, the step of determining the light spot center and positioning information corresponding to the left and right handles respectively based on the correspondence and coordinates between the light spots on the left and right handles and the centers of light spots of each category includes:

[0094] Step S31: Based on the correspondence and coordinates between the center of each type of light spot and the light spots on the left and right handles, predict the handle pose corresponding to the center of each type of light spot respectively;

[0095] Step S32: Calculate the average reprojection error of each handpiece pose relative to the left and right handpieces based on the handpiece pose, camera intrinsic parameters, two-dimensional coordinates of each light spot center, and three-dimensional coordinates of the light spots in the three-dimensional models of the left and right handpieces, respectively.

[0096] Step S33: Set the centers of the two categories of light spots with the smallest average reprojection errors corresponding to the left and right handles as the centers of the light spots corresponding to the left and right handles respectively.

[0097] Step S34: Set the handle pose corresponding to the center of the light spot corresponding to the left and right handles as the positioning information of the left and right handles respectively.

[0098] In this embodiment, it should be noted that the left and right controllers of the VR device include multiple LED light spots, and the coordinates of each LED light spot on the corresponding controller model are fixed, i.e., the three-dimensional coordinates of a 3D point. The coordinates of the center of each type of light spot are two-dimensional coordinates on the infrared image, i.e., the coordinates of a 2D point. Since the correspondence between each 2D point (i.e., the light spot center obtained in the above steps) and the 3D point can be obtained, the PNP method can be used to solve the controller pose, thereby locating the controller and obtaining the controller pose predicted by the light spot centers of each type. Then, based on the corresponding controller poses of different types, the average reprojection error relative to the left and right controllers is calculated to select the two types of light spot centers and controller poses closest to the left and right controllers.

[0099] In one feasible embodiment, the number of consecutive light spots on the left and right handles are N, respectively. In this embodiment of the invention, N can be 14, and they are divided into two staggered rings, with 7 spots on each ring, numbered P1, P2, ..., P... N That is, the 3D points mentioned above.

[0100] As a feasible implementation, for example, if there are three categories of light spot centers with average reprojection errors of 1.5, 1.1, and 1.3 relative to the left handle, and 1.05, 1.6, and 1.4 relative to the right handle, then the second category of light spot centers can be selected as the light spot center for the left handle, and the first category of light spot centers can be selected as the light spot center for the right handle. Alternatively, if the minimum average reprojection error exceeds 1.2, it is not selected as the average reprojection error for the left or right handle; instead, the light spot clustering is re-performed, for example, by re-dividing the light spot centers of each category for the next frame of the infrared image.

[0101] As an example, steps S31 to S34 include: solving the handheld pose using the PNP method based on the correspondence between the light point centers of each category and the light points on the left and right handles, the two-dimensional coordinates of each light point center, and the three-dimensional coordinates of each light point on the left and right handle models, to obtain the handheld poses corresponding to the light point centers of each category; calculating the average reprojection error of each handheld pose relative to the left and right handles based on the handheld poses corresponding to the light point centers of each category, camera intrinsic parameters, the two-dimensional coordinates of each light point center, and the three-dimensional coordinates of the light points in the three-dimensional models of the left and right handles, to obtain the average reprojection error of each handheld pose relative to the left and right handles; sorting the average reprojection errors of the handheld poses corresponding to the left and right handles in ascending order, thereby determining the handheld poses with the smallest average reprojection errors corresponding to the left and right handles, and the two categories of light points corresponding to these two handheld poses, to complete the distinction between the light point centers of the left and right handles. Figure 4 "HANDLE SUCCESS" indicates that the VR device has completed the differentiation of each light spot center and the left and right controllers. The light spot centers corresponding to the left controller include 5, 6, 7, 8, and 9, while the light spot centers corresponding to the right controller include 4, 5, 6, 7, and 8. The controller pose with the smallest average reprojection error relative to the left and right controllers is further set as the positioning information of the left and right controllers to complete the positioning of the controllers.

[0102] It should also be noted that, ideally, the resulting light spot centers fall into two categories: one for the light spot centroid corresponding to the left handle and the other for the light spot center corresponding to the right handle. One category of light spot centers is randomly selected, and the average reprojection error of the left and right handles is compared. The handle with the smaller average reprojection error is chosen as the handle corresponding to that category of light spot centers, and the other category of light spot centers corresponds to the other handle. For example, if the average reprojection error of the left handle is smaller, then the light spot of this category corresponds to the left handle, and the calculated pose is the left handle pose; otherwise, it corresponds to the right handle. The average reprojection error is the average of the reprojection errors corresponding to multiple points.

[0103] Specifically, the formula for calculating the reprojection error is as follows:

[0104] e pro =K*T*P i -p i

[0105] Among them, e pro For reprojection error, K is the camera intrinsic parameter, T is the predicted or PNP-obtained handle pose, and P is the reprojection error. i p represents the three-dimensional coordinates of each light point on the handle model. i The coordinates of the center of the light spot are two-dimensional.

[0106] Prior to the step of predicting the handle pose corresponding to the center of each type of light spot based on the correspondence and coordinates between the center of each type of light spot and the light spots on the left and right handles, the method further includes:

[0107] Step A10: Sequentially determine whether the number of light spot centers of each category is less than a first preset threshold;

[0108] Step A20: If the number of light spots is less than the first preset threshold, then the center of the light spot of the category is determined to be an interfering light spot.

[0109] Step A30: If it is not less than the first preset threshold, then perform the following step: predict the handle pose corresponding to the center of the light spot of the category and the coordinates of each light spot on the left and right handles.

[0110] In this application embodiment, a method for filtering interfering light spots is disclosed. Specifically, interfering light spots are filtered by the number of light spot centers of each category. Since the number of light spots on the left and right handles is fixed, it is ensured that each side of the handle has at least a certain number of light spots. For example, in this application embodiment, the left and right handles may each include 14 light spots, and the first preset threshold may be set to 4 to exclude the influence of interfering light spots in the environment and only perform the next step on the category of non-interfering light spots.

[0111] As an example, steps A10 to A30 include: determining whether the number of light spot centers contained in each category is less than a first preset threshold; if the number of light spot centers contained in a certain category is less than the first preset threshold, then the light spot centers of that category are determined to be interference light spots, and the light spot centers of that category are discarded without any processing; if the number of light spot centers contained in a certain category is not less than the first preset threshold, then the following steps are performed: predicting the handle pose corresponding to the light spot centers of the category based on the correspondence and coordinates between the light spot centers of the category and the light spots on the left and right handles; and repeating steps A10-A20 or A30 until the light spot centers of all categories have been determined.

[0112] Furthermore, the step of predicting the positioning information of the left and right handles in the current frame based on the positioning information of the left and right handles in the previous two frames includes:

[0113] Step S41: Predict the current handheld poses corresponding to the left and right handhelds based on the handheld poses in the positioning information of the first two frames.

[0114] Step S42: Determine the two-dimensional coordinates of the light spots on the left and right handles based on the current handle pose and the two-dimensional coordinates of the light spot centers of the left and right handles in the current frame infrared image.

[0115] Step S43: Match the two-dimensional coordinates of the light spots on the left and right handles with the two-dimensional coordinates of the centers of the light spots on the left and right handles in the current frame infrared image to obtain the number of successful matches.

[0116] Step S44: When the number of successful matches is not less than the second preset threshold, predict the updated handle pose corresponding to the left and right handles according to the correspondence and coordinates between each light spot on the left and right handles and the center of the light spot in the current frame infrared image.

[0117] Step S45: If the average reprojection error of the updated handle pose is less than a third preset threshold, then the updated handle pose is set as the positioning information of the left and right handles in the current frame.

[0118] In this embodiment, the pose prediction method is mainly performed after the poses of two frames have been stably obtained through the differentiation and localization method in steps S10 to S30. This method primarily uses a motion model to predict the handheld pose in the current frame. This eliminates the need for spot clustering and left / right handheld spot segmentation in steps S10 to S30, and allows for stable prediction of the left and right handhelds. In this case, even if the handhelds overlap in the infrared image or are affected by interfering point light sources when the user uses the handhelds, it will not affect the subsequent prediction of the handheld pose, thus obtaining stable left and right handheld positioning information and improving the environmental adaptability of the infrared-based left and right handheld differentiation and localization method.

[0119] Specifically, the current handheld poses corresponding to the left and right handhelds can be predicted based on the handheld poses in the positioning information of the first two frames using the following pose prediction formula:

[0120]

[0121] Among them, T Q Let T be the controller pose in frame Q, i.e., the controller pose in the current frame, and T be the controller pose in frame Q. Q-2 and T Q-1 This is the controller pose of the two frames preceding the Q-th frame.

[0122] Additionally, it should be noted that after predicting the controller pose, it is necessary to further determine whether the prediction was successful. This is mainly determined by the number of successful matches and the average reprojection error. Moreover, the accuracy of the controller pose predicted by the motion model is not high enough. The two-dimensional coordinates corresponding to each 2D point can be further determined using the controller pose, and PNP can be solved again to predict a more accurate updated controller pose. If the average reprojection error of the updated controller pose meets the requirements, the updated controller pose can be set as the positioning information of the left and right controllers in the current frame.

[0123] As an example, steps S41 to S45 include: inputting the handle pose from the positioning information of the first two frames into the pose prediction formula of the motion model to obtain the predicted current handle pose; determining the predicted LED spot position based on the predicted pose, wherein the LED spot position is the two-dimensional coordinate of the light spots of the left and right handles, which can be calculated using the following formula: p pre =K*T*P i , where p pre Let K be the two-dimensional coordinates of the light points on the left and right hand controllers, T be the camera intrinsic parameters, and P be the current hand controller pose. i The three-dimensional coordinates of each light point on the handle model are given. After determining the two-dimensional coordinates of the light points on the left and right handles, the nearest match is performed with the center of each light point in the actual acquired infrared image to obtain the number of regular matches that can be successfully matched, i.e., matching pairs. Specifically, if there is a point with a matching distance of less than 3 pixels in the nearest match, the match is considered successful. If the number of successful matches is greater than a second preset threshold, the match is considered successful, where the second preset threshold is 4. After the match is successful, PNP solution is performed according to the correspondence and coordinates between each light point on the left and right handles and the center of the light point in the current frame infrared image to predict the updated handle poses corresponding to the left and right handles respectively. Further, the average reprojection error corresponding to the updated handle poses is calculated. The method for calculating the average reprojection error is similar to step S32 and will not be described in detail here. It is determined whether the average reprojection error of the updated handle pose is less than a third preset threshold. If it is, the handle pose prediction of the current frame is determined to be successful, and the updated handle poses corresponding to the left and right handles are updated, where the third threshold is 1.

[0124] It should also be noted that, after the step of matching the two-dimensional coordinates of the light spots corresponding to the left and right handles with the two-dimensional coordinates of the centers of the light spots of the left and right handles in the current frame infrared image to obtain the number of successful matches, the method further includes:

[0125] Step S46: When the number of successful matches is less than the second preset threshold, the prediction is determined to be unsuccessful, and the process returns to the execution step: perform light spot clustering on each of the light spot centers to divide each of the light spot centers into different categories;

[0126] After predicting the updated handle poses corresponding to the left and right handles based on the correspondence and coordinates between the light spots on the left and right handles and the centers of the light spots in the current frame infrared image, the method further includes:

[0127] Step S47: If the average reprojection error of the updated handle pose is not less than the third preset threshold, the prediction is determined to be unsuccessful, and the execution steps are returned: perform light spot clustering on each of the light spot centers to divide each of the light spot centers into different categories.

[0128] In this embodiment of the application, it should be noted that this embodiment provides a method for judging prediction failure and processing after prediction failure. It mainly judges whether prediction has failed by the number of successful matches and the average reprojection error. After prediction failure, it returns to step S20 to re-distinguish and locate the left and right handles through light spot clustering, thereby obtaining stable output left and right handle positioning information.

[0129] Reference Figure 5 The overall concept of this invention is as follows: First, infrared images of the left and right handles are acquired. Then, LED light spots in the images are extracted. By determining whether there are stable predicted poses in the previous two frames, it is determined whether the predicted pose is usable. If not, the light spot clustering method is used to distinguish the left and right handles. Combined with two-hand PNP settlement, the left and right handles are located, and the positioning is successful. If yes, the pose is directly predicted based on the predicted poses of the previous two frames and the coordinates of the LED light spots in the image, and the positioning is successful.

[0130] This application provides a method for distinguishing and locating left and right hand controllers. First, infrared images of the left and right hand controllers are acquired. The infrared images are then processed to obtain the center of each light spot. Next, the light spot centers are clustered to divide them into different categories. Then, based on the correspondence and coordinates between each light spot on the left and right hand controllers and the center of each category, the light spot center and positioning information corresponding to the left and right hand controllers are determined. After obtaining the positioning information of the left and right hand controllers in the previous two frames, the positioning information of the left and right hand controllers in the current frame is predicted based on the positioning information of the left and right hand controllers in the previous two frames. The technical solution of this application extracts the center of each light spot and then divides the center of each light spot into different categories using a clustering method. Then, it calculates the corresponding relationship between each light spot on the left and right handles and the center of each category of light spot to obtain the light spot center and corresponding positioning information that is more consistent with the left and right handles. The technical solution of this application only requires a single frame image to complete the division of the light spot center of the left and right handles and the prediction of the handle pose to obtain positioning information. It has few dependencies, requires no input parameters, has a simple algorithm, and strong anti-interference ability. After completing the pose prediction of the first two frames using the above method, the prediction of subsequent frames can be directly performed based on the positioning information of the first two frames. After obtaining the positioning information of the left and right handles in the first two frames, it is not necessary to cluster and distinguish the center of each light spot in the infrared image. Therefore, it can continue to distinguish and locate the left and right handles even when encountering interference from point-like strong light sources or interference from partial overlap of handles, thus improving the environmental adaptability of the infrared light-based left and right handle differentiation positioning method.

[0131] Example 2

[0132] This application embodiment also provides a left and right handle differentiation and positioning device, which is applied to a left and right handle differentiation and positioning device, as described above.Figure 6 The left and right handle differentiation and positioning device includes:

[0133] The light spot extraction module 101 is used to acquire infrared images of the left and right handles, perform image processing on the infrared images, and obtain the center of each light spot.

[0134] The light spot clustering module 102 is used to perform light spot clustering on each of the light spot centers, so as to divide each of the light spot centers into different categories;

[0135] The differentiation and positioning module 103 is used to determine the light spot center and positioning information corresponding to the left and right handles respectively, based on the correspondence and coordinates between each light spot on the left and right handles and the center of each type of light spot;

[0136] The prediction and positioning module 104 is used to predict the positioning information of the left and right handles in the current frame based on the positioning information of the left and right handles in the previous two frames after obtaining the positioning information of the left and right handles in the previous two frames.

[0137] Optionally, the light spot extraction module 101 is further configured to:

[0138] Contour extraction is performed on each light spot in the infrared image to obtain the contour corresponding to each light spot;

[0139] The contours of each light point are fitted to obtain the shape of each target contour.

[0140] Extract the geometric center of each of the contour shapes to obtain the center of each of the light spots.

[0141] Optionally, the light spot clustering module 102 is further configured to:

[0142] Select the first cluster center from the centers of each light spot, and determine the cluster interval based on the ratio of the light spacing to the hand length on the handle model and the camera focal length;

[0143] The center of the light spot that is farthest from the first cluster center and at a distance greater than the cluster interval is set as another cluster center;

[0144] Repeat the following steps: If the distance between the center of the light spot and the farthest cluster center is greater than the clustering interval, then set the center of the light spot as the cluster center, until the distance between all light spot centers and the nearest cluster center is less than the clustering interval;

[0145] Each of the light spot centers is assigned to the category of the nearest cluster center.

[0146] Optionally, the light spot clustering module 102 is further configured to:

[0147] The center of the light spot in the upper left corner of the infrared image is selected as the first cluster center;

[0148] Calculate the ratio of the lamp spacing to the hand length on the handle model, and multiply the ratio by the camera focal length to obtain the clustering interval.

[0149] Optionally, the differentiation and positioning module 103 is further configured to:

[0150] Based on the correspondence and coordinates between the center of each type of light spot and the light spots on the left and right handles, predict the handle pose corresponding to the center of each type of light spot respectively;

[0151] Based on the handle pose, camera intrinsic parameters, two-dimensional coordinates of each light spot center, and three-dimensional coordinates of the light spots in the three-dimensional models of the left and right handles, the average reprojection error of each handle pose relative to the left and right handles is calculated.

[0152] Set the centers of the two categories of light spots with the smallest average reprojection errors corresponding to the left and right handles as the centers of the light spots corresponding to the left and right handles, respectively.

[0153] The handle poses corresponding to the centers of the light spots on the left and right handles are respectively set as the positioning information of the left and right handles.

[0154] Optionally, the differentiation and positioning module 103 is further configured to:

[0155] Sequentially determine whether the number of light spot centers of each category is less than a first preset threshold;

[0156] If the number is less than the first preset threshold, then the center of the light spot of the category is determined to be an interference light spot;

[0157] If it is not less than the first preset threshold, then the following step is performed: predict the handle pose corresponding to the center of the light spot of the category and the coordinates of each light spot on the left and right handles.

[0158] Optionally, the prediction and positioning module 104 is further configured to:

[0159] Predict the current hand poses of the left and right hand controllers based on the hand controller poses in the positioning information of the first two frames.

[0160] The two-dimensional coordinates of the light spots on the left and right handles are determined based on the current handle pose and the two-dimensional coordinates of the light spot centers of the left and right handles in the current frame infrared image.

[0161] The two-dimensional coordinates of the light spots on the left and right handles are matched with the two-dimensional coordinates of the centers of the light spots on the left and right handles in the current frame of the infrared image to obtain the number of successful matches.

[0162] When the number of successful matches is not less than the second preset threshold, the updated handle poses of the left and right handles are predicted according to the correspondence and coordinates between each light spot on the left and right handles and the center of the light spot in the current frame infrared image.

[0163] If the average reprojection error of the updated handle pose is less than a third preset threshold, then the updated handle pose is set to the positioning information of the left and right handles in the current frame.

[0164] Optionally, the prediction and positioning module 104 is further configured to:

[0165] When the number of successful matches is less than the second preset threshold, the prediction is determined to be unsuccessful, and the execution step is returned: perform light spot clustering on each of the light spot centers to divide each of the light spot centers into different categories.

[0166] Optionally, the prediction and positioning module 104 is further configured to:

[0167] If the average reprojection error of the updated handle pose is not less than the third preset threshold, the prediction is determined to be unsuccessful, and the execution step is returned: perform light spot clustering on each of the light spot centers to divide each of the light spot centers into different categories.

[0168] The left and right handle differentiation and positioning device provided in this application adopts the left and right handle differentiation and positioning method in the above embodiments, solving the technical problem of poor environmental adaptability of the infrared light-based left and right handle differentiation and positioning method. Compared with the prior art, the beneficial effects of the left and right handle differentiation and positioning device provided in this application are the same as the beneficial effects of the left and right handle differentiation and positioning method provided in the above embodiments, and other technical features in this left and right handle differentiation and positioning device are the same as the features disclosed in the previous embodiment method, and will not be repeated here.

[0169] Example 3

[0170] This application provides an electronic device, which includes: at least one processor; and a memory communicatively linked to the at least one processor; wherein the memory stores instructions executable by the at least one processor, which are executed by the at least one processor to enable the at least one processor to perform the left and right handle differentiation positioning method in the first embodiment described above.

[0171] The following is for reference. Figure 7The diagram illustrates a structural schematic of an electronic device suitable for implementing embodiments of the present disclosure. The electronic devices in the embodiments of the present disclosure may include, but are not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable media players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 7 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0172] like Figure 7 As shown, the electronic device may include a processing unit 1001 (e.g., a central processing unit, a graphics processor, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory 1002 (ROM) or a program loaded from a storage device 1003 into a random access memory 1004 (RAM). The RAM 1004 also stores various programs and data required for the operation of the electronic device. The processing unit 1004, ROM 1002, and RAM 1004 are interconnected via a bus 1005. An input / output (I / O) interface 1006 is also linked to the bus 1005.

[0173] Typically, the following systems can be linked to I / O interface 1006: input devices 1007 including, for example, touchscreens, touchpads, keyboards, mice, image sensors, microphones, accelerometers, gyroscopes, etc.; output devices 1008 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 1003 including, for example, magnetic tapes, hard disks, etc.; and communication devices 1009. Communication device 1009 allows the electronic device to communicate wirelessly or wiredly with other devices to exchange data. Although electronic devices with various systems are shown in the figures, it should be understood that it is not required to implement or possess all the systems shown. More or fewer systems may be implemented alternatively.

[0174] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from a storage device, or installed from a ROM. When the computer program is executed by a processing device, it performs the functions defined above in the methods of embodiments of this disclosure.

[0175] The electronic device provided in this application employs the left and right handle differentiation and positioning method in the above embodiments, solving the technical problem of poor environmental adaptability of the infrared light-based left and right handle differentiation and positioning method. Compared with the prior art, the beneficial effects of the electronic device provided in this application are the same as those of the left and right handle differentiation and positioning method provided in Embodiment 1 above, and other technical features of this electronic device are the same as those disclosed in the previous embodiment method, and will not be repeated here.

[0176] It should be understood that various parts of this disclosure can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics may be combined in any suitable manner in one or more embodiments or examples.

[0177] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0178] Example 4

[0179] This embodiment provides a computer-readable storage medium having computer-readable program instructions stored thereon, which are used to execute the left and right handle differentiation and positioning method in the first embodiment described above.

[0180] The computer-readable storage medium provided in this application embodiment may be, for example, a USB flash drive, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical link having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM, or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, system, or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination thereof.

[0181] The aforementioned computer-readable storage medium may be included in an electronic device or may exist independently without being assembled into an electronic device.

[0182] The aforementioned computer-readable storage medium carries one or more programs. When the aforementioned one or more programs are executed by an electronic device, the electronic device acquires infrared images of the left and right handles, performs image processing on the infrared images to obtain the centers of each light spot; performs light spot clustering on each light spot center to divide each light spot center into different categories; determines the light spot centers and positioning information corresponding to the left and right handles respectively based on the correspondence and coordinates between each light spot on the left and right handles and the light spot centers of each category; and after obtaining the positioning information of the left and right handles in the previous two frames, predicts the positioning information of the left and right handles in the current frame based on the positioning information of the left and right handles in the previous two frames.

[0183] Computer program code for performing the operations of this disclosure can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be linked to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be linked to an external computer (e.g., via the Internet using an Internet service provider).

[0184] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0185] The modules described in the embodiments of this disclosure can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.

[0186] The computer-readable storage medium provided in this application stores computer-readable program instructions for executing the above-described left and right handle differentiation and positioning method, thus solving the technical problem of poor environmental adaptability of the infrared light-based left and right handle differentiation and positioning method. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as the beneficial effects of the left and right handle differentiation and positioning method provided in the above-described embodiments, and will not be repeated here.

[0187] Example 5

[0188] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the left and right handle differentiation and positioning method described above.

[0189] The computer program product provided in this application solves the technical problem of poor environmental adaptability of the infrared light-based left and right handle positioning method. Compared with the prior art, the beneficial effects of the computer program product provided in this application are the same as the beneficial effects of the left and right handle positioning method provided in the above embodiments, and will not be repeated here.

[0190] The above are merely preferred embodiments of this application and do not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent scope of this application.

Claims

1. A left and right handle discrimination positioning method, characterized by, The left and right handle distinguishing positioning method comprises: acquiring infrared images of the left and right handles, performing image processing on the infrared images, and obtaining light point centers; performing light point clustering on the light point centers to divide the light point centers into different categories; determining light point centers and positioning information corresponding to the left and right handles according to the correspondence and coordinates of the light points on the left and right handles and the light point centers of the categories; after obtaining the positioning information of the left and right handles in the previous two frames, predicting the positioning information of the left and right handles in the current frame according to the positioning information of the left and right handles in the previous two frames; wherein the step of determining the light point centers and the positioning information corresponding to the left and right handles according to the correspondence and coordinates of the light points on the left and right handles and the light point centers of the categories comprises: predicting the handle poses corresponding to the light point centers of each category according to the correspondence and coordinates of the light point centers of each category and the light points on the left and right handles; calculating the average re-projection error of each handle pose relative to the left and right handles according to the handle poses corresponding to the light point centers of each category, the camera intrinsic parameters, the two-dimensional coordinates of the light point centers, and the three-dimensional coordinates of the light points in the three-dimensional model of the left and right handles; setting the light point centers of the two categories with the minimum average re-projection error of the left and right handles as the light point centers corresponding to the left and right handles respectively; setting the handle poses corresponding to the light point centers corresponding to the left and right handles as the positioning information of the left and right handles respectively.

2. The left and right handle discrimination positioning method according to claim 1, wherein The step of performing image processing on the infrared images to obtain the light point centers comprises: performing contour extraction on each light point in the infrared images to obtain the contours corresponding to each light point; fitting the contours of each light point respectively to obtain the shapes of each target contour; extracting the geometric centers of each contour shape to obtain the light point centers.

3. The left and right handle discrimination positioning method according to claim 1, wherein The step of performing light point clustering on the light point centers to divide the light point centers into different categories comprises: selecting a first clustering center from the light point centers, and determining a clustering interval according to the ratio of the lamp spacing and the hand length on the handle model and the focal length of the camera; setting the light point center farthest from the first clustering center and having a distance greater than the clustering interval as another clustering center; repeating the following steps: if the distance between a light point center and the farthest clustering center is greater than the clustering interval, setting the light point center as a clustering center, until the distance between all light point centers and the nearest clustering center is less than the clustering interval; dividing each light point center into the category to which the nearest clustering center belongs.

4. The left and right handle discrimination positioning method according to claim 3, wherein The step of selecting a first clustering center from the light point centers and determining a clustering interval according to the ratio of the lamp spacing and the hand length on the handle model and the focal length of the camera comprises: selecting the light point center at the upper left corner in the infrared image as the first clustering center; calculating the ratio of the lamp spacing and the hand length on the handle model, multiplying the ratio by the focal length of the camera to obtain the clustering interval.

5. The left and right handle discrimination positioning method of claim 1, wherein Before the step of predicting the handle pose corresponding to the light point center of each category according to the correspondence and coordinates of the light point center of each category and each light point on the left and right handles, the method further comprises: sequentially judging whether the number of light point centers of each category is less than a first preset threshold value; if less than the first preset threshold value, determining that the light point center of the category is an interference light point; if not less than the first preset threshold value, performing the step of predicting the handle pose corresponding to the light point center of the category according to the correspondence and coordinates of the light point center of the category and each light point on the left and right handles.

6. The left and right handle discrimination positioning method of claim 1, wherein The step of predicting the positioning information of the left and right handles of the current frame according to the positioning information of the left and right handles of the previous two frames comprises: predicting the current handle pose corresponding to the left and right handles respectively according to the handle pose in the positioning information of the previous two frames; determining the two-dimensional coordinates of the light points of the left and right handles according to the current handle pose and the two-dimensional coordinates of the light point centers of the left and right handles in the infrared image of the current frame; matching the two-dimensional coordinates of the light points of the left and right handles with the two-dimensional coordinates of the light point centers of the left and right handles in the infrared image of the current frame to obtain the number of successful matches; when the number of successful matches is not less than a second preset threshold value, predicting the updated handle pose corresponding to the left and right handles respectively according to the correspondence and coordinates of each light point on the left and right handles and the light point centers in the infrared image of the current frame; if the average re-projection error of the updated handle pose is less than a third preset threshold value, setting the updated handle pose as the positioning information of the left and right handles of the current frame.

7. The left and right handle discrimination positioning method of claim 6, wherein After the step of matching the two-dimensional coordinates of the light points of the left and right handles with the two-dimensional coordinates of the light point centers of the left and right handles in the infrared image of the current frame to obtain the number of successful matches, the method further comprises: when the number of successful matches is less than the second preset threshold value, determining that the prediction fails, and returning to perform the step of performing light point clustering on each light point center to divide each light point center into different categories; After predicting the updated handle pose corresponding to the left and right handles respectively according to the correspondence and coordinates of each light point on the left and right handles and the light point centers in the infrared image of the current frame, the method further comprises: if the average re-projection error of the updated handle pose is not less than the third preset threshold value, determining that the prediction fails, and returning to perform the step of performing light point clustering on each light point center to divide each light point center into different categories.

8. An electronic device, comprising: The electronic device comprises: at least one processor; and a memory communicatively linked with the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the steps of the left and right handle distinguishing positioning method in any one of claims 1 to 7.

9. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a program for implementing a left and right handle distinguishing positioning method, and the program is executed by a processor to implement the steps of the left and right handle distinguishing positioning method in any one of claims 1 to 7.

Images