Handle light spot matching method and electronic equipment

Abstract

The invention provides a handle light spot matching method and electronic equipment, which are used for ensuring that each light spot can be matched with a correct lamp bead identifier. Comprising the following steps: randomly selecting a specified number of target light spots from light spots obtained by performing light spot detection on a handle image; according to the image position coordinate of each light spot in the light spot matching pair obtained by the lamp bead identifier in each target light spot distribution handle and the actual position coordinate of each lamp bead corresponding to each lamp bead identifier in the handle coordinate system, the current actual angle of the light spot matching pair is obtained; determining an angle difference value by using the current actual angle and a predicted angle obtained by predicting the angle of the handle at the previous moment; the angle difference value is verified, and whether matching succeeds or not is determined; if yes, randomly selecting a specified number of target light spots from the light spots again until the light spots are successfully matched; and if not, the lamp bead identifiers are distributed to the target light spots again until the light spot matching pairs are matched successfully.

Description

[0001] This application claims the priority of the Chinese patent application filed with the Intellectual Property Office of the People's Republic of China on September 26, 2024, with application number 202411355226.3 and invention name "A method for matching handle light spots and electronic equipment", all contents of which are incorporated by reference in this application. Background Art

[0003] At present, head-mounted devices such as virtual reality and mixed reality are considered to be the hardware entrance to the metaverse industry. The handle has irreplaceable advantages in terms of accuracy, latency, cost, and physical feedback. Therefore, virtual reality interaction through the handle has become the main interaction method adopted by virtual reality head-mounted devices.

[0004] In order to enable the head-mounted device to locate and track the handle, the relevant technology mainly uses the head-mounted device to analyze the captured handle image through a preset visual algorithm to obtain the 3DOF (Degree of Freedom) displacement information of the handle. At the same time, the IMU (Inertial Measurement Unit) configured in the handle obtains the acceleration and angular velocity of the three axes of the handle, and then outputs the 3DOF rotation information of the handle, so that the head-mounted device can estimate the 6DOF posture of the handle. However, when using this solution, it is necessary to match the infrared lamp beads of the handle with the light spots in the image, and then determine the handle posture based on the matching results. Therefore, how to match the light spots with the infrared lamp beads has become a technical problem that needs to be solved urgently in the industry. Summary of the invention

[0005] The present application provides a method and electronic device for matching handle light spots, which are used to screen light spot matching pairs by estimating the current actual angle corresponding to the gravity direction of the light spot matching pair and performing chi-square verification with the current predicted angle predicted by using an extended Kalman filter to ensure that each light spot can be matched to the correct lamp bead identification, thereby further improving the accuracy of subsequent handle positioning.

[0006] In a first aspect, an embodiment of the present application provides a method for matching a handle light spot, the method comprising:

[0007] Performing light spot detection on the handle image acquired at the current moment to obtain the image position coordinates of each light spot in the handle image;

[0008] Randomly select a specified number of target light spots from the light spots, assign lamp bead identifiers in the handle to the target light spots, and obtain light spot matching pairs; wherein any one of the light spot matching pairs includes a specified number of target light spots and respective lamp bead identifiers of the target light spots;

[0009] According to the image position coordinates of each light spot in the light spot matching pair and the actual position coordinates of each lamp bead corresponding to each lamp bead identifier in the handle coordinate system, a current actual angle corresponding to the light spot matching pair is obtained, wherein the current actual angle is used to represent the current actual angle of the handle coordinate system relative to the world coordinate system;

[0010] Determine an angle difference value using the current actual angle and the current predicted angle; wherein the current predicted angle is predicted based on the predicted angle of the handle at the previous moment;

[0011] Determining whether the light spot matching pair is successfully matched according to the angle difference value;

[0012] If yes, after deleting the target light spot in the light spot matching pair from the light spots, a specified number of target light spots are randomly selected from the light spots again until each light spot has a lamp bead identifier that is successfully matched;

[0013] If not, a lamp bead identifier is reassigned to each target light spot until the light spot matching pair is successfully matched.

[0014] In the embodiment of the present application, the light spot matching pairs are screened by estimating the current actual angle corresponding to the gravity direction of the light spot matching pair and verifying the current predicted angle obtained based on the predicted angle of the handle at the previous moment to ensure that each light spot can match the correct lamp bead identification, thereby further improving the accuracy of subsequent handle positioning.

[0015] In one embodiment, obtaining the current actual angle corresponding to the light spot matching pair according to the image position coordinates of each light spot in the light spot matching pair and the actual position coordinates of each lamp bead corresponding to each lamp bead identifier in the handle coordinate system includes:

[0016] Determine the current gravity direction in the handle coordinate system according to the image position coordinates of each light spot in the light spot matching pair and the actual position coordinates of each lamp bead corresponding to each lamp bead identifier in the handle coordinate system;

[0017] Based on the current gravity direction, a current actual angle corresponding to the spot matching pair is obtained.

[0018] In the embodiment of the present application, the current gravity direction in the handle coordinate system is determined by the image position coordinates of each light spot in the light spot matching pair and the actual position coordinates of each lamp bead corresponding to each lamp bead identification in the handle coordinate system, and then based on the current gravity direction, the current actual angle corresponding to the light spot matching pair is obtained, thereby ensuring the accuracy of the current actual angle.

[0019] In one embodiment, determining the current gravity direction in the handle coordinate system according to the image position coordinates of each light spot in the light spot matching pair and the actual position coordinates of each lamp bead marker in the handle coordinate system includes:

[0020] Input the image position coordinates of each light spot in the light spot matching pair and the actual position coordinates of each lamp bead marker in the handle coordinate system into the P3P algorithm to obtain a first rotation matrix, wherein the first rotation matrix is ​​used to represent the rotation matrix of the handle coordinate system relative to the camera coordinate system at the current moment;

[0021] According to the first rotation matrix, the second rotation matrix and the matrix corresponding to the standard gravity direction, the current gravity direction in the handle coordinate system is obtained, wherein the camera coordinate system at the current moment obtained by the second rotation matrix is ​​equivalent to the rotation matrix of the world coordinate system, and the matrix corresponding to the standard gravity direction is used to represent the gravity direction in the world coordinate system.

[0022] In the embodiment of the present application, the first rotation matrix of the handle coordinate system relative to the camera coordinate system at the current moment is obtained through the image position coordinates of each light spot in the light spot matching pair and the actual position coordinates of each lamp bead marker in the handle coordinate system, and the current gravity direction is determined by the first rotation matrix and the camera coordinate system at the current moment, which is equivalent to the second rotation matrix of the world coordinate system, to ensure that the determined current gravity direction is the gravity direction corresponding to the current handle, thereby improving the accuracy of the determined current gravity direction.

[0023] In one embodiment, obtaining the current gravity direction in the handle coordinate system according to the first rotation matrix, the second rotation matrix, and a matrix corresponding to the standard gravity direction includes:

[0024] Multiplying the second rotation matrix by the first rotation matrix to obtain a third rotation matrix, wherein the third rotation matrix is ​​used to represent the rotation matrix of the handle coordinate system relative to the world coordinate system at the current moment;

[0025] The inverse matrix of the third rotation matrix is ​​multiplied by the matrix corresponding to the standard gravity direction to obtain the current gravity direction.

[0026] In the embodiment of the present application, the third rotation matrix of the handle coordinate system at the current moment relative to the world coordinate system is obtained by multiplying the second rotation matrix with the first rotation matrix, and the current gravity direction corresponding to the current state of the handle is obtained by multiplying the inverse matrix of the third rotation matrix with the matrix corresponding to the standard gravity direction, thereby ensuring the accuracy of the determined current gravity direction.

[0027] In one embodiment, the current actual angle includes a current actual roll angle and a current actual pitch angle; the current actual roll angle is used to represent the actual roll angle of the handle coordinate system relative to the world coordinate system at the current moment, and the current pitch angle is used to represent the actual pitch angle of the handle coordinate system relative to the world coordinate system at the current moment;

[0028] The current predicted angle includes a current predicted roll angle and a current predicted pitch angle; the current predicted roll angle is used to represent the predicted roll angle of the handle coordinate system relative to the world coordinate system at the current moment, and the current predicted pitch angle is used to represent the predicted pitch angle of the handle coordinate system relative to the world coordinate system at the current moment.

[0029] In the embodiment of the present application, the relationship between the two coordinate systems is represented by the roll angle and the pitch angle, so that the positional relationship between the two coordinate systems is more accurate.

[0030] In one embodiment, obtaining a current actual angle corresponding to the spot matching pair based on the current gravity direction includes:

[0031] The current actual rolling angle is obtained by the following formula:

[0032]

[0033] in, is the current actual rolling angle, is the horizontal position coordinate of the current gravity direction, is the vertical position coordinate of the current gravity direction, is the vertical position coordinate of the current gravity direction.

[0034] The current actual pitch angle is obtained by the following formula:

[0035]

[0036] in, is the current actual pitch angle.

[0037] In the embodiment of the present application, the current actual roll angle and the current actual pitch angle of the handle are determined by the current gravity direction of the handle to ensure the accuracy of the determined current actual pitch angle and the current actual roll angle.

[0038] In one embodiment, determining the angle difference value using the current actual angle and the current predicted angle includes:

[0039] Subtracting the current actual roll angle from the current predicted roll angle to obtain a first angle difference value; and,

[0040] Subtracting the current actual pitch angle from the current predicted pitch angle to obtain a second angle difference value;

[0041] The angle difference value is obtained according to the first angle difference value and the second angle difference value.

[0042] In the embodiment of the present application, the angle difference value is determined by the difference between the current actual roll angle and the current predicted roll angle, and the difference between the current actual pitch angle and the current predicted pitch angle, so that the determined angle difference value is more comprehensive and the accuracy of spot matching is further improved.

[0043] In one embodiment, the current prediction angle is obtained by:

[0044] Obtaining an intermediate predicted roll angle according to the predicted roll angle at the last moment and the preset roll angle noise; and,

[0045] Obtaining an intermediate predicted pitch angle according to the predicted pitch angle at the previous moment and the preset pitch angle noise;

[0046] The intermediate predicted roll angle is optimized using the handle speed at the current moment and the Kalman filter gain at the current moment to obtain the current predicted roll angle, and the intermediate predicted pitch angle is optimized using the handle speed at the current moment and the Kalman filter gain to obtain the current predicted pitch angle, wherein the Kalman filter gain at the current moment is obtained based on the intermediate predicted roll angle and the intermediate predicted pitch angle, and the handle speed at the current moment is predicted based on the handle speed at the previous moment using the extended Kalman filter EKF.

[0047] In the embodiment of the present application, an extended Kalman filter EKF is used to determine the intermediate predicted roll angle and the intermediate predicted pitch angle based on the predicted roll angle at the last moment and the predicted pitch angle at the last moment, and the intermediate predicted roll angle and the intermediate predicted pitch angle are optimized using the Kalman filter gain, so that the determined current predicted pitch angle and current predicted roll angle are more accurate, further improving the accuracy of spot matching.

[0048] In one embodiment, determining whether the spot matching pair is matched successfully according to the angle difference value includes:

[0049] The confidence of the spot matching pair is obtained by using the angle difference value of the spot matching pair and the covariance matrix of the handle state corresponding to the current moment, wherein the handle state corresponding to the current moment includes the current predicted angle and the handle speed at the current moment, and the covariance matrix of the handle state corresponding to the current moment is obtained based on the covariance matrix of the handle state corresponding to the previous moment;

[0050] If the confidence level is greater than a preset confidence level, it is determined that the spot matching pair is matched successfully;

[0051] If the confidence is not greater than the preset confidence value, it is determined that the spot matching pair is unsuccessful.

[0052] In the embodiment of the present application, the confidence of the spot matching pair is obtained by the angle difference value of the spot matching pair and the handle state covariance matrix corresponding to the current moment, and whether the spot matching pair is successfully matched is determined by comparing the confidence with the preset confidence threshold, thereby ensuring the accuracy of the determined spot matching pair.

[0053] In one embodiment, the using the angle difference value of the spot matching pair and the handle state covariance matrix corresponding to the current moment to obtain the confidence of the spot matching pair includes:

[0054] The confidence of the spot matching pair is obtained by:

[0055]

[0056] Where S is the confidence of the spot matching pair, Δ is the angle difference value of the spot matching pair, and P t|t,ΔΔ The rows and columns corresponding to the predicted angles in the covariance matrix of the handle state corresponding to the current moment.

[0057] In the embodiment of the present application, the confidence of the spot matching pair is determined by the angle difference value of the spot matching pair and the handle state covariance matrix corresponding to the current moment, that is, the angle difference values ​​are unified and then compared with the threshold, thereby making the subsequently determined spot matching result more accurate.

[0058] A second aspect of the present application provides an electronic device, including a processor and a memory, wherein the processor and the memory are connected via a bus;

[0059] The memory stores a computer program, and the processor is configured to perform the following operations based on the computer program:

[0060] Performing light spot detection on the handle image acquired at the current moment to obtain the image position coordinates of each light spot in the handle image;

[0061] Randomly select a specified number of target light spots from the light spots, assign lamp bead identifiers in the handle to the target light spots, and obtain light spot matching pairs; wherein any one of the light spot matching pairs includes a specified number of target light spots and respective lamp bead identifiers of the target light spots;

[0062] According to the image position coordinates of each light spot in the light spot matching pair and the actual position coordinates of each lamp bead corresponding to each lamp bead identifier in the handle coordinate system, a current actual angle corresponding to the light spot matching pair is obtained, wherein the current actual angle is used to represent the current actual angle of the handle coordinate system relative to the world coordinate system;

[0063] Determine an angle difference value using the current actual angle and the current predicted angle; wherein the current predicted angle is predicted based on the predicted angle of the handle at the previous moment;

[0064] Determining whether the light spot matching pair is successfully matched according to the angle difference value;

[0065] If yes, after deleting the target light spot in the light spot matching pair from the light spots, a specified number of target light spots are randomly selected from the light spots again until each light spot has a lamp bead identifier that is successfully matched;

[0066] If not, a lamp bead identifier is reassigned to each target light spot until the light spot matching pair is successfully matched.

[0067] According to a third aspect provided by an embodiment of the present invention, a computer storage medium is provided, wherein the computer storage medium stores a computer program, and the computer program is used to execute the method as described in the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0069] Figure 1 The flowchart of the method for matching the handle light spot provided in the embodiment of the present application is exemplarily shown;

[0070] Figure 2 A schematic diagram of a handle image provided by an embodiment of the present application is exemplarily shown;

[0071] Figure 3 The schematic diagram of the process for determining the current gravity direction provided by the embodiment of the present application is exemplarily shown;

[0072] Figure 4 A schematic diagram of a pitch angle and a roll angle provided in an embodiment of the present application is exemplarily shown;

[0073] Figure 5 The flowchart for determining the current prediction angle provided by the embodiment of the present application is exemplarily shown;

[0074] Figure 6 The flowchart of determining the handle speed at the current moment provided by the embodiment of the present application is exemplarily shown;

[0075] Figure 7 The schematic diagram of the process of determining the Kalman filter algorithm provided in the embodiment of the present application is exemplarily shown;

[0076] Figure 8 The schematic diagram of the process of card-square verification provided by the embodiment of the present application is exemplarily shown;

[0077] Fig. 9 The second flowchart of the method for matching the handle light spot provided in the embodiment of the present application is exemplarily shown;

[0078] Fig.10 A schematic diagram of a handle light spot matching device provided in an embodiment of the present application is exemplarily shown;

[0079] Fig.11 The structural diagram of the electronic device provided in the embodiment of the present application is exemplified. DETAILED DESCRIPTION

[0080] In order to make the purpose, implementation mode and advantages of the present application clearer, the exemplary implementation mode of the present application will be clearly and completely described below in conjunction with the drawings in the exemplary embodiments of the present application. Obviously, the described exemplary embodiments are only part of the embodiments of the present application, rather than all the embodiments.

[0081] Based on the exemplary embodiments described in this application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the claims attached to this application. In addition, although the disclosure in this application is introduced according to one or several exemplary examples, it should be understood that each aspect of the disclosure can also constitute a complete implementation method separately.

[0082] It should be noted that the brief description of terms in this application is only for the convenience of understanding the embodiments described below, and is not intended to limit the embodiments of this application. Unless otherwise specified, these terms should be understood according to their common and usual meanings.

[0083] The terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover but not exclude inclusion, for example, a product or device containing a series of components is not necessarily limited to those components clearly listed, but may include other components that are not clearly listed or inherent to these products or devices.

[0084] The term "module" as used in this application refers to any known or later developed hardware, software, firmware, artificial intelligence, fuzzy logic, or combination of hardware and / or software code that is capable of performing the functions associated with that element.

[0085] The idea of ​​the embodiments of the present application is summarized below.

[0086] At present, in order to enable the head-mounted device to locate and track the handle, the relevant technology mainly uses the head-mounted device to analyze the captured handle image through a preset visual algorithm to obtain the 3DOF (Degree of Freedom) displacement information of the handle. At the same time, the IMU (Inertial Measurement Unit) configured in the handle obtains the acceleration and angular velocity of the three axes of the handle, and then outputs the 3DOF rotation information of the handle, so that the head-mounted device estimates the 6DOF posture of the handle. However, when using this solution, it is necessary to match the infrared lamp beads of the handle with the light spots in the image, and then determine the handle posture based on the matching results. Therefore, how to match the light spots with the infrared lamp beads has become a technical problem that needs to be solved urgently in the industry.

[0087] Therefore, an embodiment of the present application provides a method for matching handle light spots, which screens light spot matching pairs by estimating the current actual angle corresponding to the gravity direction of the light spot matching pair and verifying the current predicted angle obtained based on the predicted angle of the handle at the previous moment, to ensure that each light spot can be matched to the correct lamp bead identification, thereby further improving the accuracy of subsequent handle positioning.

[0088] Next, the matching method of the handle light spot in the present application is described in conjunction with the accompanying drawings. Figure 1 As shown, it is a flow chart of a method for determining a matching method of a handle light spot, which may specifically include the following steps:

[0089] Step 101: performing light spot detection on the handle image acquired at the current moment to obtain the image position coordinates of each light spot in the handle image;

[0090] like Figure 2 As shown, it is a schematic diagram of the handle image. Figure 2 The white dot in the middle is the light spot in the handle.

[0091] In the embodiment of the present application, a light spot detection algorithm is used to detect the light spot in the handle image, but the embodiment of the present application does not limit the light spot detection algorithm. The light spot detection algorithm in the embodiment of the present application can be set according to the specific actual situation.

[0092] Step 102: randomly selecting a specified number of target light spots from the light spots, assigning lamp bead identifiers in the handle to the target light spots, and obtaining light spot matching pairs; wherein any one of the light spot matching pairs includes a specified number of target light spots and respective lamp bead identifiers of the target light spots;

[0093] The specified number in the embodiment of the present application is 3, but the embodiment of the present application does not limit the specific value of the specified number. The specific value of the specified number in the embodiment of the present application can be set according to the specific actual situation.

[0094] In the embodiment of the present application, the sets corresponding to the lamp bead identifiers in the handles assigned to the specified number of target light spots each time are all different. For example, take the three handle light spots, namely, spot a, spot b, and spot c, as an example for explanation. If the lamp bead identifiers include lamp bead 1, lamp bead 2, and lamp bead 3. Then the light spot matching pairs corresponding to the three light spots are: (spot a-lamp bead 1, spot b-lamp bead 2, spot c-lamp bead 3), (spot a-lamp bead 2, spot b-lamp bead 1, spot c-lamp bead 3), (spot a-lamp bead 2, spot b-lamp bead 3, spot c-lamp bead 1), (spot a-lamp bead 3, spot b-lamp bead 1, spot c-lamp bead 2), (spot a-lamp bead 3, spot b-lamp bead 2, spot c-lamp bead 1), (spot a-lamp bead 1, spot b-lamp bead 3, spot c-lamp bead 2).

[0095] It should be noted that the above description is only for illustration and does not limit the number of lamp beads in the handle. The number of lamp beads in the handle in the embodiment of the present application can be set according to specific actual conditions.

[0096] Step 103: according to the image position coordinates of each light spot in the light spot matching pair and the actual position coordinates of each lamp bead corresponding to each lamp bead identifier in the handle coordinate system, obtain the current actual angle corresponding to the light spot matching pair, wherein the current actual angle is used to represent the current actual angle of the handle coordinate system relative to the world coordinate system;

[0097] In one embodiment, step 103 can be specifically implemented as follows: determine the current gravity direction in the handle coordinate system according to the image position coordinates of each light spot in the light spot matching pair and the actual position coordinates of each lamp bead corresponding to each lamp bead identification in the handle coordinate system; based on the current gravity direction, obtain the current actual angle corresponding to the light spot matching pair.

[0098] Next, the method for determining the current gravity direction in the embodiment of the present application is introduced. Figure 3 As shown, it is a schematic diagram of the process of determining the current gravity direction, which may specifically include the following steps:

[0099] Step 301: input the image position coordinates of each light spot in the light spot matching pair and the actual position coordinates of each lamp bead marker in the handle coordinate system into the P3P algorithm to obtain a first rotation matrix, wherein the first rotation matrix is ​​used to represent the rotation matrix of the handle coordinate system relative to the camera coordinate system at the current moment;

[0100] Using the P3P algorithm to determine the first rotation matrix is ​​not an innovation of the present invention, and the embodiments of the present application will not be described in detail here.

[0101] Step 302: Obtain the current gravity direction in the handle coordinate system according to the first rotation matrix, the second rotation matrix and the matrix corresponding to the standard gravity direction, wherein the camera coordinate system at the current moment obtained by the second rotation matrix is ​​equivalent to the rotation matrix of the world coordinate system, and the matrix corresponding to the standard gravity direction is used to represent the gravity direction in the world coordinate system.

[0102] The current gravity direction in the handle coordinate system in the embodiment of the present application is represented by a matrix.

[0103] In one embodiment, step 302 can be specifically implemented as follows: multiplying the second rotation matrix with the first rotation matrix to obtain a third rotation matrix, wherein the third rotation matrix is ​​used to represent the rotation matrix of the handle coordinate system relative to the world coordinate system at the current moment; multiplying the inverse matrix of the third rotation matrix with the matrix corresponding to the standard gravity direction to obtain the current gravity direction. The current gravity direction can be obtained by formula (1):

[0104]

[0105] in, is the matrix corresponding to the current gravity direction, is the second rotation matrix, is the first rotation matrix, g w is the matrix corresponding to the standard gravity direction.

[0106] The matrix of the standard gravity direction in the embodiment of the present application is: Where g = 9.8. However, the specific value of g is not limited in the embodiment of the present application, and the specific value of g in the implementation of the present application can be set according to the specific actual situation.

[0107] The current actual angle in the embodiment of the present application includes the current actual roll angle and the current actual pitch angle; the current actual roll angle is used to represent the actual roll angle of the handle coordinate system relative to the world coordinate system at the current moment, and the current pitch angle is used to represent the actual pitch angle of the handle coordinate system relative to the world coordinate system at the current moment. Figure 4 As shown in the figure, it is a schematic diagram of the pitch angle and the roll angle. Figure 4 It can be seen that the pitch angle is the angle obtained by rotating the handle around the Y axis of the world coordinate system, that is, the angle between the X axis of the handle coordinate system and the X axis of the world coordinate system. The roll angle is the angle obtained by rotating the handle around the X axis of the world coordinate system, that is, the angle between the Y axis of the handle coordinate system and the Y axis of the world coordinate system. Among them, the current actual roll angle can be obtained by formula (2):

[0108]

[0109] in, is the current actual rolling angle, is the horizontal position coordinate of the current gravity direction, is the vertical position coordinate of the current gravity direction, is the vertical position coordinate of the current gravity direction.

[0110] The current actual pitch angle can be obtained by formula (3):

[0111]

[0112] in, is the current actual pitch angle.

[0113] Step 104: using the current actual angle and the current predicted angle, determining an angle difference value; wherein the current predicted angle is predicted based on the predicted angle of the handle at the previous moment;

[0114] In the embodiment of the present application, an extended Kalman filter (EKF) is used to predict the current predicted angle based on the predicted angle of the handle at the previous moment. However, the embodiment of the present application does not limit the method for determining the current predicted angle. In the embodiment of the present application, a pre-trained angle prediction model can also be used to predict the current predicted angle based on the predicted angle of the handle at the previous moment.

[0115] The current predicted angle in the embodiment of the present application includes a current predicted roll angle and a current predicted pitch angle; the current predicted roll angle is used to represent the predicted roll angle of the handle coordinate system relative to the world coordinate system at the current moment, and the current predicted pitch angle is used to represent the predicted pitch angle of the handle coordinate system relative to the world coordinate system at the current moment.

[0116] In one embodiment, step 104 can be specifically implemented as follows: subtracting the current actual roll angle from the current predicted roll angle to obtain a first angle difference value; and subtracting the current actual pitch angle from the current predicted pitch angle to obtain a second angle difference value; and obtaining the angle difference value according to the first angle difference value and the second angle difference value. The angle difference value can be obtained by formula (4):

[0117]

[0118] in, is the current actual rolling angle, α t is the current predicted roll angle, is the current actual pitch angle, β t is the current predicted pitch angle, and Δ is the angle difference value.

[0119] Next, the method of using the extended Kalman filter EKF in the embodiment of the present application to predict and determine the current predicted angle based on the predicted angle of the handle at the previous moment is described. Figure 5 As shown, it is a schematic diagram of a process for determining the current prediction angle, which may specifically include the following steps:

[0120] Step 501: obtaining an intermediate predicted roll angle according to the predicted roll angle at the last moment and a preset roll angle noise;

[0121] In one embodiment, step 501 can be specifically implemented as follows: adding the matrix corresponding to the predicted roll angle at the previous moment to the roll angle noise to obtain the intermediate predicted roll angle. The intermediate predicted roll angle can be obtained by formula (5):

[0122] α t ′=α t-1 +n α,t ……(5);

[0123] Among them, α t ′ is the intermediate predicted roll angle, α t-1 is the matrix corresponding to the predicted roll angle at the previous moment, n a,t is the matrix corresponding to the roll angle noise.

[0124] It should be noted that the preset roll angle noise in the embodiment of the present application may be the same or different at different times. Each moment has a corresponding roll angle noise, but the specific value of the roll angle noise corresponding to each moment can be set according to actual conditions. The embodiment of the present application does not limit the specific value of the roll angle noise corresponding to each moment.

[0125] Step 502: Obtain an intermediate predicted pitch angle according to the predicted pitch angle at the previous moment and the preset pitch angle noise; wherein the intermediate predicted pitch angle can be obtained by formula (6):

[0126] β t ′=β t-1 +n β,t ……(6);

[0127] Among them, β t ′ is the intermediate predicted pitch angle, β t-1 is the matrix corresponding to the predicted pitch angle at the previous moment, n β,t is the matrix corresponding to the pitch angle noise.

[0128] In the embodiment of this application, n α,t , The σ in the embodiment of the present application ω The value of is preset. τ is the value of the specified time length preset in the embodiment of the present application.

[0129] β t-1 For example, if the corresponding predicted pitch angle is β, the corresponding matrix can be [0, 0, β]. Similarly, the matrix corresponding to the predicted roll angle at the previous moment is also the same, but the form of the matrix corresponding to the angle is not limited in the embodiment of the present application. The specific form can be set according to the specific actual situation.

[0130] It should be noted that the preset pitch angle noise in the embodiment of the present application may be the same or different at different times. Each moment has a corresponding pitch angle noise, but the specific value of the pitch angle noise corresponding to each moment can be set according to actual conditions. The embodiment of the present application does not limit the specific value of the pitch angle noise corresponding to each moment.

[0131] It should be noted that the execution sequence of step 501 and step 502 in the embodiment of the present application is not limited in the embodiment of the present application. Step 501 may be executed first, and then step 502. Step 502 may also be executed first, and then step 501. Step 501 and step 502 may also be executed simultaneously.

[0132] Step 503: Optimize the intermediate predicted roll angle using the current handle speed and the current Kalman filter gain to obtain the current predicted roll angle; wherein the current predicted roll angle can be obtained by formula (7):

[0133]

[0134] Among them, α t is the current predicted roll angle, α t ′ is the intermediate predicted roll angle, K t is the Kalman filter gain at the current time t, is the speed of the handle in the world coordinate system at the current moment t, that is, the handle speed in the embodiment of the present application.

[0135] Step 504: Optimize the intermediate predicted pitch angle using the current handle speed and the Kalman filter gain to obtain the current predicted pitch angle. The current predicted pitch angle can be obtained by formula (8):

[0136]

[0137] Among them, β t is the current predicted pitch angle, β t ' is the intermediate predicted pitch angle.

[0138] It should be noted that step 503 and step 504 in the embodiment of the present application can be executed simultaneously, or step 503 can be executed first and then step 504; or step 504 can be executed first and then step 503. The execution sequence of step 503 and step 504 is not limited in the embodiment of the present application, and the execution sequence of step 503 and step 504 can be set according to the specific actual situation.

[0139] The Kalman filter gain at the current moment in the embodiment of the present application is obtained based on the intermediate predicted roll angle and the intermediate predicted pitch angle, and the handle speed at the current moment in the embodiment of the present application is predicted based on the handle speed at the previous moment using the extended Kalman filter EKF. The specific method of determining the Kalman filter gain and the handle speed at the current moment in the embodiment of the present application is introduced below:

[0140] like Figure 6 As shown in FIG. 1 , it is a schematic diagram of a process for determining the handle speed at the current moment, which may specifically include the following steps:

[0141] Step 601: Obtain the angular velocity of the handle at the last moment;

[0142] In the embodiment of the present application, the angular velocity of the handle can be directly measured by a gyroscope in the electronic device.

[0143] Step 602: Using the angular velocity of the handle at the previous moment and the fourth rotation matrix between the estimated world coordinate system and the handle coordinate system at the previous moment, obtain the fourth rotation matrix at the current moment; wherein the fourth rotation matrix at the current moment can be obtained by formula (9):

[0144]

[0145] in, is the fourth rotation matrix at the current time t, is the fourth rotation matrix corresponding to the previous time t-1, τ is the specified time length, is the angular velocity of the handle at the previous moment.

[0146] In the embodiment of the present application, there is an initial fourth rotation matrix, so that the fourth rotation matrix at the subsequent moment can be determined by the formula (9) described above. Among them, the initial fourth rotation matrix can be obtained by formula (10):

[0147]

[0148] in, is the fourth rotation matrix at the initial time t0, is the acceleration of the handle in the handle coordinate system at the initial moment, The modulus of the matrix corresponding to the acceleration of the handle in the handle coordinate system at the initial moment, g w is the preset standard gravity direction.

[0149] The acceleration of the handle in the handle coordinate system and the standard gravity direction in the embodiment of the present application can be represented by a matrix, and the embodiment of the present application does not limit its representation form.

[0150] Step 603: using the fourth rotation matrix at the current moment and the acceleration of the handle in the handle coordinate system at the current moment, obtaining the acceleration of the handle in the estimated world coordinate system at the current moment;

[0151] In one embodiment, step 603 may be specifically implemented as follows:

[0152] The fourth rotation matrix at the current moment is multiplied by the acceleration of the handle in the handle coordinate system at the current moment to obtain the acceleration of the handle in the estimated world coordinate system at the current moment. The acceleration of the handle in the estimated world coordinate system at the current moment can be obtained by formula (11):

[0153] a tu =R b u ,t ×a t b ……(11);

[0154] Among them, a t u is the acceleration of the handle in the estimated world coordinate system at the current time t, a t b It is the acceleration of the handle in the handle coordinate system at the current moment.

[0155] In the embodiment of the present application, the acceleration of the handle in the handle coordinate system at the current moment is measured by the handle imu.

[0156] Step 604: Obtain the handle speed at the current moment according to the acceleration of the handle at the current moment in the estimated world coordinate system, the handle speed at the previous moment, the angle rotation matrix corresponding to the predicted pitch angle at the previous moment, and the angle rotation matrix corresponding to the predicted roll angle at the previous moment. The handle speed at the current moment can be obtained by formula (12):

[0157]

[0158] in, is the handle speed at the current moment, is the handle speed at the last moment, is the angle rotation matrix corresponding to the predicted roll angle at the previous moment, is the angle rotation matrix corresponding to the predicted pitch angle at the previous moment, n A,t The preset acceleration noise corresponding to the current moment, τ is the specified duration.

[0159] In the embodiments of this application Among them, σ A The value of is preset. In the embodiments of the present application, the corresponding acceleration noises may be the same or different. The specific value may be set according to the actual situation. The embodiments of the present application do not limit the specific value of the acceleration noise.

[0160] In one embodiment, the angle rotation matrix corresponding to the predicted roll angle at the previous moment can be obtained by formula (13):

[0161]

[0162] Among them, α t-1 is the predicted roll angle at the previous moment.

[0163] In one embodiment, the angle rotation matrix corresponding to the predicted pitch angle at the previous moment can be obtained by formula (14):

[0164]

[0165] Among them, β t-1 is the predicted pitch angle at the previous moment.

[0166] like Figure 7 As shown, a flow chart of determining a Kalman filter algorithm may include the following steps:

[0167] Step 701: Obtain an intermediate rotation matrix according to the derivative of the angle rotation matrix corresponding to the predicted pitch angle at the previous moment and the derivative of the angle rotation matrix corresponding to the predicted roll angle at the previous moment; wherein the intermediate rotation matrix can be obtained by formula (15):

[0168]

[0169] Where F is the intermediate rotation matrix, I 3×3 is the preset identity matrix, is the derivative of the angle rotation matrix corresponding to the predicted roll angle at the previous moment, It is the derivative of the angle rotation matrix corresponding to the predicted pitch angle at the previous moment.

[0170] In the embodiments of this application

[0171] Next, the method for determining the derivative of the angle rotation matrix corresponding to the predicted pitch angle at the previous moment and the derivative of the angle rotation matrix corresponding to the predicted roll angle at the previous moment in the embodiment of the present application is introduced.

[0172] In one embodiment, the derivative of the angle rotation matrix corresponding to the predicted roll angle at the previous moment can be obtained by formula (16):

[0173]

[0174] In one embodiment, the derivative of the angle rotation matrix corresponding to the predicted pitch angle at the previous moment can be obtained by formula (17):

[0175]

[0176] Step 702: Using the intermediate rotation matrix and the Kalman filter covariance matrix of the handle state at the previous moment, obtain the intermediate Kalman filter covariance matrix of the handle state at the current moment; wherein the intermediate Kalman filter covariance matrix at the current moment can be obtained by formula (18):

[0177]

[0178] Among them, P t|t-1 is the intermediate Kalman filter covariance matrix corresponding to the current time t, P t-1|t-1 is the Kalman filter covariance matrix of the previous moment, is the variance of the preset acceleration noise, is the variance of the preset angular noise.

[0179] In the embodiment of the present application, the initial matrix of the Kalman filter covariance matrix of the handle state is a matrix whose parameters are all 0. However, the embodiment of the present application does not limit the initial matrix of the Kalman filter matrix of the handle state, and the initial matrix of the Kalman filter matrix of the handle state in the embodiment of the present application can be set according to the specific actual situation.

[0180] The handle state in the embodiment of the present application includes handle speed, current predicted roll angle, and current predicted pitch angle.

[0181] Step 703: Obtain the Kalman filter gain at the current moment according to the intermediate Kalman filter covariance matrix at the current moment. The Kalman filter gain at the current moment can be obtained by formula (19):

[0182]

[0183] Among them, K t is the Kalman filter gain at the current time t, P t|t-1,*v is the intermediate Kalman filter covariance matrix P corresponding to the current time t t|t-1 The speed column corresponds to the submatrix, P t|t-1,vv is the intermediate Kalman filter covariance matrix P corresponding to the current time t t|t-1 The parameter values ​​corresponding to the speed rows and speed columns are is the preset variance of the speed noise.

[0184] Step 105: Determine whether the spot matching pair is matched successfully according to the angle difference value, if so, execute step 106, if not, execute step 107;

[0185] In one embodiment, step 105 may be specifically implemented as: performing a chi-square verification on the angle difference value to determine whether the spot matching pair is successfully matched.

[0186] Next, the chi-square verification method of step 105 is introduced. Figure 8 The figure shows a schematic diagram of the process of chi-square verification, which may specifically include the following steps:

[0187] Step 801: Utilizing the angle difference value of the spot matching pair and the covariance matrix of the handle state corresponding to the current moment, the confidence of the spot matching pair is obtained, wherein the handle state corresponding to the current moment includes the current predicted angle and the handle speed at the current moment, and the covariance matrix of the handle state corresponding to the current moment is obtained based on the covariance matrix of the handle state corresponding to the previous moment;

[0188] In one embodiment, the confidence of the spot matching pair is obtained by formula (20):

[0189]

[0190] Where S is the confidence of the spot matching pair, Δ is the angle difference value of the spot matching pair, and P t|t,ΔΔ The rows and columns corresponding to the predicted angles in the covariance matrix of the handle state corresponding to the current moment.

[0191] The covariance matrix of the handle state at the current moment in the embodiment of the present application is obtained based on the intermediate Kalman filter covariance matrix of the handle state at the current moment described above. Specifically, the Kalman filter covariance matrix of the handle state at the current moment can be obtained by formula (21):

[0192] P t|t =P t|t-1 -K t ×P t|t-1, *v……(21);

[0193] Among them, P t|t is the Kalman filter covariance matrix of the handle state at the current moment.

[0194] Step 802: Determine whether the confidence level is greater than a preset confidence level. If yes, execute step 803; if no, execute step 804.

[0195] Step 803: Determine whether the light spot matching pair is matched successfully;

[0196] Step 804: Determine whether the spot matching pair is unsuccessful.

[0197] The confidence threshold in the embodiment of the present application is Where p is the quantile, which is usually 0.95. However, the embodiment of the present application does not limit the specific value of the confidence threshold. The specific value of the confidence threshold in the embodiment of the present application can be set according to the specific actual situation.

[0198] In one embodiment, step 106 can also be implemented as: comparing the angle difference value with a specified angle, if the angle difference value is greater than the specified angle, determining that the spot matching pair is unsuccessful; if the angle difference value is not greater than the specified angle, determining that the spot matching pair is successful.

[0199] The specified angle in the embodiment of the present application can be set according to the specific actual situation, and the specific value of the specified angle is not limited in the embodiment of the present application.

[0200] Step 106: after deleting the target light spot in the light spot matching pair from the light spots, determine whether each light spot has a lamp bead identifier that has been successfully matched. If not, return to step 102; if yes, end;

[0201] Step 107: After re-assigning lamp bead identifiers to the target light spots, return to step 103.

[0202] It should be noted that the light spot matching pairs obtained after re-assigning lamp bead identifiers to each target light spot are different from the light spot matching pairs obtained before for each target light spot.

[0203] In order to further understand the matching method of the handle light spot in the embodiment of the present application, Fig. 9 FIG. 1 is a flow chart of a method for matching a handle light spot, which may specifically include the following steps:

[0204] Step 901: performing light spot detection on the handle image acquired at the current moment to obtain the image position coordinates of each light spot in the handle image;

[0205] Step 902: randomly selecting a specified number of target light spots from the light spots, assigning lamp bead identifiers in the handle to the target light spots, and obtaining light spot matching pairs; wherein any one of the light spot matching pairs includes a specified number of target light spots and respective lamp bead identifiers of the target light spots;

[0206] Step 903: input the image position coordinates of each light spot in the light spot matching pair and the actual position coordinates of each lamp bead marker in the handle coordinate system into the P3P algorithm to obtain a first rotation matrix, wherein the first rotation matrix is ​​used to represent the rotation matrix of the handle coordinate system relative to the camera coordinate system at the current moment;

[0207] Step 904: Obtain the current gravity direction in the handle coordinate system according to the first rotation matrix, the second rotation matrix and the matrix corresponding to the standard gravity direction, wherein the camera coordinate system at the current moment obtained by the second rotation matrix is ​​equivalent to the rotation matrix of the world coordinate system, and the matrix corresponding to the standard gravity direction is used to represent the gravity direction in the coordinate system;

[0208] Step 905: Based on the current gravity direction, obtain a current actual angle corresponding to the spot matching pair, wherein the current actual angle is used to represent the current actual angle of the handle coordinate system relative to the world coordinate system;

[0209] Step 906: Determine an angle difference value using the current actual angle and the current predicted angle; wherein the current predicted angle is predicted based on the predicted angle of the handle at the previous moment;

[0210] Step 907: Utilizing the angle difference value of the spot matching pair and the covariance matrix of the handle state corresponding to the current moment, the confidence of the spot matching pair is obtained, wherein the handle state corresponding to the current moment includes the current predicted angle and the handle speed at the current moment, and the covariance matrix of the handle state corresponding to the current moment is obtained based on the covariance matrix of the handle state corresponding to the previous moment;

[0211] Step 908: Determine whether the light spot matching pair is matched successfully based on the confidence of the light spot matching pair, if so, execute step 909, if not, execute step 910;

[0212] Step 909: after deleting the target light spot in the light spot matching pair from the light spots, determine whether each light spot has a lamp bead identifier that has been successfully matched. If not, return to step 902; if yes, end;

[0213] Step 910: After re-assigning lamp bead identifiers to the target light spots, return to step 903.

[0214] Based on the same inventive concept, the handle light spot matching method disclosed above can also be implemented by a handle light spot matching device. The effect of the handle light spot matching device is similar to that of the aforementioned method, and will not be described in detail here.

[0215] Fig.10 It is a schematic diagram of the structure of a matching device for a handle light spot according to an embodiment of the present disclosure.

[0216] like Fig.10 As shown, the handle spot matching device 1000 of the present disclosure may include a spot detection module 1010, a spot matching module 1020, a current actual angle determination module 1030, an angle difference value determination module 1040, a verification module 1050, a judgment module 1060 and a re-matching module 1070.

[0217] The light spot detection module 1010 is used to perform light spot detection on the handle image acquired at the current moment to obtain the image position coordinates of each light spot in the handle image;

[0218] The light spot matching module 1020 is used to randomly select a specified number of target light spots from the light spots, assign a lamp bead identifier in the handle to each target light spot, and obtain a light spot matching pair; wherein any one of the light spot matching pairs includes a specified number of target light spots and the lamp bead identifier of each target light spot;

[0219] The current actual angle determination module 1030 is used to obtain the current actual angle corresponding to the light spot matching pair according to the image position coordinates of each light spot in the light spot matching pair and the actual position coordinates of each lamp bead corresponding to each lamp bead identifier in the handle coordinate system, wherein the current actual angle is used to represent the current actual angle of the handle coordinate system relative to the world coordinate system;

[0220] An angle difference value determination module 1040, used to determine an angle difference value using the current actual angle and the current predicted angle; wherein the current predicted angle is predicted based on the predicted angle of the handle at the previous moment;

[0221] A verification module 1050 is used to determine whether the light spot matching pair is successfully matched according to the angle difference value;

[0222] The judgment module 1060 is used for, if yes, deleting the target light spot in the light spot matching pair from the light spots, and randomly selecting a specified number of target light spots from the light spots again, until each light spot has a lamp bead identifier that is successfully matched;

[0223] The re-matching module 1070 is used to re-assign lamp bead identifiers to the target light spots if no, until the light spot matching pair is successfully matched.

[0224] In one embodiment, the current actual angle determination module 1030 is specifically configured to:

[0225] Determine the current gravity direction in the handle coordinate system according to the image position coordinates of each light spot in the light spot matching pair and the actual position coordinates of each lamp bead corresponding to each lamp bead identifier in the handle coordinate system;

[0226] Based on the current gravity direction, a current actual angle corresponding to the spot matching pair is obtained.

[0227] In one embodiment, the current actual angle determination module 1030 is further configured to:

[0228] Input the image position coordinates of each light spot in the light spot matching pair and the actual position coordinates of each lamp bead marker in the handle coordinate system into the P3P algorithm to obtain a first rotation matrix, wherein the first rotation matrix is ​​used to represent the rotation matrix of the handle coordinate system relative to the camera coordinate system at the current moment;

[0229] According to the first rotation matrix, the second rotation matrix and the matrix corresponding to the standard gravity direction, the current gravity direction in the handle coordinate system is obtained, wherein the camera coordinate system at the current moment obtained by the second rotation matrix is ​​equivalent to the rotation matrix of the world coordinate system, and the matrix corresponding to the standard gravity direction is used to represent the gravity direction in the coordinate system.

[0230] In one embodiment, the current actual angle determination module 1030 is further configured to:

[0231] Multiplying the second rotation matrix by the first rotation matrix to obtain a third rotation matrix, wherein the third rotation matrix is ​​used to represent the rotation matrix of the handle coordinate system relative to the world coordinate system at the current moment;

[0232] The inverse matrix of the third rotation matrix is ​​multiplied by the matrix corresponding to the standard gravity direction to obtain the current gravity direction.

[0233] In one embodiment, the current actual angle includes a current actual roll angle and a current actual pitch angle; the current actual roll angle is used to represent the actual roll angle of the handle coordinate system relative to the world coordinate system at the current moment, and the current pitch angle is used to represent the actual pitch angle of the handle coordinate system relative to the world coordinate system at the current moment;

[0234] The current predicted angle includes a current predicted roll angle and a current predicted pitch angle; the current predicted roll angle is used to represent the predicted roll angle of the handle coordinate system relative to the world coordinate system at the current moment, and the current predicted pitch angle is used to represent the predicted pitch angle of the handle coordinate system relative to the world coordinate system at the current moment.

[0235] In one embodiment, the current actual angle determination module 1030 is further configured to:

[0236] The current actual rolling angle is obtained by the following formula:

[0237]

[0238] in, is the current actual rolling angle, is the horizontal position coordinate of the current gravity direction, is the vertical position coordinate of the current gravity direction, is the vertical position coordinate of the current gravity direction;

[0239] The current actual pitch angle is obtained by the following formula:

[0240]

[0241] in, is the current actual pitch angle.

[0242] In one embodiment, the angle difference value determination module 1040 is specifically used to:

[0243] Subtracting the current actual roll angle from the current predicted roll angle to obtain a first angle difference value; and,

[0244] Subtracting the current actual pitch angle from the current predicted pitch angle to obtain a second angle difference value;

[0245] The angle difference value is obtained according to the first angle difference value and the second angle difference value.

[0246] In one embodiment, the apparatus further comprises:

[0247] The angle prediction module 1080 is used to obtain the current predicted angle by:

[0248] Obtaining an intermediate predicted roll angle according to the predicted roll angle at the last moment and the preset roll angle noise; and,

[0249] Obtaining an intermediate predicted pitch angle according to the predicted pitch angle at the previous moment and the preset pitch angle noise;

[0250] The intermediate predicted roll angle is optimized using the handle speed at the current moment and the Kalman filter gain at the current moment to obtain the current predicted roll angle, and the intermediate predicted pitch angle is optimized using the handle speed at the current moment and the Kalman filter gain to obtain the current predicted pitch angle, wherein the Kalman filter gain at the current moment is obtained based on the intermediate predicted roll angle and the intermediate predicted pitch angle, and the handle speed at the current moment is predicted based on the handle speed at the previous moment using the extended Kalman filter EKF.

[0251] In one embodiment, the verification module 1050 is specifically used to:

[0252] The confidence of the spot matching pair is obtained by using the angle difference value of the spot matching pair and the covariance matrix of the handle state corresponding to the current moment, wherein the handle state corresponding to the current moment includes the current predicted angle and the handle speed at the current moment, and the covariance matrix of the handle state corresponding to the current moment is obtained based on the covariance matrix of the handle state corresponding to the previous moment;

[0253] If the confidence level is greater than a preset confidence level, it is determined that the spot matching pair is matched successfully;

[0254] If the confidence is not greater than the preset confidence value, it is determined that the spot matching pair is unsuccessful.

[0255] In one embodiment, the verification module 1050 is further used to:

[0256] The confidence of the spot matching pair is obtained by:

[0257]

[0258] Where S is the confidence of the spot matching pair, Δ is the angle difference value of the spot matching pair, and P t|t,ΔΔ The rows and columns corresponding to the predicted angles in the covariance matrix of the handle state corresponding to the current moment.

[0259] After introducing a handle light spot matching method and device according to an exemplary embodiment of the present invention, next, an electronic device according to another exemplary embodiment of the present invention is introduced.

[0260] It will be appreciated by those skilled in the art that various aspects of the present invention may be implemented as a system, method or program product. Therefore, various aspects of the present invention may be specifically implemented in the following forms, namely: a complete hardware implementation, a complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software, which may be collectively referred to herein as a "circuit", "module" or "system".

[0261] In some possible implementations, the electronic device according to the present invention may include at least one processor and at least one computer storage medium. The computer storage medium stores program code, and when the program code is executed by the processor, the processor executes the steps of the handle spot matching method according to various exemplary embodiments of the present invention described above in this specification. For example, the processor may execute the following steps: Figure 1 Steps 101-107 shown in .

[0262] Refer to the following Fig.11 The electronic device 1100 according to this embodiment of the present invention is described. Fig.11The electronic device 1100 shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0263] like Fig.11 As shown, the components of the electronic device 1100 may include but are not limited to: the above-mentioned at least one processor 1101, the above-mentioned at least one computer storage medium 1102, and a bus 1103 connecting different system components (including the computer storage medium 1102 and the processor 1101).

[0264] Bus 1103 represents one or more of several types of bus structures, including a computer storage media bus or computer storage media controller, a peripheral bus, a processor, or a local bus using any of a variety of bus architectures.

[0265] Computer storage media 1102 may include readable media in the form of volatile computer storage media, such as random access computer storage media (RAM) 1121 and / or cache storage media 1122 , and may further include read-only computer storage media (ROM) 1123 .

[0266] The computer storage medium 1102 may also include a program / utility 1125 having a set (at least one) of program modules 1124, such program modules 1124 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0267] The electronic device 1100 may also communicate with one or more external devices 1104 (e.g., keyboards, pointing devices, etc.), and may also communicate with one or more devices that enable a user to interact with the electronic device 1100, and / or communicate with any device that enables the electronic device 1100 to communicate with one or more other AR devices (e.g., routers, modems, etc.). Such communication may be performed through an input / output (I / O) interface 1105. In addition, the electronic device 1100 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) through a network adapter 1106. As shown, the network adapter 1106 communicates with other modules for the electronic device 1100 through a bus 1103. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 1100, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0268] In some possible embodiments, various aspects of a handle light spot matching method provided by the present invention may also be implemented in the form of a program product, which includes a program code. When the program product runs on a computer device, the program code is used to enable the computer device to execute the steps of the handle light spot matching method according to various exemplary embodiments of the present invention described above in this specification.

[0269] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A method for matching a handle light spot, characterized in that: The method comprises: Performing light spot detection on the handle image acquired at the current moment to obtain the image position coordinates of each light spot in the handle image; Randomly select a specified number of target light spots from the light spots, assign lamp bead identifiers in the handle to the target light spots, and obtain light spot matching pairs; wherein any one of the light spot matching pairs includes a specified number of target light spots and respective lamp bead identifiers of the target light spots; According to the image position coordinates of each light spot in the light spot matching pair and the actual position coordinates of each lamp bead corresponding to each lamp bead identifier in the handle coordinate system, a current actual angle corresponding to the light spot matching pair is obtained, wherein the current actual angle is used to represent the current actual angle of the handle coordinate system relative to the world coordinate system; Determine an angle difference value using the current actual angle and the current predicted angle; wherein the current predicted angle is predicted based on the predicted angle of the handle at the previous moment; Determining whether the light spot matching pair is successfully matched according to the angle difference value; If yes, after deleting the target light spot in the light spot matching pair from the light spots, a specified number of target light spots are randomly selected from the light spots again until each light spot has a lamp bead identifier that is successfully matched; If not, a lamp bead identifier is reassigned to each target light spot until the light spot matching pair is successfully matched.

2. The method according to claim 1, characterized in that The obtaining of the current actual angle corresponding to the light spot matching pair according to the image position coordinates of each light spot in the light spot matching pair and the actual position coordinates of each lamp bead corresponding to each lamp bead identifier in the handle coordinate system comprises: Determine the current gravity direction in the handle coordinate system according to the image position coordinates of each light spot in the light spot matching pair and the actual position coordinates of each lamp bead corresponding to each lamp bead identifier in the handle coordinate system; Based on the current gravity direction, a current actual angle corresponding to the spot matching pair is obtained.

3. The method according to claim 2, characterized in that The determining the current gravity direction in the handle coordinate system according to the image position coordinates of each light spot in the light spot matching pair and the actual position coordinates of each lamp bead marker in the handle coordinate system includes: Input the image position coordinates of each light spot in the light spot matching pair and the actual position coordinates of each lamp bead marker in the handle coordinate system into the P3P algorithm to obtain a first rotation matrix, wherein the first rotation matrix is ​​used to represent the rotation matrix of the handle coordinate system relative to the camera coordinate system at the current moment; According to the first rotation matrix, the second rotation matrix and the matrix corresponding to the standard gravity direction, the current gravity direction in the handle coordinate system is obtained, wherein the camera coordinate system at the current moment obtained by the second rotation matrix is ​​equivalent to the rotation matrix of the world coordinate system, and the matrix corresponding to the standard gravity direction is used to represent the gravity direction in the world coordinate system.

4. The method according to claim 3, characterized in that The obtaining the current gravity direction in the handle coordinate system according to the first rotation matrix, the second rotation matrix and the matrix corresponding to the standard gravity direction includes: Multiplying the second rotation matrix by the first rotation matrix to obtain a third rotation matrix, wherein the third rotation matrix is ​​used to represent the rotation matrix of the handle coordinate system relative to the world coordinate system at the current moment; The inverse matrix of the third rotation matrix is ​​multiplied by the matrix corresponding to the standard gravity direction to obtain the current gravity direction.

5. The method according to claim 1, characterized in that The current actual angle includes a current actual roll angle and a current actual pitch angle; the current actual roll angle is used to represent the actual roll angle of the handle coordinate system relative to the world coordinate system at the current moment, and the current pitch angle is used to represent the actual pitch angle of the handle coordinate system relative to the world coordinate system at the current moment; The current predicted angle includes a current predicted roll angle and a current predicted pitch angle; the current predicted roll angle is used to represent the predicted roll angle of the handle coordinate system relative to the world coordinate system at the current moment, and the current predicted pitch angle is used to represent the predicted pitch angle of the handle coordinate system relative to the world coordinate system at the current moment.

6. The method according to claim 5, characterized in that Based on the current gravity direction, obtaining a current actual angle corresponding to the spot matching pair includes: The current actual rolling angle is obtained by the following formula: in, is the current actual rolling angle, is the horizontal position coordinate of the current gravity direction, is the vertical position coordinate of the current gravity direction, is the vertical position coordinate of the current gravity direction; The current actual pitch angle is obtained by the following formula: in, is the current actual pitch angle.

7. The method according to claim 5, characterized in that Determining an angle difference value using the current actual angle and the current predicted angle includes: Subtracting the current actual roll angle from the current predicted roll angle to obtain a first angle difference value; and, Subtracting the current actual pitch angle from the current predicted pitch angle to obtain a second angle difference value; The angle difference value is obtained according to the first angle difference value and the second angle difference value.

8. The method according to claim 5, characterized in that The current prediction angle is obtained by: Obtaining an intermediate predicted roll angle according to the predicted roll angle at the last moment and the preset roll angle noise; and, Obtaining an intermediate predicted pitch angle according to the predicted pitch angle at the previous moment and the preset pitch angle noise; The intermediate predicted roll angle is optimized using the handle speed at the current moment and the Kalman filter gain at the current moment to obtain the current predicted roll angle, and the intermediate predicted pitch angle is optimized using the handle speed at the current moment and the Kalman filter gain to obtain the current predicted pitch angle, wherein the Kalman filter gain at the current moment is obtained based on the intermediate predicted roll angle and the intermediate predicted pitch angle, and the handle speed at the current moment is predicted based on the handle speed at the previous moment using the extended Kalman filter EKF.

9. The method according to claim 1, characterized in that: The step of determining whether the light spot matching pair is successfully matched according to the angle difference value includes: The confidence of the spot matching pair is obtained by using the angle difference value of the spot matching pair and the covariance matrix of the handle state corresponding to the current moment, wherein the handle state corresponding to the current moment includes the current predicted angle and the handle speed at the current moment, and the covariance matrix of the handle state corresponding to the current moment is obtained based on the covariance matrix of the handle state corresponding to the previous moment; If the confidence level is greater than a preset confidence level, it is determined that the spot matching pair is matched successfully; If the confidence is not greater than the preset confidence value, it is determined that the spot matching pair is unsuccessful.

10. An electronic device, characterized in that: comprising a processor and a memory, wherein the processor and the memory are connected via a bus; The memory stores a computer program, and the processor is configured to perform the following operations based on the computer program: Performing light spot detection on the handle image acquired at the current moment to obtain the image position coordinates of each light spot in the handle image; Randomly select a specified number of target light spots from the light spots, assign lamp bead identifiers in the handle to the target light spots, and obtain light spot matching pairs; wherein any one of the light spot matching pairs includes a specified number of target light spots and respective lamp bead identifiers of the target light spots; According to the image position coordinates of each light spot in the light spot matching pair and the actual position coordinates of each lamp bead corresponding to each lamp bead identifier in the handle coordinate system, a current actual angle corresponding to the light spot matching pair is obtained, wherein the current actual angle is used to represent the current actual angle of the handle coordinate system relative to the world coordinate system; Determine an angle difference value using the current actual angle and the current predicted angle; wherein the current predicted angle is predicted based on the predicted angle of the handle at the previous moment; Determining whether the light spot matching pair is successfully matched according to the angle difference value; If yes, after deleting the target light spot in the light spot matching pair from the light spots, a specified number of target light spots are randomly selected from the light spots again until each light spot has a lamp bead identifier that is successfully matched; If not, a lamp bead identifier is reassigned to each target light spot until the light spot matching pair is successfully matched.

Images