Remote controller positioning method and remote controller

Abstract

The invention discloses a remote controller positioning method and a remote controller, and the method comprises the steps: if IMU data is detected at a first moment, adjusting first state information of the remote controller at the first moment by applying the IMU data and a first adjustment strategy, and obtaining second state information at a second moment; if the UWB data is detected at the second moment, adjusting the second state information by applying the UWB data and a second adjustment strategy to obtain target state information at the second moment; the target state information comprises a target position, a target attitude, a target speed and target offset information at a second moment; the first adjustment strategy and the second adjustment strategy are determined according to a predetermined extended Kalman filter; the focus of the remote controller on the display device is calculated by applying the target posture and the actual position of the remote controller at the second moment, and the actual position is determined according to the UWB data detected at the second moment. According to the embodiment of the invention, the positioning precision of the remote controller is improved.

Description

Technical Field

[0001] This application relates to the field of positioning technology, and in particular, to a positioning method and a remote controller for a remote controller. Background Art

[0002] In the field of remote controller positioning, the position of the remote controller can be calculated through UWB data, or the position of the remote controller can be calculated through IMU data, and then the position of the cursor of the remote controller on the display device can be calculated according to the position of the remote controller, so as to realize the positioning of the remote controller.

[0003] However, for the solution of positioning through UWB data, multiple UWB base stations need to be deployed in the positioning area, resulting in high costs. Moreover, the positioning accuracy is easily affected by the propagation range of UWB signals and metals. For the solution of positioning through IMU data, the positioning error will accumulate over time, and the positioning accuracy will decrease significantly after long-term use; it is easily affected by external vibrations, impacts and other interferences, affecting the measurement accuracy. In addition, the absolute position of the remote controller cannot be determined only relying on IMU data. Summary of the Invention

[0004] In an exemplary embodiment of the present application, a positioning method and a remote controller for a remote controller are provided to improve the positioning accuracy of the remote controller while ensuring that the cost is not very high.

[0005] According to a first aspect in the exemplary embodiment, a positioning method for a remote controller is provided, which is applied to the remote controller. The method includes:

[0006] If IMU data is detected at a first moment, the IMU data and a first adjustment strategy are used to adjust the first state information of the remote controller at the first moment to obtain second state information at a second moment; wherein, the second state information includes a reference position, a reference attitude, and a reference speed at the second moment, and bias information at the first moment; the bias information includes accelerometer bias information and gyroscope bias information;

[0007] If UWB data is detected at the second moment, the UWB data and a second adjustment strategy are used to adjust the second state information to obtain target state information at the second moment; wherein, the target state information includes a target position, a target attitude, a target speed, and target bias information at the second moment; the first adjustment strategy and the second adjustment strategy are determined according to a pre-determined extended Kalman filter;

[0008] The focus of the remote controller on the display device is calculated by applying the target attitude and the actual position of the remote controller at the second moment; wherein, the actual position of the remote controller at the second moment is determined according to the UWB data detected at the second moment.

[0009] In an embodiment of the present application, by using an extended Kalman filter, the IMU data and UWB data are fused. When the IMU data is detected, a state advancement operation is performed, and when the UWB data is detected, a state observation operation is performed. Then, the target attitude in the state information obtained by the state observation and the actual position of the remote controller are used to calculate the focus of the remote controller on the display device. By positioning the remote controller in this way, the defects of positioning only with IMU data and positioning only with UWB data are overcome, and the positioning accuracy of the remote controller is improved.

[0010] In an alternative embodiment, if the UWB data is not detected at the second moment, the method further includes:

[0011] Adjust the second state information by applying a third adjustment strategy to obtain the target state information at the second moment; wherein, the actual position of the remote controller at the second moment is determined according to the position calculated by the remote controller based on the UWB data at the first moment, the IMU data at the first moment, and the IMU data at the second moment.

[0012] In the above embodiment, if the UWB data is not detected at the second moment, the adjustment strategy applied at this time is distinguished from the adjustment strategy when the UWB data is detected. And in this case, due to the lack of UWB data, the actual position of the remote controller applied at this time is calculated according to the position calculated by the UWB data at the first moment and the IMU data of these two moments respectively. Such a design can handle different situations according to whether the UWB data is received, further improving the positioning accuracy.

[0013] In an alternative embodiment, the first adjustment strategy is the adjustment strategy indicated by the state advancement equation in the extended Kalman filter;

[0014] Adjust the first state information of the remote controller at the first moment by applying the IMU data and the first adjustment strategy to obtain the second state information at the second moment, including:

[0015] Calculate the state transition matrix and the process noise drive matrix of the state advancement equation according to the first state information of the remote controller at the first moment;

[0016] Apply the angular velocity and acceleration in the IMU data, and based on the state transition matrix and the process noise drive matrix, perform a state advancement operation on the position, attitude, and velocity in the first state information to obtain the second state information.

[0017] In the above embodiments, during the state advancement process, the state transition matrix and the process noise driving matrix can be calculated first. Then, based on the angular velocity and acceleration in the IMU data, and using the state transition matrix and the process noise driving matrix, a state advancement operation is performed on the position, attitude, and velocity in the first state information to obtain the second state information. This state advancement process makes full use of the IMU data and improves the positioning accuracy.

[0018] In an alternative embodiment, the state advancement equation includes:

[0019]

[0020] where t is the first moment, t + τ is the second moment, b is the coordinate system where the remote controller is located, and w is the world coordinate system;

[0021] is the rotation of the remote controller at the second moment, is the rotation of the remote controller at the first moment, ω b is the angular velocity of the remote controller, b ω is the angular velocity bias, n ω is the noise corresponding to the angular velocity;

[0022] is the velocity of the remote controller at the second moment, is the velocity of the remote controller at the first moment, is the set rotation matrix, a b is the acceleration of the remote controller, n a is the noise corresponding to the acceleration, g w is the direction of gravity;

[0023] is the reference position of the remote controller at the second moment, is the reference position of the remote controller at the first moment;

[0024] b a,t+τ is the acceleration bias at the second moment, b a,t is the acceleration bias at the first moment, is the noise corresponding to the acceleration bias;

[0025] b ω,t+τ is the angular velocity bias at the second moment, b ω,t is the angular velocity bias at the first moment, is the noise corresponding to the angular velocity bias.

[0026] In the above embodiments, by applying such a state advancement equation to perform the state advancement operation, the accuracy of the state advancement is improved, the accuracy of the extended Kalman filter is further improved, and the positioning accuracy is further improved.

[0027] In an alternative embodiment, the second adjustment strategy is the adjustment strategy indicated by the state observation equation in the extended Kalman filter;

[0028] Adjust the second state information using the UWB data and the second adjustment strategy to obtain the target state information at the second moment, including:

[0029] Calculate the observation matrix and the process noise covariance matrix of the state observation equation according to the second state information;

[0030] Apply the distance, pitch angle, and horizontal angle in the UWB data, and perform a state observation operation on the second state information based on the observation matrix and the process noise covariance matrix to obtain the target state information.

[0031] In the above embodiment, during the state observation process, first calculate the observation matrix and the process noise covariance matrix, and then apply the distance, pitch angle, and horizontal angle in the UWB data to perform a state observation operation on the second state information based on the observation matrix and the process noise covariance matrix to obtain the target state information. This state advancement process makes full use of the UWB data, further improving the accuracy of the extended Kalman filter and the positioning accuracy.

[0032] In an alternative embodiment, the state observation equation includes:

[0033]

[0034] Where is the transformation matrix from the world coordinate system to the external device UWB coordinate system, is the reference position of the remote control at the first moment, d is the distance between the external device UWB unit and the remote control UWB unit, n d is the noise corresponding to the distance, θ R is the first horizontal angle of the remote control relative to the external device, is the noise corresponding to the first horizontal angle, is the first elevation angle of the remote control relative to the external device, is the noise corresponding to the first elevation angle, represents the direction of the remote control relative to the external device; where the external device is a display device or a Dongle device communicating with the display device;

[0035] is the transformation matrix from the remote control IMU coordinate system to the display device coordinate system, is the transformation matrix from the remote control IMU coordinate system to the world coordinate system, θ I is the second horizontal angle of the external device relative to the remote control, the noise corresponding to the second horizontal angle, is the second elevation angle of the external device relative to the remote controller, is the noise corresponding to the second elevation angle, represents the direction of the external device relative to the remote controller.

[0036] In the above embodiment, the state observation operation is performed using such a state observation equation, which improves the accuracy of state observation, further improves the accuracy of the extended Kalman filter, and further improves the positioning accuracy.

[0037] In an alternative embodiment, the third adjustment strategy is the adjustment strategy indicated by the updated state observation equation in the extended Kalman filter;

[0038] The updated state observation equation is a state observation equation fabricated according to the noise characteristics.

[0039] In the above embodiment, when UWB data cannot be detected, in order to ensure the smooth progress of the state observation process, a state observation equation can be fabricated according to the noise characteristics. Such a design takes different situations into account, further improves the accuracy of the extended Kalman filter, and thus improves the positioning accuracy.

[0040] In an alternative embodiment, the application target pose and the actual position of the remote controller at the second moment are used to calculate the focus of the remote controller on the display device, including:

[0041] Calculate the coordinates of the remote controller on the display device according to the application target pose and the actual position of the remote controller at the second moment;

[0042] Determine the pixel coordinates of the remote controller on the display device according to the coordinates of the remote controller on the display device; wherein, the position indicated by the pixel coordinates on the display device is the focus.

[0043] In the above embodiment, after obtaining the coordinates of the remote controller on the display device, the pixel coordinates of the calculator on the display device can be calculated, so that the cursor of the remote controller on the display device can be determined more accurately according to the pixel coordinates.

[0044] In an alternative embodiment, the method further includes:

[0045] Perform smoothing filtering on the pixel coordinates using a first-order filter; wherein, the filtering coefficient in the first-order filter is determined according to the rotational speed in the speed of the remote controller.

[0046] In the above embodiment, after performing the smoothing filtering, the jitter situation can be effectively removed.

[0047] According to the second aspect of the exemplary embodiment, a remote controller is provided, which is applied to a remote controller. The remote controller includes a data transmission unit and a processor;

[0048] A data transmission unit, configured to perform:

[0049] Obtain IMU data and UWB data;

[0050] A processor, configured to perform:

[0051] If IMU data is detected at a first moment, apply the IMU data and a first adjustment strategy to adjust the first state information of the remote controller at the first moment to obtain second state information at a second moment; wherein, the second state information includes a reference position, a reference attitude, and a reference speed at the second moment, and bias information at the first moment; the bias information includes accelerometer bias information and gyroscope bias information;

[0052] If UWB data is detected at the second moment, apply the UWB data and a second adjustment strategy to adjust the second state information to obtain target state information at the second moment; wherein, the target state information includes a target position, a target attitude, a target speed, and target bias information at the second moment; the first adjustment strategy and the second adjustment strategy are determined according to a pre-determined extended Kalman filter;

[0053] Apply the target attitude and the actual position of the remote controller at the second moment to calculate the focus of the remote controller on the display device; wherein, the actual position of the remote controller at the second moment is determined according to the UWB data detected at the second moment.

[0054] According to a third aspect in the exemplary embodiment, there is provided a positioning device for a remote controller, applied to the remote controller, the device includes:

[0055] A state update unit, configured to: if IMU data is detected at a first moment, apply the IMU data and a first adjustment strategy to adjust the first state information of the remote controller at the first moment to obtain second state information at a second moment; wherein, the second state information includes a reference position, a reference attitude, and a reference speed at the second moment, and bias information at the first moment; the bias information includes accelerometer bias information and gyroscope bias information;

[0056] The state update unit is further configured to: if UWB data is detected at the second moment, apply the UWB data and a second adjustment strategy to adjust the second state information to obtain target state information at the second moment; wherein, the target state information includes a target position, a target attitude, a target speed, and target bias information at the second moment; the first adjustment strategy and the second adjustment strategy are determined according to a pre-determined extended Kalman filter;

[0057] A positioning unit is configured to calculate the focus of the remote controller on the display device by applying the target pose and the actual position of the remote controller at the second moment; wherein, the actual position of the remote controller at the second moment is determined according to the UWB data detected at the second moment.

[0058] According to a fourth aspect of the exemplary embodiments, there is provided a computer storage medium storing computer program instructions which, when run on a computer, cause the computer to execute the positioning method of the remote controller as in the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] To more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for the description of the embodiments. Obviously, the following drawings are only some embodiments of the present application, and those of ordinary skill in the art can obtain other drawings without creative efforts based on these drawings.

[0060] Figure 1a Exemplarily shows a schematic diagram of a UWB data provided by an embodiment of the present application;

[0061] Figure 1b Exemplarily shows an application scenario diagram of remote controller positioning provided by an embodiment of the present application;

[0062] Figure 1c Exemplarily shows a schematic diagram of the positioning principle of a remote controller provided by an embodiment of the present application;

[0063] Figure 1d Exemplarily shows a schematic diagram of a UWB and IMU fusion positioning scheme for a pointing remote controller provided by an embodiment of the present application;

[0064] Figure 2 Exemplarily shows a flowchart of a positioning method of a remote controller provided by an embodiment of the present application;

[0065] Figure 3 Exemplarily shows a flowchart of a method for determining second state information;

[0066] Figure 4 Exemplarily shows a flowchart of a method for determining target state information;

[0067] Figure 5 Exemplarily shows a schematic diagram of calculating pixel coordinates provided by an embodiment of the present application;

[0068] Figure 6 Exemplarily shows a schematic structural diagram of a positioning device of a remote controller provided by an embodiment of the present application;

[0069] Figure 7 The figure schematically shows the structure of a remote controller provided by an embodiment of the present application. Detailed implementation manners

[0070] To make the objectives, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application.

[0071] For the convenience of understanding, the names and symbols involved in the embodiments of the present application are explained below:

[0072] (1) The Ultra Wideband (UWB) directional remote controller (hereinafter referred to as the remote controller) is a remote controller that uses UWB technology to achieve precise pointing and control functions.

[0073] The positioning principle of UWB is as follows:

[0074] The UWB technology transmits data by emitting ultra-wideband pulse signals. In the remote controller, the UWB chip uses the Angle of Arrival (AoA) technology to measure the distance between the remote controller and the display device with millimeter-level precision and determine the pointing direction of the remote controller through angle measurement. For example, certain types of UWB chips support UWB channels 5 and 9, with a frequency range of 6250 MHz to 8250 MHz. Through their UWB functions, precise positioning of the remote controller can be achieved, supporting a 3-antenna specification and enabling the measurement of 2D AoA and 3D AoA, providing strong hardware support for the remote controller, so that users can achieve a mouse-like operation experience on the display device by pointing the remote controller.

[0075] Therefore, the UWB directional remote controller can achieve high-precision pointing and positioning functions, bringing a brand-new control experience to users, and easily realizing operations such as clicking, selecting, sliding, and dragging with the remote controller, greatly expanding the usage scenarios of the device.

[0076] (2) The Inertial Measurement Unit (IMU) is a device used to measure the motion state of an object and consists of an accelerometer and a gyroscope. The accelerometer is used to measure the acceleration of an object in three axial directions. The working principle of the accelerometer is based on Newton's second law, and the acceleration value is calculated by detecting the force generated by the mass block under the action of acceleration. The gyroscope mainly measures the angular velocity of an object around three coordinate axes. The gyroscope utilizes the principle of conservation of angular momentum. When the object rotates, the rotor inside the gyroscope will generate corresponding changes in angular momentum, and the angular velocity information is obtained by detecting this change.

[0077] The working principle of the IMU is to measure the acceleration and angular velocity information of an object in real time through an accelerometer and a gyroscope. The acceleration data measured by the accelerometer can be integrated to obtain velocity and displacement information. However, due to the cumulative error in the integration process, the error will be relatively large when using the accelerometer alone to measure displacement for a long time. The angular velocity information measured by the gyroscope can be integrated to obtain the attitude angle of the object, but there is also a problem of error accumulation. Therefore, a fusion algorithm is usually adopted to fuse the data of the accelerometer and the gyroscope to improve the measurement accuracy and reliability. For example, the Kalman filter algorithm is a commonly used fusion algorithm. It can perform an optimal estimation on the data of the accelerometer and the gyroscope based on the measurement error of the sensor and the dynamic model of the system, so as to obtain more accurate information on the motion state of the object.

[0078] (3) The Dongle device, a small external device, is also known as a dongle, a hardware key, or an adapter, etc. In some devices such as smart TVs and game consoles, the Dongle can be used to expand the functions of the device, such as adding Bluetooth function, Wi-Fi function, etc., to improve the performance and user experience of the device. In the embodiments of the present application, a UWB chip can be set in the Dongle device, or a UWB chip can also be set in the display device to communicate with the UWB chip in the remote control, and then locate the remote control.

[0079] (4) In the embodiments of the present application, the reference position refers to the position of the remote control obtained when the state is advanced, the target position is the position of the remote control obtained when the state is observed, and the actual position is the actual position of the remote control.

[0080] (5) Scalars are represented by lowercase letters, vectors are represented by lowercase letters, and matrices are represented by uppercase letters.

[0081] q represents quaternion rotation;

[0082] v represents the velocity vector;

[0083] p represents the position vector;

[0084] b represents the bias vector;

[0085] n represents noise or the pointing vector;

[0086] R represents a 3x3 rotation matrix;

[0087] P represents the covariance matrix;

[0088] The subscript s represents the display device coordinate system, with the origin at the upper left corner of the display device, the horizontal right direction as the x-axis, and the vertical downward direction as the y-axis;

[0089] The subscript w represents the world coordinate system, with the origin at the UWB unit of the Dongle and the coordinate axes parallel to those of s;

[0090] The subscript b represents the IMU coordinate system of the remote control;

[0091] The subscript R represents the Dongle UWB coordinate system;

[0092] The subscript I represents the UWB coordinate system of the remote control;

[0093] The subscript t represents the moment; the subscript τ represents the time interval; the superscript T represents the matrix transpose;

[0094] exp and log represent the exponential function and the logarithmic function;

[0095] represents quaternion multiplication;

[0096] The superscript ∧ represents the skew operation, which converts a 3x1 vector into a 3x3 skew-symmetric matrix;

[0097] The superscript ∨ represents the inverse operation of the skew operation;

[0098] The FromTwoVector(v1, v2) function returns a rotation that rotates the vector v1 to coincide with another vector v2;

[0099] g w =[0, -9.81, 0] T represents the direction of gravity;

[0100] Rotation from the UWB R-end coordinate system to the display device coordinate system and translation

[0101] Rotation from the IMU coordinate system of the remote control to the UWB I-end coordinate system

[0102] The current smart TV interface is complex. Traditional remote controls are operated through buttons, which are cumbersome and have a high learning cost, and are not user-friendly, especially for elderly users. Even smart remote controls with voice functions cannot completely replace buttons.

[0103] In the field of projectors, there are pointing remote controls. The remote control emits infrared light that forms a light spot on the screen. The camera of the projector recognizes this light spot and calculates the position of the light spot on the screen through algorithms, thereby realizing UI interaction, with direct and clear operations. Therefore, adding a pointing function to the TV remote control can well solve the above problems. However, the above scheme cannot be directly transplanted to the TV remote control. Thus, the UWB ultra-wideband positioning scheme came into being.

[0104] UWB ultra-wideband positioning consists of two base stations, each base station consists of 1 to 3 antennas. The base stations send encoded radio pulses with high frequency and short time to measure the distance and azimuth angle of each other's base stations; 1 antenna can only measure distance, 2 antennas can measure distance and 1 azimuth angle, and 3 antennas can measure distance and 2 azimuth angles. Install a base station on the TV and a base station on the remote control. Knowing each other's positions can calculate the pointing direction of the remote control. However, UWB positioning has fatal defects. First, it is difficult to increase the measurement frequency. A certain company can achieve 50Hz, which is still relatively low for smooth operation. Second, the power consumption is large. The power consumption is proportional to the measurement frequency. To meet the battery life of the remote control, the frequency is required not to exceed 10Hz. Third, it is vulnerable to external environmental interference. Therefore, there has been no mature and commercially available UWB positioning pointing remote control solution for a long time. In the embodiments of the present application, a positioning solution of 3 antennas (UWB unit in the remote control) + 3 antennas (UWB unit in the display device) can be adopted, with a measurement frequency of 10Hz, integrating an IMU sensor, realizing a commercially available pointing remote control solution, and having better accuracy and range.

[0105] Figure 1a FIG. is a schematic diagram of UWB data provided by an embodiment of the present application. Among them, d is the distance between two UWB units, and θ is the horizontal angle. is the elevation angle.

[0106] After introducing the design concept of the embodiments of the present application, the following briefly introduces the application scenarios applicable to the technical solutions of the embodiments of the present application. It should be noted that the following introduced application scenarios are only used to illustrate the embodiments of the present application rather than to limit. In specific implementation, the technical solutions provided by the embodiments of the present application can be flexibly applied according to actual needs.

[0107] Reference Figure 1b , shows an application scenario diagram of remote control positioning. In this schematic, it includes a dongle device. In the actual application process, it is also possible to directly apply the display device without using the dongle device. This is just an example and does not form a specific limitation.

[0108] Reference Figure 1c , shows a schematic diagram of the positioning principle of a remote control. In this schematic, the dongle device includes a UWB unit (referred to as the R end), a main control unit, and Bluetooth. The remote control includes a UWB unit (referred to as the I end), a main control unit, Bluetooth, an IMU, an infrared emitter, and buttons; the display device (TV) includes an infrared receiver.

[0109] Among them, the UWB R terminal and the UWB I terminal measure each other's azimuth, obtain the distance, 4 azimuth angles (two groups of horizontal angles and pitch angles), and confidence, and send them to the main control unit of the remote control. The IMU sensor measures the acceleration and angular velocity of the remote control and sends them to the main control unit of the remote control. The fusion algorithm calculates the focus between the pointing direction of the remote control and the display device based on these measurement data, sends it to the dongle device via Bluetooth, and then sends it to the operating system of the TV via USB, and finally displays the cursor.

[0110] To further illustrate the technical solutions provided by the embodiments of the present application, the following will be described in detail in conjunction with the accompanying drawings and specific implementation manners. Although the embodiments of the present application provide method operation steps as shown in the following embodiments or drawings, based on routine or non-creative labor, more or fewer operation steps may be included in the method. In steps where there is no necessary causal relationship logically, the execution order of these steps is not limited to the execution order provided by the embodiments of the present application.

[0111] Before introducing the technical solutions of the present application, the principle of UWB and IMU fusion positioning of the pointing remote control will be described first.

[0112] Figure 1d It is a schematic diagram of a UWB and IMU fusion positioning scheme for a pointing remote control provided by an embodiment of the present application. Refer to Figure 1d It can be seen that the IMU data and the UWB data are fused and filtered using an Extended Kalman Filter (EKF). The attitude data in the result is fused with the result of the combined filtering of the acceleration and angular velocity in the IMU data for 3DoF attitude fusion. The fused attitude information is combined with the anchor point position determined according to the UWB data for 6DoF pose merging to obtain a 2D plane coordinate, and then converted into a pixel coordinate on the display device.

[0113] Next, in combination with Figure 1d , the process of EKF fusion filtering and the positioning process will be described.

[0114] Next, in combination with Figures 1a - 1d The application scenario shown, refer to Figure 2 The flowchart of a positioning method of a remote control shown, this method is applied to the remote control. In combination with Figure 2 , the technical solutions provided by the embodiments of the present application will be described.

[0115] S201: If IMU data is detected at the first moment, then the IMU data and the first adjustment strategy are used to adjust the first state information of the remote control at the first moment to obtain the second state information at the second moment.

[0116] Among them, the second state information includes the reference position, reference attitude, and reference speed at the second moment, as well as the offset information at the first moment; the offset information includes accelerometer offset information and gyroscope offset information.

[0117] S202: If UWB data is detected at the second moment, apply the UWB data and the second adjustment strategy to adjust the second state information to obtain the target state information at the second moment.

[0118] Among them, the target state information includes the target position, target attitude, target speed, and target offset information at the second moment; the first adjustment strategy and the second adjustment strategy are determined according to a pre-determined extended Kalman filter.

[0119] S203: Apply the target attitude and the actual position of the remote control at the second moment to calculate the focus of the remote control on the display device.

[0120] Among them, the actual position of the remote control at the second moment is determined according to the UWB data detected at the second moment.

[0121] In the embodiments of the present application, by using an extended Kalman filter, by fusing IMU data and UWB data, when IMU data is detected, a state advancement operation is performed, and when UWB data is detected, a state observation operation is performed. Then, apply the target attitude in the state information obtained by state observation and the actual position of the remote control to calculate the focus of the remote control on the display device. Positioning the remote control in this way overcomes the defects of positioning only with IMU data and positioning only with UWB data, and improves the positioning accuracy of the remote control.

[0122] First, explain the three elements of the extended Kalman filter: state variables, state advancement model, and state observation model. In the embodiments of the present application, the state variables are:

[0123]

[0124] Among them, the 5 vectors respectively represent the rotation (angle change), speed, position, accelerometer offset, and gyroscope offset of the remote control in the world coordinate system. Optionally, the size of the state variables is 4 + 3 + 3 + 3 + 3 = 16.

[0125] Optionally, the process of extended Kalman filtering is a process of estimating and optimizing the state variables at the next moment according to the state information at the current moment.

[0126] Secondly, explain the initialization process of the Kalman filter:

[0127] It is necessary to provide the initial values of the state s and covariance P to the EKF, and their calculation methods are as follows

[0128] Initialization process for rotation :

[0129] Rotation should satisfy two constraints. In the first constraint, assuming that the remote control is stationary during initialization, the direction of the IMU acceleration in the world coordinate system is the direction of gravity. Therefore, the velocity direction can be integrated for a period of time t0 to t1 (such as 0.5 seconds), and the velocity direction is the direction of gravity. The formula is written as:

[0130]

[0131] The first constraint is:

[0132]

[0133] According to the constraint conditions, it can be determined that:

[0134]

[0135]

[0136] In the second constraint, the R end and I end of UWB measure each other's directions coincide and are in opposite directions. It can be deduced that they also coincide and are in opposite directions in the plane projection, and we can get:

[0137]

[0138] Position initialization directly uses the measurement value of the R end of UWB, that is:

[0139]

[0140] The remaining states are set to 0:

[0141]

[0142] Covariance is set to a fixed diagonal matrix, and the value can be preset according to experience.

[0143] The meanings of the variables can be determined by referring to the definitions of the above symbols, which will not be elaborated here.

[0144] In summary, during initialization, IMU integration is performed and UWB data is recorded. If valid UWB data exists within 0.5 s, rotation and position are initialized and the system enters the 6DoF state; if no valid UWB data exists after 0.5 s, rotation is initialized, the position is fixed, and the system enters the 3DoF state. In the 6DoF state, there is continuous UWB angle constraint or the system is stationary; if there is no UWB angle constraint for 30 s of accumulated stationary time, the system enters the 3DoF state; in the 3DoF state, there is no UWB constraint and the direction is not reliable; when there is a UWB angle constraint, the system enters the 6DoF state and the position is re-initialized.

[0145] Regarding S201, the sampling frequency of the IMU unit (which can also be called the IMU sensor) is 200 Hz, and the sampling frequency of the UWB unit is 10 Hz. Therefore, at some moments, UWB data cannot be directly detected. At this time, it can be processed as if there is no UWB, or the estimated UWB data can be used for calculation according to other data filling schemes. This is not limited here.

[0146] In the embodiments of the present application, the first moment is represented by t. At the moment t, if IMU data is detected, the IMU data and the first adjustment strategy can be used to adjust the state information at the first moment to obtain the second state information.

[0147] Among them, the second state information includes the reference position, reference attitude, and reference speed at the second moment, as well as the bias information at the first moment; the bias information includes accelerometer bias information and gyroscope bias information.

[0148] Exemplarily, the first adjustment strategy is the adjustment strategy indicated by the state propagation equation in the extended Kalman filter. The process of using the IMU data and the first adjustment strategy to adjust the first state information of the remote controller at the first moment to obtain the second state information can be implemented through Figure 3 steps S201-1 to S201-2 in

[0149] S201-1: Calculate the state transition matrix and process noise drive matrix of the state propagation equation according to the first state information of the remote controller at the first moment.

[0150] S201-2: Apply the angular velocity and acceleration in the IMU data, and perform a state propagation operation on the position, attitude, and speed in the first state information based on the state transition matrix and process noise drive matrix to obtain the second state information.

[0151] Next, a specific example is used to illustrate the state propagation equation and the state propagation process:

[0152] Optionally, the state propagation equation is as follows:

[0153]

[0154]

[0155] Among them, \(t\) is the first moment, \(t + \tau\) is the second moment, \(b\) is the coordinate system where the remote control is located, and \(w\) is the world coordinate system; is the rotation of the remote control at the second moment, is the rotation of the remote control at the first moment, \(\omega\) b The angular velocity of the remote control, \(b\) ω is the angular velocity bias, \(n\) ω is the noise corresponding to the angular velocity; is the velocity of the remote control at the second moment, is the velocity of the remote control at the first moment, is the set rotation matrix, \(a\) b The acceleration of the remote control, \(n\) a is the noise corresponding to the acceleration, \(g\) w is the direction of gravity; is the reference position of the remote control at the second moment, is the reference position of the remote control at the first moment; \(b\) a,t+τ is the acceleration bias at the second moment, \(b\) a,t is the acceleration bias at the first moment, is the noise corresponding to the acceleration bias; \(b\) ω,t+τ is the angular velocity bias at the second moment, \(b\) ω,t is the angular velocity bias at the first moment, is the noise corresponding to the angular velocity bias.

[0156] The five equalities in the above state propagation equation are uniformly expressed as:

[0157] \(s\) t+τ = f(s t , n imu );

[0158] It is necessary to transform it into an error state propagation equation, that is:

[0159] \(\delta s\) t+τ = F\(\delta s\) t + Gn imu ;

[0160]

[0161] Because the rotation has 3 degrees of freedom and the size of its error state is 3, so the size of the error state \(\delta s\) t is 3 + 3 + 3 + 3 + 3 = 15.

[0162] In the case where the sizes of the state and the error state are different, the manifold addition and subtraction are defined as follows:

[0163] Manifold addition:

[0164]

[0165] Manifold subtraction:

[0166]

[0167] Among them, the manifold addition and subtraction satisfy the relationship:

[0168]

[0169] Therefore, the state transition matrix F and the process noise driving matrix G can be calculated as follows:

[0170]

[0171]

[0172] Among them, F is a matrix of size 15x15, and G is a matrix of size 15x12.

[0173] In this way, the state advancement process can be executed using this state advancement equation.

[0174] In a specific example, t represents the first moment, t+τ represents the second moment. At the first moment, IMU data is received, and the state is advanced, and the following operations are performed

[0175]

[0176] P t+τ = FP t F T + GP imu G T

[0177] Among them, P imu is the IMU covariance, which is a known fixed value.

[0178] Regarding S202, if UWB data is detected at the second moment, the UWB data and the second adjustment strategy are used to adjust the second state information.

[0179] Among them, the second adjustment strategy is the adjustment strategy indicated by the state observation equation in the extended Kalman filter. In this way, the process of adjusting the second state information using the UWB data and the second adjustment strategy to obtain the second state information at the second moment can be achieved through Figure 4 the steps S202-1 to S202-2 in:

[0180] S202-1: Calculate the observation matrix and process noise covariance matrix of the state observation equation according to the second state information.

[0181] S202-2: Apply the distance, pitch angle, and horizontal angle in the UWB data, and perform a state observation operation on the second state information based on the observation matrix and process noise covariance matrix to obtain the target state information.

[0182] Next, a specific example is used to illustrate the state observation equation and the observation process:

[0183] Among them, according to whether UWB data can be detected at the second moment, the state observation equation includes the following two categories:

[0184] The first case is when UWB data is detected at the second moment.

[0185] This case is also the 6DoF case, and the UWB data includes the distance d, horizontal angle θ, and altitude angle The direction of the object to be measured (remote control) is expressed as follows:

[0186]

[0187] The position of the remote control observed at the Dongle is:

[0188]

[0189] The position of the Dongle observed at the remote control is:

[0190]

[0191] Among them, is the conversion matrix from the world coordinate system to the external device UWB coordinate system, is the reference position of the remote control at the first moment, d is the distance between the external device UWB unit and the remote control UWB unit, n d is the noise corresponding to the distance, θ R is the first horizontal angle of the remote control relative to the external device, is the noise corresponding to the first horizontal angle, is the first altitude angle of the remote control relative to the external device, is the noise corresponding to the first altitude angle, represents the direction of the remote control relative to the external device; among them, the external device is a display device or a Dongle device communicating with the display device;

[0192] is the conversion matrix from the remote control IMU coordinate system to the display device coordinate system, is the conversion matrix from the remote control IMU coordinate system to the world coordinate system, θI is the second horizontal angle of the external device relative to the remote control, the noise corresponding to the second horizontal angle, is the second height angle of the external device relative to the remote control, is the noise corresponding to the second height angle, represents the direction of the external device relative to the remote control.

[0193] It is uniformly written in the form of residuals, that is:

[0194]

[0195] The residual is non-linear. Linearizing the estimated value gives:

[0196]

[0197] Therefore, the observation matrix H and the process noise covariance matrix Q can be calculated as follows:

[0198]

[0199] In this way, the state observation process can be performed using this state observation equation.

[0200] The second case is when UWB data is not detected at the second moment.

[0201] In this case, if UWB data is not detected at the second moment, the third adjustment strategy is applied to adjust the second state information to obtain the target state information at the second moment. Here, the third strategy refers to the strategy represented by the forged state observer.

[0202] This case is also the 3DoF case. Since no UWB data is detected, there is no observation. Therefore, a reasonable observation can be forged (the state observation equation forged according to the noise characteristics). The position of the remote control is not too far from the initial position, that is:

[0203]

[0204] Written in the form of residuals, that is:

[0205]

[0206] The residual is linear. Taking the derivative gives:

[0207]

[0208] When UWB data is received at time t+τ, perform state update and execute the following operations to obtain the gain K, the updated state and the updated covariance P t+τ :

[0209] K = P t+τ H T (HP t+τ H T + QP uwb / p Q T ) -1

[0210]

[0211] P t+τ ← (I - KH)P t+τ

[0212] Regarding S203, for the result of the extended Kalman filter, the process of calculating the focus of the remote controller on the display device by applying the target attitude (i.e., the first - dimensional information of the state variable) in the obtained state variables and the actual position of the remote controller at the second moment can be achieved through steps A1 - A2:

[0213] A1: Calculate the coordinates of the remote controller on the display device according to the target attitude and the actual position of the remote controller at the second moment.

[0214] In a specific example, the calculated coordinates of the remote controller on the display device are: (x p , y p , z p ).

[0215] A2: Determine the pixel coordinates of the remote controller on the display device according to the coordinates of the remote controller on the display device.

[0216] According to Figure 5 the geometric relationship shown, the pixel coordinates of the focus of the remote controller on the display device, u and v, are:

[0217]

[0218] where w p , h p are the width and height of the pixel resolution of the display device, w, h are the physical dimensions of the display device, and x a , y a , z a is the position of the remote controller in the display device coordinate system.

[0219] After obtaining the pixel coordinates, a first - order filter can also be applied to perform smoothing filtering on the pixel coordinates. Among them, the filtering coefficient in the first - order filter is determined according to the rotational speed in the speed of the remote controller.

[0220] Optionally, the process of filtering using a first - order filter is as follows:

[0221]

[0222] The filtering coefficient α ∈ (0, 1) can vary according to the change in motion speed. The smaller the value, the more obvious the filtering effect. One of the strategies can be written as:

[0223]

[0224] Adjust α according to the rotation speed of the remote control. When stationary, ω b = 0, α = 0, and the parameters β and γ are preset according to experience.

[0225] To make the technical solution of this application more perfect, the following gives a complete example to illustrate the complete process of positioning, which can be achieved through steps B1 - B7:

[0226] B1: Turn on the pointing switch / wake up.

[0227] There is a DIP switch on the side of the remote control, and the user can control whether to use the pointing function; when the user picks up the remote control, the main control can be awakened, and when the pointing function is turned on, the pointing function is initialized.

[0228] B2: Initialize UWB and IMU.

[0229] Wake up the UWB chip, enable the IMU chip, and send the measurement data to the main control through the SPI interface.

[0230] B3: Initialization of the fusion algorithm

[0231] When 0.5 seconds of IMU data and 1 reliable UWB data are received cumulatively, initialize the pose of the remote control to achieve 6DoF initialization; if no reliable UWB data is received within 0.5 seconds, initialize the rotation of the remote control, and the translation is set to a preset value for 3DoF initialization.

[0232] B4: EKF filter.

[0233] EKF is divided into two stages: state propagation and update. Each time an IMU data is received, a state propagation is performed to update the pose and covariance of the remote control, and the measurement error will accumulate; each time a UWB data is received, an update is performed to reduce the pose error of the remote control, and this is repeated.

[0234] B5: Smooth filtering.

[0235] Calculate the cursor coordinates based on the position of the remote control, and apply a smooth filter to reduce jitter.

[0236] B6: Send the cursor coordinates via Bluetooth.

[0237] The cursor coordinates are sent to the Dongle at a frequency of 200 Hz via a private Bluetooth protocol, and finally sent to the TV UI system to display the cursor.

[0238] B7: Stop the algorithms and UWB and IMU

[0239] When the user toggles the pointing switch to turn off the pointing function or the remote control goes into sleep after being stationary for 15 seconds, the pointing algorithm stops, and the UWB and IMU chips are turned off to reduce power consumption.

[0240] In summary, in the embodiments of the present application, a UWB 3 - antenna + 3 - antenna positioning scheme is adopted. The 3 - antenna has better positioning accuracy and range than the 2 - antenna. The EKF filter algorithm is used to fuse the IMU sensor, which can provide an output of up to 200 Hz, while the UWB measurement frequency only needs to be 10 Hz, meeting the battery life requirements of the remote control. The fusion algorithm is implemented with an EKF (Kalman) filter, which runs on the main control of the remote control, fuses the UWB and IMU data, estimates the pose (translation + rotation) of the remote control, calculates the cursor position, and sends it to the display device via Bluetooth after smoothing filtering.

[0241] As Figure 6 shown, based on the same inventive concept, the embodiments of the present application provide a positioning device for a remote control, which is applied to the remote control. The device includes a state update unit 61 and a positioning unit 62.

[0242] The state update unit 61 is configured to: if IMU data is detected at a first moment, apply the IMU data and a first adjustment strategy to adjust the first state information of the remote control at the first moment to obtain the second state information at a second moment; wherein, the second state information includes the reference position, reference attitude, and reference speed at the second moment, and the bias information at the first moment; the bias information includes accelerometer bias information and gyroscope bias information.

[0243] The state update unit 61 is further configured to: if UWB data is detected at the second moment, apply the UWB data and a second adjustment strategy to adjust the second state information to obtain the target state information at the second moment; wherein, the target state information includes the target position, target attitude, target speed, and target bias information at the second moment; the first adjustment strategy and the second adjustment strategy are determined according to a pre - determined extended Kalman filter.

[0244] The positioning unit 62 is configured to calculate the focus of the remote control on the display device by applying the target attitude and the actual position of the remote control at the second moment; wherein, the actual position of the remote control at the second moment is determined according to the UWB data detected at the second moment.

[0245] In an optional implementation manner, the state update unit 61 is further configured to:

[0246] If UWB data is not detected at the second moment, the third adjustment strategy is applied to adjust the second state information to obtain the target state information at the second moment; wherein, the actual position of the remote controller at the second moment is determined according to the position calculated based on the UWB data at the first moment, the IMU data at the first moment, and the IMU data at the second moment.

[0247] In an alternative embodiment, the first adjustment strategy is the adjustment strategy indicated by the state propagation equation in the extended Kalman filter;

[0248] The state update unit 61 is specifically configured to:

[0249] Calculate the state transition matrix and the process noise drive matrix of the state propagation equation according to the first state information of the remote controller at the first moment;

[0250] Apply the angular velocity and acceleration in the IMU data, and perform a state propagation operation on the position, attitude, and velocity in the first state information based on the state transition matrix and the process noise drive matrix to obtain the second state information.

[0251] In an alternative embodiment, the state propagation equation includes:

[0252]

[0253] where t is the first moment, t + τ is the second moment, b is the coordinate system where the remote controller is located, and w is the world coordinate system;

[0254] is the rotation of the remote controller at the second moment, is the rotation of the remote controller at the first moment, ω b the angular velocity of the remote controller, b ω is the angular velocity bias, n ω is the noise corresponding to the angular velocity;

[0255] is the velocity of the remote controller at the second moment, is the velocity of the remote controller at the first moment, is the set rotation matrix, a b is the acceleration of the remote controller, n a is the noise corresponding to the acceleration, g w is the direction of gravity;

[0256] is the reference position of the remote controller at the second moment, is the reference position of the remote controller at the first moment;

[0257] b a,t+τ is the acceleration bias at the second moment, b a,tis the acceleration bias at the first moment, is the noise corresponding to the acceleration bias;

[0258] b ω,t+τ is the angular velocity bias at the second moment, b ω,t is the angular velocity bias at the first moment, is the noise corresponding to the angular velocity bias.

[0259] In an alternative implementation, the second adjustment strategy is the adjustment strategy indicated by the state observation equation in the extended Kalman filter;

[0260] The state update unit 61 is specifically configured to:

[0261] Calculate the observation matrix and the process noise covariance matrix of the state observation equation according to the second state information;

[0262] Apply the distance, pitch angle, and horizontal angle in the UWB data, and perform a state observation operation on the second state information based on the observation matrix and the process noise covariance matrix to obtain the target state information.

[0263] In an alternative implementation, the state observation equation includes:

[0264]

[0265] where, is the conversion matrix from the world coordinate system to the external device UWB coordinate system, is the reference position of the remote control at the first moment, d is the distance between the external device UWB unit and the remote control UWB unit, n d is the noise corresponding to the distance, θ R is the first horizontal angle of the remote control relative to the external device, is the noise corresponding to the first horizontal angle, is the first elevation angle of the remote control relative to the external device, is the noise corresponding to the first elevation angle, represents the direction of the remote control relative to the external device; where the external device is a display device or a Dongle device communicating with the display device;

[0266] is the conversion matrix from the remote control IMU coordinate system to the display device coordinate system, is the conversion matrix from the remote control IMU coordinate system to the world coordinate system, θ I is the second horizontal angle of the external device relative to the remote control, the noise corresponding to the second horizontal angle, is the second elevation angle of the external device relative to the remote control, is the noise corresponding to the second elevation angle, represents the direction of the external device relative to the remote control.

[0267] In an alternative embodiment, the third adjustment strategy is the adjustment strategy indicated by the updated state observation equation in the extended Kalman filter;

[0268] The updated state observation equation is a state observation equation fabricated according to the noise characteristics.

[0269] In an alternative embodiment, the positioning unit 62 is specifically configured to:

[0270] calculate the coordinates of the remote control on the display device according to the target attitude and the actual position of the remote control at the second moment;

[0271] determine the pixel coordinates of the remote control on the display device according to the coordinates of the remote control on the display device; wherein, the position indicated by the pixel coordinates on the display device is the focus.

[0272] In an alternative embodiment, it further includes a processing unit, and the processing unit is configured to:

[0273] perform smoothing filtering on the pixel coordinates by applying a first-order filter; wherein, the filtering coefficient in the first-order filter is determined according to the rotational speed in the speed of the remote control.

[0274] Since this device is the device in the method of the embodiments of the present application, and the principle of the device to solve the problem is similar to that of the method, the implementation of this device can refer to the implementation of the method, and the repeated parts will not be elaborated.

[0275] As Figure 7 shown, based on the same inventive concept, the embodiments of the present application provide a remote control, which is applied to the remote control. The remote control includes a data transmission unit 71 and a processor 72.

[0276] The data transmission unit 71 is configured to execute:

[0277] acquire IMU data and UWB data;

[0278] The processor 72 is configured to execute:

[0279] If IMU data is detected at the first moment, apply the IMU data and the first adjustment strategy to adjust the first state information of the remote control at the first moment to obtain the second state information at the second moment; wherein, the second state information includes the reference position, reference attitude and reference speed at the second moment, and the bias information at the first moment; the bias information includes accelerometer bias information and gyroscope bias information;

[0280] If UWB data is detected at the second moment, the UWB data and the second adjustment strategy are applied to adjust the second state information to obtain the target state information at the second moment; wherein, the target state information includes the target position, target attitude, target speed, and target offset information at the second moment; the first adjustment strategy and the second adjustment strategy are determined according to a pre-determined extended Kalman filter;

[0281] Apply the target attitude and the actual position of the remote control at the second moment to calculate the focus of the remote control on the display device; wherein, the actual position of the remote control at the second moment is determined according to the UWB data detected at the second moment. In an alternative embodiment, the processor 72 is further configured to:

[0282] If UWB data is not detected at the second moment, apply a third adjustment strategy to adjust the second state information to obtain the target state information at the second moment; wherein, the actual position of the remote control at the second moment is determined according to the position calculated based on the UWB data at the first moment, the IMU data at the first moment, and the IMU data at the second moment of the remote control.

[0283] In an alternative embodiment, the first adjustment strategy is the adjustment strategy indicated by the state propagation equation in the extended Kalman filter;

[0284] The processor 72 is specifically configured to:

[0285] Calculate the state transition matrix and the process noise driving matrix of the state propagation equation according to the first state information of the remote control at the first moment;

[0286] Apply the angular velocity and acceleration in the IMU data, and based on the state transition matrix and the process noise driving matrix, perform a state propagation operation on the position, attitude, and speed in the first state information to obtain the second state information.

[0287] In an alternative embodiment, the state propagation equation includes:

[0288]

[0289] wherein, t is the first moment, t + τ is the second moment, b is the coordinate system where the remote control is located, and w is the world coordinate system;

[0290] is the rotation of the remote control at the second moment, is the rotation of the remote control at the first moment, ω b the angular velocity of the remote control, b ω is the angular velocity bias, n ω is the noise corresponding to the angular velocity;

[0291] is the speed of the remote control at the second moment, is the speed of the remote control at the first moment, is the set rotation matrix, a b is the acceleration of the remote control, n a is the noise corresponding to the acceleration, g w is the direction of gravity;

[0292] is the reference position of the remote control at the second moment, is the reference position of the remote control at the first moment;

[0293] b a,t+τ is the acceleration bias at the second moment, b a,t is the acceleration bias at the first moment, is the noise corresponding to the acceleration bias;

[0294] b ω,t+τ is the angular velocity bias at the second moment, b ω,t is the angular velocity bias at the first moment, is the noise corresponding to the angular velocity bias.

[0295] In an alternative embodiment, the second adjustment strategy is the adjustment strategy indicated by the state observation equation in the extended Kalman filter;

[0296] The processor 72 is specifically configured to:

[0297] Calculate the observation matrix and the process noise covariance matrix of the state observation equation according to the second state information;

[0298] Apply the distance, pitch angle, and horizontal angle in the UWB data, and perform a state observation operation on the second state information based on the observation matrix and the process noise covariance matrix to obtain the target state information.

[0299] In an alternative embodiment, the state observation equation includes:

[0300]

[0301] Wherein, is the conversion matrix from the world coordinate system to the external device UWB coordinate system, is the reference position of the remote control at the first moment, d is the distance between the external device UWB unit and the remote control UWB unit, n d is the noise corresponding to the distance, θ R is the first horizontal angle of the remote control relative to the external device, is the noise corresponding to the first horizontal angle, is the first elevation angle of the remote control relative to the external device, is the noise corresponding to the first elevation angle, represents the direction of the remote control relative to the external device; wherein, the external device is a display device or a Dongle device communicating with the display device;

[0302] is the transformation matrix from the IMU coordinate system of the remote control to the coordinate system of the display device, is the transformation matrix from the IMU coordinate system of the remote control to the world coordinate system, θ I is the second horizontal angle of the external device relative to the remote control, the noise corresponding to the second horizontal angle, is the second elevation angle of the external device relative to the remote control, is the noise corresponding to the second elevation angle, represents the direction of the external device relative to the remote control.

[0303] In an alternative embodiment, the third adjustment strategy is the adjustment strategy indicated by the updated state observation equation in the extended Kalman filter;

[0304] The updated state observation equation is a state observation equation fabricated according to the noise characteristics.

[0305] In an alternative embodiment, the processor 72 is specifically configured to:

[0306] Calculate the coordinates of the remote control on the display device according to the target attitude and the actual position of the remote control at the second moment;

[0307] Determine the pixel coordinates of the remote control on the display device according to the coordinates of the remote control on the display device; wherein, the position indicated by the pixel coordinates on the display device is the focus.

[0308] In an alternative embodiment, it further includes a processing unit, and the processor 72 is configured to:

[0309] Apply a first-order filter to perform smoothing filtering on the pixel coordinates; wherein, the filtering coefficient in the first-order filter is determined according to the rotational speed in the speed of the remote control.

[0310] The embodiment of the present application further provides a computer storage medium, in which computer program instructions are stored. When the instructions run on a computer, the computer is made to execute the steps of the above-mentioned positioning method of the remote control.

[0311] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, a system, or a computer program product. Therefore, the present application can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0312] The present application is described with reference to the flowcharts and / or block diagrams of methods, apparatuses (systems), and computer program products according to the present application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, as well as the combination of flows and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to generate a machine, such that the instructions executed by the processor of the computer or other programmable data processing devices generate means for implementing the functions specified in one or more flows in the flowchart and / or one or more blocks in the block diagram.

[0313] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, such that the instructions stored in the computer-readable memory generate a manufactured article including an instruction device that implements the functions specified in one or more flows in the flowchart and / or one or more blocks in the block diagram.

[0314] These computer program instructions can also be loaded onto a computer or other programmable data processing device, such that a series of operation steps are executed on the computer or other programmable device to generate a computer-implemented process, and thus the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one or more flows in the flowchart and / or one or more blocks in the block diagram.

[0315] Obviously, those skilled in the art can make various modifications and variations to the present application without departing from the scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalent technologies, the present application is also intended to include these modifications and variations.

Claims

1. A positioning method for a remote control, characterized in that, Applied to a remote controller, the method includes: If IMU data is detected at a first moment, applying the IMU data and a first adjustment strategy to adjust first state information of the remote controller at the first moment to obtain second state information at a second moment; wherein, the second state information includes a reference position, a reference attitude, and a reference speed at the second moment, and bias information at the first moment; the bias information includes accelerometer bias information and gyroscope bias information; If UWB data is detected at the second moment, applying the UWB data and a second adjustment strategy to adjust the second state information to obtain target state information at the second moment; wherein, the target state information includes a target position, a target attitude, a target speed, and target bias information at the second moment; the first adjustment strategy and the second adjustment strategy are determined according to a pre-determined extended Kalman filter; Calculating a focus of the remote controller on a display device by using the target attitude and an actual position of the remote controller at the second moment; wherein, the actual position of the remote controller at the second moment is determined according to the UWB data detected at the second moment.

2. The method according to claim 1, wherein If UWB data is not detected at the second moment, the method further includes: Applying a third adjustment strategy to adjust the second state information to obtain the target state information at the second moment; wherein, the actual position of the remote controller at the second moment is determined according to a position calculated based on the UWB data at the first moment, the IMU data at the first moment, and the IMU data at the second moment of the remote controller.

3. The method according to claim 1, wherein The first adjustment strategy is the adjustment strategy indicated by a state propagation equation in the extended Kalman filter; The applying the IMU data and the first adjustment strategy to adjust the first state information of the remote controller at the first moment to obtain the second state information at the second moment includes: Calculating a state transition matrix and a process noise driving matrix of the state propagation equation according to the first state information of the remote controller at the first moment; Applying the angular velocity and acceleration in the IMU data, and performing a state propagation operation on the position, attitude, and speed in the first state information based on the state transition matrix and the process noise driving matrix to obtain the second state information.

4. The method according to claim 3, characterized in that, The state propagation equation includes: wherein, t is the first moment, t + τ is the second moment, b is the coordinate system where the remote controller is located, and w is the world coordinate system; For the rotation of the remote controller at the second moment, For the rotation of the remote controller at the first moment, ω b The angular velocity of the remote controller, b ω Is the angular velocity bias, n ω Is the noise corresponding to the angular velocity; is the speed of the remote controller at the second moment, is the speed of the remote controller at the first moment, is the set rotation matrix, a b is the acceleration of the remote controller, n a is the noise corresponding to the acceleration, g w is the direction of gravity; is the reference position of the remote controller at the second moment, is the reference position of the remote controller at the first moment; b a,t+τ is the acceleration bias at the second moment, b a,t is the acceleration bias at the first moment, is the noise corresponding to the acceleration bias; b ω,t+τ is the angular velocity bias at the second moment, b ω,t is the angular velocity bias at the first moment, is the noise corresponding to the angular velocity bias.

5. The method according to claim 1, wherein The second adjustment strategy is the adjustment strategy indicated by a state observation equation in the extended Kalman filter; The applying the UWB data and the second adjustment strategy to adjust the second state information to obtain the target state information at the second moment includes: Calculating an observation matrix and a process noise covariance matrix of the state observation equation according to the second state information; Applying the distance, pitch angle, and horizontal angle in the UWB data, and performing a state observation operation on the second state information based on the observation matrix and the process noise covariance matrix to obtain the target state information.

6. The method according to claim 5, characterized in that The state observation equation includes: Among them, is the conversion matrix from the world coordinate system to the external device UWB coordinate system, is the reference position of the remote control at the first moment, d is the distance between the external device UWB unit and the remote control UWB unit, and n d is the noise corresponding to the distance, and θ R is the first horizontal angle of the remote control relative to the external device, is the noise corresponding to the first horizontal angle, is the first elevation angle of the remote control relative to the external device, is the noise corresponding to the first elevation angle, represents the direction of the remote control relative to the external device; among them, the external device is the display device or a Dongle device communicating with the display device; is the transformation matrix from the IMU coordinate system of the remote control to the coordinate system of the display device, is the transformation matrix from the IMU coordinate system of the remote control to the world coordinate system, θ I is the second horizontal angle of the external device relative to the remote control, the noise corresponding to the second horizontal angle, is the second altitude angle of the external device relative to the remote control, is the noise corresponding to the second altitude angle, ) represents the direction of the external device relative to the remote control.

7. The method according to claim 2, wherein The third adjustment strategy is the adjustment strategy indicated by the updated state observation equation in the extended Kalman filter; The updated state observation equation is a state observation equation fabricated according to the noise characteristics.

8. The method according to any one of claims 1, characterized in that, Applying the target attitude and the actual position of the remote controller at the second moment to calculate the focus of the remote controller on the display device includes: Calculating the coordinates of the remote controller on the display device according to the target attitude and the actual position of the remote controller at the second moment; Determining the pixel coordinates of the remote controller on the display device according to the coordinates of the remote controller on the display device; wherein, the position indicated by the pixel coordinates on the display device is the focus.

9. The method according to any one of claims 1 to 8, characterized in that, The method further includes: Applying a first-order filter to perform smoothing filtering on the pixel coordinates; wherein, the filtering coefficient in the first-order filter is determined according to the rotational speed in the speed of the remote controller.

10. A remote control, characterized in that, Applied to a remote controller, the remote controller includes a data transmission unit and a processor; The data transmission unit is configured to execute: Obtain IMU data and UWB data; The processor is configured to execute: If IMU data is detected at the first moment, applying the IMU data and the first adjustment strategy to adjust the first state information of the remote controller at the first moment to obtain the second state information at the second moment; wherein, the second state information includes the reference position, reference attitude, and reference speed at the second moment, and the bias information at the first moment; the bias information includes accelerometer bias information and gyroscope bias information; If UWB data is detected at the second moment, applying the UWB data and the second adjustment strategy to adjust the second state information to obtain the target state information at the second moment; wherein, the target state information includes the target position, target attitude, target speed, and target bias information at the second moment; the first adjustment strategy and the second adjustment strategy are determined according to a pre-determined extended Kalman filter; Applying the target attitude and the actual position of the remote controller at the second moment to calculate the focus of the remote controller on the display device; wherein, the actual position of the remote controller at the second moment is determined according to the UWB data detected at the second moment.

Images