Remote control method and remote control device

By combining IMU and OMT sensors, the coordinates of the remote control device are measured in real time and the position of the TV cursor is updated, which solves the problem of insufficient cursor movement accuracy on smart TVs, and achieves a low-cost and high-precision remote control effect.

CN118301396BActive Publication Date: 2024-11-01HONOR DEVICE CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410527018.0
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-04-29
Publication Date
2024-11-01
Estimated Expiration
2044-04-29

AI Technical Summary

Technical Problem

It is difficult for existing remote control devices to quickly move the cursor on smart TVs, especially when selecting web page content or entering information, traditional key remote control devices cannot meet the user's interaction needs, and there is a cumulative error in IMU measurement, which affects the accuracy.

Method used

A low-cost optical dynamic tracking OMT sensor is introduced. Through the combination of IMU and OMT sensors, the coordinates of remote control devices are measured in real time, accumulated errors are improved, and coordinates are sent to the TV for update cursor position through wireless communication.

Benefits of technology

It effectively improves the cumulative error caused by the IMU measurement position, improves the accuracy and user experience of the remote control device, and reduces costs, avoiding the use of high-cost UWB modules and antennas.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118301396B_ABST
    Figure CN118301396B_ABST
Patent Text Reader

Abstract

The embodiments of the present application provide a remote control method and a remote control device. The remote control device may include: an IMU, a first OMT sensor, and a second OMT sensor; the first OMT sensor is arranged on the opposite side of the user operation surface, and the second OMT sensor is arranged on the surface perpendicular to the user operation surface. The remote control method may include: when the acceleration change of the remote control device is detected by the IMU, the position at the time of the acceleration change may be calibrated as the coordinate origin, and the displacement of large-amplitude movement may be measured by the OMT sensor; at the same time, the relative displacement of small-amplitude movement may be measured by the IMU. The coordinates of the remote control device are determined by combining the measurement results of the OMT sensor and the measurement results of the IMU, and finally the coordinates are sent to the TV to update the cursor position on the TV screen. In this way, positioning can be achieved at a low cost, and the influence of the IMU cumulative error can also be reduced.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of electronic technology, and in particular to a remote control method and a remote control device. Background Art

[0002] With the development of smart TV technology, the interaction needs between users and TV sets can no longer be met by traditional button remote control devices. Users need to quickly move the cursor to select TV content, such as selecting web content, selecting products on smart TVs, and even inputting information on TV. How to achieve the control of the cursor movement of a TV set through a remote control device is a problem that technicians in this field are concerned about. Summary of the invention

[0003] The embodiments of the present application provide a remote control method and a remote control device, which can improve the cumulative error caused by measuring the position only by IMU, and only requires the introduction of low-cost OMT sensors, which has cost advantages.

[0004] In the first aspect, the embodiment of the present application provides a remote control method, which can be applied to a remote control device. The remote control device may include: an inertial measurement unit IMU, a first optical dynamic tracking OMT sensor and a second OMT sensor; the remote control device includes a first surface and a second surface arranged opposite to each other, the first surface is a user operation surface, the second surface is provided with the first OMT sensor, and the remote control device also includes a third surface, the third surface is perpendicular to the first surface, and the third surface is provided with the second OMT sensor.

[0005] The remote control method may include the following steps:

[0006] The remote control device detects the acceleration change of the remote control device through the IMU;

[0007] The remote control device calibrates the position where the acceleration changes as the coordinate origin;

[0008] Based on the coordinate origin, the remote control device controls the first OMT sensor to measure the X and Y coordinate values ​​of the remote control device, and controls the second OMT sensor to measure the Z coordinate value of the remote control device;

[0009] The remote control device also controls the IMU to measure the displacement of the remote control device;

[0010] The remote control device determines the coordinates of the remote control device according to the coordinates output by the first OMT sensor and the second OMT sensor and the movement displacement measured by the IMU;

[0011] The remote control device sends the coordinates of the remote control device to the television set, and the television set updates the cursor position on the screen of the television set according to the coordinates of the remote control device.

[0012] In the first aspect, the first OMT sensor and the second OMT sensor may be the first OMT sensor and the second OMT sensor mentioned in the embodiment, respectively, the IMU may be the IMU mentioned in the embodiment, and the first surface, the second surface, and the third surface may be the first surface 21, the second surface 22, and the third surface 23 mentioned in the subsequent embodiments, respectively. The remote control device may be the remote control device mentioned in the subsequent embodiments, and the television may be the television mentioned in the subsequent embodiments.

[0013] In the first aspect, the remote control device measures the large-scale movement of the remote control device by introducing an OMT sensor, and combines it with an IMU to measure the small-scale movement of the remote control device, thereby feeding back the movement of the remote control device to the TV in real time, so that the TV updates the cursor position according to the latest coordinates of the remote control device. In this way, the cumulative error caused by measuring the position only by the IMU can be improved, and only low-cost OMT sensors need to be introduced, which has cost advantages.

[0014] In combination with the first aspect, in some embodiments, the remote control device can specifically control the first OMT sensor to measure the X, Y coordinate values ​​of the remote control device when performing the first movement, and control the second OMT sensor to measure the Z coordinate value of the remote control device when performing the first movement. The remote control device can also specifically control the IMU to measure the movement displacement of the remote control device when performing the second movement. The displacement of the first movement is greater than the first displacement, and the displacement of the second movement is less than or equal to the first displacement.

[0015] That is, the first movement is a large-range movement, and the second movement is a small-range movement; the first OMT sensor and the second OMT sensor can be used to measure position information for large-range movement, and the IMU can be used to measure position information for small-range movement.

[0016] In combination with the first aspect, in some embodiments, in order to identify whether the remote control device is moving over a large range, the remote control device can also detect the displacement amount during each movement, and determine the movement with a displacement greater than the first displacement amount as the first movement, and determine the movement with a displacement less than or equal to the first displacement amount as the second movement.

[0017] In combination with the first aspect, in some embodiments, the remote control device determines the coordinates of the remote control device based on the coordinates output by the first OMT sensor and the second OMT sensor, and the movement displacement measured by the IMU. Specifically, it may include: the remote control device can obtain the coordinates of the remote control device based on the coordinates output by the first OMT sensor and the second OMT sensor, and superimpose the movement displacement measured by the IMU.

[0018] In combination with the first aspect, in some embodiments, the remote control device can control the first OMT sensor to capture a first image, control the second OMT sensor to capture a second image, and determine X, Y coordinate values ​​based on the first image, and determine the Z coordinate value based on the second image.

[0019] The coordinates of the remote control device may include three-dimensional coordinate values ​​X, Y, and Z, wherein X and Y may be determined based on the image content of the first image captured by the first OMT sensor and output by the first OMT sensor; Z may be determined based on the image content of the second image captured by the second OMT sensor and output by the second OMT sensor. The first image refers to the first image captured by the first OMT sensor, and the second image refers to the photo captured by the second OMT sensor. Specifically, the processor 35 may start the first OMT sensor and the second OMT sensor after the IMU detects a change in acceleration, and control the OMT sensors to capture images. Accordingly, if the IMU does not detect a change in acceleration for a long time, the remote control device may turn off the OMT sensor to save power consumption.

[0020] In combination with the first aspect, in some embodiments, the remote control device can determine X and Y based on the first image in the following manner: the remote control device can determine the coordinate origins of the X-axis and the Y-axis using the first image when the acceleration changes, that is, calibrate the first image when the acceleration changes as the coordinate origin of the X-axis and the Y-axis; then, determine X and Y based on the image content difference between the first image at the first moment and the first image when the acceleration changes. The acceleration change may be a moment earlier than the first moment, such as the moment when the first frame of the first image is captured after the acceleration change is detected. In other words, X and Y can be specifically determined based on the image content difference between the first image at the first moment and the first image when the acceleration changes, wherein the first image when the acceleration changes is used to calibrate the coordinate origins of the X-axis and the Y-axis.

[0021] In combination with the first aspect, in some embodiments, the difference in image content between the first image at the first moment and the first image when the acceleration changes can be specifically used to determine the image displacement of the first object in the first image at the first moment relative to the first object when the acceleration changes, and the image displacement is directly used to determine X and Y. Here, the first object can be an object that always exists in the first image from the time when the acceleration changes to the first moment, and the displacement of the remote control device on the X axis and the Y axis can be calculated through its image displacement. For example, for multiple first images taken by the first OMT sensor, the first object can be a piece of floor or the user's feet captured on the ground, because when the user holds the remote control device in a standard posture, the first OMT sensor faces the ground.

[0022] In combination with the first aspect, in some embodiments, the remote control device may determine Z based on the second image in the following manner: the remote control device may determine the coordinate origin of the Z axis using the second image when the acceleration changes, that is, calibrate the second image when the acceleration changes as the coordinate origin of the Z axis; and then determine Z based on the image content difference between the second image at the first moment and the second image when the acceleration changes. In other words, Z may be specifically determined based on the image content difference between the second image at the first moment and the second image when the acceleration changes, wherein the second image when the acceleration changes is used to calibrate the coordinate origin of the Z axis.

[0023] In combination with the first aspect, in some embodiments, the image content difference between the second image at the first moment and the second image when the acceleration changes can be specifically used to determine the image displacement of the second object in the second image at the first moment relative to the second object when the acceleration changes, and the image displacement is directly used to determine Z. Here, the second object can be an object that always exists in the second image from the time when the acceleration changes to the first moment, and the displacement of the remote control device on the Z axis can be calculated through its image displacement. For example, for the multiple second images taken by the second OMT sensor, the second object can be the front abdomen of the user or the TV in front of the user.

[0024] In combination with the first aspect, in some embodiments, the remote control device may specifically mark the position at the starting moment when the acceleration change occurs as the coordinate origin.

[0025] In combination with the first aspect, in some embodiments, in order to reduce the cumulative error of the IMU, at a set first time, if the displacement of the remote control device is greater than a first threshold, the remote control device controls the IMU to perform initialization calibration.

[0026] In the second aspect, the present application provides a remote control device, which may include one or more memories, one or more processors; the remote control device may also include: an inertial measurement unit IMU, a first optical dynamic tracking OMT sensor and a second OMT sensor. The remote control device may include a first surface and a second surface arranged opposite to each other, the first surface is a user operation surface, and the second surface is provided with a first OMT sensor. The remote control device may also include a third surface, the third surface is perpendicular to the first surface, and the third surface is provided with a second OMT sensor. The memory is coupled to one or more processors, the memory is used to store a computer program, and the processor calls the computer program to implement the method described in the first aspect and any possible implementation method of the first aspect.

[0027] In a third aspect, the present application provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the method described in the first aspect and any possible implementation manner of the first aspect is implemented.

[0028] In a fourth aspect, the present application provides a computer program product, comprising a computer program, which, when executed by a processor, implements the method described in the first aspect and any possible implementation manner of the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 A remote control system is shown;

[0030] Figure 2 A scheme for controlling a cursor of a television set using a remote control device is shown;

[0031] Figure 3 The hardware architecture of a remote control device provided by an embodiment of the present application is shown;

[0032] Figure 4 The measurement principle of the OMT sensor is shown;

[0033] Figure 5 To illustrate the appearance of a remote control device provided by an embodiment of the present application;

[0034] Figure 6 A standard posture for holding a remote control device is shown;

[0035] Figure 7 An application scenario of using the remote control device provided by the present application to control the cursor of a television is shown;

[0036] Figure 8 The remote control device provided in the embodiment of the present application is shown Figure 7 Hardware interaction process of application scenarios;

[0037] Fig. 9 The remote control device provided in the embodiment of the present application is shown Figure 7 Software flow of application scenarios;

[0038] Fig.10 The overall process of a remote control method provided by an embodiment of the present application is shown;

[0039] Fig.11 The figure shows the method flow of calibrating IMU provided in an embodiment of the present application. DETAILED DESCRIPTION

[0040] The terms used in the following embodiments of the present application are only for the purpose of describing specific embodiments and are not intended to limit the present application.

[0041] Figure 1 A remote control system is shown. Figure 1As shown, the remote control system may include a television 10 and a remote control device 20. The remote control device 20 may be configured with one or more buttons, such as a power button 11, a confirmation button (OK) 12, a return button 13, a home button 14, a menu button 15, a volume up button 16, a volume down button 17, and a left direction button 12c, a right direction button 12d, an up direction button 12a, a down direction button 12b, etc. The remote control device 20 may communicate with the television 10 via a communication method such as Bluetooth to transmit a key signal to the television 10. The key signal may be generated by triggering a key operation on the remote control device 20 by a user.

[0042] In addition to controlling the TV set 10 by buttons, the remote control device 20 can also detect its own position change in real time and transmit the position change to the TV set 10 in a timely manner, so that the TV set 10 determines the position of the cursor 100 on the screen according to the position of the remote control device 20. In this way, the user can control the cursor 100 on the TV set 10 by rotating and moving the remote control device 20.

[0043] The television set 10 may be a matching television set for the remote control device 20 , and may map the coordinates of the remote control device 20 transmitted from the remote control device 20 into the position of the cursor 100 on the screen.

[0044] One solution for controlling the cursor 100 of the TV 10 by using a remote control device is to detect the displacement of the remote control device by an inertial measurement unit (IMU) in the remote control device 20 to achieve positioning, such as detecting the relative position by a six-axis IMU, and then the TV 10 determines the cursor position based on the positioning. However, the disadvantage of this solution is that the IMU has cumulative errors, which will affect the measurement accuracy.

[0045] Another solution for controlling the cursor of the TV set 10 using a remote control device is as follows: Figure 2 As shown, the remote control device 20 is configured with an ultra wide band (UWB) module and an IMU at the same time, and the absolute position of the remote control device measured by the UWB module and the displacement of the remote control device measured by the IMU are used to achieve positioning, and then the TV 10 determines the cursor position based on the positioning. Specifically, the remote control device measures its own position through UWB, and then measures the displacement through the IMU to determine the coordinates of the remote control device after movement; then the remote control device can send the coordinates to the TV 10 through low-power Bluetooth (bluetooth low energy, BLE), so that the TV 10 maps the coordinates to the cursor position. However, this solution is costly, and requires the configuration of UWB modules and miniaturized antennas in both the TV 10 and the remote control device, and UWB also causes high power consumption.

[0046] To solve some or all of the above problems, embodiments of the present application provide a remote control device and a remote control method.

[0047] Figure 3 and Figure 5 A remote control device 20 provided in an embodiment of the present application is shown.

[0048] like Figure 3 As shown, the remote control device 20 may include: an inertial measurement unit IMU 31, an optical dynamic tracking OMT sensor 32, an optical dynamic tracking OMT sensor 33, one or more buttons 34, a processor 35, a memory 40, a wireless communication module 37, and an antenna 39. The various parts of the remote control device 20 may be connected via a bus.

[0049] Among them, IMU 31 can be used to detect acceleration, tilt, impact, vibration, rotation and multi-degree-of-freedom motion. IMU 31 can refer to a device that uses an accelerometer and a gyroscope to measure the single-axis, dual-axis or three-axis attitude angle (or angular rate) and acceleration of an object. IMU 31 can be, for example, a 6-axis IMU sensor, which can be equipped with a gyroscope and an accelerometer on three orthogonal axes, with a total of 6 degrees of freedom, to measure the angular velocity and acceleration of an object in three-dimensional space; IMU can also be a 9-axis IMU, that is, a magnetometer can be added to the 6-axis IMU, which can further improve the measurement accuracy of the heading angle.

[0050] OMT sensors 32 and 33 can be used to detect the movement of objects in a large range (e.g., 80 mm to infinity). OMT sensors can continuously and dynamically measure the coordinate information of moving objects. OMT sensors can include a shooting module and a calculation module, which can be used to detect the coordinate value of the remote control device 20. The main working principle of the OMT sensor is to use a low-resolution high-speed camera to take pictures and compare the two frames before and after to calibrate the displacement and direction. Figure 4 As shown, the OMT sensor captures a first frame of chessboard image and uses it to calibrate the coordinate origin; when the chessboard moves, the OMT sensor captures its second frame of image and outputs the coordinates based on the image content difference between the second frame and the first frame. For example, the image content difference can be used to determine the image displacement of the chessboard in the first frame relative to the chessboard in the second frame, and the image displacement can be used to determine the X-axis, Y-axis, or Z-axis coordinates of the chessboard relative to the origin.

[0051] In the embodiment of the present application, the OMT sensor can be controlled to capture images in real time when the remote control device 20 moves. The OMT sensor may include a capturing module and a computing module. Based on the image frames captured in real time by the capturing module, the computing module may determine the origin of the measurement coordinate system and determine the real-time coordinates of the remote control device 20. Finally, the remote control device may feed back the coordinates output by the OMT sensor to the TV 10.

[0052] In the embodiment of the present application, the OMT sensor may be any optical sensor (also called a camera) that can measure the position information of a moving object by continuously capturing multiple images.

[0053] like Figure 5 As shown, the remote control device 20 may include a first surface 21 and a second surface 22 disposed opposite to each other. The first surface 21 may be a user operation surface, on which components for receiving user operations may be arranged, such as a button 34, such as a menu button, a volume button, etc. The button 34 may be a physical button or a virtual button displayed on a touch screen. Figure 5 As shown, the OMT sensor 32 may be disposed on the second surface 22. The battery cover of the remote control device 20 may also be disposed on the second surface 22. Generally, the first surface 21 may be referred to as the front surface of the remote control device 20, and the second surface 22 may be referred to as the back surface of the remote control device 20. Figure 6 2 shows a standard posture of a user holding the remote control device 20. Figure 6 As shown, when the user holds the remote control device 20 in a standard posture, the first surface 21 faces upward (such as the sky or ceiling above), which facilitates the user's fingers (especially the thumb) to operate the buttons on the first surface 21; at this time, the second surface 22 faces downward (such as the bottom surface) and is held by the user's palm. Figure 5 As shown, the remote control device 20 may further include a third surface 23, the third surface 23 may be perpendicular to the first surface 21, and the OMT sensor 33 may be disposed on the third surface 23. Figure 5 When the remote control device 20 is held in the standard posture shown, the third surface 23 may face the user, such as opposite to the user's abdomen; but not limited to this, the third surface 23 may also be the side facing the TV 10, that is, the OMT sensor 33 may be set on the side facing the TV 10.

[0054] Specifically, the image captured by the OMT sensor 32 can be used to determine the X-axis and Y-axis coordinates of the moving object, and the image captured by the OMT sensor 33 can be used to determine the Z-axis coordinate of the moving object. This will be described in detail in subsequent embodiments and will not be expanded here. The OMT sensors 32 and 33 can use a high frame rate to perform fast and accurate object tracking.

[0055] In the embodiment of the present application, the moving object is relative to the remote control device 20, which means an object that has moved relative to the remote control device 20, not an object that has moved absolutely relative to the earth's coordinate system. For example, an absolutely stationary object relative to the moving remote control device 20 is also a moving object. In the application scenarios involved in the embodiments of the present application, the TV 10, the ground, the roof, etc. are usually stationary, and the user controls the cursor by moving the remote control device 20. In this example, the stationary TV 10, the ground, the roof, etc. can be moving objects relative to the moving remote control device 20.

[0056] The processor 35 may be a micro controller unit (MCU). The processor 35 may include one or more interfaces, such as an inter-integrated circuit (I2C) interface, a serial peripheral interface (SPI), a general purpose I / O (GPIO) interface, etc. For example, the processor 35 may communicate with the OMT sensor and the IMU using an SPI interface, the processor 35 may communicate with the button 34 using a GPIO interface, the processor 35 may communicate with the PMU 36 via a power management bus, and the processor 35 may communicate with the touch panel 38 using an I2C bus.

[0057] The memory 40 can store the computer program / instructions of the remote control method provided in the embodiment of the present application. When the processor 35 reads the computer program / instructions from the memory 40 and runs the computer program / instructions, the remote control device 20 can execute the steps of the remote control method provided in the embodiment of the present application.

[0058] The wireless communication module 37 may include a Bluetooth module, a wireless fidelity (Wi-Fi) module, a ZigBee module, etc. The wireless communication module 37 may be used to communicate with a television (such as a television set) that is compatible with the remote control device 20. Figure 1 The data may include coordinate information of the remote control device 20, such as the X-axis, Y-axis, and Z-axis coordinates of the remote control device 20. In this way, the television 10 can determine the position of the cursor 100 according to the coordinate information, thereby updating the cursor position on the screen.

[0059] The antenna 39 can be coupled to the wireless communication module 37, so that the remote control device 20 can communicate with other devices through wireless communication technology. The wireless communication module 37 receives electromagnetic waves via the antenna 39, modulates and filters the electromagnetic wave signals, and sends the processed signals to the processor 35. The wireless communication module 37 can also receive the signal to be sent from the processor 35, modulate the frequency of the signal, amplify it, and convert it into electromagnetic waves for radiation through the antenna 39.

[0060] Figure 3 The schematic structure does not constitute a specific limitation on the remote control device 20. The remote control device 20 may include more or fewer parts than shown in the figure, or combine some parts, or separate some parts, or arrange the parts differently. The various parts shown in the figure may be implemented in hardware, software, or a combination of software and hardware. For example, Figure 3 As shown, the remote control device 20 may further include a power management unit (PMU) 36 .

[0061] Figure 7 FIG. 2 shows an application scenario in which a user uses the remote control device 20 provided in an embodiment of the present application to control the cursor of the television 10. Figure 7 As shown, the user can pick up the remote control device 20 from a desktop or other places, and move or rotate the remote control device 20 to control the cursor 100 on the TV set 10 to move from one position to another. Figure 7 The dashed line indicates the position of the cursor or remote control device before the movement, and the solid line indicates the position of the cursor or remote control device after the movement.

[0062] Figure 8-Figure 9 It shows how the remote control device provided in the embodiment of the present application can be implemented Figure 7 Application scenario shown.

[0063] like Figure 8 as well as Fig. 9 As shown in S81 to S82 in FIG. 1 , when the user picks up or puts down the remote control from a desktop or other place, the IMU in the remote control device 20 can detect the acceleration change, and the position where the acceleration change occurs is the picking position. Figure 8 as well as Fig. 9 As shown in S83 in FIG. 1 , the OMT sensor 32 and the OMT sensor 33 can calibrate the pick-up position as the coordinate origin (x0, y0, z0). Figure 8 as well as Fig. 9 As shown in S84 in FIG. 1 , when the remote control device 20 moves, the OMT sensor 32 and the OMT sensor 33 can output the X, Y, and Z coordinates of the remote control device 20; at the same time, as shown in FIG. Figure 8 as well as Fig. 9As shown in S85 in FIG. 1 , the IMU can measure the relative position. Specifically, the OMT sensor can measure the position information of a large range of movement, and the IMU can measure the position information of a small range of movement. Then, as Figure 8 as well as Fig. 9 As shown in S86 in FIG. 1 , the OMT sensor and the IMU can transmit the measurement results to the processor 35 via the SPI bus, and the processor 35 can perform mixed calculations on the measurement results of the OMT sensor and the measurement results of the IMU to obtain the latest coordinates (x1, y1, z1) of the remote control device 20. Next, as Figure 8 as well as Fig. 9 As shown in S87 in FIG. 1 , the remote control device 20 can send the coordinates (x1, y1, z1) to the TV 10 via the wireless communication module 37 such as low-power Bluetooth. Here, the hybrid calculation may refer to: controlling the OMT sensor to measure the coordinate value of the remote control device 20 when performing a large displacement movement, and controlling the IMU to measure the displacement of the remote control device 20 when performing a small displacement movement; then, based on the coordinates output by the OMT sensor, the displacement measured by the IMU is superimposed to obtain the coordinates of the remote control device 20.

[0064] like Figure 8 as well as Fig. 9 As shown in S88 to S89, after receiving the coordinates through the communication module, the television 10 can map the cursor position through the processor, and determine the latest position of the cursor as (x1, y1, z1) based on the latest position (x1, y1, z1) of the remote control device 20. b , Y b ), and finally move the cursor 100 from position (X a , Y a ) moves to position (X b , Y b ), that is, update the cursor position.

[0065] like Fig. 9 As shown, specifically as shown in S90 to S91, if the IMU does not detect acceleration changes for a long time (such as 10 minutes), the remote control device 20 can enter the sleep mode and turn off high-power peripherals such as the OMT sensor, buttons, wireless communication module, touch panel, etc. to save power consumption.

[0066] The remote control device 20 provided in the embodiment of the present application introduces OMT sensors 32 and OMT sensors 33 to measure the large-scale movement of the remote control device, and combines IMU to measure the small-scale movement of the remote control device, so as to feed back the movement of the remote control device to the TV set 10 in real time, so that the TV set 10 updates the cursor position according to the latest coordinates of the remote control device 20. In this way, the cumulative error caused by measuring the position only by IMU can be improved, and by introducing low-cost OMT sensors, it has a significant cost advantage compared to UWB chips and antennas, and the software algorithm complexity is low.

[0067] The following describes the remote control method provided by the embodiments of the present application.

[0068] Fig.10 The overall process of a remote control method provided by an embodiment of the present application is shown. The method can be applied to the remote control device 20 described in the previous embodiment, and is described below.

[0069] S101 : The remote control device 20 may detect, through the IMU 31 , a change in acceleration of the remote control device 20 .

[0070] Specifically, the IMU 31 may send the acceleration information detected in real time to the processor 35 via an interface such as SPI or I2C.

[0071] S102: The remote control device 20 may calibrate the position where the acceleration changes as the coordinate origin.

[0072] For example, when the user picks up the remote control from a desktop or other place, the IMU 31 in the remote control device 20 can detect the acceleration change. The position when the acceleration change occurs is the picking-up position, and the images captured by the OMT sensor 32 and the OMT sensor 33 at the picking-up position can be calibrated as the coordinate origin (x0, y0, z0).

[0073] Specifically, the remote control device 20 may mark the position at the starting moment when the acceleration change occurs as the coordinate origin.

[0074] S103 , based on the coordinate origin, the remote control device 20 can control the OMT sensor 32 to measure the X and Y coordinate values ​​of the remote control device 20 , and control the OMT sensor 33 to measure the Z coordinate value of the remote control device 20 .

[0075] Specifically, the remote control device 20 may control the OMT sensor 32 to capture the first image, control the OMT sensor 33 to capture the second image, and determine the X, Y coordinate values ​​based on the first image, and determine the Z coordinate value based on the second image.

[0076] In the implementation of the present application, the coordinates of the remote control device may include three-dimensional coordinate values ​​X, Y, and Z, wherein X and Y may be determined based on the image content of the first image captured by the OMT sensor 32 and output by the OMT sensor 32; Z may be determined based on the image content of the second image captured by the OMT sensor 33 and output by the OMT sensor 33. The first image refers to the first image captured by the OMT sensor 32, and the second image refers to the photo captured by the OMT sensor 33. Specifically, the processor 35 may start the OMT sensor 32 and the OMT sensor 33 after the IMU 31 detects a change in acceleration, and control the OMT sensor to capture an image. Accordingly, if the IMU 31 does not detect a change in acceleration for a long time, the remote control device 20 may turn off the OMT sensor to save power consumption.

[0077] In some embodiments, the remote control device 20 can determine X and Y based on the first image in the following manner: the remote control device 20 can use the first image when the acceleration changes to determine the coordinate origins of the X-axis and the Y-axis, that is, calibrate the first image when the acceleration changes as the coordinate origins of the X-axis and the Y-axis; then, determine X and Y based on the image content difference between the first image at the first moment and the first image when the acceleration changes. The acceleration change may be a moment earlier than the first moment, such as the moment when the first frame of the first image is captured after the acceleration change is detected. In other words, X and Y can be specifically determined based on the image content difference between the first image at the first moment and the first image when the acceleration changes, wherein the first image when the acceleration changes is used to calibrate the coordinate origins of the X-axis and the Y-axis.

[0078] The image content difference between the first image at the first moment and the first image when the acceleration changes can be specifically used to determine the image displacement of the first object in the first image at the first moment relative to the first object when the acceleration changes, and the image displacement is directly used to determine X and Y. Here, the first object can be an object that always exists in the first image from the time when the acceleration changes to the first moment, and the displacement of the remote control device 20 on the X axis and the Y axis can be calculated through its image displacement. For example, for multiple first images taken by the OMT sensor 32, the first object can be a piece of floor or the user's feet on the ground, because when the user holds the remote control device 20 in a standard posture, the OMT sensor 32 faces the ground.

[0079] In some embodiments, the remote control device 20 may determine Z based on the second image in the following manner: the remote control device 20 may determine the coordinate origin of the Z axis using the second image when the acceleration changes, that is, calibrate the second image when the acceleration changes as the coordinate origin of the Z axis; and then determine Z based on the image content difference between the second image at the first moment and the second image when the acceleration changes. In other words, Z may be specifically determined based on the image content difference between the second image at the first moment and the second image when the acceleration changes, wherein the second image when the acceleration changes is used to calibrate the coordinate origin of the Z axis.

[0080] The image content difference between the second image at the first moment and the second image when the acceleration changes can be specifically used to determine the image displacement of the second object in the second image at the first moment relative to the second object when the acceleration changes, and the image displacement is directly used to determine Z. Here, the second object can be an object that always exists in the second image from the time when the acceleration changes to the first moment, and the displacement of the remote control device 20 on the Z axis can be calculated through its image displacement. For example, for the multiple second images taken by the OMT sensor 33, the second object can be the front abdomen of the user or the TV 10 in front of the user.

[0081] S104 : The remote control device 20 may also control the IMU 31 to measure the displacement of the remote control device 20 .

[0082] In the embodiment of the present application, the remote control device 20 can control the OMT sensor 32 to measure the X, Y coordinate values ​​of the remote control device 20 when performing the first movement, and control the OMT sensor 33 to measure the Z coordinate value of the remote control device 20 when performing the first movement; and, the IMU 31 can be controlled to measure the movement displacement of the remote control device 20 when performing the second movement. Among them, the displacement amount of the first movement is greater than the first displacement amount, and the displacement amount of the second movement is less than or equal to the first displacement amount. That is, the first movement is a large-range movement, and the second movement is a small-range movement; the OMT sensor can be used to measure the position information of a large-range movement, and the IMU can be used to measure the position information of a small-range movement.

[0083] In order to identify whether the remote control device 20 is moving in a large range, the remote control device 20 can also detect the displacement at each movement, determine the movement with a displacement greater than the first displacement as the first movement, and determine the movement with a displacement less than or equal to the first displacement as the second movement.

[0084] S105 : The remote control device 20 may determine the coordinates of the remote control device 20 according to the coordinates output by the OMT sensor 32 and the OMT sensor 33 and the displacement measured by the IMU 31 .

[0085] Specifically, the remote control device 20 may obtain the coordinates of the remote control device 20 based on the coordinates output by the OMT sensor 32 and the OMT sensor 33 and superimposed with the movement displacement measured by the IMU 31 .

[0086] S106 : The remote control device 20 may send the coordinates of the remote control device 20 to the television set, and the television set 10 may be used to update the cursor position on the screen of the television set 10 according to the coordinates of the remote control device 20 .

[0087] Specifically, after receiving the coordinates through the communication module, the TV 10 can map the cursor position through the processor, determine the latest position of the cursor 100 based on the latest position of the remote control device 20, and finally move the cursor 100 from the most recent historical position to the latest cursor position on the screen.

[0088] It can be seen that by implementing Fig. 9 In the method shown, the remote control device 20 avoids measuring the position of the remote control device only through the IMU 31, thereby reducing the impact of the IMU cumulative error on the measurement result; moreover, the introduction of low-cost OMT sensors to measure large-range movements combined with IMU measurement of small-range movements can reduce costs and simplify the algorithm.

[0089] Furthermore, in order to reduce the cumulative error of the IMU 31, the remote control device 20 provided in the embodiment of the present application can also use a timer to set a time. If a large-scale movement of the remote control device 20 is detected within the set time, the IMU can be controlled to enter initialization calibration. For the specific process, please refer to Fig.11 :

[0090] S91. The set time of the timer is reached.

[0091] S92: During the set time, determine whether the displacement of the remote control device 20 is greater than the first displacement. That is, determine whether a large-scale movement occurs. If yes, execute S93-S95; otherwise, execute S96.

[0092] S93 , measuring the coordinates of the remote control device 20 through the OMT sensor 32 and the OMT sensor 33 .

[0093] Regarding how to use the OMT sensor to measure the coordinates of the remote control device 20, please refer to the previous text, which will not be repeated here.

[0094] S94, control the IMU 31 to enter initialization calibration.

[0095] S95. The calibration of IMU31 is completed.

[0096] S96, determine the coordinates of the remote control device 20 through the coordinates measured by the OMT sensor 32, the OMT sensor 33 and the displacement measured by the IMU 31. For how to determine the coordinates of the remote control device 20 by combining the coordinates measured by the OMT sensor and the displacement measured by the IMU, please refer to the previous text, which will not be repeated here.

[0097] In the embodiment of the present application, the setting time of the timer can be a specific duration, such as 8 hours, that is, the IMU calibration is attempted every 8 hours; not limited thereto, the setting time can also be a specific time point, such as 12 o'clock every night, that is, the IMU calibration is attempted at 12 o'clock every night. In practical applications, the setting time can be selected according to needs, and the embodiment of the present application does not limit this.

[0098] The present application also provides a computer-readable storage medium, which stores a computer program (also referred to as code or instruction). When the computer program is executed, the computer executes the method executed by the electronic device in any of the above embodiments.

[0099] The various implementation modes of the present application can be combined arbitrarily to achieve different technical effects.

[0100] In the foregoing embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in this application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) mode. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk Solid State Disk), etc.

[0101] Those skilled in the art can understand that to implement all or part of the processes in the aforementioned embodiments, the processes can be completed by computer programs to instruct related hardware, and the programs can be stored in computer-readable storage media. When the programs are executed, they can include the processes in the aforementioned method embodiments. The aforementioned storage media include: ROM or random access memory RAM, magnetic disk or optical disk and other media that can store program codes.

[0102] In short, the above description is only an embodiment of the technical solution of the present invention, and is not intended to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made according to the disclosure of the present invention shall be included in the protection scope of the present invention.

Claims

1. A remote control method, characterized in that: The invention is applied to a remote control device, the remote control device comprises: an inertial measurement unit IMU, a first optical dynamic tracking OMT sensor and a second OMT sensor; the remote control device comprises a first surface and a second surface arranged opposite to each other, the first surface is a user operation surface, the second surface is provided with the first OMT sensor, the remote control device further comprises a third surface, the third surface is perpendicular to the first surface, the third surface is provided with the second OMT sensor, and the method comprises: The remote control device detects a change in acceleration of the remote control device through the IMU; The remote control device calibrates the position where the acceleration change occurs as the coordinate origin; Based on the coordinate origin, the remote control device controls the first OMT sensor to measure the X and Y coordinate values ​​of the remote control device, and controls the second OMT sensor to measure the Z coordinate value of the remote control device; The remote control device also controls the IMU to measure the displacement of the remote control device; The remote control device determines the coordinates of the remote control device according to the coordinates output by the first OMT sensor and the second OMT sensor and the movement displacement measured by the IMU; The remote control device sends the coordinates of the remote control device to the television set, and the television set is used to update the cursor position on the screen of the television set according to the coordinates of the remote control device.

2. The method according to claim 1, characterized in that The remote control device controls the first OMT sensor to measure the X and Y coordinate values ​​of the remote control device, and controls the second OMT sensor to measure the Z coordinate value of the remote control device, specifically including: The remote control device controls the first OMT sensor to measure the X and Y coordinate values ​​of the remote control device when the remote control device performs the first movement, and controls the second OMT sensor to measure the Z coordinate value of the remote control device when the remote control device performs the first movement; The remote control device further controls the IMU to measure the movement displacement of the remote control device, specifically including: The remote control device further controls the IMU to measure the displacement of the remote control device when performing a second movement; The displacement amount of the first movement is greater than the first displacement amount, and the displacement amount of the second movement is less than or equal to the first displacement amount.

3. The method according to claim 2, characterized in that The remote control device determines the coordinates of the remote control device according to the coordinates output by the first OMT sensor and the second OMT sensor and the movement displacement measured by the IMU, specifically including: The remote control device obtains the coordinates of the remote control device based on the coordinates output by the first OMT sensor and the second OMT sensor and superimposed with the movement displacement measured by the IMU.

4. The method according to claim 3, characterized in that The remote control device calibrates the position when the acceleration change occurs as the coordinate origin, which specifically includes: the remote control device calibrates the position at the starting moment when the acceleration change occurs as the coordinate origin.

5. The method according to claim 4, characterized in that Based on the coordinate origin, the remote control device controls the first OMT sensor to measure the X and Y coordinate values ​​of the remote control device, and controls the second OMT sensor to measure the Z coordinate value of the remote control device, specifically including: The remote control device controls the first OMT sensor to capture a first image, controls the second OMT sensor to capture a second image, and determines the X, Y coordinate values ​​based on the first image, and determines the Z coordinate value based on the second image.

6. The method according to claim 5, characterized in that The determining the X, Y coordinate values ​​based on the first image specifically includes: The X and Y coordinate values ​​are determined based on the image content difference between the first image at a first moment and the first image when the acceleration change occurs, wherein the first moment is later than when the acceleration change occurs.

7. The method according to claim 6, characterized in that The image content difference between the first image at the first moment and the first image when the acceleration change occurs is used to determine the image displacement of the first object in the first image at the first moment relative to the first object when the acceleration change occurs, and the image displacement is used to determine the X, Y coordinate values.

8. The method according to claim 5, characterized in that The determining the Z coordinate value based on the second image specifically includes: The remote control device determines the Z coordinate value based on an image content difference between the second image at a first time point later than when the acceleration change occurs and the second image when the acceleration change occurs.

9. The method according to claim 8, characterized in that The image content difference between the second image at the first moment and the second image when the acceleration change occurs is used to determine the image displacement of the second object in the second image at the first moment relative to the second object when the acceleration change occurs, and the image displacement is used to determine the Z coordinate value.

10. The method according to any one of claims 1 to 9, characterized in that Also includes: The remote control device detects a displacement amount of the remote control device.

11. The method according to claim 1, characterized in that: The method further comprises: At a set first time, if the displacement of the remote control device is greater than a first threshold, the remote control device controls the IMU to perform initialization calibration.

12. The method according to claim 1, characterized in that The first surface is provided with one or more buttons, and the buttons are used to receive user operations.

13. A remote control device, characterized in that: The remote control device includes one or more memories and one or more processors; the remote control device also includes: an inertial measurement unit IMU, a first optical dynamic tracking OMT sensor and a second OMT sensor; the remote control device includes a first surface and a second surface arranged opposite to each other, the first surface is a user operation surface, and the second surface is provided with the first OMT sensor, the remote control device also includes a third surface, the third surface is perpendicular to the first surface, and the third surface is provided with the second OMT sensor, the memory is coupled to the one or more processors, the memory is used to store a computer program, and the processor calls the computer program to implement the steps of the method described in any one of claims 1-12.

14. A computer-readable storage medium having a computer program stored thereon; characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 12 are implemented.

15. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 12 are implemented.