Equipment control method, equipment control system and equipment

Abstract

The invention discloses an equipment control method, an equipment control system and equipment. When the remote controller points to the equipment, the equipment can display a cursor on a screen, and the cursor is used for identifying the pointing position of the remote controller on the screen of the equipment. The remote controller may receive a first input of a user such as a zoom gesture, a rotation gesture, etc. According to the first input, the device can execute a corresponding first operation on an object corresponding to the cursor. In some embodiments, the remote controller may also receive a second input of the user, such as a return gesture, a return desktop gesture, a multi-task interface entry gesture, a pull-down menu gesture, a pull-down search gesture, etc. According to the second input, the device can execute a corresponding second operation, and an operation object of the second operation is not determined by the position of the cursor. In some embodiments, the remote controller can also realize screen capture operations such as full screen capture, scrolling screen capture, regional screen capture and the like through click operation. Through the device control method, a user can control the device more conveniently and efficiently.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of terminal, and in particular, to a device control method, a device control system and a device. BACKGROUND

[0002] With the development of terminal technology, the services and contents on the smart TV are increasing. The traditional remote controller only controls the smart TV through physical buttons, which is difficult to meet the increasing interactive needs of the smart TV. Therefore, an innovative interactive mode is urgently needed between the smart TV and the remote controller, so that the user can operate the smart TV with large screen and mass applications more conveniently. SUMMARY

[0003] The present application provides a device control method, a device control system and a device. In the method, when the remote controller points to the device, the device can display a cursor on the screen to identify the position pointed by the remote controller on the screen of the device. The remote controller can receive a first input of a zoom gesture, a rotation gesture, etc. of the user. According to the first input, the device can perform a corresponding first operation on the object corresponding to the cursor. In some embodiments, the remote controller can also receive a second input of a return gesture, a return desktop gesture, an entering multi-task interface gesture, a pull-down menu gesture, a pull-down search gesture, etc. of the user. According to the second input, the device can perform a corresponding second operation, and the operation object of the second operation is not determined by the position of the cursor. In some embodiments, the remote controller can also realize the screenshot operation such as full-screen screenshot, scrolling screenshot, and region screenshot through the point operation. Through the above device control method, the user can control the device more conveniently and efficiently.

[0004] In a first aspect, the present application provides a device control method applied to a remote controller in a control system, the control system further comprising a device, a touch area of the remote controller comprising a first touch area and a second touch area, the first touch area being an annular touch area, the first touch area surrounding the periphery of the second touch area, the method comprising: receiving, by the remote controller, a first input of the user through the touch area, the first input comprising a zoom gesture or a rotation gesture, the first input being a multi-point touch gesture, a touch point of the first input being located in one or more of the following: the first touch area, the second touch area, the first input being used to trigger the device to perform a first operation on an object corresponding to a cursor, the first operation comprising a zoom operation or a rotation operation, the cursor being used to identify the position pointed by the remote controller on the screen of the device.

[0005] Implementing the method provided in the first aspect, the remote control can receive multi-touch gestures (i.e., the first input) from the user on the remote control's touch area to trigger a device (such as a smart TV) to perform a corresponding operation (i.e., the first operation) with the object currently occupied by the cursor as the operation object. Compared to the solution of switching between displayed content by swiping left and right on the touch area, this solution of triggering the device to perform a corresponding operation with the object currently occupied by the cursor through multi-touch first input can provide functions such as zooming and rotation, thereby providing users with a richer and more convenient interactive experience.

[0006] In conjunction with the method provided in the first aspect, in some embodiments, the first touch area in the remote control includes a first sub-area, a second sub-area, a third sub-area, and a fourth sub-area. The first sub-area and the fourth sub-area are arranged opposite each other, and the second sub-area and the third sub-area are arranged opposite each other. The first sub-area and the fourth sub-area are respectively used as the up and down directional keys of the remote control, and the second sub-area and the third sub-area are respectively used as the left and right directional keys of the remote control. The second touch area is also used as the confirmation key of the remote control.

[0007] By implementing the method provided in the above embodiments, the remote control in the above control system can be compatible with the design of the confirmation key and directional key in the traditional remote control. On the one hand, this can reduce the learning cost for users and conform to their usage habits. On the other hand, the remote control can also receive more user input by combining multiple operations of the confirmation key, directional key and cursor position, thereby providing users with a richer and more convenient interactive experience.

[0008] In conjunction with the method provided in the first aspect, in some embodiments, the first input is a zoom gesture, the touch points of the zoom gesture include a first touch point and a second touch point, the zoom gesture includes a shrink gesture and a zoom gesture, wherein the shrink gesture is a sliding operation in which the first touch point and the second touch point move towards each other, and the zoom gesture is a sliding operation in which the first touch point and the second touch point move away from each other; or, the first input is a rotation gesture, the touch points of the rotation gesture include a third touch point and a fourth touch point, the rotation gesture is a sliding operation in which both the third touch point and the fourth touch point move in a clockwise or counterclockwise direction.

[0009] By implementing the method provided in the above embodiments, the remote control can control the device by receiving different sliding gestures (i.e., first inputs) on the touch area. The sliding gestures include zoom gestures and rotation gestures.

[0010] In conjunction with the method provided in the first aspect, in some embodiments, a second input from a user is received through a touch area. The second input is used to trigger the device to perform a second operation. The object of the second operation is not determined by the position of the cursor. The type of the second operation is determined based on one or more of the following: the starting point of the second input, the ending point of the second input, the sliding direction of the second input, the hovering position of the second input, and the hovering time of the second input.

[0011] Implementing the method provided in the above embodiments, the remote control can also receive other swiping gestures (i.e., second inputs) from the user through the remote control's touch area, and then trigger the device (such as a smart TV) to execute the corresponding second operation. The second operation is not determined by the cursor's position. The specific type of the second operation is determined based on one or more of the user's swiping gesture's start point, end point, direction, hovering position, and time. By changing the parameters of these indicators, the user can perform richer gesture operations on the remote control to control the device.

[0012] In conjunction with the method provided in the first aspect, in some embodiments, the second input includes a return gesture, which is a gesture of sliding from the left or right edge of the first touch area towards the center of the second touch area; or, the second input includes a return to desktop gesture, which is an operation of sliding from the lower edge of the first touch area towards the center of the second touch area; or, the second input includes an entry to a multitasking interface gesture, which is an operation of sliding from the lower edge of the first touch area towards the center of the second touch area and pausing in the second touch area for a first time; or, the second input includes a drop-down menu gesture, which is an operation of sliding from the upper edge of the first touch area towards the center of the second touch area; or, the second input includes a drop-down search gesture, which is an operation of sliding from a first sub-region of the first touch area excluding the upper edge towards the center of the second touch area.

[0013] By implementing the method provided in the above embodiments, the remote control can control the device by receiving different swiping gestures (i.e., second inputs) on the touch area. The swiping gestures include a return gesture, a return to the desktop gesture, a multitasking interface gesture, a drop-down menu gesture, and a drop-down search gesture.

[0014] In conjunction with the method provided in the first aspect, in some embodiments, the method further includes: receiving a user's double-click operation on the second touch area, the double-click operation on the second touch area being used to trigger the device to perform a screenshot operation on the entire screen; or, the remote control receiving a third input from the user by long-pressing the second touch area and moving the cursor position, the third input being used to trigger the device to perform a screenshot operation, the screenshot area being determined by the movement trajectory of the cursor.

[0015] By implementing the method provided in the above embodiments, the remote control can trigger a full-screen screenshot operation by receiving a double-click operation on the touch area. The remote control can also trigger a scrolling screenshot or a region screenshot operation based on the cursor's movement trajectory after receiving a long-press operation.

[0016] In conjunction with the method provided in the first aspect, in some embodiments, the method further includes: the remote controller sending touch data to the device, the touch data including the position coordinates of a touch point detected by the remote controller through the touch area, the touch data being used by the device to identify a first input.

[0017] By implementing the method provided in the above embodiments, after the remote control receives user input through the touch area, it can report the touch data to the device for gesture recognition. By fully utilizing the device's computing power, users can control the device using richer input methods (i.e., gestures or taps). Furthermore, compared to the scheme that uses touch data to recognize gestures on the remote control side, the scheme that uses the device side for gesture recognition only requires updating the definition of the first input in the gesture recognition algorithm on the device side, eliminating the need for updates on both sides and simplifying the algorithm optimization process.

[0018] In conjunction with the method provided in the first aspect, in some embodiments, the cursor movement trajectory is an S-shaped movement trajectory, the screenshot operation is a scrolling screenshot operation, and the screenshot area includes the content of multiple pages during the scrolling screenshot operation.

[0019] In conjunction with the method provided in the first aspect, in some embodiments, the cursor movement trajectory is a closed trajectory, and the screenshot area is the area within the closed trajectory.

[0020] Secondly, this application provides a device control method applied to a device in a control system. The control system further includes a remote controller. The touch area of ​​the remote controller includes a first touch area and a second touch area. The first touch area is a ring-shaped touch area surrounding the second touch area. The method includes: detecting that the remote controller is pointing at the device; the device displays a cursor on the screen, the cursor being used to identify the position pointed to by the remote controller on the screen of the device; and, according to a first input, the device performs a first operation on the object corresponding to the cursor, the first operation including a zoom operation or a rotation operation. The first input is received by the remote controller through the touch area, the first input including a zoom gesture or a rotation gesture, the first input being a multi-touch gesture, and the touch point of the first input being located in one or more of the following: the first touch area and the second touch area.

[0021] By implementing the method provided in the second aspect, the device can display a cursor at the location pointed to by the remote control when it detects that the remote control is pointing at the device screen. Subsequently, the device can execute the corresponding first operation (such as zoom and rotate operation) based on the first multi-touch input (including zoom and rotate gestures) received by the remote control, thereby realizing the control of the device and providing users with a richer and more convenient interactive experience.

[0022] In conjunction with the method provided in the second aspect, in some embodiments, the first touch area in the remote control includes a first sub-area, a second sub-area, a third sub-area, and a fourth sub-area. The first and fourth sub-areas are arranged opposite each other, and the second and third sub-areas are arranged opposite each other. The first and fourth sub-areas are also used as the up and down directional keys of the remote control, and the second and third sub-areas are also used as the left and right directional keys of the remote control, respectively. The second touch area is also used as the confirmation key of the remote control.

[0023] The method provided by the above embodiments enables the remote control that receives user input and triggers the device to perform corresponding operations to be compatible with the hardware design of traditional remote controls. As a result, the device (such as a smart TV) can recognize more user input based on the input received by the remote control (including touch input, such as pressing the confirmation key and directional keys) and various operations of the cursor position, thereby providing users with a richer and more convenient interactive experience.

[0024] In conjunction with the method provided in the second aspect, in some embodiments, the first input is a zoom gesture. The touch points of the zoom gesture include a first touch point and a second touch point. The zoom gesture includes a shrink gesture and a zoom gesture. The shrink gesture is a sliding operation in which the first touch point and the second touch point move towards each other, and the zoom gesture is a sliding operation in which the first touch point and the second touch point move away from each other. Based on the first input, the device performs a first operation on the object corresponding to the cursor, specifically including: based on the shrink gesture, the device shrinks the content on the device screen with the cursor position as the center, and the content on the device screen includes the object corresponding to the cursor; or, based on the zoom gesture, the device zooms the content on the device screen with the cursor position as the center, and the content on the device screen includes the object corresponding to the cursor.

[0025] By implementing the method provided in the above embodiments, a device (such as a smart TV) can perform a zoom operation based on a zoom gesture received by the remote control, and the zoom operation is centered on the location of the cursor (i.e., the cursor itself).

[0026] In conjunction with the method provided in the second aspect, in some embodiments, the first input is a rotation gesture, the touch points of the rotation gesture include a third touch point and a fourth touch point, and the rotation gesture is a sliding operation in which both the third touch point and the fourth touch point slide in a clockwise or counterclockwise direction. According to the first input, the device performs a first operation on the object corresponding to the cursor, specifically including: according to the rotation gesture, the device rotates the content on the device screen with the cursor position as the center, and the rotation direction of the content on the device screen is the same as the sliding direction of the third touch point and the fourth touch point in the rotation gesture.

[0027] By implementing the method provided in the above embodiments, a device (such as a smart TV) can perform a rotation operation based on a rotation gesture received by the remote control. The rotation operation is centered on the position of the cursor and the object corresponding to the cursor is the object to be rotated. The direction of rotation of the object is the same as the direction of rotation in the user's rotation gesture.

[0028] In conjunction with the method provided in the second aspect, in some embodiments, the method further includes: the device performing a second operation based on a second input, the second input being received by a remote control through a touch area, the object of the second operation not being determined by the position of the cursor, and the type of the second operation being determined based on one or more of the following of the second input: the starting point of the second input, the ending point of the second input, the sliding direction of the second input, the hovering position of the second input, and the hovering time of the second input.

[0029] By implementing the method provided in the above embodiments, a device (such as a smart TV) can perform a corresponding second operation based on other user inputs (i.e., second inputs) received by the remote control. The second operation is not determined by the cursor's position. The specific type of the second operation is determined based on one or more of the user's swipe gesture's start point, end point, direction, hovering position, and time. By changing the parameters of these indicators, users can perform richer gesture operations on the remote control to control the device.

[0030] In conjunction with the method provided in the second aspect, in some embodiments, the device performs a second operation based on a second input, specifically including: the second input including a return gesture, which is a gesture of sliding from the left or right edge of the first touch area towards the center of the second touch area, and the device cancels the previous operation step or returns to the previous interface based on the return gesture; or, the second input including a return to desktop gesture, which is an operation of sliding from the lower edge of the first touch area towards the center of the second touch area, and the device returns to the desktop page based on the return to desktop gesture; or, the second input including an entry to a multitasking interface gesture, which is an operation of sliding from the lower edge of the first touch area towards the center of the second touch area and staying on the second touch area for a first time. The operation can be as follows: The device opens a multitasking interface based on a gesture to enter the multitasking interface, which includes thumbnails of one or more running applications; or, the second input includes a drop-down menu gesture, which is a sliding operation from the top edge of the first touch area to the center of the second touch area, and the device opens a drop-down menu based on the gesture, which includes one or more of the following: the device's notification center, quick function settings interface; or, the second input includes a drop-down search gesture, which is a sliding operation from the area excluding the top edge of the first sub-area of ​​the first touch area to the center of the second touch area, and the content displayed on the device screen is the desktop page, and the device opens the system search interface based on the drop-down search gesture.

[0031] By implementing the method provided in the above embodiments, a device (such as a smart TV) can perform operations such as returning, returning to the desktop, entering a multitasking session, entering a drop-down menu, and entering a drop-down search based on the second input received by the remote control.

[0032] In conjunction with the method provided in the second aspect, in some embodiments, the method further includes: the device performing a screenshot operation on the entire screen based on the operation of double-clicking the second touch area, wherein the operation of double-clicking the second touch area is received by the remote control through the touch area; or, the device performing a screenshot operation based on a third input, wherein the screenshot area is determined by the movement trajectory of the cursor, and the third input is the input received by the remote control from the user who long-presses the second touch area and moves the cursor position.

[0033] By implementing the method provided in the above embodiments, the device can perform a full-screen screenshot operation after the remote control receives a double-click operation. The device can also determine the cursor's movement trajectory and trigger a scrolling screenshot or a region screenshot operation after the remote control receives a long-press operation.

[0034] In conjunction with the method provided in the second aspect, in some embodiments, the cursor movement trajectory is an S-shaped movement trajectory, the screenshot operation is a scrolling screenshot operation, and the screenshot area includes the content of multiple pages during the scrolling screenshot operation.

[0035] In conjunction with the method provided in the second aspect, in some embodiments, the cursor movement trajectory is a closed trajectory, and the screenshot area is the area within the closed trajectory.

[0036] In conjunction with the method provided in the second aspect, in some embodiments, based on the first input, the device performs a first operation on the object corresponding to the cursor, specifically including: receiving touch data from a remote controller, the touch data including the position coordinates of the touch point detected by the remote controller through the touch area; the device identifying the first input based on the touch data; and the device performing the first operation based on the identified first input and the position of the cursor.

[0037] By implementing the method provided in the above embodiments, the device can recognize gestures based on the reported touch data. While fully utilizing the device's computing power, users can control the device using richer input methods (i.e., gestures or taps) via the remote control. Furthermore, compared to a scheme that uses touch data to recognize gestures on the remote control side, this device-side gesture recognition scheme only requires updating the definition of the first input in the gesture recognition algorithm on the device side, eliminating the need for updates on both sides and simplifying the algorithm optimization process.

[0038] In conjunction with the method provided in the second aspect, in some embodiments, before the device identifies the first input based on the touch data, the method further includes: the device obtaining first converted data based on the touch data, the first converted data being the coordinate data corresponding to the touch data magnified proportionally from the touch area of ​​the remote control to the device screen; the first electronic data processing of the first converted data, the processed first converted data having higher accuracy and continuity compared to the first converted data before processing; the device identifying the first input based on the touch data specifically includes: the device identifying the first input based on the processed first converted data.

[0039] By implementing the method provided in the above embodiments, the device (such as a smart TV) can preprocess the touch data before recognizing gestures using the touch data. The data preprocessing may include methods such as coordinate transformation, dithering filtering, and polynomial fitting interpolation, thereby making the preprocessed touch data smoother and more accurate, and thus more accurate in the subsequent gesture recognition process.

[0040] Thirdly, a device control method is applied to a control system, the control system including a remote controller and a device, the remote controller having a touch area including a first touch area and a second touch area, the first touch area being a ring-shaped touch area surrounding the second touch area, the method including: detecting that the remote controller is pointing at the device, the device displaying a cursor on the screen, the cursor being used to identify the position pointed to by the remote controller on the device's screen; the remote controller receiving a first input from a user through the touch area, the first input including a zoom gesture or a rotation gesture, the first input being a multi-touch gesture, the touch point of the first input being located in one or more of the following: the first touch area, the second touch area; according to the first input, the device performing a first operation on the object corresponding to the cursor, the first operation including a zoom operation or a rotation operation.

[0041] In conjunction with the method provided in the third aspect, in some embodiments, the first touch area in the remote control includes a first sub-area, a second sub-area, a third sub-area, and a fourth sub-area. The first sub-area and the fourth sub-area are arranged opposite each other, and the second sub-area and the third sub-area are arranged opposite each other. The first sub-area and the fourth sub-area are respectively used as the up and down directional keys of the remote control, and the second sub-area and the third sub-area are respectively used as the left and right directional keys of the remote control. The second touch area is also used as the confirmation key of the remote control.

[0042] In conjunction with the method provided in the third aspect, in some embodiments, the first input is a zoom gesture, the touch points of the zoom gesture include a first touch point and a second touch point, the zoom gesture includes a shrink gesture and a zoom gesture, wherein the shrink gesture is a sliding operation in which the first touch point and the second touch point move towards each other, and the zoom gesture is a sliding operation in which the first touch point and the second touch point move away from each other; or, the first input is a rotation gesture, the touch points of the rotation gesture include a third touch point and a fourth touch point, the rotation operation is a sliding operation in which both the third touch point and the fourth touch point move in a clockwise or counterclockwise direction.

[0043] In conjunction with the method provided in the third aspect, in some embodiments, based on the first input, the device performs a first operation on the object corresponding to the cursor, specifically including: based on a zoom-out gesture, the device zooms out on the content on the device screen with the cursor's location as the center, the content on the device screen including the object corresponding to the cursor; or, based on a zoom-in gesture, the device zooms out on the content on the device screen with the cursor's location as the center, the content on the device screen including the object corresponding to the cursor; or, based on a rotation gesture, the device rotates the content on the device screen with the cursor's location as the center, the rotation direction of the content on the device screen being the same as the sliding direction of the third touch point and the fourth touch point in the rotation gesture.

[0044] In conjunction with the method provided in the third aspect, in some embodiments, the method further includes: a remote control receiving a second input from a user through a touch area; the device performing a second operation based on the second input, wherein the object of the second operation is not determined by the position of the cursor, and the type of the second operation is determined based on one or more of the following of the second input: the starting point of the second input, the ending point of the second input, the sliding direction of the second input, the hovering position of the second input, and the hovering time of the second input.

[0045] In conjunction with the method provided in the third aspect, in some embodiments, the second input includes a return gesture, a return to desktop gesture, a multitasking interface gesture, a drop-down menu gesture, and a drop-down search gesture. The return gesture is a gesture that slides from the left or right edge of the first touch area towards the center of the second touch area. The return to desktop gesture is an operation that slides from the bottom edge of the first touch area towards the center of the second touch area. The multitasking interface gesture is an operation that slides from the bottom edge of the first touch area towards the center of the second touch area and stays in the second touch area for a first time. The drop-down menu gesture is an operation that slides from the top edge of the first touch area towards the center of the second touch area. The drop-down search gesture is an operation that slides from the area excluding the top edge of the first sub-area of ​​the first touch area towards the center of the second touch area.

[0046] In conjunction with the method provided in the third aspect, in some embodiments, based on the first input, the device performs a first operation on the object corresponding to the cursor, specifically including: the device canceling the previous operation step or returning to the previous interface based on a return gesture; or, the device returning to the desktop page based on a return-to-desktop gesture; or, the device opening a multitasking interface based on an enter-multitasking interface gesture, the multitasking interface including thumbnails of one or more running applications; or, the device opening a drop-down menu based on a drop-down menu gesture, the drop-down menu including one or more of the following: the device's notification center, a quick function settings interface; or, when the content displayed on the device screen is the desktop page, the device opening the system search interface based on a drop-down search gesture.

[0047] In conjunction with the method provided in the third aspect, in some embodiments, the method further includes: the remote control receiving a user's double-click operation on the second touch area; and the device performing a screenshot operation on the entire screen based on the double-click operation on the second touch area.

[0048] In conjunction with the method provided in the third aspect, in some embodiments, the method further includes: the remote control receiving a third input from a user who long-presses the second touch area and moves the cursor position; the device performing a screenshot operation based on the third input, wherein the screenshot area is determined by the movement trajectory of the cursor.

[0049] In conjunction with the method provided in the third aspect, in some embodiments, the cursor movement trajectory is an S-shaped movement trajectory, the screenshot operation is a scrolling screenshot operation, and the screenshot area includes the content of multiple pages during the scrolling screenshot operation.

[0050] In conjunction with the method provided in the third aspect, in some embodiments, the cursor movement trajectory is a closed trajectory, and the screenshot area is the area within the closed trajectory.

[0051] In conjunction with the method provided in the third aspect, in some embodiments, based on the first input, the device performs a first operation on the object corresponding to the cursor, specifically including: the remote controller sending touch data to the device, the touch data including the position coordinates of the touch point detected by the remote controller through the touch area; the device recognizing the first input based on the touch data; and the device performing the first operation based on the recognized first input and the position of the cursor.

[0052] In conjunction with the method provided in the third aspect, in some embodiments, before the device identifies the first input based on the touch data, the method further includes: the device obtaining first converted data based on the touch data, the first converted data being the coordinate data corresponding to the touch data magnified proportionally from the touch area of ​​the remote control to the device screen; the first electronic data processing of the first converted data, the processed first converted data having higher accuracy and continuity compared to the first converted data before processing; the device identifying the first input based on the touch data specifically includes: the device identifying the first input based on the processed first converted data.

[0053] Understandably, the device control method provided in the third aspect is executed by the control system, which consists of a remote controller and the device. Therefore, the device control method provided in the third aspect can be integrated from the device control method provided in the first aspect and the device control method provided in the second aspect. Thus, the beneficial effects it can achieve can be referred to the beneficial effects in the corresponding methods, which will not be elaborated here.

[0054] Fourthly, this application provides a remote control, the touch area of ​​which includes a first touch area and a second touch area. The first touch area is a ring-shaped touch area surrounding the periphery of the second touch area. The remote control includes one or more processors and one or more memories. The one or more memories are used to store a program, which, when executed by the one or more processors, causes the execution of the method described in the first aspect and any possible implementation thereof.

[0055] Fifthly, this application provides an apparatus including a memory, a processor, and a computer program stored in the memory; the processor executes the computer program to implement the method described in the second aspect and any possible implementation thereof.

[0056] In a sixth aspect, this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method described in the first aspect and any possible implementation thereof, or the method described in the second aspect and any possible implementation thereof.

[0057] In a seventh aspect, this application provides a computer program product, including a computer program that, when executed by a processor, implements the method described in the first aspect and any possible implementation thereof, or the method described in the second aspect and any possible implementation thereof.

[0058] Understandably, the remote control provided in the fourth aspect, the device provided in the fifth aspect, the computer storage medium provided in the sixth aspect, and the computer program product provided in the seventh aspect are all used to execute the method provided in this application. Therefore, the beneficial effects they can achieve can be referred to the beneficial effects in the corresponding methods, and will not be repeated here. Attached Figure Description

[0059] Figure 1 This is a schematic diagram of a conventional remote control provided in an embodiment of this application;

[0060] Figure 2 This application provides an embodiment of a smart TV and Figure 1 The diagram shows the interface of a traditional remote control during interaction.

[0061] Figure 3 This is a schematic diagram of another remote control provided in an embodiment of this application;

[0062] Figure 4 This application provides an embodiment of a smart TV and Figure 3 The diagram shows how the remote control interacts.

[0063] Figure 5 This is a schematic diagram of the sub-region distribution in the first touch area of ​​a remote control provided in an embodiment of this application;

[0064] Figure 6 This is a schematic diagram of an interface for displaying images on a smart TV, provided in an embodiment of this application.

[0065] Figure 7 This is a schematic diagram illustrating the operation of a magnification gesture provided in an embodiment of this application;

[0066] Figure 8 This is a schematic diagram of an enlarged image of an interface provided in an embodiment of this application;

[0067] Figure 9 This is a schematic diagram illustrating a shrinking gesture provided in an embodiment of this application;

[0068] Figure 10 This is a schematic diagram of an interface after the image has been reduced in size, as provided in an embodiment of this application;

[0069] Figure 11 This is a schematic diagram of a rotation gesture provided in an embodiment of this application;

[0070] Figure 12 This is a schematic diagram of an interface after image rotation provided in an embodiment of this application;

[0071] Figure 13 This is another schematic diagram of the interface after image rotation provided in the embodiments of this application;

[0072] Figure 14 This is a schematic diagram of a return gesture provided in an embodiment of this application;

[0073] Figure 15 This is a schematic diagram of a gesture for returning to the desktop provided in an embodiment of this application;

[0074] Figure 16 This is a schematic diagram of an operation for entering a multitasking interface provided in an embodiment of this application;

[0075] Figure 17 This is a schematic diagram of a multitasking interface for a smart TV provided in an embodiment of this application;

[0076] Figure 18 This is a schematic diagram illustrating the operation of a drop-down menu gesture provided in an embodiment of this application;

[0077] Figure 19 This is a schematic diagram of a drop-down menu interface in a smart TV provided in an embodiment of this application;

[0078] Figure 20 This is a schematic diagram illustrating the operation of a pull-down search gesture provided in an embodiment of this application;

[0079] Figure 21 This is a schematic diagram of a pull-down search interface in a smart TV provided in an embodiment of this application;

[0080] Figure 22 This is a schematic diagram illustrating a full-screen screenshot gesture provided in an embodiment of this application;

[0081] Figure 23 This is a schematic diagram illustrating a scrolling screenshot gesture provided in an embodiment of this application;

[0082] Figure 24A This is a scrollable display interface for a smart TV provided in this application embodiment;

[0083] Figure 24B This is provided by the embodiments of this application. Figure 24A The interface shown is the one displayed after scrolling down.

[0084] Figure 24C The embodiments provided in this application are based on Figure 24A and Figure 24B The screenshot shown is obtained after performing a scrolling screenshot on the displayed interface.

[0085] Figure 25 This is a schematic diagram illustrating the operation of a region screenshot gesture provided in an embodiment of this application;

[0086] Figure 26 This is a schematic diagram of the architecture of a control system provided in an embodiment of this application;

[0087] Figure 27 This is an internal implementation flowchart of a device control method provided in an embodiment of this application;

[0088] Figure 28 This is a schematic diagram of a touch data reporting mechanism provided in an embodiment of this application;

[0089] Figure 29 This is a schematic diagram of a coordinate transformation provided in an embodiment of this application;

[0090] Figure 30 This is a schematic diagram of a jitter filtering process provided in an embodiment of this application;

[0091] Figure 31 This is a schematic diagram of a polynomial fitting interpolation provided in an embodiment of this application;

[0092] Figure 32 This is a schematic diagram of touch data reporting when a magnification gesture is detected, provided in an embodiment of this application;

[0093] Figure 33 This is a schematic diagram of the structure of a device provided in an embodiment of this application;

[0094] Figure 34 This is a schematic diagram of the structure of a remote control provided in an embodiment of this application. Detailed Implementation

[0095] The terminology used in the embodiments of this application is for the purpose of describing particular embodiments only and is not intended to be a limitation of this application.

[0096] A smart TV is a television with internet connectivity, allowing users to access and use various online services and content through its built-in operating system and application platform, thus providing a richer and more convenient entertainment experience. Users can control the smart TV using a remote control.Figure 1 This is a schematic diagram of a conventional remote control provided in an embodiment of this application. For example... Figure 1 As shown, traditional remote controls primarily control smart TVs by operating physical buttons (such as the confirmation button and directional buttons). Figure 2 This application provides an embodiment of a smart TV and Figure 1 The diagram illustrates the interface used when interacting with a traditional remote control. The content selected by the user can be highlighted on the smart TV's display screen, for example... Figure 2 In the scenario shown, the user has selected the "My Home" control in the top navigation bar of the current interface. The smart TV can select and display the "My Home" control, thus highlighting the selected content on the homepage. Subsequently, the user can... Figure 1 Use the arrow keys shown to switch between selecting content up, down, left, and right.

[0097] However, with the development of terminal technology, the services and content on smart TVs are increasing daily. Figure 2 Taking the scenario shown as an example, if a user wants to enter the online store, due to the numerous functional controls on the display interface, the user needs to perform multiple operations with the directional keys, such as operating in the order of "down-right-down-right," to switch the selected content from "My Home" to "Online Store," and then click the confirmation button to enter the online store to make purchases. It is evident that this method of controlling a smart TV solely through physical buttons is insufficient to meet the ever-increasing interactive needs of smart TVs. Therefore, an innovative interaction method between smart TVs and remote controls is urgently needed to enable users to more conveniently operate smart TVs with large screens and a vast array of applications.

[0098] Figure 3 This is a schematic diagram of another remote control provided in an embodiment of this application, compared to Figure 1 The traditional remote control shown here has its directional and confirmation buttons designed to be touch-sensitive. These buttons function as a touchpad to receive user input, allowing users to operate the remote control via touch. This design significantly enhances the functionality of smart TVs and similar devices. Figure 3 The interactivity between the remote controls shown allows users to control the TV with gestures, thus enriching the user experience and enhancing the functionality of the remote controls.

[0099] besides, Figure 3The remote control shown employs an inertial measurement unit (IMU) and ultra-wideband (UWB) technology to achieve precise positioning of the remote's pointer. This allows users to use the remote like a laser pointer to accurately point at target content on the smart TV, eliminating the need for step-by-step icon switching via physical buttons, as with traditional remotes. This provides users with an efficient, convenient, and smooth control experience. Specifically, Figure 3 The remote control shown can transmit ultra-wideband wireless signals via a UWB module; correspondingly, the smart TV is equipped with a UWB receiver to receive the signals transmitted by the remote control. By measuring the arrival time or time difference of the aforementioned ultra-wideband wireless signal, the smart TV can calculate the position information of the remote control relative to the smart TV. This position information refers to a point in three-dimensional space, i.e., a position with x, y, and z coordinates. The smart TV can determine the precise position of the remote control relative to itself using the remote control's spatial coordinates and its own coordinates in the same coordinate system (e.g., the smart TV can use itself as the origin of the three-dimensional coordinate system). This position information of the remote control relative to the smart TV is also called UWB positioning data. Although the smart TV can know the precise spatial position of the remote control relative to itself through the aforementioned UWB technology, determining the pointing angle of the remote control (i.e., whether the remote control is horizontal, vertical, or tilted in space, and the specific angle of tilt) requires the assistance of other types of sensors or algorithms. Therefore, the aforementioned remote control can also collect IMU data in real time through its built-in IMU sensors (such as accelerometers and gyroscopes), including but not limited to the remote control's tilt angle and acceleration data, and send this IMU data to the smart TV. This allows the smart TV to determine the remote control's pointing angle in space. Thus, the smart TV can combine IMU data and UWB positioning data to determine the location the remote control is pointing at, so that when the remote control is pointed at the smart TV, the smart TV's interface will display... Figure 4 The cursor is displayed at the location pointed to by the remote control (i.e., Figure 4 The cursor (a circular ring) can be used to indicate the target content selected by the user on the smart TV interface. In some embodiments, the smart TV can highlight the target content selected by the cursor. For example, in... Figure 4 In a smart TV, the border of a module can be thickened when the cursor is selected. Another example is that a smart TV can slightly enlarge a control when it is selected, thus distinguishing the selected control from other surrounding controls.

[0100] pass Figure 3The remote control shown allows users to perform five gesture operations on the smart TV: selection, clicking, dragging, swiping, and selecting. These gesture operations refer to the ways users interact with the device through specific hand movements.

[0101] Specifically, smart TVs can change the cursor's position on the screen after detecting a change in the user's remote control direction, thus enabling the remote control to select target content on the smart TV. For example, to select target content from... Figure 2 To switch from "My Home" to "Shop," the user only needs to use the aforementioned remote control as shown. Figure 4 The cursor pointing to the "Shop" control allows you to switch between selected content (i.e., select the target content) without needing to press multiple buttons. When the cursor is located in a clickable area on the smart TV interface (such as the "Shop" control or various links), the smart TV can switch the displayed interface after detecting that the user has pressed the confirmation button on the remote control, thus enabling the remote control to click on the selected content on the smart TV. For example, as shown... Figure 4 As shown, after detecting that the user has pressed the confirmation button on the remote control, the smart TV can... Figure 4 The main interface shown leads to the "Shop" display interface. If the content currently selected by the cursor is draggable (such as a video progress bar), after detecting that the user has long-pressed the confirmation button on the remote control and the remote control's direction has changed, the smart TV can use the content selected by the cursor when the confirmation button was pressed as the drag object, and then drag the drag object along the cursor's movement trajectory, thus realizing the drag operation. For example, the user can drag control A from the first area to the second area on the smart TV display interface by long-pressing and moving the remote control. If the content currently selected by the cursor is not draggable, after detecting that the user has long-pressed the confirmation button on the remote control and the remote control's direction has changed, the smart TV can display the cursor's movement trajectory accordingly, thus realizing the selection function. In addition, since the remote control also has touch functionality, after detecting that the user has long-pressed the confirmation button on the remote control and the remote control's direction has changed, the smart TV can display the cursor's movement trajectory accordingly, thus realizing the selection function. Figure 3 When the touchpad is swipeable left or right, the smart TV can also switch the display interface accordingly by swiping left or right. This new interaction method between the remote control and the smart TV makes the user experience similar to using a smartphone, thereby not only improving the control efficiency of the smart TV, but also reducing the learning curve for users to some extent.

[0102] However, the aforementioned remote control only supports five operations: pointing, clicking, swiping, dragging, and selecting. As various functions in smart TVs continue to improve and develop, these five gestures are insufficient to meet the ever-increasing interactive needs of smart TVs.

[0103] Therefore, this application provides a device control method. This method can be applied to a control system, which includes devices such as smart TVs equipped with a display screen and a remote control equipped with a touchpad and having a pointing function. The aforementioned pointing function refers to... Figure 4 The example shown illustrates the function of controlling the selection of content on the device by changing the direction of the remote control.

[0104] When the device control method provided in this application is implemented, the device can display a cursor on the display interface when the remote control is pointed at the device (e.g., ...). Figure 4 The circular cursor shown indicates the position the remote control is pointing at on the device's screen. The remote control can then receive a first input from the user via a touch area. Based on the first input, the device can perform a corresponding operation on the object corresponding to the cursor, i.e., the first operation. The first input can be a multi-touch gesture related to the cursor's location, including but not limited to zoom gestures (including enlarge and shrink gestures) and rotation gestures on the object corresponding to the cursor. In some embodiments, the remote control can also receive a second input from the user via a touchpad. Based on the second input, the device can perform a second operation unrelated to the current cursor position. This second input includes, but is not limited to, a return gesture, a return to the desktop gesture, a multitasking interface gesture, a drop-down menu gesture, and a drop-down search gesture. In some embodiments, the remote control can also perform screenshot operations such as full-screen screenshot, scrolling screenshot, and area screenshot via tap operations.

[0105] The above-mentioned devices are not limited to smart TVs; they can also be artificial intelligence (AI) devices, in-vehicle devices, smart home devices, and / or smart city devices equipped with displays. This application does not impose any special restrictions on the specific type of such device. The following examples illustrate this. Figure 3 The remote control shown is used as a remote control, and the smart TV is used as a device to illustrate the device control method provided in the embodiments of this application.

[0106] like Figure 3 As shown, the areas outlined in the dashed lines are all touchpads. Figure 3 The area covered by the touchpad shown is the touch area. Specifically, the touch area of ​​the remote control includes a first touch area and a second touch area. The first touch area is a ring-shaped touch area, surrounding the second touch area, thus forming the following... Figure 3 The continuous touch areas are shown. The touch points involved in the above multi-touch gestures can be located in one or more of the following: a first touch area, a second touch area. The first touch area can also function as a directional key when pressed, and the second touch area can also be used as a confirmation button on the remote control. The first touch area can also... Figure 5The area shown is divided into four sub-regions: sub-region 1, sub-region 2, sub-region 3, and sub-region 4. Sub-region 1 and sub-region 4 are positioned opposite each other, and sub-region 2 and sub-region 3 are positioned opposite each other. Sub-region 1 and sub-region 4 are also used as... Figure 3 The up and down arrow keys on the remote control shown are also used as the left and right arrow keys in the second and third sub-areas, respectively. (Not limited to...) Figure 3 As shown in the embodiments, this application does not impose any special restrictions on the specific shape of the first touch area and the first touch area.

[0107] First, the device control method provided in the embodiments of this application will be introduced with the help of the display interface of a smart TV.

[0108] Figure 6 This is a schematic diagram of an interface for displaying images on a smart TV, provided in an embodiment of this application. For example... Figure 6 As shown, the interface includes a status bar and a content display area, where the content display area may include an image display area. This is understandable. Figure 6 The image shown is in portrait orientation (i.e., the height of the image is greater than its width). To fully display the image, the image display area may only be a portion of the content display area. This layout is not limited to this; if the width-to-height ratio of the image is the same as the width-to-height ratio of the content display area, or if the image is displayed in a tiled mode, then the image display area may occupy the entire area of ​​the content display area. This application does not impose any special limitations on this. In the following description, we will temporarily use... Figure 6 The interface display scheme shown is described below. In addition, the image display area may also include a cursor, the position of which is determined by the direction the remote control is pointing.

[0109] The remote control can receive the user's initial input via its touch area, thereby triggering the smart TV to respond based on the object corresponding to the cursor (e.g., ...). Figure 6 (The cursor corresponds to an image) and performs operations such as zooming in, zooming out, and rotating. The following sections will elaborate on each gesture in the first input.

[0110] Zooming in

[0111] Figure 7 This is a schematic diagram illustrating the operation of a magnification gesture provided in an embodiment of this application. It can be understood that... Figure 7 The diagrams illustrating subsequent gesture operations only show a portion of the touch area in the remote control.

[0112] Depend on Figure 7 As can be seen, when the remote control detects a presence such as... Figure 7As shown, when two fingers slide in opposite directions, the remote control triggers the smart TV to zoom in on the content on the smart TV screen, using the current cursor position as the zoom center. The content on the smart TV screen includes the object corresponding to the cursor. Specifically, the touch points for the zoom gesture can include a first touch point and a second touch point. These two touch points are the points where the user's two fingers contact the touch area of ​​the remote control. Therefore, the zoom gesture is a sliding operation where the first and second touch points move in opposite directions. Taking the cursor on an image on the smart TV screen as an example, when the remote control detects... Figure 7 After making the zoom-in gesture shown, the remote control can trigger the smart TV to... Figure 6 The image shown is enlarged to the size of... Figure 8 The size shown. (By) Figure 6 and Figure 8 As the comparison shows, this zoom-in is centered on the cursor's location; therefore, the farther the position is from the cursor, the greater the displacement. The zoom ratio is determined by the distance of the swipe gesture; the farther the swipe in the opposite direction, the greater the zoom ratio; conversely, the shorter the swipe in the opposite direction, the smaller the zoom ratio. Understandably, after the zoom-in operation, some content in the original image will exceed the image display area; therefore, the smart TV can choose not to display the excess portion on the interface.

[0113] Understandable, such as Figure 6 or Figure 8 In the scenario shown, a cursor appears on the smart TV interface indicating that the remote control is currently pointing at the smart TV. However, this is not limited to this scenario; if the remote control detects... Figure 7 When the zoom-in gesture is shown, the remote control is not pointing at the smart TV, meaning the cursor is not currently displayed on the smart TV's screen. In this case, the smart TV can zoom in on the displayed content using the first coordinate as the zoom center. The first coordinate is a point preset by the developers on the smart TV's display interface. Optionally, the first coordinate can be the center point of the smart TV's display interface.

[0114] The above magnification operation is not limited to, for example, Figure 6 The image shown can be enlarged, and the remote control can also trigger the smart TV to directly enlarge the displayed page or other content after detecting the aforementioned enlargement gesture. This application embodiment does not specifically limit the object on which the smart TV performs the enlargement operation.

[0115] Shrink operation

[0116] Figure 9 This is a schematic diagram of a shrinking gesture provided in an embodiment of this application.

[0117] When the remote control detects a presence such as in the remote control's touch area. Figure 9As shown in the diagram, when a two-finger swipe gesture is made, the remote control triggers the smart TV to zoom out on the content on the smart TV screen, using the current cursor position as the zoom-out center. Similarly, the zoom-out gesture is a swipe gesture where the first touch point and the second touch point move towards each other. For example, when the remote control detects... Figure 9 After making the zoom-out gesture shown, the remote control can trigger the smart TV to... Figure 6 The image shown is shrunk to the size of... Figure 10 The size shown. (By) Figure 6 and Figure 10 As can be seen from the comparison, this shrinking is centered on the cursor position; therefore, the further away from the cursor, the greater the content displacement. The shrinking ratio is determined by the distance of the swipe gesture in the gesture operation. The greater the swipe distance in the opposite direction, i.e., the closer the distance between the user's two fingers, the greater the shrinking ratio; conversely, the shorter the swipe distance in the opposite direction, i.e., the farther the distance between the user's two fingers, the smaller the shrinking ratio. Optionally, after detecting that the user has finished executing the shrinking gesture, the smart TV can directly display... Figure 10 The image shown. (By...) Figure 10 As shown, after performing the above shrinking operation, the shrunk image cannot fill the entire image display area. Therefore, the smart TV can fill in the empty parts of the image display area (i.e.,...) Figure 10 The area marked with a diagonal line (in the center) can be filled with color. Optionally, the smart TV can use the same color as the background color of the content display area to fill in the aforementioned empty areas. For example, Figure 6 If the background outside the image display area is black, then the image will be scaled down. Figure 10 The text also uses black to fill in the missing parts (i.e., the parts with diagonal lines). Optionally, smart TVs can use... Figure 10 The image display area is limited. If the shrunk image size is smaller than this display area, the smart TV will be triggered to switch off from the previous zoom-out gesture after the remote control detects that the user has finished executing the zoom-out gesture. Figure 10 The displayed interface has been restored to Figure 6 The display interface shown.

[0118] Similar to the zoom-in operation, if the remote control detects the zoom-out gesture and there is no cursor on the smart TV's display interface, the smart TV can zoom out on the displayed content using the second coordinate as the zoom-out center. This second coordinate is a point preset by the developers on the smart TV's display interface, and it can be the same as the first coordinate. Likewise, the zoomed-out content is not limited to images but also includes text pages and other displayed content.

[0119] Rotation operation

[0120] Figure 11This is a schematic diagram of a rotation gesture provided in an embodiment of this application.

[0121] like Figure 11 As shown, when the remote control detects that the user is simultaneously sliding two fingers counter-clockwise on the touchpad, the remote control triggers the smart TV to rotate the content on the smart TV screen counter-clockwise around the current cursor position. Specifically, the touch points for the rotation gesture (i.e., the points where the user's finger and the touch area of ​​the remote control come into contact during rotation) include the third touch point and the fourth touch point, as shown below. Figure 11 As shown, a rotation gesture is a sliding operation where both the third and fourth touch points move counter-clockwise. It can be understood that the rotation direction of the content on the smart TV screen is the same as the sliding direction of the third and fourth touch points in the rotation gesture. For example, when the remote control detects... Figure 11 After performing the counter-clockwise rotation gesture shown, the remote control can trigger the smart TV to... Figure 6 The image shown is rotated to Figure 12 The angle shown. It is understandable that the above... Figure 12 The rotation angle of the image depends on the distance the two fingers slide counter-clockwise simultaneously. The greater the counter-clockwise distance, the larger the rotation angle; conversely, the shorter the counter-clockwise distance, the smaller the rotation angle. Optionally, the smart TV can directly display the image when the user stops sliding. Figure 12 The interface shown. Optionally, the smart TV can be set with a first rotation threshold. If, through the above rotation operation, the rotation angle of the image is less than the first rotation threshold, then after detecting that the user has finished executing the above rotation gesture, the remote control can trigger the smart TV to return to the center display of the image, that is, from the above... Figure 12 The interface shown has been restored. Figure 6 The interface shown; if, through the above rotation operation, the rotation angle is greater than the first rotation threshold but less than or equal to 90 degrees, then after detecting the user's gesture to stop the rotation operation, the remote control can trigger the smart TV to rotate the above image by 90 degrees, that is, from the above... Figure 12 The interface shown switches to Figure 13 The interface shown is similar to the one described above. Similarly, there is a second rotation threshold between 90° and 180°, a third rotation threshold between 180° and 270°, and a fourth rotation threshold between 270° and 360°, which can achieve similar functions, so that the image rotation is in units of 90 degrees or multiples of 90 degrees, which is more in line with the user's operating and browsing habits.

[0122] Understandably, during user gesture operations with the remote control, image rotation is centered on the cursor position on the smart TV interface. However, if the smart TV has a rotation threshold (such as a first rotation threshold), and the rotation angle exceeds the first rotation threshold but is less than or equal to 90 degrees, the smart TV can automatically enlarge the image proportionally and display it in the center. Therefore, the final displayed image (such as...) Figure 13 It cannot be directly based on the original image (such as...) Figure 6 The image is obtained by rotating it directly around the cursor's position. The aforementioned automatic proportional enlargement of images on smart TVs refers to enlarging the image according to its original height and width ratio. Figure 13 The image is enlarged proportionally until the height of the enlarged image equals the height of the content display area or the width of the image equals the width of the content display area.

[0123] Similarly, if the remote control detects such Figure 11 During the rotation operation shown, the remote control is not pointed at the smart TV, meaning the aforementioned cursor is not displayed on the smart TV. The smart TV can perform the rotation operation with the third coordinate as the rotation center. The aforementioned third coordinate is a point preset by the developers in the smart TV display interface, and this third coordinate can be the same as the aforementioned first coordinate or second coordinate.

[0124] Understandable. Figure 11 and Figure 12 Only a counter-clockwise rotation example is given. In addition, the remote control can also detect when the user simultaneously rotates two fingers clockwise on the touchpad (i.e.,...). Figure 11 When the remote control slides in the opposite direction (as shown), it triggers the smart TV to rotate the displayed content clockwise around the current cursor position. This application embodiment does not limit the direction of rotation during the rotation operation.

[0125] In some embodiments, the remote control can also receive a second input from the user via the touchpad, including but not limited to gestures such as returning, returning to the desktop, entering the multitasking interface, pulling down the menu, and pulling down to search, thereby enabling control operations on the smart TV (i.e., the second operation). The object of the second operation is not determined by the cursor position. The type of the second operation is determined based on one or more of the following: the starting point of the second input, the ending point of the second input, the sliding direction of the second input, the hovering position of the second input, and the hovering time of the second input. The above-mentioned hovering refers to the state in which the user remains in contact with the touch area of ​​the remote control without releasing it. The following will elaborate on each of the above-mentioned second inputs triggering the second operation.

[0126] Return operation

[0127] Figure 14This is a schematic diagram of a return gesture provided in an embodiment of this application. When the remote control detects a sliding motion from the left edge of the first touch area of ​​the remote control towards the center of the second touch area, the remote control triggers the smart TV to perform a return operation. The return operation refers to the smart TV canceling previous operations or returning to the previous level interface. Optionally, the previous level interface is the interface displayed before entering the current interface. It is not limited to the interface displayed before entering the current interface; the previous level interface may also include a previous level interface defined by the system or application. It is understood that the remote control can only trigger the smart TV to perform the above-mentioned return operation if the smart TV currently has uncancellable operations or if the currently displayed interface has a previous level interface.

[0128] Not limited to Figure 14 The sliding operation shown is from the left edge of the first touch area of ​​the remote control to the center of the second touch area. When the remote control detects the sliding operation from the right edge of the first touch area of ​​the remote control to the center of the second touch area, the remote control can also trigger the smart TV to perform a return operation.

[0129] Return to desktop

[0130] Figure 15 This is a schematic diagram illustrating a gesture for returning to the home screen, provided in an embodiment of this application. When the remote control detects a swipe gesture from the lower edge of the first touch area of ​​the remote control towards the center of the second touch area, the remote control triggers the smart TV to return to the home screen. It is understood that the smart TV can only perform this return-to-home screen operation if it is currently displaying a non-home screen, such as the interface of a first application or the negative one screen. If the smart TV is currently displaying a home screen, even if the swipe gesture from the lower edge of the first touch area of ​​the remote control towards the center of the second touch area is detected, the remote control cannot trigger the smart TV to return to the home screen.

[0131] Preferably, the desktop may include multiple pages. For example, the desktop may include three pages: a first desktop, a second desktop, and a third desktop. The second desktop page may contain a first application. In this case, the return-to-desktop operation may switch from the user interface of the first application to the second desktop interface. Optionally, regardless of which screen the first application is displayed on, the return-to-desktop operation always switches from the user interface of the first application to a fixed screen, such as the first desktop page. This fixed screen can be preset by the developers.

[0132] Enter the multitasking interface

[0133] Figure 16This is a schematic diagram illustrating a gesture for entering a multitasking interface, provided in an embodiment of this application. When the remote control detects a swipe from the lower edge of the first touch area of ​​the remote control towards the center of the second touch area and lingers at the center for a first moment, the remote control triggers the smart TV to enter the multitasking interface. The multitasking interface includes thumbnails of one or more running applications.

[0134] For example, such as Figure 17 As shown, the multitasking interface can include thumbnails of the Weibo and camera applications. This indicates that these applications (i.e., Weibo and camera) are still running in the background. Even though they are not displayed in the foreground or directly operated by the user, they continue to perform their functions or tasks within the operating system and still consume system resources on the smart TV. By accessing the multitasking interface, users can switch to other background applications and clear background applications by swiping up on the selected application (also called a task), thus facilitating multitasking switching and management.

[0135] Drop-down menu operation

[0136] Figure 18 This is a schematic diagram illustrating a drop-down menu gesture provided in an embodiment of this application. When the remote control detects a sliding motion from the upper edge of the first touch area of ​​the remote control to the center of the second touch area, the remote control can trigger the smart TV to execute a drop-down menu operation. The drop-down menu may include one or more of the following: the smart TV's notification center, quick function settings interface, etc. Figure 19 As shown, the aforementioned shortcut function settings interface may include control shortcut keys for functions such as wireless local area network (WLAN), Bluetooth, and flashlight. Users can quickly turn the corresponding functions on or off using these control shortcut keys. The aforementioned notification center is used to display notification messages pushed by various applications on the smart TV to the user. This application embodiment does not impose any special limitations on the specific content included in the drop-down menu.

[0137] Drop-down search operation

[0138] Figure 20 This is a schematic diagram illustrating a pull-down search gesture provided in an embodiment of this application. When the remote control detects a swipe operation from a non-edge position in a first sub-region of the first touch area of ​​the remote control towards the center of the second touch area, the remote control can trigger the smart TV to perform a pull-down search operation. For example, Figure 21 This shows the interface of a smart TV after performing a drop-down search. Figure 21The interface shown can display a search box on the smart TV, which can be used for online searches or searches of files and applications within the device. Optionally, the drop-down search interface can also display search history. Optionally, the drop-down search interface can also include trending applications pushed by the system based on big data, such as the top n applications searched by system users in the most recent second time period. The values ​​of the aforementioned second time period and n can be preset by the developers. It is understood that the smart TV will only trigger the display of the aforementioned search bar when the drop-down search operation is detected and the smart TV is displaying a desktop page. Figure 21 The search box shown. The aforementioned desktop page can also include the first desktop, the second desktop, and the third desktop page. Otherwise, even if the remote control detects the aforementioned drop-down search operation, it will not trigger the smart TV to display the search box shown. Figure 21 The drop-down search box shown.

[0139] In some embodiments, the remote control can also perform screenshot operations such as full-screen screenshot, scrolling screenshot, and area screenshot by tapping. Among these, the user's long-press on the second touch area and movement of the cursor position, such as scrolling screenshot and area screenshot, is also referred to as a third input. When a smart TV performs a screenshot operation based on the third input, the screenshot area is determined by the cursor's movement trajectory. This will be explained in more detail below.

[0140] Full-screen screenshot operation

[0141] Figure 22 This is a schematic diagram illustrating a full-screen screenshot gesture provided in an embodiment of this application. Figure 22 As can be seen, when the remote control detects a double-click operation on the second touch area, it can trigger the TV to perform a screenshot operation on the entire screen. In other words, the content captured by the full-screen screenshot is limited to the content that can be displayed on the current page.

[0142] Scrolling screenshot operation

[0143] Scrolling screenshot is a feature that automatically scrolls the screen and continuously captures page content, suitable for scenarios such as long web pages, chat logs, and long microblog posts. With scrolling screenshot, users can capture the entire page content at once without manually stitching together multiple screenshots.

[0144] Figure 23 This is a schematic diagram illustrating the operation of a scrolling screenshot gesture provided in an embodiment of this application. Figure 23As can be seen, when controlling a smart TV, after the remote control detects a long press on the second touch area (i.e., the confirmation button), if the object selected by the cursor is non-draggable, the long press can trigger the smart TV to track the cursor's movement trajectory during the long press. If the detected movement trajectory is an "S" shape, the smart TV will execute the aforementioned scrolling screenshot operation.

[0145] For example, Figure 24A This is a scrollable display interface for a smart TV provided in this application embodiment. Figure 24B yes Figure 24A The page shown is the display interface obtained after scrolling down the right scroll bar. When the remote control detects the above long press operation, since there is no draggable object at the current cursor position, the long press operation can trigger the smart TV to... Figure 24A The interface shown tracks the cursor's movement trajectory. If the current movement trajectory is detected as... Figure 23 The "S" shape shown indicates that the smart TV can perform the aforementioned scrolling screenshot operation to obtain... Figure 24C .like Figure 24C As shown, the figure is composed of Figure 24A and Figure 24B The interface is obtained by splicing together and removing duplicate parts.

[0146] Area screenshot operation

[0147] Region screenshot refers to the function of capturing a selected area within the displayed interface. With this function, users can directly capture a portion of the page content without further cropping. Furthermore, compared to traditional rectangular cropping, the region screenshot function allows users to crop the page into free-form shapes, thus better adapting to user needs and preferences.

[0148] like Figure 25 As shown, similar to the scrolling screenshot operation, for smart TVs, after the remote control detects a long press on the second touch area, it can trigger the smart TV to track the cursor's movement trajectory during the long press. If the detected cursor movement trajectory forms a closed trajectory, the smart TV can perform a screenshot operation on that closed trajectory area. Similar to the scrolling screenshot operation, the prerequisite for tracking the cursor's movement trajectory is that no draggable object is detected at the cursor's location during the long press operation.

[0149] by Figure 25For example, if the cursor's movement trajectory forms a closed loop resembling a cloud, then the screenshot will capture the content within that cloud-shaped area. Understandably, when the remote control detects the end of the long-press operation, it can send a notification message to the smart TV. Optionally, after receiving the notification message indicating the end of the long-press operation, the smart TV can identify a preset screenshot shape most similar to the cursor's movement trajectory using methods such as neural network models, and use this preset screenshot shape to replace the area defined by the cursor's movement trajectory to take a screenshot of the page. In this process, the final screenshot shape used for the screenshot can completely cover the area involved in the cursor's movement trajectory, and the area covered by the final screenshot shape is close to the area defined by the cursor's movement trajectory, with an error range less than a first error threshold. This first error threshold is preset by the developers.

[0150] In some embodiments, when the smart TV receives a notification message indicating the end of the long-press operation, and detects that the cursor's movement trajectory during the long-press operation fails to form a closed trajectory, the smart TV can complete the cursor's movement trajectory during the long-press operation. Optionally, the smart TV can directly connect the start and end points of the cursor's movement trajectory to complete the cursor's movement trajectory during the long-press operation. Optionally, the smart TV can identify the shape most similar to the cursor's movement trajectory based on methods such as neural network models, and then use that shape to complete the cursor's movement trajectory. Not limited to the above methods, this application embodiment does not impose special restrictions on how to specifically complete the cursor's movement trajectory during the long-press operation.

[0151] The gesture operations described above are all based on, for example... Figure 26 The architecture of the control system shown is implemented.

[0152] like Figure 26 As shown, the remote control may contain a touch chip, which can be used to report detected touch data to the smart TV. The smart TV includes a gesture recognition algorithm and an event processing system. The gesture recognition algorithm is responsible for parsing the touch data and recognizing the user's gestures based on the data, while the event processing system is responsible for executing corresponding actions or commands based on the gesture recognition results. It is understood that the gesture recognition algorithm and event processing system can be provided by the smart TV's operating system.

[0153] Figure 27 This is a flowchart illustrating the internal implementation of a device control method provided in an embodiment of this application. The following is a description of the process... Figure 27 Let's introduce in detail the basis Figure 26 The architecture of the control system shown demonstrates how to control a smart TV using the gestures described above.

[0154] S101, The remote control reports touch data to the smart TV.

[0155] Specifically, the aforementioned touch data may include the position coordinates of touch points. A touch point refers to a specific touch unit on the touchpad that the user touches or clicks when interacting with the smart TV via the touchpad on the remote control. The position coordinates of these touch points are captured by the remote control and sent (i.e., reported) to the smart TV as touch data in order to perform corresponding operations or execute commands.

[0156] Optionally, the remote control can report touch data to the smart TV at a certain frequency. Figure 28 This is a schematic diagram of a touch data reporting mechanism provided in an embodiment of this application. Figure 28 The following introduction will use a 4x4 touch area as an example. Figure 28 One of the squares is a touch unit, then Figure 28 The touch area shown consists of 16 touch units. A coordinate system is established with the horizontal axis as the x-axis, the vertical axis as the y-axis, and the top-left corner of the touch area as the origin. This allows us to determine the position coordinates of each touch unit. For example, the coordinates of the first touch unit in the first row are (1, 1), and the coordinates of the fourth touch unit in the fourth row are (4, 4). If the user swipes across the touch units in the second row from left to right, the remote control can detect the touch unit at position coordinates (1, 2) being touched at time T1. At this time, the remote control's touch chip can report the touch data (1, 2) to the smart TV. Similarly, the remote control can detect the touch unit at position coordinates (2, 2) being touched at time T2, the touch unit at position coordinates (3, 2) being touched at time T3, and the touch unit at position coordinates (4, 2) being touched at time T4. The time intervals from T1 to T4 can be the same. Therefore, the final touch data reported by the remote control to the smart TV is a sequence of position coordinates: (1, 2), (2, 2), (3, 2), (4, 2). Understandably, if a user touches a particular touch unit for an extended period (e.g., when entering a multitasking interface and needing to stay in the second touch area), the position coordinates of that touch point will appear multiple times in the touch data reported by the remote control to the smart TV.

[0157] The touch data reported by the remote control can be either single-channel or multi-channel. For example, when a user operates on the touchpad using two fingers, such as zooming in or out, the remote control can report dual-channel touch data, with each channel corresponding to the touch data of one finger.

[0158] When the first touch area is pressed, the directional keys on the remote control are also pressed, and when the second touch area is pressed, the confirmation key on the remote control is also pressed. Therefore, in addition to position coordinates, the touch data also includes notification data. For example, when the confirmation key is detected as pressed, the remote control can send notification data A to the smart TV; correspondingly, when the confirmation key is detected as released, the remote control can send notification data B to the smart TV. For another example, the first touch area can be divided into a first sub-area, a second sub-area, a third sub-area, and a fourth sub-area. When the first sub-area is pressed, the remote control can send notification data C to the TV to notify the smart TV that the up directional key on the remote control has been pressed; when the second sub-area is pressed, the remote control can send notification data D to the TV to notify the smart TV that the left directional key on the remote control has been pressed; when the third sub-area is pressed, the remote control can send notification data E to the TV to notify the smart TV that the right directional key on the remote control has been pressed; and when the fourth sub-area is pressed, the remote control can send notification data F to the TV to notify the smart TV that the down directional key on the remote control has been pressed.

[0159] S102, The smart TV preprocesses the touch data.

[0160] The above preprocessing of touch data includes coordinate transformation, jitter filtering, and polynomial fitting interpolation.

[0161] The coordinate transformation described above refers to proportionally converting the position coordinates reported by the remote control, based on the remote control's coordinate system and resolution, to position coordinates based on the TV's coordinate system and resolution. Notification data within the touch data does not require transformation. The transformed position coordinates are also referred to as the first transformed data. It is understandable that touchpads in different remote controls may have different resolutions. Since gesture recognition based on touch data is performed by the smart TV, transforming the position coordinates in the touch data to a unified position coordinate system and resolution based on the smart TV is more beneficial for the smart TV to perform subsequent processing and gesture recognition. If the coordinates of the touch point on the remote control are (X1, Y1), the resolution of the touchpad on the remote control is Xresl×Yresl, and the resolution of the smart TV display is Xres2×Yres2, where the units of the resolution and coordinates are pixels, then the smart TV can obtain the proportionally transformed position coordinates (X2, Y2) using the following formula:

[0162]

[0163]

[0164] by Figure 29 The coordinate conversion shown is an example; the remote control's touchpad resolution is 400 pixels × 400 pixels, and according to... Figure 29As shown in (b), the resolution of the smart TV is 3840 pixels × 2160 pixels. If the coordinates of the touch point on the remote control are (200, 200), then the coordinates of the touch point after conversion calculated by the above formula are (1920, 1080).

[0165] Subsequently, the smart TV can process the aforementioned first transformation data, including dithering filtering and polynomial fitting interpolation. Compared to the first transformation data before processing, the processed first transformation data has higher accuracy and continuity.

[0166] It's understandable that data may fluctuate or jitter during sensor sampling or user touch operations. This is especially true for devices like remote controls with lower touchpad resolution, where the amount of data collected may be insufficient. Fluctuations in some data can have a greater impact on subsequent gesture recognition results. Therefore, if... Figure 30 As shown, the jitter filtering process described above is used to eliminate these random or unnecessary variations, making the data more stable, thereby improving the smoothness of the data and ultimately improving the accuracy of the data.

[0167] For example, a smart TV can perform the above jitter filtering process using the following formula (Gaussian filter formula):

[0168]

[0169] In the above formula, x represents the horizontal coordinate of the touch point in the smart TV coordinate system after the coordinate transformation. Continuing the example above, for the transformed coordinates (1920, 1080), x is 1920. σ in the above formula is the standard deviation, which determines the smoothness of the processed data. The larger σ is, the smoother the filtered data. Data smoothing refers to the amplitude of data fluctuations in the data sequence; the smaller the fluctuation amplitude, the smoother the data. Understandably, the value of σ can be preset by the developers before performing the above jitter filtering process. π and e are mathematical constants. G(x) obtained through the above formula is the vertical coordinate y of the touch point in the smart TV coordinate system after filtering. Not limited to the above Gaussian filtering formula, smart TVs can also use other formulas to perform the above jitter filtering process; this application embodiment does not impose special limitations on this.

[0170] Because remote controls have low resolution, they report a small amount of touch data. In contrast, televisions have high resolution, and the coordinates converted to touch data on a smart TV might be filtered out as noise. Therefore, the converted touch data needs to be processed as follows: Figure 31 The polynomial fitting interpolation shown expands the amount of data for the touch trajectory. This is understandable. Figure 31This is only to demonstrate the effect of polynomial fitting interpolation and does not restrict the distribution of the data.

[0171] Specifically, a smart TV can first determine the order of the polynomial to be fitted. For example, a smart TV can determine the polynomial to be fitted as follows:

[0172] p(x) = w0 + w1x + w2x 2 +w3x 3

[0173] The aforementioned polynomial (i.e., the order of the polynomial) can be pre-set in the smart TV by the developers. Subsequently, the smart TV can solve the polynomial based on the coordinates of the received touch point to obtain the coefficients w0, w1, w2, and w3. Optionally, the smart TV can use methods such as the least squares method to solve the polynomial. Not limited to the least squares method, other methods that can obtain the coefficients w0, w1, w2, and w3 (such as Gaussian elimination) can also be used to solve the polynomial here; this application embodiment does not impose any special limitations on this. After obtaining the aforementioned polynomial p(x), the smart TV can use the aforementioned polynomial p(x) to interpolate the new horizontal coordinate x value to obtain new data points (such as...). Figure 31 (The black dot in the center blank) thereby increasing the amount of data of the touch trajectory.

[0174] Understandably, preferably, the smart TV can sequentially perform the aforementioned coordinate transformation, data jitter filtering, and polynomial fitting interpolation operations to improve the accuracy and continuity of touch data. Optionally, the smart TV can perform only one or more of the aforementioned data preprocessing steps.

[0175] S103, the smart TV recognizes gestures based on processed touch data.

[0176] Figure 32 This is a schematic diagram illustrating the reporting of touch data upon detecting a magnification gesture, provided in an embodiment of this application. Figure 32 As shown, the smart TV can determine that the gesture is a swipe in the opposite direction by using the data reported by the two channels of the remote control at times T1 and T2, and thus determine that the gesture is a magnification gesture. Furthermore, based on the ratio of the distance between the touch points at time T1 to the distance between the touch points at time T2, the smart TV can calculate the magnification factor.

[0177] The above process for recognizing gestures based on processed touch data takes the zoom-in operation as an example. The recognition process for other gesture operations can also refer to the above process. Based on the position coordinates of the touch points in the touch data reported by the remote control and the timing relationship between the position coordinates of each touch point, the starting position, sliding direction, touch frequency, hover position, and hover time of the sliding operation are determined, thereby determining what specific gesture it is. The correspondence between the starting position, sliding direction, touch frequency, hover position, and hover time of the sliding operation and the gesture can be referred to the previous text, and will not be repeated here.

[0178] Understandably, the m touch units at the edge of the first touch area of ​​the remote control are considered the edge touch area. For gesture operations such as drop-down menu operations and drop-down search operations, the swiping direction is the same. The smart TV can determine whether the swiping operation starts from the upper edge of the first touch area by whether the starting position of the swipe operation (i.e., the first touch point) falls within the aforementioned edge touch area. If it does, it can be identified as a drop-down menu operation; otherwise, it can be identified as a drop-down search operation. The value of m can be preset by the R&D personnel.

[0179] Understandably, for operations such as returning, returning to the desktop, entering the multitasking interface, drop-down menus, drop-down search, and double-tap to take a screenshot, smart TVs can directly recognize the gestures based on the processed touch data. However, for operations such as zooming in, zooming out, rotating, scrolling to take a screenshot, and taking a screenshot of a region, smart TVs still need to obtain the current cursor position to further determine the gestures made by the user on the remote control.

[0180] Specifically, for scrolling screenshot and area screenshot operations, the smart TV can determine that the remote control received a long press operation based on the touch data reported by the remote control. The smart TV can then obtain the cursor's movement trajectory during the long press operation. If the cursor's movement trajectory is S-shaped, the gesture is a scrolling screenshot gesture; if the cursor's movement trajectory forms a first closed trajectory, the gesture is an area screenshot gesture. During this process, the smart TV can determine whether the remote control received a long press operation by the time difference between receiving notification data B and notification data A. If the time difference is greater than or equal to a third time interval, the smart TV determines that the remote control received a long press operation; conversely, if the time difference is less than the third time interval, the smart TV determines that the remote control did not receive a long press operation. For zooming in, the smart TV can obtain the current cursor position as the zoom center. Similarly, in zooming out, the smart TV can determine the zoom center by obtaining the cursor position; in rotating, the smart TV can determine the rotation center by obtaining the cursor position.

[0181] Understandably, the cursor's location (i.e., its coordinates on the smart TV screen) is determined by the remote control's direction. Specifically, the cursor's coordinates on the smart TV screen are calculated by the smart TV based on IMU data and UWB positioning data. The coordinate transformation performed during the aforementioned touch data preprocessing is only used for gesture recognition and does not represent the cursor's location. Therefore, if the smart TV needs to know the cursor's location when recognizing gestures, the gesture recognition algorithm needs to additionally obtain the calculated cursor coordinates from the smart TV.

[0182] S104. The smart TV performs the corresponding operation based on the recognition result report.

[0183] Understandably, regardless of the interface the smart TV is on when the user performs the gesture operation, the smart TV can perform operations such as zooming in, zooming out, various screenshot operations (including double-tap screenshot, scroll screenshot, and area screenshot), entering the multitasking interface, and drop-down menu operations. Therefore, for these gestures that do not have special requirements on the smart TV's page state, the smart TV can report the event to the event processing system based on the recognition result after performing the above step S103, and then the smart TV can perform the corresponding operation.

[0184] However, for some gesture operations, such as the aforementioned back, return to desktop, pull-down search, and rotation gestures, these actions will only trigger the smart TV to execute the corresponding operation when the smart TV is displaying a specific page. In other words, after recognizing these gestures, the smart TV should also check the current page state. Only if the current page state meets the preset triggering conditions can the smart TV report the recognition results and execute the corresponding operation.

[0185] Taking page A currently displayed on the smart TV as an example, for the return operation, page A should be a non-desktop page. For example, if page A can be the user interface of the first application, then page B can be displayed on the smart TV. This page B can be a desktop page or a return page from page A in the first application. The return page from page A in the first application (i.e., page B) is not limited to the parent page of page A. This embodiment does not impose any special restrictions on which page B the smart TV jumps to when returning. In other words, if page A is a desktop page, then performing a return operation on the desktop page will not result in a corresponding page B on the smart TV. Therefore, the page state does not meet the triggering condition, and the smart TV will not report and execute the return to desktop operation. Similarly, for the return to desktop operation, the triggering condition is that the currently displayed page (such as page A) is a non-desktop page. For the pull-down search operation, the triggering condition is that the currently displayed page on the smart TV is a desktop page; for the rotation operation, the triggering condition is that the currently displayed page is an image preview page.

[0186] In the above method steps S101 to S104, step S102 is optional, that is, the smart TV can directly recognize gestures based on the touch data reported by the remote control, thereby simplifying the complexity of the method execution and reducing the occupation of the smart TV's computing resources.

[0187] Understandably, the aforementioned remote control is developed based on traditional remote controls; therefore, it is compatible with the physical buttons found in traditional remote controls. For example... Figure 3 As shown, the touchpad in the aforementioned remote control is actually a continuous touch area composed of two independent hardware components. This is because in the traditional industrial design of remote controls, the touchpad is located where the confirmation button and directional buttons are positioned, and these two physical buttons are independent of each other. Therefore, in situations like... Figure 3 In the remote control shown, the touchpad of the remote control continues the above design in terms of hardware design, that is, the first touch area and the second touch area of ​​the remote control touch area are provided by two hardware modules respectively. However, in terms of software implementation, the touch area of ​​the remote control is a whole continuous touch area. That is, when the remote control detects a gesture in the touch area of ​​the remote control, the touch data reported to the smart TV by the touch points of both the first touch area and the second touch area adopts a unified coordinate system.

[0188] like Figure 29(a) exemplarily illustrates a unified coordinate representation of the touch area of ​​a remote control. The origin of the coordinate system is the top-left corner of a rectangle formed by the diameter of the circle containing the touch area of ​​the remote control, with the top edge of the rectangle as the x-axis and the left edge as the y-axis. It is understandable that, using this method to construct the coordinate system, the position coordinates of some areas within the system (such as the portion of the rectangle excluding the circle containing the touch area of ​​the remote control), although existing in the coordinate system, will never be detected by the touchpad and thus never reported to the smart TV. Furthermore, although the touch area can be circular overall, the arrangement of the touch units within the touch area can be... Figure 28 or Figure 32 The touch units in the rectangular touch areas shown are arranged identically. Therefore, optionally, the aforementioned remote control can be designed with this in mind during the hardware design phase. Figure 29 The coordinate system (a) in the above-mentioned touch area is used to assign coordinates to each touch unit, so that continuously changing touch data can be reported to the smart TV when the touch point crosses different hardware structures. Optionally, the two independent touchpads in the remote control can report data to the central processing unit of the remote control in their respective selected coordinate systems. Then, the central processing unit calculates the touch data in the same coordinate system based on the touch data in different coordinate systems. The central processing unit can then send the calculated touch data to the touch chip, so that the touch chip can report the calculated touch data in the unified coordinate system to the smart TV.

[0189] Understandable, because Figure 3 The remote control shown is compatible with the physical buttons of traditional remote controls. Therefore, the first and second touch areas of the remote control are actually provided by two independent touchpads in terms of hardware design. Thus, compared to using a single touchpad to implement the touch areas of the remote control, by precisely distinguishing the starting point, ending point, and direction of the slide, such as... Figure 3 The method shown, which uses two independent touchpads connected to form a continuous touch area, provides more space for gesture operation, such as... Figure 3 The remote control shown can support more gestures.

[0190] The following example illustrates the swipe-down gesture. If the remote control's touch area consists of a touchpad, swiping down on this touchpad can be done in two ways: from the top edge or from a non-top edge. Understandably, for a touchpad, swiping down from the non-top edge and from the center is not clearly distinguishable because, unlike a display screen, the touchpad doesn't explicitly show the user the boundary between the non-top edge and the center. Therefore, the user can only distinguish the action by whether it crosses the edge of the touchpad. However, if the remote control's touch area is like... Figure 3 As shown, the device consists of two touchpads. For the user, the boundary between the two touchpads provides a clear distinction between actions. Therefore, even with the same swipe gesture, the difference is noticeable for actions like... Figure 3 The remote control shown can more precisely distinguish multiple functions: for example, sliding from the outer edge of the first touch area to near the edge of the second touch area can adjust the volume; sliding from the outer edge of the first touch area to the center of the second touch area can access the drop-down menu; and sliding down within the second touch area can access the drop-down search box. Therefore, as can be seen... Figure 3 The remote control shown uses two touchpads to report data, which can greatly enrich the ways users can control smart TVs with gestures.

[0191] Understandably, in the above device control methods, the gesture recognition step is implemented on the smart TV side. However, it's not limited to the smart TV side; gesture recognition can also be performed on the remote control side, thereby reducing the computational burden on the smart TV and allowing it to focus more on display and other tasks.

[0192] Specifically, the remote control can recognize gestures based on detected touch data and send a gesture notification to the smart TV after gesture recognition. This gesture notification may include information indicating the recognized gesture. For example, if the remote control detects a gesture such as... Figure 28 The touch data shown indicates that the position coordinates (1, 2) fall within the edge touch area. The remote control can then recognize this gesture as a "return" gesture based on its built-in gesture recognition algorithm. The remote control can then send a gesture notification to the smart TV, including the gesture number 01 used for the return gesture. The smart TV can then execute the return operation based on the received gesture notification. It is understood that the information of the gesture recognized (such as the gesture number) can be pre-set by the developers in the remote control.

[0193] Understandably, for scrolling screenshots and area screenshots, since the remote control doesn't capture the cursor's movement trajectory, the gesture notification sent by the remote to the smart TV indicates a long press operation. After receiving this notification, the smart TV still needs to further analyze the cursor's movement trajectory to determine whether the user's gesture is a scrolling screenshot or an area screenshot. Similarly, for zooming in, zooming out, and rotating operations, after the remote control sends the corresponding gesture notification to the smart TV, the smart TV can further confirm the zoom center, zoom out center, or rotation center based on the cursor's current position.

[0194] Not limited to this, similar to implementing gesture recognition on smart TVs, for gestures that can only trigger the smart TV to perform the corresponding operation when a specific page is displayed, the smart TV also needs to detect the current page state after receiving the gesture notification. Only if the current page state meets the preset triggering conditions can the smart TV report and execute the corresponding operation based on the recognition result. Specific details can be found in the previous descriptions and will not be repeated here.

[0195] Figure 33 This is a schematic diagram of the structure of a device provided in an embodiment of this application, which may include the aforementioned smart TV.

[0196] like Figure 33 As shown, the device may include components such as a processor 211, a memory 212, a wireless communication processing module 213, a power switch 214, a display screen 215, an audio module 216, and a speaker 217. The components within the device are connected via a bus and communicate with each other based on the bus.

[0197] Processor 211 may include one or more processing units, such as application processors, modem processors, graphics processors, image signal processors, controllers, video codecs, digital signal processors, baseband processors, and / or neural network processors. These different processing units may be independent devices or integrated into one or more processors. The controller can generate operation control signals based on instruction opcodes and timing signals to control instruction fetching and execution.

[0198] Memory 212 is coupled to processor 211 and is used to store various software programs and / or multiple sets of instructions. Memory 212 can be used to store computer executable program code, which includes instructions. Processor 211 executes various functional applications and data processing of the device by running the instructions stored in memory 212. Memory may also be provided in processor 211 for storing instructions and data.

[0199] The memory 212 may include one or more random access memories (RAMs) and one or more non-volatile memories. The RAMs can be directly read and written by the processor 211. The RAMs can be used to store executable programs (e.g., machine instructions) of the operating system or other running programs, as well as user and application data. The non-volatile memories can also store executable programs and user and application data. Executable programs, i.e., user data, stored in the non-volatile memories can be pre-loaded into the RAMs for direct reading and writing by the processor 211.

[0200] The executable program code and user data used to implement the device control method provided in the embodiments of this application can be stored in non-volatile memory. During the implementation of the above-described device control method, the device can load the non-volatile memory executable program code and user data into random access memory to provide the user with an efficient and convenient device control method.

[0201] The wireless communication processing module 213 can provide wireless communication solutions including WLAN, such as Wi-Fi, Bluetooth communication, ZigBee communication, NFC communication, infrared communication, and UWB communication. In this embodiment, the device can receive UWB signals sent by the remote control based on the wireless communication function provided by the wireless communication processing module 213, thereby determining the precise position of the remote control relative to the device and thus determining the direction of the remote control. In addition, the wireless communication processing module is also used to receive touch data reported by the remote control, which is then used for subsequent gesture recognition.

[0202] The power switch 214 can be used to control the power supply to the device, thereby supplying power to the processor 211, memory 212, wireless communication processing module 213, display screen 215, audio module 216, speaker 217, etc.

[0203] Display screen 215 is used for display. Display screen 215 includes a display panel. The device can implement display functions through GPU, display screen 215, and application processor, etc. In the embodiments of this application, the device can display various pages and simultaneously display user interactions, such as page switching after remote control operation, through the display functions provided by GPU, display screen 215, and application processor, etc.

[0204] Audio module 216 can be used to convert digital audio signals into analog audio signals for output, and can also be used to convert analog audio input into digital audio signals. Speaker 217 can be used to convert the audio signals transmitted by audio module 216 into sound signals. The device can implement audio playback functions through audio module 216, speaker 217, etc. In some embodiments, audio module 216 may also include a microphone for converting sound signals into electrical signals.

[0205] It is understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on the device. In other embodiments of this application, the device may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. Components may be implemented in hardware, software, or a combination of software and hardware.

[0206] Figure 34 This is a schematic diagram of the structure of a remote control provided in an embodiment of this application.

[0207] like Figure 34 As shown, the remote control may include components such as a processor 311, a memory 312, a wireless communication processing module 313, a power switch 314, a touch chip 315, and an IMU 316. The components in the device are connected via a bus and communicate with each other based on the bus.

[0208] Processor 311 may include one or more processing units, such as application processors, modem processors, graphics processors, image signal processors, controllers, video codecs, digital signal processors, baseband processors, and / or neural network processors. These different processing units may be independent devices or integrated into one or more processors. The controller can generate operation control signals based on instruction opcodes and timing signals to control instruction fetching and execution.

[0209] Memory 312 is coupled to processor 311 and is used to store various software programs and / or multiple sets of instructions. Memory 312 can be used to store computer executable program code, which includes instructions. Processor 311 executes various functional applications and data processing of the device by running the instructions stored in memory 312. Memory may also be provided in processor 311 for storing instructions and data.

[0210] The memory 312 may include one or more random access memories (RAMs) and one or more non-volatile memories. The RAMs can be directly read and written by the processor 311. The RAMs can be used to store executable programs (e.g., machine instructions) of the operating system or other running programs, as well as user and application data. The non-volatile memories can also store executable programs and user and application data. Executable programs, i.e., user data, stored in the non-volatile memories can be pre-loaded into the RAMs for direct reading and writing by the processor 311.

[0211] The executable program code and user data used to implement the device control method provided in the embodiments of this application can be stored in non-volatile memory. During the implementation of the above-described device control method, the remote controller can load the non-volatile memory executable program code and user data into random access memory to provide the user with an efficient and convenient device control method.

[0212] Corresponding to wireless communication processing module 313 Figure 33 The wireless communication processing module 213 shown can provide wireless communication solutions including WLAN, such as Wi-Fi, Bluetooth communication, ZigBee communication, NFC communication, infrared communication, and UWB communication. It is understood that in this embodiment, the remote control can transmit UWB signals based on the wireless communication function provided by the aforementioned wireless communication processing module 213, enabling the device to determine the precise position of the remote control relative to the device, and thus determine the direction of the remote control. In addition, the aforementioned wireless communication processing module is also used to transmit touch data for subsequent gesture recognition.

[0213] The power switch 314 can be used to control the power supply to the device, thereby supplying power to the processor 311, memory 312, wireless communication processing module 313, touch chip 315, etc.

[0214] The touch chip 315 is used to detect the user's touch behavior on the touchpad, thereby generating touch data. In this embodiment, the touch chip 315 is used to generate touch data for various gestures, and sends the touch data to the device's wireless communication processing module 213 via the wireless communication processing module 313, thereby completing the reporting of touch data.

[0215] The IMU316 includes an accelerometer and a gyroscope, used to measure the acceleration and angular velocity of the remote control in three-dimensional space, respectively. In some embodiments, the IMU316 also includes a magnetometer for measuring the magnetic field around an object. In this embodiment, the IMU316 is used to collect IMU data, including but not limited to the tilt angle and acceleration data of the remote control, which is transmitted to the wireless communication processing module 313. Figure 33 The device shown can determine the pointing angle of the remote control in space, and then combine IMU data and UWB positioning data to determine the position pointed to by the remote control and display a cursor at that position.

[0216] Similarly, Figure 34 The schematic diagram shown is only an example and should not be construed as a specific limitation on the remote control.

[0217] Those skilled in the art will recognize that the functions described in the embodiments of this application in one or more of the above examples can be implemented using hardware, software, firmware, or any combination thereof. When implemented in software, these functions can be stored in a readable medium or transmitted as one or more instructions or code on a readable medium.

[0218] The above detailed embodiments further illustrate the purpose, technical solution, and beneficial effects of the embodiments of this application. It should be understood that the above are merely specific embodiments of the embodiments of this application and are not intended to limit the protection scope of the embodiments of this application. Any modifications, equivalent substitutions, improvements, etc., made on the basis of the technical solutions of the embodiments of this application should be included within the protection scope of the embodiments of this application.

Claims

1. A device control method characterized by, A remote controller applied in a control system, the control system further comprising a device, a touch area of the remote controller comprising a first touch area and a second touch area, the first touch area being a ring-shaped touch area, the first touch area surrounding the second touch area, the method comprising: The remote controller receiving a first input of a user through the touch area, the first input comprising a zoom gesture or a rotation gesture, the first input being a multi-point touch gesture, touch points of the first input being located in one or more of the following: the first touch area, the second touch area, the first input being used to trigger the device to perform a first operation on an object corresponding to a cursor, the first operation comprising a zoom operation or a rotation operation, the cursor being used to identify a position pointed by the remote controller on a screen of the device.

2. The method of claim 1, wherein, The first touch area in the remote controller comprising a first sub-area, a second sub-area, a third sub-area, a fourth sub-area, the first sub-area and the fourth sub-area being oppositely arranged, the second sub-area and the third sub-area being oppositely arranged, the first sub-area and the fourth sub-area being further used as up and down direction keys of the remote controller respectively, the second sub-area and the third sub-area being further used as left and right direction keys of the remote controller respectively; The second touch area being further used as a confirmation key of the remote controller.

3. The method according to claim 1 or 2, characterized in that, The first input being a zoom gesture, touch points of the zoom gesture comprising a first touch point and a second touch point, the zoom gesture comprising a zoom-out gesture and a zoom-in gesture, wherein the zoom-out gesture is a sliding operation of the first touch point and the second touch point moving towards each other, the zoom-in gesture being a sliding operation of the first touch point and the second touch point moving away from each other. Or, the first input being a rotation gesture, touch points of the rotation gesture comprising a third touch point and a fourth touch point, the rotation gesture being a sliding operation of the third touch point and the fourth touch point both moving along a clockwise or counterclockwise direction.

4. The method according to any one of claims 1 to 3, characterized in that, The method further comprising: Receiving a second input of a user through the touch area, the second input being used to trigger the device to perform a second operation, an operation object of the second operation not being determined by a position of the cursor, a type of the second operation being determined according to one or more of the following of the second input: a starting point of the second input, an ending point of the second input, a sliding direction of the second input, a position of a hovering of the second input, a time of the hovering of the second input.

5. The method of claim 4, wherein, The second input being a return gesture, the return gesture being a sliding gesture from a left edge or a right edge of the first touch area to a center of the second touch area; Or, the second input comprising a return desktop gesture, the return desktop gesture being a sliding operation from a lower edge of the first touch area to the center of the second touch area; Or, the second input comprising an entering multi-task interface gesture, the entering multi-task interface gesture being a sliding operation from the lower edge of the first touch area to the center of the second touch area and staying in the second touch area for a first time; Or, the second input includes a pull-down menu gesture, the pull-down menu gesture is a sliding operation from the upper edge of the first touch area to the center of the second touch area; Or, the second input includes a pull-down search gesture, the pull-down search gesture is a sliding operation from a region other than the upper edge in the first sub-region in the first touch area to the center of the second touch area.

6. The method according to any one of claims 1 to 5, characterized in that, The method further comprises: Receiving an operation of double-clicking the second touch area by the user, the operation of double-clicking the second touch area being used to trigger the device to perform a screenshot operation on the entire screen; Or, the remote controller receives a third input of long-pressing the second touch area by the user and moving the cursor position, the third input being used to trigger the device to perform a screenshot operation, and the screenshot region being determined by the moving track of the cursor.

7. The method according to any one of claims 1 to 6, characterized in that, The method further comprises: The remote controller sends touch data to the device, the touch data including the position coordinates of the touch point detected by the remote controller through the touch area.

8. A device control method characterized by, Applied to a device in a control system, the control system further comprising a remote controller, a touch area of the remote controller comprising a first touch area and a second touch area, the first touch area being a ring-shaped touch area, the first touch area surrounding the periphery of the second touch area, the method comprising: Detecting that the remote controller points to the device, the device displaying a cursor on the screen, the cursor being used to identify the position pointed to by the remote controller on the screen of the device; According to a first input, the device performs a first operation on the object corresponding to the cursor, the first operation including a zoom operation or a rotation operation, the first input being received by the remote controller through the touch area, the first input including a zoom gesture or a rotation gesture, the first input being a multi-point touch gesture, the touch point of the first input being located in one or more of the following: the first touch area, the second touch area.

9. The method of claim 8, wherein, The first input is a zoom gesture, the touch point of the zoom gesture including a first touch point and a second touch point, the zoom gesture including a zoom-out gesture and a zoom-in gesture, wherein the zoom-out gesture is a sliding operation of the first touch point and the second touch point moving towards each other, and the zoom-in gesture is a sliding operation of the first touch point and the second touch point moving away from each other, and according to the first input, the device performs a first operation on the object corresponding to the cursor, comprising: According to the zoom-out gesture, the device zooms out the content on the screen of the device with the position of the cursor as the center, the content on the screen of the device including the object corresponding to the cursor; Or, according to the zoom-in gesture, the device zooms in the content on the screen of the device with the position of the cursor as the center, the content on the screen of the device including the object corresponding to the cursor.

10. The method of claim 8, wherein, The first input is a rotation gesture, a touch point of the rotation gesture includes a third touch point and a fourth touch point, the rotation gesture is a sliding operation of the third touch point and the fourth touch point in a clockwise or counterclockwise direction, and the device performs a first operation on an object corresponding to the cursor according to the first input, including: According to the rotation gesture, the device rotates the content on the screen of the device with the position of the cursor as the center, and the rotation direction of the content on the screen of the device is the same as the sliding direction of the third touch point and the fourth touch point in the rotation gesture.

11. The method according to any one of claims 8-10, characterized in that, The method further includes: The device performs a second operation based on a second input received by the remote controller through the touch area, the operation object of the second operation is not determined by the position of the cursor, and the type of the second operation is determined according to one or more of the following: the starting point of the second input, the ending point of the second input, the sliding direction of the second input, the position of the hovering of the second input, and the time of the hovering of the second input.

12. The method according to any one of claims 8-11, characterized in that, The method further includes: According to the operation of double-clicking the second touch area, the device performs a screenshot operation on the entire screen, and the operation of double-clicking the second touch area is received by the remote controller through the touch area; Alternatively, the device performs a screenshot operation according to a third input, and the screenshot area is determined by the moving track of the cursor, and the third input is that the remote controller receives an input of a user long-pressing the second touch area and moving the position of the cursor.

13. A remote control, characterized in that The touch area of the remote controller includes a first touch area and a second touch area, the first touch area is an annular touch area, the first touch area surrounds the periphery of the second touch area, the remote controller includes one or more processors and one or more memories; the one or more memories are used to store a program, and when the one or more processors execute the program, the method of any one of claims 1-7 is executed.

14. An apparatus, comprising: including one or more processors and one or more memories; the one or more memories are used to store a program, and the program causes the method of any one of claims 8-12 to be executed when the one or more processors execute the program.

Images