Interaction method, readable storage medium, program product, device, and system

By transmitting control parameters between smart home devices and other devices and dynamically updating the interactive interface, the problems of limited remote control operation and high network resource consumption are solved, thereby improving user experience and device efficiency.

WO2024227366A1PCT designated stage expired Publication Date: 2024-11-07HUAWEI TECH CO LTD
View PDF 5 Cites 0 Cited by

Patent Information

Application Number
PCT/CN2024/071432
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Priority Date
2023-05-04
Filing Date
2024-01-09
Publication Date
2024-11-07

AI Technical Summary

Technical Problem

In existing technologies, the operation methods of remote controls for smart home devices are limited, and when users simulate remote controls using other devices, network resources are consumed significantly, affecting the user experience.

Method used

By transmitting control parameters between the first and second devices, rather than the entire display interface, dynamic updates of the interactive interface are achieved, reducing network resource consumption and supporting convenient control of cursor and navigation operations.

Benefits of technology

It improved the user experience, reduced network resource consumption, and avoided impacting other services.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN2024071432_07112024_PF_FP_ABST
    Figure CN2024071432_07112024_PF_FP_ABST
Patent Text Reader

Abstract

The present application relates to the technical field of communications. Disclosed are an interaction method, a readable storage medium, a program product, a device, and a system. In the method, a first device does not need to send specific content of the current display interface to a second device, but sends control parameters of some or all controls in the current display interface of the first device to the second device; after receiving the control parameters of the controls, the second device can display an operation frame and an operation prompt for each control; and if the second device detects a touch-control operation of a user for the operation frame of a certain control, the second device can send to the first device a touch-control event corresponding to the touch-control operation, such that the first device can trigger, in response to the touch-control event, a corresponding interaction event, for example, an event of switching display content in the control. In this way, network resources which are occupied during an interaction process can be reduced, and an impact on other services of the second device and the first device is avoided.
Need to check novelty before this filing date? Find Prior Art

Description

Interaction method, readable storage medium, program product, device and system

[0001] This application claims priority to the Chinese patent application filed with the China Patent Office on May 4, 2023, with application number 202310503346.2 and application name “Interactive Method, Readable Storage Medium, Program Product, Device and System”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of communication technology, and in particular to an interaction method, a readable storage medium, a program product, a device, and a system. Background Art

[0003] Smart home devices (such as smart TVs, smart screens, and projectors) are becoming increasingly popular. These devices are often equipped with remote controls, allowing users to interact with them.

[0004] To meet users' control needs for smart home devices, users can usually simulate the remote control of smart home devices through other devices (such as mobile phones, tablets, etc.), thereby enabling users to interact with smart home devices through other devices. For example, referring to Figure 1, the physical buttons on the remote control 30 of the smart screen 10 can be mapped to virtual buttons on the mobile phone 20, so that users can send control commands to the smart screen 10 by clicking the virtual buttons on the mobile phone 20, thereby interacting with the smart screen 10.

[0005] Generally speaking, the remote controls of smart home devices are usually in the form of buttons, so when users simulate the remote control through other devices, they only interact by clicking on the display screen of other devices, and the operation method is simple.

[0006] Summary of the Invention

[0007] In view of this, the present application provides an interactive method, a readable storage medium, a program product, a device, and a system.

[0008] A first aspect provides an interaction method, which includes: a first device displays a first interface, where the first interface includes a first control; the first device detects that a connection is established with a second device, and sends a control parameter corresponding to the first control to the second device, where the control parameter is used to instruct the second device to display a first operation box on the second interface; the first device receives a first event sent by the second device, where the first event is generated when the second device detects at least one operation of a user within the first operation box; in response to the first event, the first device displays a third interface, which is different from the first interface.

[0009] In this method, after the first device and the second device establish a connection, the control parameters of the first control in the first interface, such as the operation box, operation prompt, etc., can be sent to the second device, so as to instruct the second device to display the first operation box of the first control. The first device can display a third interface different from the first interface based on the first event (such as the first touch event below) received from the second device. In this way, the first device does not need to send the specific display content of the first interface to the second device, which helps to reduce the network resources occupied during the interaction process.

[0010] It should be understood that the first control is a control that can respond to touch events (such as a slide event, a pinch event, a pull event, a click event, etc.) to update the content in the control.

[0011] In a possible implementation of the first aspect, the third interface includes content in the first control updated in response to the first event. That is, the third interface is obtained by updating the content in the first control according to the first event.

[0012] In a possible implementation of the first aspect above, the above method also includes: the first device receives a second event sent by the second device, the second event is generated by the second device detecting that the user drags the indicator control in the second operation box in the second interface along the first direction; in response to the second event, the first device moves the cursor along the first direction; the first device receives a third event sent by the second device, the third event is generated by the second device detecting that the user releases the indicator control; in response to the third event, the first device stops moving the cursor.

[0013] That is, the first device can, based on the second event (e.g., the cursor move event below) sent by the second device, keep the mouse moving in the first direction until a third event (e.g., the cursor stop moving event below) is received. In this way, the user can move the cursor of the first device over a long distance by dragging the indicator control in the first direction to a certain position in the second operation box (e.g., the cursor operation box below) without having to repeatedly drag the indicator control, which helps to improve the user's interactive experience.

[0014] In one possible implementation of the first aspect, the speed of the cursor movement increases as the distance between the location of the indicator control and the center of the second operation box increases. In other words, the user can adjust the speed of the cursor movement on the first device by controlling the distance between the location of the indicator control and the center of the second operation box, which is convenient.

[0015] In a possible implementation of the first aspect above, the above method also includes: the first device receives a fourth event sent by the second device, and the fourth event is generated by the second device detecting a user's click operation in the second operation box; in response to the fourth event, the first device displays a fourth interface, and the fourth interface is different from the first interface.

[0016] That is to say, the user can control the second device to trigger a click event at the cursor position by clicking in the second operation box, thereby generating and displaying the fourth interface.

[0017] In a possible implementation of the first aspect above, the first operation box is located inside the third operation box; and the method also includes: the first device receives a fifth event sent by the second device, the fifth event being generated by the second device detecting that the user performs at least one operation from the edge of the third operation box or from outside the third operation box to inside the third operation box; in response to the fifth event, the first device displays a fifth interface, which is different from the first interface.

[0018] That is to say, the user can trigger the second device to display the fifth event (such as the second touch event below) and display the fifth interface by performing at least one operation from the edge of the third operation box (such as the touch operation box below) or from outside the third operation box to inside the third operation box.

[0019] It should be understood that the fifth interface can be realized by triggering a navigation interaction event corresponding to the fifth event on the first interface, such as triggering a return to the previous level event, a return to the desktop event, a recent task event, a display drop-down menu event, a stop running currently running application event, etc.

[0020] It should be understood that at least one operation from the edge of the third operation box or outside the third operation box to the inside of the third operation box may include any one of the following operations: a single-finger / multi-finger swiping down operation from the upper boundary of the third operation box or outside the upper boundary of the third operation box to the inside of the third operation box, a single-finger / multi-finger swiping up operation from the lower boundary of the third operation box or outside the lower boundary of the third operation box to the inside of the third operation box, a single-finger / multi-finger swiping right operation from the left boundary of the third operation box or outside the left boundary of the third operation box to the inside of the third operation box, and a single-finger / multi-finger swiping left operation from the right boundary of the third operation box or outside the right boundary of the third operation box to the inside of the third operation box.

[0021] It should be understood that the above-mentioned first control is automatically selected by the first device from one or more controls in the first interface, that is, only controls that meet the definition of the first control are selected.

[0022] In a possible implementation of the first aspect, the at least one operation within the first operation box may include at least one of the following operations: a sliding operation, a pinching operation, a pulling operation, and a clicking operation.

[0023] In a possible implementation of the first aspect above, the first device includes any one of a smart screen, a TV, a projector, and a display.

[0024] The second aspect provides an interaction method, which includes: a second device receives a control parameter sent by a first device, where the control parameter is a control parameter corresponding to a first control in a first interface displayed by the first device; the second device displays a second interface in response to the control parameter, where the second interface includes a first operation box corresponding to the first control; the second device detects at least one operation of the user within the first operation box, and sends a first event to the first device, where the first event is used to instruct the first device to display a third interface, which is different from the first interface.

[0025] In this method, after establishing a connection with the first device, the second device can receive control parameters for the first control in the first interface from the first device and display a first operation box for the first control based on the received control parameters. When the second device detects a user operation in the first operation box, it can send a corresponding first event to the first device, instructing the first device to display a third interface different from the first interface. This eliminates the need for the first device to receive the specific display content of the first interface from the second electronic device, which helps reduce network resources consumed during the interaction process.

[0026] It should be understood that the first control is a control that can respond to touch events (such as a slide event, a pinch event, a pull event, a click event, etc.) to update the content in the control.

[0027] In a possible implementation of the above-mentioned second aspect, the above-mentioned second interface also includes a second operation box, the second operation box includes an indicator control, and the above-mentioned method also includes: the second device detects that the user drags the indicator control along the first direction, sends a second event to the first device, and the second event is used to instruct the first device to move the cursor along the first direction; the second device detects that the user releases the indicator control, and sends a third event to the first device, and the third event is used to instruct the first device to stop moving the cursor.

[0028] That is, after detecting that the user is dragging the indicator control in the first direction, the second device can send a second event (e.g., a cursor move event) to the first device, instructing the first device to continue moving the cursor in the first direction until a third event (e.g., a cursor stop movement event) is received. This allows the user to move the cursor of the first device over long distances without repeatedly dragging the indicator control, which improves the user's interactive experience.

[0029] In a possible implementation of the second aspect above, the above method also includes: the second device detects a user's click operation in the second operation box, and sends a fourth event to the first device, and the fourth event is used to instruct the first device to display a fourth interface, which is different from the first interface.

[0030] In a possible implementation of the second aspect above, the above-mentioned second interface also includes a third operation box, and the first operation box is located inside the third operation box. The above-mentioned method also includes: the second device detects at least one operation of the user from the edge of the third operation box or outside the third operation box to the inside of the third operation box, and sends a fifth event to the first device. The fifth event is used to instruct the first device to display a fifth interface, and the fifth interface is different from the first interface.

[0031] That is, the user can trigger the second device to display the fifth event and the fifth interface by performing at least one operation from the edge of the third operation box or from outside the third operation box to inside the third operation box.

[0032] It should be understood that the fifth interface can be realized by triggering a navigation interaction event corresponding to the fifth event on the first interface, such as triggering a return to the previous level event, a return to the desktop event, a recent task event, a display drop-down menu event, a stop running currently running application event, etc.

[0033] It should be understood that at least one operation from the edge of the third operation box or outside the third operation box to the inside of the third operation box may include any one of the following operations: a single-finger / multi-finger swiping down operation from the upper boundary of the third operation box or outside the upper boundary of the third operation box to the inside of the third operation box, a single-finger / multi-finger swiping up operation from the lower boundary of the third operation box or outside the lower boundary of the third operation box to the inside of the third operation box, a single-finger / multi-finger swiping right operation from the left boundary of the third operation box or outside the left boundary of the third operation box to the inside of the third operation box, and a single-finger / multi-finger swiping left operation from the right boundary of the third operation box or outside the right boundary of the third operation box to the inside of the third operation box.

[0034] In a possible implementation of the second aspect, the first operation box includes an operation prompt corresponding to the first control.

[0035] By displaying the operation prompt in the first operation box, the user is helped to understand the touch event types that the first operation box can respond to, thereby improving the user's interactive experience.

[0036] It should be understood that the above-mentioned first control is automatically selected by the first device from one or more controls in the first interface, that is, only controls that meet the definition of the first control are selected.

[0037] In a possible implementation of the second aspect, the at least one operation in the first operation box includes at least one of the following operations: a sliding operation, a pinching operation, a pulling operation, and a clicking operation.

[0038] In a possible implementation of the second aspect, the second device includes any one of a mobile phone, a tablet computer, and a wearable device.

[0039] A third aspect provides a readable storage medium, which includes instructions. When the instructions are executed by a device, the device implements any one of the interaction methods provided in the first aspect or the second aspect.

[0040] A fourth aspect provides a first device, which includes: a memory for storing instructions; a processor for executing instructions so that the first device implements the above-mentioned first aspect and any one of the possible implementations of the above-mentioned first aspect.

[0041] The fifth aspect provides a second device, which includes: a memory for storing instructions; a processor for executing instructions so that the second device implements the above-mentioned second aspect and any one of the possible implementations of the above-mentioned second aspect.

[0042] The sixth aspect provides a system, which includes any one of the first devices provided by the fourth aspect and any one of the second devices provided by the fifth aspect.

[0043] A seventh aspect provides a program product, which, when running on a device, enables the device to implement any one of the interaction methods provided in the first aspect or the second aspect.

[0044] It should be understood that the beneficial effects of the third to seventh aspects mentioned above can be referred to the description of the first and second aspects mentioned above, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] FIG1 is a schematic diagram showing a scenario in which a user uses a mobile phone 20 to interact with a smart screen 10 by simulating a remote control 30 according to some embodiments of the present application;

[0046] FIG2 shows a schematic diagram of a scenario in which interaction is achieved by transmitting the display interface of the smart screen 10 to the mobile phone 20 according to some embodiments of the present application;

[0047] FIG3 shows a schematic diagram of entering an interactive scenario by transmitting control parameters of a control according to some embodiments of the present application;

[0048] FIG4A is a schematic diagram showing an interactive scenario of adjusting the content in the menu control 11 of the smart screen 10 by sliding the mobile phone 20 according to some embodiments of the present application;

[0049] FIG4B is a schematic diagram showing an interactive scenario of adjusting the content in the recommendation control 12 of the smart screen 10 by sliding on the mobile phone 20 according to some embodiments of the present application;

[0050] FIG5A is a schematic diagram showing another method of entering an interactive scenario by transmitting control parameters of a control according to some embodiments of the present application;

[0051] FIG5B is a schematic diagram showing an interactive scenario of enlarging the playback control W1 in the smart screen 10 by pulling out the control in the mobile phone 20 according to some embodiments of the present application;

[0052] FIG5C is a schematic diagram showing an interactive scenario of reducing the playback control W1 on the smart screen 10 by a pinch operation on the mobile phone 20 according to some embodiments of the present application;

[0053] FIG6 is a schematic diagram showing another method of entering an interactive scenario by transmitting control parameters of a control according to some embodiments of the present application;

[0054] FIG7A is a schematic diagram illustrating an interactive scenario of moving the cursor position of a smart screen 10 by dragging the mobile phone 20 according to some embodiments of the present application;

[0055] FIG7B is a schematic diagram illustrating an interaction scenario in which a click operation on a mobile phone 20 triggers an interaction event corresponding to the click event on a smart screen 10 according to some embodiments of the present application;

[0056] FIG8 is a schematic diagram showing another method of entering an interactive scenario by transmitting control parameters of a control according to some embodiments of the present application;

[0057] FIG9A is a schematic diagram illustrating a scenario of implementing navigation interaction with a smart screen 10 through a sliding operation on a mobile phone 20 according to some embodiments of the present application;

[0058] FIG9B is a schematic diagram showing another scenario of implementing navigation interaction with the smart screen 10 through a sliding operation on the mobile phone 20 according to some embodiments of the present application;

[0059] FIG9C is a schematic diagram illustrating another scenario of implementing navigation interaction with the smart screen 10 through a sliding operation on the mobile phone 20 according to some embodiments of the present application;

[0060] FIG10 is a schematic diagram showing an interaction flow of an interaction method according to some embodiments of the present application;

[0061] FIG11A shows a schematic diagram of establishing a communication connection between a mobile phone 20 and a smart screen 10 according to some embodiments of the present application;

[0062] FIG11B shows another schematic diagram of establishing a communication connection between a mobile phone 20 and a smart screen 10 according to some embodiments of the present application;

[0063] FIG12 is a schematic diagram showing an interaction flow of another interaction method according to some embodiments of the present application;

[0064] FIG13 is a schematic diagram showing an interaction flow of another interaction method according to some embodiments of the present application;

[0065] FIG14 shows a schematic diagram of a software architecture for implementing interaction between a mobile phone 20 and a smart screen 10 according to some embodiments of the present application;

[0066] FIG15 shows a schematic structural diagram of a mobile phone 20 according to some embodiments of the present application;

[0067] Figure 16 shows a structural diagram of a smart screen 10 according to some embodiments of the present application. DETAILED DESCRIPTION

[0068] Illustrative embodiments of the present application include, but are not limited to, interactive methods, readable storage media, program products, and devices.

[0069] The technical solution of this application is introduced below with reference to the accompanying drawings.

[0070] It should be understood that a smart home device can be any device that can display images, including but not limited to smart TVs, smart screens, projectors, monitors, etc. For ease of description, the following description takes the smart home device as an example of a smart screen 10.

[0071] It should be understood that the device used to interact with smart home devices can be any device that includes a display and supports user operations, including but not limited to mobile phones, tablet computers, wearable devices (such as smart watches, smart bracelets), etc. For ease of description, the following description uses the mobile phone 20 as an example of the device used to interact with smart home devices.

[0072] To improve the user experience of interacting with the smart screen 10 through the mobile phone 20, in some embodiments, the interface displayed on the smart screen 10 can be transmitted to the mobile phone 20, so that the mobile phone 20 can display the interface on the smart screen 10. When the mobile phone 20 detects that the user performs an operation in the area corresponding to a control on the mobile phone 20, the mobile phone 20 sends a touch event corresponding to the operation to the smart screen 10. After receiving the touch event, the smart screen 10 triggers an interaction event corresponding to the touch event.

[0073] For example, referring to Figure 2, the smart screen 10 displays a display interface of a video application, which includes a menu control 11 that can respond to a sliding operation. When the smart screen 10 detects a left swipe event for the menu control 11, it can switch the label displayed in the menu control 11. The smart screen 10 can send the display interface to the mobile phone 20, and then the mobile phone 20 can display the received display interface. When the mobile phone 20 detects a left swipe operation by the user in the area 21 corresponding to the menu control 11 in the mobile phone 20, the mobile phone 20 can send a left swipe event for the menu control 11 to the smart screen 10. After receiving the left swipe event, the smart screen 10 can switch the label in the menu control 11 from "ABCD" to "BCDE".

[0074] However, in the above method, the smart screen 10 will continue to send the display interface of the smart screen to the mobile phone 20, which will occupy a large amount of network bandwidth of the smart screen 10 and the mobile phone 20, which may affect other services of the smart screen 10 and the mobile phone 20 and affect the user experience.

[0075] In view of this, an embodiment of the present application provides an interaction method, in which the smart screen 10 does not need to send the content of the current display interface of the smart screen 10 to the mobile phone 20, but instead sends the control parameters (such as operation boxes, operation prompts, etc.) of the controls (hereinafter referred to as first controls) in the current display interface of the smart screen 10 that can respond to touch events (such as sliding events, pinching events, pulling events, clicking events, etc.) and update the content in the controls to the mobile phone 20. After receiving the control parameters of one or more first controls, the mobile phone 20 can display the operation boxes and operation prompts of the one or more first controls. If the mobile phone 20 detects a touch event of the user on the operation box of a first control, the touch event can be sent to the smart screen 10, so that the smart screen 10 can respond to the touch event and trigger a corresponding interactive event in the first control, such as switching the display content in the first control, and keeping other controls from responding to the touch event.

[0076] In this way, users can interact with the smart screen 10 through operations on the mobile phone 20 (such as sliding, pinching, pulling, clicking, etc.), which is convenient and helps to improve the user experience. In addition, since the smart screen 10 does not need to send the content of the current display interface of the smart screen 10 to the mobile phone 20, the network resources occupied when the user interacts with the smart screen 10 through the mobile phone 20 can be reduced, which can avoid affecting other services of the mobile phone 20 and the smart screen 10.

[0077] For example, referring to Figure 3, the first control in the display interface of the smart screen 10 includes a menu control 11 and a recommendation control 12. After the mobile phone 20 establishes a connection with the smart screen 10, the smart screen sends the control parameters of the menu control 11 and the recommendation control 12 to the mobile phone 20.

[0078] It should be understood that in some implementations, after the mobile phone 20 establishes a communication connection with the smart screen 10 , the smart screen 10 can also display a cursor 13 to indicate the cursor position of the smart screen 10 .

[0079] It should be understood that in some embodiments, the smart screen 10 can also send the position of the cursor 13 to the mobile phone 20.

[0080] After receiving the control parameters of the menu control 11 and the recommendation control 12, the mobile phone 20 can display the operation box 11A of the menu control 11, the arrow 11B for indicating the touch direction (left and right sliding) of the menu control 11, the icon 11C for indicating the operation method (single-finger sliding) of the menu control 11, the operation box 12A of the recommendation control 12, the arrow 12B for indicating the touch direction (left and right sliding) of the recommendation control 12, the icon 12C for indicating the touch method (single-finger sliding) of the recommendation control 12, and the cursor 23 in the touch operation box 22 in corresponding proportions. Among them, the position of the cursor 23 in the touch operation box 22 corresponds to the position of the cursor 13 in the smart screen 10.

[0081] Referring to Figure 4A, when the mobile phone 20 detects the user swiping left on the operation box 11A, it can detect a left swipe event for the menu control 11, and the mobile phone 20 sends the left swipe event to the smart screen 10. After receiving the left swipe event for the menu control 11, the smart screen 10 triggers an interaction event for the left swipe event in the menu control 11, switching the label in the menu control 11 from "ABCD" to "BCDE".

[0082] Referring to Figure 4B, when the mobile phone 20 detects the user swiping left on the operation box 12A, it can detect a left swipe event for the recommendation control 12, and the mobile phone 20 sends the left swipe event to the smart screen 10. After receiving the left swipe event for the recommendation control 12, the smart screen 10 triggers an interaction event for the left swipe event in the recommendation control 12, switching the content in the recommendation control 12 from "thumbnail of video V1, thumbnail of video V2, thumbnail of video V3" to "thumbnail of video V2, thumbnail of video V3, and thumbnail of video V4."

[0083] For another example, referring to FIG5A , the display interface of the smart screen 10 includes a playback control W1. The smart screen 10 can send the control parameters of the playback control W1 to the mobile phone 20. After receiving the control parameters of the playback control W1, the mobile phone 20 can display an operation box W1A of the playback control W1 and an icon W1B for indicating the touch mode (pinch / pull) of the playback control W1 in the touch operation box 22.

[0084] It should be understood that in some embodiments, the touch operation box 22 corresponds to the edge of the display interface of the smart screen 10.

[0085] 5B , when the mobile phone 20 detects the user's pull-out operation on the operation box W1A, the mobile phone 20 sends a pull-out event to the smart screen 10. After receiving the pull-out event, the smart screen 10 can trigger an interaction event corresponding to the pull-out event on the playback control W1, enlarge the playback control W1, and display the enlarged playback control.

[0086] 5C , when the mobile phone 20 detects a user pinching operation in the operation box W1A, the mobile phone 20 sends a pinch event to the smart screen 10. After receiving the pinch event, the smart screen 10 can trigger an interaction event corresponding to the pinch event on the playback control W1, shrink the playback control W1, and display the shrunken playback control.

[0087] It should be understood that the interaction events corresponding to the above touch operations are only examples. In other embodiments, each touch operation may also correspond to other interaction events defined by the application or operating system in the smart screen 10, which is not limited here.

[0088] It should be understood that in some embodiments, the operation box and operation prompt of each first control displayed on the mobile phone 20 can be located in a touch operation box in the display interface of the mobile phone 20 (for example, the touch operation box 22 in Figure 3), and the aspect ratio of the touch operation box can be the same as the aspect ratio of the smart screen 10. In addition, the position and size of the touch operation box of each first control corresponds to the position and size of each first control in the display interface of the smart screen 10 (for example, proportional scaling).

[0089] It should be understood that the control parameters of the first control may include at least one of an operation box, an operation prompt, and a control identifier.

[0090] The operation box is used to indicate the area in the first control that can respond to touch operations. In some embodiments, the operation box can be represented by the coordinates of multiple vertices of the control, or by the center position, length, or width of the control, which is not limited here.

[0091] The operation prompt is used to prompt the user the type of touch event that the first control corresponding to the operation box can respond to. For example, the operation prompt may include but is not limited to the touch method corresponding to the touch event (such as single-finger / multi-finger sliding, pinching, stretching, single / multiple clicks, etc.), touch direction (such as left, right, left and right, up, down, up and down, sliding down the upper edge, sliding up the lower edge, sliding right along the left edge, sliding left along the right edge, etc.).

[0092] The control identifier is used to uniquely identify a first control, such as a control identification code, a control name, etc.

[0093] In some embodiments, the operation prompt may be represented by text, such as swipe up, swipe down, swipe left, swipe right, single-finger swipe, multi-finger swipe, swipe up and down, swipe left and right, etc. In other embodiments, the operation prompt may be represented by an icon or animation that can indicate the touch event to which the first control corresponding to the operation box can respond. The embodiment of the present application does not limit the specific form of the operation prompt for the first control displayed by the mobile phone 20.

[0094] In some embodiments, when the mobile phone 20 displays the operation prompt of the first control, it may only display the touch direction of the first control without displaying the touch method of the control, which is not limited here.

[0095] In some embodiments, for other controls in the smart screen 10 that are not the first control, the smart screen 10 does not need to send the control parameters of these controls to the mobile phone 20, which can reduce the amount of data sent by the smart screen 10. In addition, the mobile phone 20 does not need to display the operation box and operation prompts of these controls, which can reduce the amount of content displayed in the touch operation box 22, making operation easier.

[0096] Cursor interaction scenario

[0097] In some embodiments, the mobile phone 20 can also display a cursor operation box so that the user can trigger corresponding cursor interaction events, such as cursor movement events, cursor stop movement events, cursor click events, etc., by sliding, dragging, clicking, etc. in the cursor operation box, so as to realize the movement of the cursor position in the smart screen 10 or the selection and confirmation of a certain control.

[0098] Referring to FIG6 , compared to the situation shown in FIG3 , the mobile phone 20 can also display a cursor operation box 24 . The outline of the cursor operation box 24 is circular, and the center of the cursor operation box includes an indicator block 24A (indicator control). The user can drag the indicator block 24A to any position in the cursor operation box 24 by operating in the cursor operation box 24 , such as dragging. When the indicator block 24A is dragged to a certain position in the cursor operation box 24 , the cursor in the smart screen 10 can move accordingly.

[0099] In some embodiments, the direction of the cursor movement in the smart screen 10 can be along the direction of the position of the indicator block 24A pointed to by the center of the cursor operation box 24, and the speed of the cursor movement increases with the increase of the distance between the position of the indicator block 24A and the center of the cursor operation box 24.

[0100] In other embodiments, the outline of the cursor operation box 24 may also be other shapes, such as an ellipse, a polygon, an irregular shape, etc., which is not limited here.

[0101] For example, referring to FIG7A , when the mobile phone 20 detects that the user drags the indicator block 24A from the center O of the cursor operation box 24 to the O' position, it can send a cursor movement event (such as movement direction, movement speed, etc.) to the smart screen 10. After receiving the cursor movement event, the smart screen 10 can move the cursor 13, and the direction of movement of the cursor 13 can be along the OO' direction, and the speed of movement of the cursor 13 can increase with the increase of the distance between O and O'. In this way, the user can realize the continuous movement of the cursor 13 by dragging the indicator block 24A to a certain position, without having to repeatedly drag the indicator block 24A to realize the continuous movement of the cursor 13, which is conducive to improving the user experience.

[0102] In some embodiments, the smart screen 10 can send the position of the cursor 13 to the mobile phone 20 during the movement of the cursor 13, so that the mobile phone 20 can update the cursor 23 displayed in the area 22 to the corresponding position.

[0103] In some embodiments, when the mobile phone 20 detects that the user releases the indicator block 24, it sends a cursor stop movement event to the smart screen 10. In response to the cursor stop movement event, the smart screen 10 stops moving the cursor 13.

[0104] In some embodiments, when the mobile phone 20 detects a user clicking on the cursor operation box, it can send a click event to the smart screen 10. After receiving the click event, the smart screen 10 can trigger an interaction event corresponding to the click event at the location of the cursor 13.

[0105] For example, referring to FIG7B , when the mobile phone 20 detects a click operation by the user in the cursor operation box 24, it can send a click event to the smart screen 10. After receiving the click event, the smart screen 10 can trigger an interaction event corresponding to the click event at the location of the cursor 13 (i.e., the thumbnail of "V2") and play the video "V2".

[0106] In some embodiments, if the display interface of the smart screen 10 changes, the smart screen 10 can send the control parameters of the first control in the changed display interface to the mobile phone 20, and then the mobile phone 20 can update the operation box and operation prompts of the control displayed in area 22 according to the received control parameters.

[0107] For example, referring to FIG8 , after the smart screen 10 switches to the display interface for playing the video “V2”, the control parameters of the brightness adjustment control 14 (transparent display), the volume adjustment control 15 (transparent display), and the progress bar control 16 in the display interface can be sent to the mobile phone 20. Thus, the mobile phone 20 can display in area 22 an operation box 14A of the brightness adjustment control 14, an arrow 14B for indicating the touch direction (up and down sliding) of the brightness adjustment control 14, an icon 14C for indicating the touch mode (single-finger sliding) of the brightness adjustment control 14, an operation box 15A of the volume adjustment control 15, an arrow 15B for indicating the touch direction (up and down sliding) of the volume adjustment control 15, an icon 15C for indicating the touch mode (single-finger sliding) of the volume adjustment control 15, an operation box 16A of the progress bar control 16, an arrow 16B for indicating the touch direction (left and right sliding) of the progress bar control 16, and an icon 16C for indicating the touch mode (single-finger sliding) of the progress bar control 16.

[0108] After the mobile phone 20 displays the operation boxes and operation prompts of the brightness adjustment control 14, volume adjustment control 15, and progress bar control 16, the user can adjust the display screen brightness of the smart screen 10, the volume of the played sound, and the progress of the played video by operating in the operation box 14A, operation box 15A, or operation box 16A.

[0109] When the Smart Screen 10 adjusts the brightness of the display screen, the volume of the sound being played, or the progress of the video being played, it can also display corresponding content, such as a brightness adjustment icon, a volume adjustment icon, and a progress bar control.

[0110] Navigation interaction scene

[0111] In some embodiments, the operating system or application of the smart screen 10 further defines navigation interaction events, including but not limited to return to the previous level event, return to the desktop event, recent task event, display drop-down menu event, stop running the currently running application event, etc. Specifically, corresponding to a single-finger swipe event from the bottom border of the current display interface of the smart screen 10 upward, the smart screen 10 can switch the display interface to the desktop or main interface; corresponding to a two-finger swipe event from the bottom border of the current display interface of the smart screen 10 upward, the smart screen 10 can display the currently running application (including foreground applications and background applications); corresponding to a single-finger swipe event from the left border of the current display interface of the smart screen 10 to the right, or a single-finger swipe event from the right border of the current display interface of the smart screen 10 to the left, the smart screen 10 can return to the previous level.

[0112] Based on this, after detecting the user sliding event from the touch operation box 22 to the inside of the operation box, the mobile phone 20 can send a corresponding touch event to the smart screen 10. After receiving the touch event, the smart screen 10 triggers the corresponding navigation interaction event.

[0113] For example, referring to FIG9A , corresponding to the scenario shown in FIG8 , while the smart screen 10 is playing video “V2”, when the mobile phone 20 detects a left swipe operation from the right boundary of the touch operation box 22 or outside the right boundary to the inside of the touch operation box 22, it can send a right boundary left swipe event to the smart screen 10. After receiving the right boundary left swipe event, the smart screen 10 can trigger a “return to previous level” navigation interaction event corresponding to the event, switching the displayed interface to the interface before the user played video “V2”.

[0114] For another example, referring to FIG9B , corresponding to the scenario shown in FIG8 , while the smart screen 10 is playing video “V2”, when the mobile phone 20 detects a single-finger upward swipe operation from the lower boundary of the touch operation box 22 or outside the lower boundary to the inside of the touch operation box 22, it can send a lower boundary single-finger upward swipe event to the smart screen 10. After receiving the lower boundary single-finger upward swipe event, the smart screen 10 can trigger a “return to desktop” navigation interaction event corresponding to the event, switching the displayed interface to the desktop (or homepage) of the smart screen 10.

[0115] For another example, referring to FIG9C , corresponding to the scenario shown in FIG8 , while the smart screen 10 is playing video "V2", when the mobile phone 20 detects a two-finger upward swipe operation from the lower boundary of the touch operation box 22 or outside the lower boundary to the inside of the touch operation box 22, it can send a lower boundary two-finger upward swipe event to the smart screen 10. After receiving the lower boundary two-finger upward swipe event, the smart screen 10 can trigger a "recent tasks" navigation interaction event corresponding to the event, displaying interface thumbnails of the applications recently run on the smart screen 10.

[0116] It should be understood that the interaction events corresponding to the above operations are only examples. In other embodiments, the interaction events corresponding to the operations may also be other interaction events, which are not limited here.

[0117] It should be understood that in other scenarios, if the application or operating system in the smart screen 10 defines other operations (or touch events) that the control can respond to and other corresponding interactive events, the mobile phone 20 can also send corresponding touch events to the smart screen 10 based on the user's operation in the control operation box, so that the smart screen 10 can trigger corresponding interactive events according to the received touch events and switch the content in the display interface.

[0118] The following describes the technical solution of the present application in conjunction with the interaction scenarios shown in FIG. 2 to FIG. 9C .

[0119] First, the method of interaction between the user through the mobile phone 20 and the first control in the current display interface of the smart screen 10 is introduced.

[0120] Specifically, FIG10 shows a schematic diagram of an interaction process of an interaction method according to some embodiments of the present application. As shown in FIG10 , the method includes the following steps:

[0121] S1001: The mobile phone 20 and the smart screen 10 establish a control communication connection.

[0122] The mobile phone 20 establishes a control communication connection with the smart screen 10, triggering the interaction method provided in the embodiment of the present application.

[0123] It should be understood that the control communication connection is used to establish a control binding relationship between the mobile phone 20 and the smart screen 10. After the control communication connection is established, the mobile phone 20 can control the smart screen 10 by sending instructions to the smart screen 10, such as touch events.

[0124] For example, the mobile phone 20 and the smart screen 10 can install corresponding control applications or services, such as a smart life application, to establish a control communication connection. For example, referring to Figure 11A, the mobile phone 20 can establish a control communication connection with the smart screen 10 when it detects that the user selects the smart screen 10 in the smart life application, such as clicking the icon of the smart screen 10.

[0125] For example, the mobile phone 20 and the smart screen 10 can establish a control communication connection through a distributed system function, such as a hyperterminal function. For example, referring to FIG11B , when the mobile phone 20 detects that the user selects the smart screen 10 in the hyperterminal card in the drop-down menu, the control communication connection is established with the smart screen 10.

[0126] It should be understood that in other embodiments, the mobile phone 20 can also establish a control communication connection with the smart screen 10 in other ways, which is not limited here.

[0127] S1002: The smart screen 10 parses the current display interface to obtain the first controls and control parameters of each first control in the current display interface.

[0128] After the smart screen 10 and the mobile phone 20 establish a control communication connection, the current display interface is analyzed to obtain the first control and the control parameters of each first control in the current display interface.

[0129] For example, corresponding to the situation shown in Figure 3, the smart screen 10 can parse and obtain the first control in the current display interface, including the menu control 11 and the recommendation control 12, and obtain the control parameters of the menu control 11 and the recommendation control 12 according to the layout file of the current display interface.

[0130] For another example, for the situation shown in Figure 5A, the smart screen 10 can parse and obtain the first control in the current display interface, including the menu control 11 and the playback control W1, and obtain the control parameters of the menu control 11 and the playback control W1 according to the layout file of the current display interface.

[0131] It should be understood that the control parameters of each first control may include at least one of an operation box of each first control, an operation prompt (such as a touch mode, a touch direction, etc.), a control identifier, etc.

[0132] It should be understood that the operation box is used to indicate the range in which the first control can respond to touch events. The size of the operation box can correspond to the size of the first control, or only to the size of a part of the area in the first control, which is not limited here.

[0133] It should be understood that in some embodiments, the first control is invisible in the current display interface of the smart screen 10 (such as transparent display, etc.), such as the brightness adjustment control 14 and the volume adjustment control 15 in the situation shown in Figure 8, which is not limited here.

[0134] S1003: The smart screen 10 sends the control parameters of the first control in the current display interface to the mobile phone 20.

[0135] After obtaining the control parameters of the first controls in the current display interface, the smart screen 10 sends the control parameters of each first control to the mobile phone 20.

[0136] In some embodiments, the smart screen 10 can also send the cursor position in the smart screen 10 to the smart screen 10.

[0137] In some embodiments, when the smart screen 10 and the mobile phone 20 establish a connection, the smart screen 10 can display a cursor in the display interface.

[0138] S1004 : The mobile phone 20 displays the operation boxes of the first controls in the touch operation box 22 .

[0139] After receiving the control parameters of each first control in the current display interface of the smart screen 10 , the mobile phone 20 displays the operation box of each first control in the touch operation box 22 .

[0140] Optionally, after receiving the control parameters of each first control in the current display interface of the smart screen 10, the mobile phone 20 displays the operation box and operation prompt of each first control in the touch operation box 22.

[0141] For example, corresponding to the situation shown in FIG. 3 , the mobile phone 20 may display the operation boxes and operation prompts of the menu control 11 and the recommendation control 12 at corresponding positions in the touch operation box 22 .

[0142] For another example, corresponding to the situation shown in FIG. 5A , the mobile phone 20 may display the operation boxes and operation prompts of the menu control 11 and the play control W1 at corresponding positions in the touch operation box 22 .

[0143] For another example, corresponding to the situation shown in FIG. 8 , the mobile phone 20 may display the operation boxes and operation prompts of the brightness adjustment control 14 , the volume adjustment control 15 , and the progress bar control 16 at corresponding positions in the touch operation box 22 .

[0144] In some embodiments, the aspect ratio of the touch operation box 22 can be the same as the aspect ratio of the current display interface of the smart screen 10, and the operation box of each control can be displayed in the touch operation box 22 in a proportional scaling manner.

[0145] In other embodiments, the aspect ratio of the touch operation box 22 may be different from the aspect ratio of the current display interface of the smart screen 10. When the operation boxes of each control are scaled and displayed in the touch operation box 22, the scaling ratio in the length direction and the scaling ratio in the width direction may be different.

[0146] In some embodiments, if the mobile phone 20 receives the cursor position sent by the smart screen 10, the cursor may also be displayed at a position corresponding to the cursor position in the touch operation box 22. For example, in the scenario shown in FIG3 , the mobile phone 20 may display a cursor 23 in the touch operation box 22, and the position of the cursor 23 in the touch operation box corresponds to the position of the cursor in the current display interface of the smart screen 10.

[0147] S1005: The mobile phone 20 detects a first operation performed by the user in an operation box of a first control, and sends a first touch event for the first control to the smart screen 10.

[0148] When the mobile phone 20 detects that the user performs a first operation in the operation box of a first control in the touch operation box 22, it sends a first touch event for the first control to the smart screen 10.

[0149] For example, corresponding to the situation in Figure 4A, when the mobile phone 20 detects the user's left swipe operation in the operation box 11A of the menu control 11, it can send a left swipe event for the menu control 11 to the smart screen 10.

[0150] For another example, corresponding to the situation in Figure 4B, when the mobile phone 20 detects the user's left swipe operation in the operation box 12A of the recommendation control 12, it can send a left swipe event for the recommendation control 12 to the smart screen 10.

[0151] For another example, corresponding to the situation in Figure 5B, when the mobile phone 20 detects the user's pulling-out operation in the operation box W1A of the playback control W1, it can send a pulling-out event for the playback control W1 to the smart screen 10.

[0152] For example, corresponding to the situation in Figure 5C, when the mobile phone 20 detects the user's pinch operation in the operation box W1A of the playback control W1, it can send a pinch event for the playback control W1 to the smart screen 10.

[0153] In some embodiments, the first touch event for the first control may include the control identifier of the first control and the type of the touch event (eg, touch mode and touch direction, etc.).

[0154] S1006: The smart screen 10 triggers an interaction event corresponding to the first touch event on the first control.

[0155] After receiving the first touch event for the first control, the smart screen 10 triggers an interaction event corresponding to the first touch event on the first control, and switches the displayed content according to the interaction event, such as switching the content in the first control.

[0156] For example, corresponding to the situation shown in Figure 4A, after receiving a left swipe event for the menu control 11, the smart screen 10 can trigger an interactive event corresponding to the left swipe event on the menu control 11, and switch the label in the menu control 11 from "ABCD" to "BCDE".

[0157] For another example, corresponding to the situation shown in Figure 4B, after receiving a left-swipe event for the recommendation control 12, the smart screen 10 can trigger an interactive event corresponding to the left-swipe event on the recommendation control 12, and switch the content in the recommendation control 12 from "thumbnail of video V1, thumbnail of video V2, thumbnail of video V3" to "thumbnail of video V2, thumbnail of video V3 and thumbnail of video V4".

[0158] For another example, corresponding to the situation shown in Figure 5B, after receiving the pull-out event for the playback control W1, the smart screen 10 can trigger an interactive event corresponding to the pull-out event on the playback control W1 to enlarge the playback control W1.

[0159] For example, corresponding to the situation shown in Figure 5C, after receiving a pinch event for the playback control W1, the smart screen 10 can trigger an interactive event corresponding to the pinch event on the playback control W1 to shrink the playback control W1.

[0160] Through the method provided in the embodiment of the present application, the user can interact with the smart screen 10 through operations on the mobile phone 20, such as sliding operations, pinching operations, etc., which is convenient and conducive to improving the user experience. In addition, since the smart screen 10 does not need to send the content of the current display interface of the smart screen 10 to the mobile phone 20, the network resources occupied when the user interacts with the smart screen 10 through the mobile phone 20 can be reduced, which can avoid affecting other services of the mobile phone 20 and the smart screen 10.

[0161] The following introduces the technical solution for users to interact with the cursor of the smart screen 10 through the mobile phone 20.

[0162] Specifically, FIG12 shows a schematic diagram of an interaction process of another interaction method according to some embodiments of the present application. As shown in FIG12 , the method includes the following steps:

[0163] S1201: The mobile phone 20 and the smart screen 10 establish a control communication connection.

[0164] The mobile phone 20 establishes a control communication connection with the smart screen 10, triggering the interaction method provided in the embodiment of the present application. The manner in which the mobile phone 20 establishes a control communication connection with the smart screen 10 can refer to the aforementioned step S1001 and will not be repeated here.

[0165] S1202: The smart screen 10 parses the current display interface to obtain the first controls and control parameters of each first control in the current display interface.

[0166] After the smart screen 10 and the mobile phone 20 establish a control communication connection, the current display interface is analyzed to obtain the first control and the control parameters of each first control in the current display interface.

[0167] S1203: The smart screen 10 sends the control parameters of the first control in the current display interface to the mobile phone 20.

[0168] After obtaining the control parameters of the first controls in the current display interface, the smart screen 10 sends the control parameters of each first control to the mobile phone 20.

[0169] In some embodiments, the smart screen 10 can also send the cursor position in the smart screen 10 to the smart screen 10.

[0170] S1204 : The mobile phone 20 displays the operation box of the first control in the touch operation box 22 and displays the cursor operation box 24 .

[0171] After receiving the control parameters of each first control in the current display interface of the smart screen 10, the mobile phone 20 displays the operation box of each first control in the touch operation box 22 and displays the cursor operation box 24.

[0172] Optionally, the mobile phone 20 may also display the operation box and operation prompt of the first control in the touch operation box 22, and display the cursor operation box 24. For example, corresponding to the situation shown in FIG6, the mobile phone 20 may display the cursor operation box 24 with a circular outline and the indicator block 24A.

[0173] It should be understood that the manner in which the mobile phone 20 displays the operation box of the first control in the touch operation box 22 can refer to the aforementioned step S1004 and is not limited here.

[0174] It should be understood that in some embodiments, the mobile phone 20 may also display the cursor operation box 24 in the display interface when establishing a control communication connection with the smart screen 10, for example, when executing the aforementioned step S1201, which is not limited here.

[0175] It should be understood that in other embodiments, the outline of the cursor operation box 24 and the outline of the indicator block 24A may also be other shapes, such as ellipse, polygon, irregular shape, etc., which are not limited here.

[0176] It should be understood that in some embodiments, if the mobile phone 20 receives the cursor position sent by the smart screen 10, the cursor may also be displayed at a position corresponding to the cursor position in the touch operation box 22. For example, in the scenario shown in FIG6 , the mobile phone 20 may display a cursor 23 in the touch operation box 22, and the position of the cursor 23 in the touch operation box corresponds to the position of the cursor 13 in the smart screen 10 in the current display interface of the smart screen 10.

[0177] S1205: The mobile phone 20 detects that the user drags the indicator block 24A to the first position in the cursor operation box 24 and sends a cursor movement event to the smart screen 10.

[0178] When the mobile phone 20 detects that the user has dragged the indicator block 24A to the first position in the cursor operation box 24, it can send a cursor movement event to the smart screen 10. For example, in the scenario shown in FIG7A , when the mobile phone 20 detects that the user has dragged the indicator block 24A to the O' position, it can send a cursor movement event to the smart screen 20. The event may include the speed of the cursor movement (determined by the distance between O and O', the greater the distance, the faster the movement speed) and direction (OO' direction).

[0179] It should be understood that, in some embodiments, the cursor movement event may include the direction and speed of the cursor movement.

[0180] It should be understood that in some embodiments, the mobile phone 20 may move the cursor in the touch operation frame 22 in a corresponding proportion according to the speed and direction corresponding to the cursor movement event.

[0181] S1206: The smart screen 10 continues to move the cursor.

[0182] After receiving the cursor movement event, the smart screen 10 continues to move the cursor according to the speed and direction in the cursor movement event.

[0183] For example, in the situation shown in FIG. 7A , the smart screen 10 can continuously move the cursor along the OO′ direction.

[0184] It should be understood that in some embodiments, the smart screen 10 can send the position of the cursor to the mobile phone 20 in real time during the process of moving the cursor, so that the mobile phone 20 can adjust the position of the cursor 23 in the touch operation box 22 in real time.

[0185] S1207: The mobile phone 20 detects that the user releases the indicator block 24A and sends a cursor stop movement event to the smart screen 10.

[0186] When the mobile phone 20 detects that the user releases the indicator block 24A, it sends a cursor stop movement event to the smart screen 10.

[0187] S1208: The smart screen 10 stops moving the cursor.

[0188] After receiving the cursor stop moving event, the smart screen 10 stops moving the cursor in response to the event.

[0189] That is, from the time the user drags the indicator block 24A to a certain position until the user releases the indicator block 24A, the smart screen 10 will continue to move the cursor. This way, in some scenarios where the cursor needs to be moved over a long distance, the user does not need to drag the indicator block 24A multiple times or perform a sliding operation on the mobile phone 20, which is conducive to improving the user experience.

[0190] S1209: The mobile phone 20 detects the user's click operation in the cursor operation box 24 and sends a click event to the smart screen 10.

[0191] When the mobile phone 20 detects a click operation by the user in the cursor operation box 24 , it can send a click event to the smart screen 10 .

[0192] S1210: The smart screen 10 triggers an interactive event corresponding to the click event at the cursor location.

[0193] After receiving a click event, the smart screen 10 responds to the click event, triggers an interaction event corresponding to the click event, and switches the displayed content according to the interaction event.

[0194] For example, corresponding to the situation shown in FIG7B , the smart screen 10 can display the playback interface of the video “V2” after receiving a click event.

[0195] It should be understood that the interactive event corresponding to the click event can be defined by the application corresponding to the current display interface of the smart screen 10 or the operating system of the smart screen 10, including but not limited to executing a selection operation, confirming an operation, opening a file, switching to a certain interface, switching the content in a certain control, etc.

[0196] Through the interaction method provided in the embodiment of the present application, the user can trigger the cursor interaction event in the smart screen 10 by dragging operations, clicking operations, etc. in the cursor operation box 24, which is conducive to improving the user experience.

[0197] Next, a technical solution is introduced for a user to trigger a navigation interaction event in the smart screen 10 through operations in the mobile phone 20.

[0198] Specifically, FIG13 shows a schematic diagram of an interaction process of another interaction method according to some embodiments of the present application. As shown in FIG13 , the interaction process includes the following steps:

[0199] S1301: The mobile phone 20 and the smart screen 10 establish a control communication connection.

[0200] The mobile phone 20 establishes a control communication connection with the smart screen 10, triggering the interaction method provided in the embodiment of the present application. The manner in which the mobile phone 20 establishes a control communication connection with the smart screen 10 can refer to the aforementioned step S1001 and will not be repeated here.

[0201] S1302: The smart screen 10 parses the current display interface to obtain the first controls and control parameters of each first control in the current display interface.

[0202] After the smart screen 10 and the mobile phone 20 establish a control communication connection, the current display interface is analyzed to obtain the first control and the control parameters of each first control in the current display interface.

[0203] S1303: The smart screen 10 sends the control parameters of the first control in the current display interface to the mobile phone 20.

[0204] After obtaining the control parameters of the first controls in the current display interface, the smart screen 10 sends the control parameters of each first control to the mobile phone 20.

[0205] It should be understood that in some embodiments, the smart screen 10 can also send the cursor position in the smart screen 10 to the smart screen 10.

[0206] S1304 : The mobile phone 20 displays the operation boxes of the first controls in the touch operation box 22 .

[0207] After receiving the control parameters of each first control in the current display interface of the smart screen 10, the mobile phone 20 displays the operation box of each first control in the touch operation box 22.

[0208] Alternatively, after receiving the control parameters of each first control in the current display interface of the smart screen 10, the mobile phone 20 can also display the operation box and operation prompt of each first control in the touch operation box 22.

[0209] In some embodiments, if the mobile phone 20 receives the cursor position sent by the smart screen 10, the cursor can also be displayed at a position corresponding to the cursor position in the touch operation box 22.

[0210] S1305: The mobile phone 20 detects that the user performs a second operation from touching the boundary of the operation box 22 to the inside of the operation box, and sends a corresponding second touch event to the smart screen 10.

[0211] When the mobile phone 20 detects the user's second operation, it sends a corresponding second touch event to the smart screen 10.

[0212] It should be understood that the second operation may be a single-finger sliding operation or a multi-finger sliding operation in which the user slides from a certain boundary of the touch operation frame 22 or outside of a certain boundary to the inside of the touch operation frame 22 .

[0213] For example, corresponding to the situation shown in Figure 9A, when the mobile phone 20 detects that the user slides left from outside the right boundary of the touch operation box 22 to the inside of the touch operation box 22, it can send a right boundary left slide event to the smart screen 10.

[0214] For another example, corresponding to the situation shown in Figure 9B, when the mobile phone 20 detects that the user performs a single-finger upward swipe operation from outside the lower boundary of the touch operation box 22 to the inside of the touch operation box 22, it can send a lower boundary single-finger upward swipe event to the smart screen 10.

[0215] For another example, corresponding to the situation shown in Figure 9C, when the mobile phone 20 detects that the user performs a two-finger swipe-up operation from outside the lower boundary of the touch operation box 22 to inside the touch operation box 22, it can send a two-finger swipe-up event on the lower boundary to the smart screen 10.

[0216] In other embodiments, the second operation may also include other operations, such as a pinch operation, etc., which are not limited here.

[0217] In some embodiments, the second touch event may include the type of the touch event (eg, touch mode and touch direction, etc.).

[0218] S1306: The smart screen 10 triggers a navigation interaction event corresponding to the second touch event.

[0219] After receiving the second touch event, the smart screen 10 triggers a navigation interaction event corresponding to the second touch event.

[0220] Navigation interaction events can be defined by the operating system of the smart screen 10 and / or the application corresponding to the current interface in the smart screen 10, including but not limited to returning to the previous level event, returning to the desktop event, recent task event, displaying a drop-down menu event, stopping the currently running application event, etc.

[0221] For example, corresponding to the situation shown in Figure 9A, after receiving the left swipe event on the right boundary, the smart screen 20 can return to the previous level and switch the display screen to the screen before playing the video "V2".

[0222] For another example, corresponding to the situation shown in Figure 9B, after receiving a single-finger swipe-up event on the lower boundary, the smart screen 20 can trigger a "return to desktop" navigation interaction event and switch the displayed interface to the desktop (or homepage) of the smart screen 10.

[0223] For another example, corresponding to the situation shown in Figure 9C, after receiving the two-finger swipe up event on the lower boundary, the smart screen 20 can trigger the "recent tasks" navigation interaction event and display the interface thumbnails of the applications that the smart screen 10 has recently run.

[0224] It should be understood that in other embodiments, the navigation interaction event may also include other navigation interaction events, which are not limited here.

[0225] Through the interactive method provided in the embodiment of the present application, the user can trigger the navigation interaction event in the smart screen 10 by touching the boundary of the operation box 22 in the mobile phone 20 to operate inside the operation box. The operation is convenient and helps to improve the user experience.

[0226] The following describes the software architecture for implementing the interaction methods in the aforementioned embodiments.

[0227] Specifically, Figure 14 shows a schematic diagram of the software architecture for implementing the aforementioned interaction method in a mobile phone 20 and a smart screen 10 according to some embodiments of the present application.

[0228] As shown in Figure 14, the software architecture of the smart screen 10 may include a display subsystem 101, an input subsystem 102 and an interaction subsystem 103, and the software architecture of the mobile phone 20 may include an interaction subsystem 201.

[0229] The display subsystem 101 can be used to display images through the display screen of the smart screen 10 and send the current display interface to the interactive subsystem 103.

[0230] The display subsystem 101 is also used to receive touch events from the input subsystem 102 and trigger corresponding interaction events based on the received touch events. For example, when the display subsystem 101 receives a touch event for a first control, it can trigger a corresponding interaction event on the control. For another example, when the display subsystem 101 receives a touch event corresponding to an operation from the boundary of the touch operation box 22 to the inside of the operation box in the mobile phone 20, it can trigger a corresponding navigation interaction event. For another example, when the display subsystem 101 receives a cursor interaction event (such as a cursor movement event, a cursor stop movement event, a click event, etc.), it can trigger a corresponding cursor interaction event.

[0231] For example, in some embodiments, the display subsystem 101 may be used to perform part or all of the operations of step S1006 , step S1206 , step S1208 , step S1210 , and step S1306 .

[0232] The interaction subsystem 103 may include a parsing module 103A and a communication module 103B.

[0233] The parsing module 103A is used to parse the control parameters of each first control in the current display interface of the smart screen 10, such as at least one of an operation prompt, an operation box, a control identifier, etc., and send the obtained control parameters to the communication module 103B. For example, in some embodiments, the parsing module 103A can be used to perform all or part of the operations of the aforementioned steps S1002, S1202, and S1302.

[0234] The communication module 103B is used for communication between the smart screen 10 and the mobile phone 20. For example, the communication module 103B can be used to send control parameters and cursor position to the communication module 201B in the mobile phone 20. For another example, the communication module 103B can also be used to receive touch events from the communication module 201B and forward the touch events to the input subsystem 102. For example, in some embodiments, the communication module 103B can be used to implement all or part of the operations of the aforementioned steps S1001, S1003, S1201, S1203, S1301, and S1303.

[0235] The interactive subsystem 201 may include a display module 201A and a communication module 201B.

[0236] The communication module 201B is used for communication between the smart screen 10 and the mobile phone 20. For example, the communication module 201B can be used to receive control parameters, cursor position, etc. from the smart screen 10. For another example, the communication module 201B can also be used to forward the touch event detected by the display module 201A to the communication module 103B. For example, in some embodiments, the communication module 201B can be used to implement all or part of the operations of the aforementioned steps S1001, S1005, S1201, S1205, S1207, S1301, and S1305.

[0237] The display module 201A can be used to display the touch operation box 22 and the cursor operation box 24, and can also be used to display the operation boxes and operation prompts of each first control. It can also be used to detect the user's operation on the touch operation box 22 or the cursor operation box 24 and generate corresponding touch events. For example, in some embodiments, the display module 201 can be used to perform all or part of the operations of the aforementioned steps S1004, step S1005, step S1204, step S1205, step S1207, step S1304, and step S1305.

[0238] It should be understood that in other embodiments, the software architecture used in the mobile phone 20 and the smart screen 10 to implement the interactive method of the embodiments of the present application may include more or fewer modules, or some modules may be merged or split, and other architectures may be adopted, which are not limited here.

[0239] Based on the software architecture shown in FIG14 , the process of interaction between the mobile phone 20 and the smart screen 10 may include the following steps.

[0240] ① The display subsystem 101 sends a display interface to the analysis module 103A.

[0241] It should be understood that the display subsystem 101 may send the display interface to the decoding module 103A frame by frame, or may send the display interface to the parsing module 103A only when there is a change in the current display interface.

[0242] ② The parsing module 103A sends the control parameters of the first control in the display interface to the communication module 103B.

[0243] It should be understood that after receiving the display interface, the parsing module 103A can parse the control parameters of the first control in the display interface and send the obtained control parameters to the communication module 103B. For details, please refer to the above step S1002 and will not be repeated here.

[0244] ③ Communication module 103B sends the control parameters to communication module 201B.

[0245] ④ The communication module 201B forwards the control parameters to the display module 201A.

[0246] It should be understood that after receiving the control parameters, the display module 201 can display the operation boxes and operation prompts corresponding to each first control on the display screen of the smart phone 20 according to the received control parameters. For details, please refer to the above step S1004 and will not be repeated here.

[0247] ⑤ The display module 201 sends the detected touch event to the communication module 201B.

[0248] After detecting a user operation in an operation box of a first control, the aforementioned operation box, or the aforementioned cursor operation box, the display module 201 may send a touch event corresponding to the operation to the communication module 201B. For details, please refer to the aforementioned steps S1005, S1205, S1207, S1209, and S1305, which will not be repeated here.

[0249] ⑥ The communication module 201B sends the touch event to the communication module 103B.

[0250] ⑦ The communication module 103B sends the touch event to the input subsystem 102.

[0251] ⑧The input subsystem 102 forwards the touch event to the display subsystem 101.

[0252] It should be understood that after receiving a touch event, the display subsystem 101 can trigger a corresponding interaction event, including but not limited to an interaction event for a certain control, a navigation interaction event, a cursor interaction event, etc. For details, please refer to the aforementioned steps S1006, S1206, S1208, S1210, and S1306, which will not be repeated here.

[0253] Based on the above software architecture, users can interact with the smart screen 10 through operations on the mobile phone 20, such as sliding, pinching, dragging, and clicking. Since the smart screen 10 does not need to transmit the specific screen of the display interface to the mobile phone 20, but transmits control parameters of the control, cursor position and other data to the mobile phone 20, it is beneficial to reduce the network resources occupied during the interaction process.

[0254] Furthermore, FIG15 shows a schematic structural diagram of a mobile phone 20 according to some embodiments of the present application.

[0255] FIG15 shows a schematic structural diagram of a mobile phone 20 according to some embodiments of the present application.

[0256] As shown in Figure 15, the mobile phone 20 may include a processor 210, an external memory interface 220, an internal memory 221, a universal serial bus (USB) interface 230, a charging management module 240, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 250, a wireless communication module 260, an audio module 270, a speaker 270A, a receiver 270B, a microphone 270C, an earphone interface 270D, a sensor module 280, a button 290, a motor 291, an indicator 292, a camera 293, a display screen 294, and a subscriber identification module (SIM) card interface 295, etc. Among them, the sensor module 280 can include a pressure sensor 280A, a gyroscope sensor 280B, an air pressure sensor 280C, a magnetic sensor 280D, an acceleration sensor 280E, a distance sensor 280F, a proximity light sensor 280G, a fingerprint sensor 280H, a temperature sensor 280J, a touch sensor 280K, an ambient light sensor 280L, a bone conduction sensor 280M, and the like.

[0257] The processor 210 may include one or more processing units, for example, the processor 210 may include a central processing unit (CPU), an application processor (AP), a modem processor, a GPU, an image signal processor (ISP), a microcontroller unit (MCU), a video codec, a digital signal processor (DSP), a baseband processor, a neural-network processing unit (NPU), a field programmable gate array (FPGA), etc. In some embodiments, different processing units may be independent devices or integrated into one or more processors.

[0258] In some embodiments, the processor 210 can be used to execute instructions of the interaction methods provided by the aforementioned embodiments, such as instructions for obtaining control parameters, cursor positions, etc. from other devices, instructions for displaying the operation box and operation prompts of the first control according to the control parameters and cursor position, instructions for detecting user operations and sending corresponding touch events to other devices, etc.

[0259] USB interface 230 is an interface that complies with USB standards and specifications, and may be a Mini USB interface, a Micro USB interface, a USB Type-C interface, or the like. USB interface 230 can be used to connect a charger to charge mobile phone 20, and can also be used to transfer data between mobile phone 20 and peripheral devices. For example, this interface can also be used to connect to other devices, such as projectors, smart screens, and other display devices, to obtain data / commands from other devices or send data / commands to other devices through USB interface 230.

[0260] The charging management module 240 is configured to receive charging input from a charger. While the charging management module 240 is charging the battery 142 , it can also power the device through the power management module 141 .

[0261] The power management module 141 is used to connect the battery 142, the charging management module 240, and the processor 210. The power management module 141 receives input from the battery 142 and / or the charging management module 240 to provide power to the processor 210, the internal memory 221, the display 294, the camera 293, and the wireless communication module 260.

[0262] The wireless communication function of the mobile phone 20 can be implemented through the antenna 1, the antenna 2, the mobile communication module 250, the wireless communication module 260, the modem processor and the baseband processor.

[0263] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals.

[0264] The mobile communication module 250 can provide solutions for wireless communications including 2G / 3G / 4G / 5G applied on the mobile phone 20. The mobile communication module 250 may include at least one filter, a switch, a power amplifier, a low noise amplifier (LNA), etc. The mobile communication module 250 can receive electromagnetic waves from the antenna 1, and filter, amplify, and process the received electromagnetic waves, and transmit them to the modulation and demodulation processor for demodulation. The mobile communication module 250 can also amplify the signal modulated by the modulation and demodulation processor, and convert it into electromagnetic waves for radiation through the antenna 1. In some embodiments, at least some of the functional modules of the mobile communication module 250 can be set in the processor 210. In some embodiments, at least some of the functional modules of the mobile communication module 250 can be set in the same device as at least some of the modules of the processor 210.

[0265] The wireless communication module 260 can provide wireless communication solutions including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR), etc., which are applied to the mobile phone 20. The wireless communication module 260 can be one or more devices that integrate at least one communication processing module. The wireless communication module 260 receives electromagnetic waves via the antenna 2, frequency modulates and filters the electromagnetic wave signals, and sends the processed signals to the processor 210. The wireless communication module 260 can also receive the signal to be sent from the processor 210, frequency modulate it, amplify it, and convert it into electromagnetic waves for radiation through the antenna 2.

[0266] For example, in some embodiments, the mobile phone 20 can establish a control communication connection with other devices, such as the smart screen 10, through the mobile communication module 250 or the wireless communication module 260.

[0267] The mobile phone 20 implements display functions through a GPU, display screen 294, and an application processor. The GPU is a microprocessor for image processing and is connected to the display screen 294 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 210 may include one or more GPUs that execute program instructions to generate or change display information. For example, the mobile phone 20 can use the GPU and display screen 294 to display the touch operation box 22, the cursor operation box 24, the cursor 23, and the operation prompts and touch directions of each first control.

[0268] The display screen 294 is used to display images, videos, etc. The display screen 294 includes a display panel. In some embodiments, the mobile phone 20 may include 1 or N display screens 294, where N is a positive integer greater than 1.

[0269] The camera 293 is used to capture still images or videos. In some embodiments, the mobile phone 20 may include 1 or N cameras 293, where N is a positive integer greater than 1.

[0270] The external memory interface 220 may be used to connect an external memory card.

[0271] The internal memory 221 can be used to store executable program code, which includes instructions. The internal memory 221 may include a program storage area and a data storage area. In some embodiments, the program storage area may store an operating system, an application required for at least one function, etc., such as instructions for the interaction methods provided in the aforementioned embodiments. The data storage area may store data created during the use of the mobile phone 20, such as the aforementioned control parameters. In addition, the internal memory 221 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, etc. The processor 210 executes various functional applications of the mobile phone 20 by running instructions stored in the internal memory 221 and / or instructions stored in a memory provided in the processor 210.

[0272] The mobile phone 20 can implement audio functions through the audio module 270, the speaker 270A, the receiver 270B, the microphone 270C, the headphone jack 270D, and the application processor.

[0273] The audio module 270 is used to convert digital audio information into analog audio signal output, and is also used to convert analog audio input into digital audio signals. The audio module 270 can also be used to encode and decode audio signals. The speaker 270A, also known as the "speaker", is used to convert audio electrical signals into sound signals. The receiver 270B, also known as the "earpiece", is used to convert audio electrical signals into sound signals. The microphone 270C, also known as the "microphone" or "microphone", is used to convert sound signals into electrical signals. The headphone jack 270D is used to connect wired headphones. The pressure sensor 280A is used to sense pressure signals and can convert pressure signals into electrical signals.

[0274] Touch sensor 280K, also known as a "touch device," can be mounted on display screen 294. Together, touch sensor 280K and display screen 294 form a touch screen, also known as a "touch screen." Touch sensor 280K detects actions performed on or near it. The touch sensor transmits the detected actions to the application processor to determine the type of touch event, such as the aforementioned slide event, pinch event, drag event, or click event.

[0275] Buttons 290 include a power button and volume buttons. Motor 291 can generate vibration prompts. Indicator 292 can be an indicator light that can be used to indicate charging status, power level changes, messages, missed calls, notifications, etc. SIM card interface 295 is used to connect a SIM card.

[0276] It should be understood that the structure of the mobile phone 20 shown in the embodiment of the present application does not constitute a specific limitation on the mobile phone 20. In other embodiments of the present application, the mobile phone 20 may include more or fewer components than shown, or some components may be combined or separated, or the components may be arranged differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0277] It should be understood that the structure of the mobile phone 20 shown in the embodiment of this application is merely an example. In other embodiments, the mobile phone 20 may include more or fewer modules, or some modules may be separated or combined, without limitation herein. It should be understood that devices used to interact with smart home devices may employ the same or similar structure as the mobile phone 20, without limitation herein.

[0278] Furthermore, FIG16 shows a schematic structural diagram of a smart screen 10 according to some embodiments of the present application.

[0279] As shown in FIG16 , the smart screen 10 may include a processor 110 , a memory 120 , a display screen 130 , a communication module 140 , a power switch 150 , an input / output interface 160 , an audio module 170 , and the like.

[0280] The processor 110 may include one or more processing units. For details, please refer to the aforementioned processor 210, which will not be described in detail here.

[0281] In some embodiments, the processor 110 can be used to execute instructions of the interaction methods provided by the aforementioned embodiments, such as instructions for parsing the first control in the current display interface of the smart screen 10, instructions for sending control parameters of the first control to other devices, and instructions for triggering corresponding interaction events based on received touch events.

[0282] The memory 120 can be used to store data and instructions. For example, it can store the control parameters of the aforementioned first control, store instructions for the interaction methods of the aforementioned embodiments, etc. The specific implementation of the memory 120 can refer to the aforementioned internal memory 221 and will not be described in detail here.

[0283] The display screen 130 is used to display images. The specific implementation form of the display screen 130 can refer to the aforementioned display screen 294 and will not be described in detail here.

[0284] The communication module 140 is used to enable communication between the smart screen 10 and other devices. For example, the communication module 140 can be used to send the control parameters of the first control, the cursor position, etc. to other devices, and can also be used to receive touch events corresponding to operations from other devices. It should be understood that the specific form of the communication module 140 can refer to the aforementioned mobile communication module 250 or wireless communication module 260, and will not be described in detail here.

[0285] The power switch 150 is used to turn the smart screen 10 on or off.

[0286] The input and output interface 160 includes interfaces for receiving data from other devices or outputting data (such as audio, video, etc.) to other devices, including but not limited to high definition multimedia interface (HDMI), S / PDIF (Sony / Philips digital interconnect format, Sony TM and Philips TM digital audio interface protocol developed by the manufacturer), local area network (LAN) interface, universal serial bus (USB) interface, etc.

[0287] The audio module 170 is used to implement the audio function of the smart screen 10. For details, please refer to the aforementioned audio module 270 and will not be repeated here.

[0288] In some embodiments, the smart screen 10 itself may not have any operating functions.

[0289] The structure of the smart screen 10 shown in the embodiment of the present application is only an example. In other embodiments, the smart screen 10 may include more or fewer modules, and some modules may be merged or split, which is not limited here.

[0290] In the accompanying drawings, some structural or method features may be shown in a particular arrangement and / or order. However, it should be understood that such a particular arrangement and / or order may not be required. Rather, in some embodiments, these features may be arranged in a manner and / or order different from that shown in the illustrative drawings. In addition, the inclusion of a structural or method feature in a particular figure does not imply that such feature is required in all embodiments, and in some embodiments, such features may not be included or may be combined with other features.

[0291] It should be noted that in the examples and description of this patent, relational terms such as first and second, etc. are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the term "comprises" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or device that includes a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a" do not exclude the presence of other identical elements in the process, method, article or device that includes the elements.

Claims

1. An interactive method, characterized in that: include: The first device displays a first interface, where the first interface includes a first control; The first device detects that a connection is established with the second device, and sends a control parameter corresponding to the first control to the second device, where the control parameter is used to instruct the second device to display a first operation box on the second interface; The first device receives a first event sent by the second device, where the first event is generated when the second device detects at least one operation of a user in the first operation box; In response to the first event, the first device displays a third interface, where the third interface is different from the first interface.

2. The method according to claim 1, characterized in that The third interface includes the content in the first control updated in response to the first event.

3. The method according to claim 1 or 2, characterized in that: The method further comprises: The first device receives a second event sent by the second device, where the second event is generated when the second device detects that a user drags an indication control in a second operation box in the second interface along a first direction; In response to the second event, the first device moves a cursor along the first direction; The first device receives a third event sent by the second device, where the third event is generated when the second device detects that the user releases the indicating control; In response to the third event, the first device stops moving the cursor.

4. The method according to claim 3, characterized in that: The speed of the cursor movement increases as the distance between the position of the indicator control and the center of the second operation frame increases.

5. The method according to claim 3 or 4, characterized in that: The method further comprises: The first device receives a fourth event sent by the second device, where the fourth event is generated when the second device detects a click operation of a user in the second operation box; In response to the fourth event, the first device displays a fourth interface, where the fourth interface is different from the first interface.

6. The method according to any one of claims 1 to 5, characterized in that: The first operation frame is located inside the third operation frame; the method further includes: The first device receives a fifth event sent by the second device, where the fifth event is generated when the second device detects at least one operation performed by a user from an edge of the third operation box or from outside the third operation box to inside the third operation box; In response to the fifth event, the first device displays a fifth interface, which is different from the first interface.

7. The method according to any one of claims 1 to 5, characterized in that: The first control is automatically selected by the first device from one or more controls in the first interface.

8. An interactive method, characterized in that: include: The second device receives a control parameter sent by the first device, where the control parameter is a control parameter corresponding to a first control in a first interface displayed by the first device; The second device displays a second interface in response to the control parameter, where the second interface includes a first operation box corresponding to the first control; The second device detects at least one operation of the user in the first operation box, and sends a first event to the first device, where the first event is used to instruct the first device to display a third interface, which is different from the first interface.

9. The method according to claim 8, characterized in that The second interface further includes a second operation box, the second operation box includes an indication control, and the method further includes: The second device detects that the user drags the indicating control along the first direction, and sends a second event to the first device, where the second event is used to instruct the first device to move a cursor along the first direction; The second device detects that the user releases the indicating control, and sends a third event to the first device, where the third event is used to instruct the first device to stop moving the cursor.

10. The method according to claim 9, characterized in that The method further comprises: The second device detects a click operation of the user in the second operation box, and sends a fourth event to the first device, where the fourth event is used to instruct the first device to display a fourth interface, which is different from the first interface.

11. The method according to any one of claims 8 to 10, characterized in that The second interface further includes a third operation box, the first operation box is located inside the third operation box, and the method further includes: The second device detects at least one operation performed by the user from the edge of the third operation box or from outside the third operation box to inside the third operation box, and sends a fifth event to the first device, where the fifth event is used to instruct the first device to display a fifth interface, which is different from the first interface.

12. The method according to any one of claims 8 to 11, characterized in that The first operation box includes an operation prompt corresponding to the first control.

13. The method according to any one of claims 8 to 12, characterized in that: The first control is automatically selected by the first device from one or more controls in the first interface.

14. A readable storage medium, characterized in that: The readable storage medium includes instructions, and when the instructions are executed by a device, the device implements the interaction method described in any one of claims 1 to 7 or the interaction method described in any one of claims 8 to 13.

15. A first device, characterized in that: include: A memory for storing instructions; A processor, configured to execute the instruction so that the first device implements the interaction method according to any one of claims 1 to 7.

16. A second device, characterized in that: include: A memory for storing instructions; A processor, configured to execute the instruction so that the second device implements the interaction method according to any one of claims 8 to 13.

17. A system, characterized in that: include: The first device of claim 15 and the second device of claim 16.

18. A program product, characterized in that When the program product is run on a device, the device implements the interaction method described in any one of claims 1 to 7 or the interaction method described in any one of claims 8 to 13.

Citation Information

Patent Citations

  • Intelligent television touch remote-control terminal screen mapping method and control method and control system of touch remote-control terminal

    CN104202643A

  • Mobile terminal-based equipment control method, device and mobile terminal

    CN107145278A

  • Mobile terminal-based device control method, equipment and mobile terminal

    CN107147769A

  • Screen projection control method and device and storage medium

    CN112905136A

  • Portable terminal device and program

    JP2012185629A