Method, apparatus, and system for resolving gesture conflict across devices

Abstract

The present application provides a method for resolving a gesture conflict across devices. The method is mainly applied to a scenario where a first electronic device controls a large screen of a collaborative terminal by initiating a mirroring control request, and comprises: the first electronic device loading a first gesture list and a conflict gesture processing strategy table, and receiving a second gesture list; displaying a predefined conflict processing strategy on the interface of the first electronic device; and obtaining a response when a first operation gesture is executed. The use of the method can effectively solve the problem of gesture control conflicts across devices, and effectively improve the user experience.

Description

Method, device and system for resolving gesture conflicts across devices Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a method, apparatus, and system for resolving gesture conflicts across devices. Background Art

[0002] With the development of society and advancements in technology, meeting content is now commonly shared through large screens in conference rooms. For some online meeting attendees, users can access the meeting via mobile phones, laptops, or in offline conference rooms. The conference room presenter shares the presentation material on the large screen, or the large screen displays the content shared by remote attendees. However, some attendees may be too far away to see the content and may use their mobile phones or laptops to pull up the content displayed on the large screen. After pulling up the large screen image using a terminal, users can control the large screen by annotating and touching the image displayed on their terminal screen. However, gesture conflicts between the user's terminal screen and the remote touch screen can lead to cross-device gesture conflicts, which in turn affects the user experience and efficiency of cross-device touch control.

[0003] Based on this, how to properly resolve gesture conflicts becomes a crucial issue when performing gesture detection across devices. Summary of the Invention

[0004] This application proposes a method for resolving gesture conflicts across devices. The method effectively avoids the occurrence of gesture conflicts by predefining conflict gesture processing strategies. The method also supports users to predefine backup gestures for conflicting gestures, which can effectively improve the user experience.

[0005] On the first aspect, the present application proposes a method for resolving gesture conflicts across devices, which is applied to a scenario in which a first electronic device controls a collaborative terminal large screen by initiating a mirror control request, including the first electronic device loading a first gesture list, a conflict gesture processing strategy table, and receiving a second gesture list; displaying a predefined conflict processing strategy on the interface of the first electronic device; and obtaining a response after executing the first operation gesture.

[0006] In this solution, the first electronic device initiates a mirror control request to control the large screen of the collaborative terminal. During this process, the first electronic device will first load the first gesture list supported by the local APP, load the local predefined conflict gesture processing strategy table, and receive the second gesture list from the collaborative terminal. The predefined conflict processing strategy is displayed on the first electronic pen interface to remind the user to refer to and use the corresponding gesture. The user executes the first operation gesture and obtains the response generated by the first operation gesture, thereby effectively improving the user's operating efficiency and effectively resolving conflicting gestures.

[0007] In a possible implementation manner of the first aspect, the first gesture list is used to control the first electronic device.

[0008] In a possible implementation manner of the first aspect, the receiving the second gesture list includes: receiving the second gesture list from the collaborative terminal, where the second gesture list is used to control the collaborative terminal.

[0009] In a possible implementation manner of the first aspect, the predefined conflict handling strategy includes but is not limited to a switch switching mode and a standby gesture mode.

[0010] In a possible implementation manner of the first aspect, the switch switching mode is used to select a target operation object by switching a switch, where the target operation object is the first electronic device or the cooperative terminal.

[0011] This solution mainly targets gestures that only need to be used on the local electronic device. Users can control the gestures to control the first electronic device or the cooperative terminal by switching on and off. This can effectively realize the scene of controlling the local electronic device.

[0012] In a possible implementation of the first aspect, the backup gesture mode is used to complete the control of the collaborative terminal by using a predefined backup gesture.

[0013] This solution mainly addresses the situation where the first operation gesture responds differently on the first electronic device and the collaborative terminal. In order to better control the collaborative terminal, the collaborative terminal can be controlled through predefined backup gestures. This method can effectively resolve conflicts and thus effectively improve user efficiency.

[0014] In a possible implementation of the first aspect, obtaining a response after executing the first operation gesture includes: if there is no conflict with the first operation gesture, normally obtaining a response after executing the first operation gesture; if there is a conflict with the first operation gesture and there is no corresponding predefined conflict handling strategy, guiding the user to configure the conflict handling strategy corresponding to the first operation gesture; if there is a conflict with the first operation gesture but there is a corresponding predefined first operation gesture conflict handling strategy, obtaining a response based on the first operation gesture conflict handling strategy.

[0015] In a possible implementation of the first aspect, guiding the user to configure the conflict handling strategy corresponding to the first operation gesture includes: configuring the conflict handling strategy corresponding to the first operation gesture to update the conflict gesture handling strategy table.

[0016] In a possible implementation of the first aspect, the method further includes: loading the updated conflict gesture processing strategy table; displaying the updated predefined conflict processing strategy on the first electronic device interface; obtaining a response after executing the first operation gesture;

[0017] In this solution, the user can configure the first conflict gesture strategy for gestures that conflict and have no defined conflict handling strategy, and update the corresponding conflict gesture handling strategy table. The first electronic device loads the updated conflict gesture strategy table and displays the updated conflict gesture strategy table on the interface of the first electronic device. This solution supports users to use defined backup gestures to control the collaborative terminal. This method can effectively avoid gesture conflicts.

[0018] In a possible implementation manner of the first aspect, the first electronic device loads the gesture conflict handling strategy table in real time.

[0019] In this solution, the user's first electronic device display interface can update the displayed gesture conflict handling strategy table in real time and remind the user to refer to the predefined gesture conflict handling strategy in real time. This method configures the corresponding conflicting gesture handling strategy based on the conflict determination result. The newly configured gesture handling strategy takes effect in real time and can be used by the user, thereby improving usage efficiency and user experience.

[0020] In a second aspect, an electronic device with a touch interface is proposed, wherein the electronic device includes a processor coupled to a memory, and the processor is used to store instructions. When the instructions are executed by the processor, the electronic device executes a method as described in any one of the above-mentioned first aspects.

[0021] In a third aspect, a computer-readable storage medium is proposed, comprising computer instructions. When the computer instructions are executed on a computer system, the computer system implements the method described in any one of the first aspects above.

[0022] In a fourth aspect, a computer program product is proposed, wherein the computer program product includes instructions, and when the instructions are executed, the computer implements the method as described in any one of the first aspects above.

[0023] The beneficial effects of the second to fourth aspects mentioned above can be referred to the introduction of the first aspect mentioned above, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] FIG1 is a schematic diagram of an architecture of an application scenario provided by an embodiment of the present application;

[0025] FIG2 is a schematic diagram of a method for resolving gesture conflicts across devices provided in an embodiment of the present application. DETAILED DESCRIPTION

[0026] In order to make the purpose, technical solutions and advantages of this application more clear, the embodiments of this application are described below in conjunction with the accompanying drawings. Obviously, the described embodiments are only embodiments of a part of this application, rather than all embodiments. It is known to those skilled in the art that with the emergence of new application scenarios, the technical solutions provided in the embodiments of this application are also applicable to similar technical problems.

[0027] The terms "first", "second", etc. in the specification and claims of this application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that the descriptions used in this way can be interchangeable where appropriate so that the embodiments can be implemented in a sequence other than that illustrated or described in this application. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or modules is not necessarily limited to those steps or modules clearly listed, but may include other steps or modules that are not clearly listed or that are inherent to these processes, methods, products or devices. The naming or numbering of steps in this application does not mean that the steps in the method flow must be executed in the time / logical sequence indicated by the naming or numbering. The named or numbered process steps can change the execution order according to the technical purpose to be achieved, as long as the same or similar technical effects can be achieved.

[0028] The division of units in this application is a logical division. In actual application, there may be other division methods. For example, multiple units can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between each other shown or discussed can be through some interfaces, and the indirect coupling or communication connection between units can be electrical or other similar forms, which are not limited in this application. Moreover, the units or sub-units described as separate components may or may not be physically separated, may or may not be physical units, or may be distributed into multiple circuit units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this application.

[0029] To facilitate understanding, some technical terms involved in the embodiments of this application are first introduced below.

[0030] (1) Collaborative Terminal

[0031] A collaborative terminal refers to a large-screen device with a built-in camera, touch screen, and WIFI module. The device has the capabilities of electronic whiteboard writing, annotation, and touch.

[0032] (2) Mirror viewing

[0033] Usually, accessing shared large-screen content through a mobile phone terminal or a computer terminal is a process of viewing a mirrored image. Therefore, this method of viewing large-screen content through a mobile phone terminal or an APP is a process of mirrored viewing.

[0034] (3) Mirror touch

[0035] In this embodiment, controlling the large screen through a touchscreen phone is a mirrored touch method. This application proposes a method for resolving gesture conflicts across devices. This method supports detecting gesture conflicts between a terminal device and a large screen on the end side and provides the user with a conflict resolution method to enable the user to smoothly switch between local operations and large-screen touch. This method can effectively resolve the impact of gesture conflicts and effectively improve the user experience.

[0036] Please refer to Figure 1, which is an architectural diagram of an application scenario provided by an embodiment of the present application. As shown in Figure 1, the application scenario includes an electronic device 001 and a collaborative terminal 002. Among them, the scenario is mainly to demonstrate the interaction scenario between the electronic device 001 and the collaborative terminal 002. In this embodiment, the electronic device 001 is a mobile phone, a tablet computer or a laptop computer, wherein the mobile phone, tablet computer, and laptop computer contain a screen projection APP, which can be used to receive the video media stream from the collaborative terminal 002 and watch the mirror image through the APP on the mobile phone end, and gestures can be operated on the mobile phone 001 to realize the control of the collaborative terminal 002 through the electronic device 001, wherein the electronic device 001 mainly controls the collaborative terminal by means of a signaling control flow. In addition, the collaborative terminal 002 can also control the electronic device 001 by means of a signaling control flow.

[0037] In one implementation of this embodiment, the collaborative terminal 002 is mainly used to display shared content. The collaborative terminal usually exists in a conference room scene to display shared content to multiple participants. The electronic device 001 views the mirror image from the collaborative terminal 001 through the screen projection APP, and the user can control the content displayed in the collaborative terminal 002 through gesture touch on the electronic device 001.

[0038] Please refer to Figure 2, which is a schematic diagram of a method for resolving gesture conflicts across devices provided by this embodiment. This embodiment mainly uses a flowchart to elaborate on how to resolve gesture conflicts when gesture responses from an electronic device conflict with those from a cooperating terminal, thereby improving the user experience.

[0039] In one implementation of this embodiment, the way in which the electronic device resolves gesture conflicts with the cooperative terminal is illustrated by resolving conflicts between a mobile phone and a large screen.

[0040] The user initiates mirror control on the mobile phone APP. The specific steps are as follows:

[0041] Step 1: Load the gesture list supported by the local APP

[0042] After the electronic device initiates a mirror control request and receives a consent response from the collaborative terminal, it starts loading a list of gestures supported by the local APP. In one possible implementation, the gesture list may include commonly used gestures such as pinch-to-zoom, zoom-in followed by single-finger sliding, etc. These gestures are usually based on the fact that the display screen of the mobile phone itself is small, and it is usually necessary to zoom and slide the content to view the presented content more clearly.

[0043] Step 2: Read the predefined conflict gesture handling strategy

[0044] The mobile app reads a predefined conflict gesture handling strategy. Typically, this strategy might include: dynamically displaying a toggle switch after a two-finger zoom conflict; or reading a predefined backup gesture solution: sliding the canvas with both hands if a conflict occurs after a two-finger zoom. In this embodiment, the predefined conflict gesture handling strategy supports user customization, adapting to user habits and improving the user experience.

[0045] Step 3: Receive the gesture list from the collaborative terminal

[0046] The mobile phone app receives the touch gesture list of the current page from the collaborative terminal (such as a large screen) in real time through network signaling. Usually, the display screen of the mobile phone app is smaller, and the display screen of the large screen is larger. There are certain differences in the functional requirements of gestures between the two devices. Therefore, not only will there be slightly different response definitions for the same gesture, but there are also situations where the gestures used in the mobile phone app do not exist on the large screen, and vice versa. Based on this, it is usually necessary to compare the gesture list from the mobile phone app with the gesture list from the collaborative terminal in order to better formulate the corresponding response strategy.

[0047] In one embodiment, the large screen will push the list of gestures supported by the large screen interface in real time. Specifically, a signaling connection channel is established between the mobile phone APP and the large screen mirroring service, and the protocol used is TLS. When the foreground interface of the large screen changes, such as the whiteboard APP switches to the foreground, the whiteboard application or other third-party applications will call the open interface of the mirror component to synchronize the list of gestures supported by the current application interface in real time. The gesture list can include sliding, pinch-to-zoom, double-click, etc., and can be in JSON format. The mirroring service component will push the supported gesture list to the mobile phone APP side in real time through the signaling channel created previously.

[0048] Step 4: Compare the two gesture lists to see if there is a conflict

[0049] Compare the gesture list of the local app with the gesture list from the collaborative terminal to analyze whether there is a conflict. If there is no conflict, jump to Step 8. If there is a conflict, it is necessary to determine whether there is a predefined conflict strategy for the gesture.

[0050] Step 5: Is there a gesture conflict handling strategy table?

[0051] If there is a predefined gesture conflict handling strategy table, the predefined strategy will be displayed on the mobile phone interface to remind the user of the currently predefined gestures and the corresponding conflict handling strategy, so that the user can adaptively adjust the target operation gesture. It is worth noting that the process of loading the gesture conflict strategy table is carried out in real time or periodically to ensure that the user's mobile device interface can display the latest gesture conflict strategy table in real time to remind the user in real time; if no predefined gestures are defined, jump to Step 8.

[0052] Step 6: Mobile phone interface display control switch

[0053] For different gestures, the predefined conflict gesture processing strategies are also different. For some gestures, they may only be used on the mobile phone side, and for some gestures, they may only be used on the large screen side. It is usually more reasonable to control such gestures using the switch mode. In this step, by displaying some gestures that can be used for the switch switching mode on the interface, and reminding the user to switch the switch by any other means, it is possible to achieve only the operation of the gesture or only the operation of the large screen. The reminder method and the display method are not limited here. The user can be allowed to complete the switch switching operation on the interface, or the user can be reminded to switch the switch at other locations by jumping. Any reminder that can achieve the above gestures and support the switch switching operation falls within the protection scope of the present invention.

[0054] Step 7: The phone interface displays the backup gesture

[0055] Some gestures may exist on both the phone and the large screen, but the defined responses are different. Therefore, a predefined response processing strategy is required for these gestures. This definition is generally based on scenario-based judgment and consideration, so a response that is more suitable for the scenario is designed. Therefore, these gestures often require the definition of a backup gesture. This step is mainly used to display the corresponding responses of the currently defined backup gestures on the interface, so as to help users better select and use the target operation gesture for controlling the large screen.

[0056] Step 8: Initiate gesture operation

[0057] Based on the above judgment, according to the corresponding processing strategy obtained by analysis, whether it is a switch mode switching operation or a definition operation of a backup gesture, the user is required to initiate a gesture operation on the mobile phone interface to achieve the corresponding response and function.

[0058] Step 9: Determine again whether the gestures conflict

[0059] For gestures initiated by the user, conflict detection and judgment need to be performed again. Specifically, in one embodiment, when the gestures supported by the large screen are two-finger zoom, click, one-finger slide, etc., and the gesture supported by the local mobile phone APP is two-handed zoom, when the gesture performed by the user is two-finger zoom, the result obtained by the mobile phone APP is conflicting, so the judgment result is conflict.

[0060] Step 10: If there is no conflict, complete the response normally

[0061] If the result of the above steps is that there is no conflict, it is only necessary to execute according to the predefined response, and real-time response and processing can be performed according to the specific selection of collaborative device or local device.

[0062] Step 11: If there is a conflict but a corresponding conflict handling strategy exists, respond according to the predefined strategy.

[0063] If the judgment result of Step 9 is a conflict but there is a corresponding conflict handling strategy, then respond according to the predefined conflict handling strategy corresponding to the gesture. If the conflict handling strategy corresponding to the current gesture is the switch mode, the user can determine whether the control object of the current gesture is the mobile phone or the large screen by switching the switch; if the conflict handling strategy corresponding to the current gesture is the backup gesture mode, the user can complete the control of the large screen by using the backup gesture.

[0064] Step 12: If there is a conflict and no corresponding conflict handling strategy exists, guide the user to follow the gesture conflict settings

[0065] If the result of Step 9 is a conflict and there is no corresponding conflict handling strategy, you need to guide the user to complete the configuration of the handling strategy for the conflicting gestures. For example, for the two-finger zoom conflict, guide the user to enable the zoom switch. When the switch is turned on, the large screen can be controlled, and when the switch is turned off, the local app canvas can be controlled. For the scenario of single-finger swiping on the zoomed canvas, guide the user to enable the backup gesture. When the gesture is a single-finger swipe, it is used to control the large screen, and when the gesture is a two-finger swipe, it is used to drag and view the local canvas. After completing this step, continue to jump to Step 3 to repeat the operation.

[0066] Some of the functions or features described in the above embodiments may be implemented through software, some may be implemented through hardware, or some may be implemented through a combination of software, hardware, or firmware. When implemented through software, all or part of the functions or features may be implemented in the form of a computer program product.

[0067] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function according to the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer language, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website, computer, training equipment or data center to another website, computer, training equipment or data center via a wired (e.g., coaxial cable, optical fiber, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) method. The computer-readable storage medium can be any available medium that a computer can store or a data storage device such as a training equipment, data center, etc. that includes one or more available media integrations. Available media can be magnetic media, (e.g., floppy disk, hard disk, tape), optical media (e.g., DVD), or semiconductor media (e.g., solid-state drive (SSD)).

Claims

1. A method for resolving gesture conflicts across devices, characterized in that: Applicable to the scenario where the first electronic device controls the collaborative terminal's large screen by initiating a mirroring control request, including: The first electronic device loads the first gesture list and the conflicting gesture processing strategy table, and receives the second gesture list; Displaying a predefined conflict handling strategy on the first electronic device interface; Get the response after executing the first operation gesture.

2. The method according to claim 1, characterized in that The first gesture list is used to control the first electronic device.

3. The method according to claim 1, characterized in that Receiving the second gesture list includes: The second gesture list is received from the collaborative terminal, where the second gesture list is used to control the collaborative terminal.

4. The method according to claim 1, wherein The predefined conflict handling strategies include but are not limited to a switch switching mode and a standby gesture mode.

5. The method according to claim 1, wherein The switch switching mode is used to select a target operation object by switching a switch, wherein the target operation object is the first electronic device or the cooperative terminal.

6. The method according to claim 1, wherein The standby gesture mode is used to control the cooperative terminal by using predefined standby gestures.

7. The method according to claim 1, characterized in that The obtaining of a response after executing the first operation gesture includes: If there is no conflict with the first operation gesture, normally obtain a response after executing the first operation gesture; If the first operation gesture conflicts and there is no corresponding predefined conflict handling strategy, guiding the user to configure the conflict handling strategy corresponding to the first operation gesture; If there is a conflict with the first operation gesture but there is a corresponding predefined first operation gesture conflict handling strategy, a response is obtained based on the first operation gesture conflict handling strategy.

8. The method according to claim 7, characterized in that The guiding the user to configure a conflict handling strategy corresponding to the first operation gesture includes: A conflict handling strategy corresponding to the first operation gesture is configured to update the conflict gesture handling strategy table.

9. The method according to claim 8, characterized in that The method further comprises: Loading the updated conflict gesture processing strategy table; Displaying the updated predefined conflict handling strategy on the first electronic device interface; Obtain a response after executing the first operation gesture.

10. The method according to claim 1, characterized in that The first electronic device loads the gesture conflict handling strategy table in real time.

11. An electronic device with a touch interface, characterized in that: The electronic device comprises a processor coupled to a memory, wherein the processor is configured to store instructions. When the instructions are executed by the processor, the electronic device executes the method according to any one of claims 1 to 10.

12. A computer-readable storage medium, characterized in that The method comprises computer instructions, which, when executed on a computer system, enable the computer system to implement the method according to any one of claims 1 to 10.

13. A computer program product comprising instructions, characterized in that: When the instructions are executed, the computer implements the method according to any one of claims 1 to 10.

Images