Cross-device control method and device, electronic device and storage medium

Abstract

The invention is suitable for the technical field of equipment management, and provides a cross-equipment control method and device, electronic equipment and a storage medium, and the method comprises the steps: displaying a cross-equipment interaction area in a first operation interface of first equipment; the cross-device interaction area comprises a second operation interface and / or a first control; in response to an interaction operation initiated by a user in the cross-device interaction area, generating an operation instruction corresponding to the interaction operation; and sending the operation instruction to a second device to control the second device to execute the operation instruction corresponding to the interaction operation. According to the technical scheme provided by the invention, the operation efficiency under the condition of multi-device collaborative operation can be improved, the operation difficulty of a user is reduced, and then the use experience of the user is improved.

Description

Technical Field

[0001] The present application belongs to the technical field of device management, and in particular, relates to a cross-device control method, apparatus, electronic device and storage medium. Background Art

[0002] As the process of electronicization continues to advance, the number of electronic devices held by users has also increased. For example, the same user can hold multiple electronic devices such as smartphones, laptops, and desktop computers to cope with different usage scenarios. Therefore, how to efficiently control each electronic device in different usage scenarios has become the focus of users.

[0003] In the existing device control technology, different electronic devices are equipped with independent interaction modules, and users need to initiate interaction operations on the interaction modules corresponding to the electronic devices in order to send specified operation instructions to the electronic devices. However, in some usage scenarios, when users need to control two or more electronic devices at the same time, they need to frequently initiate interaction operations on the interaction modules on different electronic devices, which greatly reduces the user's operation efficiency and increases the user's operation difficulty. Summary of the invention

[0004] The embodiments of the present application provide a cross-device control method, apparatus, electronic device, and computer-readable storage medium, which can solve the problems of existing device control technology, that is, when two or more electronic devices need to be operated simultaneously, operations need to be performed on different interaction modules, and the operation efficiency is low and the operation difficulty is high.

[0005] In a first aspect, an embodiment of the present application provides a cross-device control method, which is applied to a first device, comprising: displaying a cross-device interaction area in a first operation interface of the first device; the cross-device interaction area includes a second operation interface and / or a first control; the second operation interface is a virtual touch panel of the second device; in response to an interaction operation initiated by a user in the cross-device interaction area, generating an operation instruction corresponding to the interaction operation; and sending the operation instruction to a second device to control the second device to execute the operation instruction corresponding to the interaction operation.

[0006] It should be noted that, in some embodiments, the cross-device interaction area can be an operation interface, such as a second operation interface, which can include one or more controls, and the interface can also include an area that can receive other interactive operations such as sliding operations and text input operations; in other embodiments, the cross-device interaction area can also be a control, such as a first control, through which cross-device interactive operations are implemented, or the second operation interface is opened through the control to perform more cross-device interactive operations; in other embodiments, the cross-device interaction area can also include both one or more operation interfaces and one or more controls outside the operation interface.

[0007] The implementation of the embodiment of the present application has the following beneficial effects: by displaying a cross-device interaction area in the first device that can realize the control of other electronic devices, by receiving the interactive operation initiated by the user in the cross-device interaction area, an operation instruction acting on other devices outside the first device, that is, the second device, is generated, and the operation instruction is sent to the second device; after the second device receives the operation instruction, the operation instruction corresponding to the interactive operation can be executed in the second device to achieve the purpose of cross-device control. Compared with the existing device control technology, when the user operates the first device and the second device at the same time, there is no need to initiate interactive operations on the interactive modules of different devices, but the first device can be controlled in the first operation interface of the first device, and the second device can be controlled through the cross-device interaction area in the first operation interface, which improves the operation efficiency in the case of multi-device collaborative operation, reduces the user's operation difficulty, and then improves the user's experience.

[0008] In a possible implementation of the first aspect, displaying the cross-device interaction area in the first operation interface of the first device includes: displaying the second operation interface in the first operation interface in response to a first click operation of a user on a first control in the first operation interface; the first control is displayed when the first device and the second device are in a state where a cross-device control connection has been established.

[0009] In a possible implementation of the first aspect, in response to the user's first click operation on the first control in the first operation interface, before displaying the second operation interface in the first operation interface, it also includes: if the first device state of the first device meets a preset first connection condition, sending a connection request to the second device; receiving a connection confirmation instruction fed back by the second device based on the connection request, establishing the cross-device control connection with the second device, and displaying the first control in the first operation interface.

[0010] In a possible implementation of the first aspect, the cross-device interaction area can be moved. The movement can be actively moved by the user, or can be moved according to the relative position change of the second device and the first device. For example, if the second device is on the left side of the first device, the cross-device interaction area can be displayed on the left part of the first device interface; if the second device is moved to the right side of the first device, the cross-device interaction area can be automatically moved to the right part of the first device interface.

[0011] In a possible implementation of the first aspect, the first device state includes: screen state, motion state and device placement posture; if the first device state of the first device satisfies a preset first connection condition, a connection request is sent to the second device, including: if at least one of the following conditions is met: the screen state is in a bright screen state, the motion state is a stationary state and the device placement posture is within a preset angle range, a connection request is sent to the second device.

[0012] In a possible implementation of the first aspect, if the first device state of the first device satisfies a preset first connection condition, sending a connection request to the second device includes: generating a first Bluetooth signal carrying the connection request through a first Bluetooth module in the first device; the first Bluetooth signal is used for the second device to receive the first Bluetooth signal through a second Bluetooth module, and determine the distance value between the second device and the first device through the first Bluetooth signal;

[0013] The receiving of a connection confirmation instruction fed back by the second device based on the connection request includes: receiving a second Bluetooth signal fed back by the second device; the second Bluetooth signal carries the connection confirmation instruction; the second Bluetooth signal is generated when the second device detects that the distance value is less than a preset distance threshold.

[0014] In a possible implementation of the first aspect, the interactive operation includes an interface sharing operation; the first operation interface is displayed in a first window; and in response to the interactive operation initiated by the user in the cross-device interactive area, generating an operation instruction corresponding to the interactive operation includes: if it is detected that the user initiates a drag operation of dragging the first window to the first control, closing the first window and generating an interface sharing instruction for the first operation interface;

[0015] The sending the operation instruction to the second device to control the second device to execute the operation instruction corresponding to the interactive operation includes: sending the interface sharing instruction to the second device to display the first operation interface in a third window on the second device.

[0016] In a possible implementation of the first aspect, displaying the cross-device interaction area in the first operation interface of the first device includes: if the first operation interface is the operation interface of a first application installed on the first device, displaying the first operation interface through a first window, and displaying the second operation interface through a second window; the first layer where the first window is located is under the second layer where the second window is located.

[0017] In a possible implementation of the first aspect, if the first operation interface is the operation interface of the first application installed on the first device, then after displaying the first operation interface through the first window and displaying the second operation interface through the second window, it also includes: in response to a second click operation of the user on the first window, adjusting the first layer to the second layer; and / or in response to a third click operation of the user on the second window, adjusting the second layer to the first layer.

[0018] In a possible implementation of the first aspect, the interactive operation includes a first sliding operation; and in response to the interactive operation initiated by the user in the cross-device interactive area, generating an operation instruction corresponding to the interactive operation includes: in response to the first sliding operation initiated by the user in the second operation interface, generating a sliding instruction corresponding to the first sliding operation;

[0019] The sending the operation instruction to the second device to control the second device to execute the operation instruction corresponding to the interactive operation includes: sending the sliding instruction to the second device to control the virtual cursor in the second device to move based on the sliding instruction.

[0020] In a possible implementation manner of the first aspect, the interactive operation includes a text input operation; before generating an operation instruction corresponding to the interactive operation in response to the interactive operation initiated by the user in the cross-device interaction area, the method may further include: in response to a fourth click operation initiated by the user on a second control in the second operation interface, setting the second operation interface to a text input mode;

[0021] The generating, in response to an interactive operation initiated by a user in the cross-device interactive area, an operation instruction corresponding to the interactive operation, comprises: in response to a text input operation initiated by a user in the second operation interface, determining at least one character corresponding to the text input operation, and generating a text input instruction for the at least one character;

[0022] The sending the operation instruction to the second device to control the second device to execute the operation instruction corresponding to the interactive operation includes: sending the text input instruction to the second device to input the at least one character in a text box corresponding to the second device.

[0023] In a possible implementation manner of the first aspect, the interactive operation includes a screenshot operation; before the generating an operation instruction corresponding to the interactive operation in response to the interactive operation initiated by the user in the cross-device interaction area, the method may further include: in response to a fifth click operation initiated by the user on a third control in the second operation interface, setting the second operation interface to a screenshot mode;

[0024] The step of generating an operation instruction corresponding to the interaction operation in response to the interaction operation initiated by the user in the cross-device interaction area includes: determining a first screenshot area corresponding to the second sliding operation in response to the second sliding operation initiated by the user in the second operation interface; determining a second screenshot area in a third operation interface of the second device according to the first screenshot area, and generating a screenshot instruction for the second screenshot area;

[0025] The sending of the operation instruction to the second device to control the second device to execute the operation instruction corresponding to the interactive operation includes: sending the screenshot instruction to the second device; and receiving a screenshot image of the second screenshot area fed back by the second device based on the screenshot instruction.

[0026] In a possible implementation manner of the first aspect, the interactive operation may also include other types of operations such as screen recording and text extraction. The implementation manner refers to the aforementioned interactive operation.

[0027] In a second aspect, an embodiment of the present application provides a cross-device control apparatus, including:

[0028] An interactive area display unit, configured to display a cross-device interactive area in a first operating interface of the first device; the cross-device interactive area includes a second operating interface and / or a first control; the second operating interface is a virtual touch panel of the second device;

[0029] an operation instruction generating unit, configured to generate an operation instruction corresponding to the interaction operation in response to an interaction operation initiated by a user in the cross-device interaction area;

[0030] An operation instruction sending unit is used to send the operation instruction to the second device to control the second device to execute the operation instruction corresponding to the interactive operation.

[0031] In a possible implementation manner of the second aspect, the interaction area display unit:

[0032] The first click operation unit is used to display the second operation interface in the first operation interface in response to the user's first click operation on the first control in the first operation interface; the first control is displayed when the first device and the second device are in a state where a cross-device control connection has been established.

[0033] In a possible implementation manner of the second aspect, the control device further includes:

[0034] a connection request sending unit, configured to send a connection request to the second device if the first device state of the first device satisfies a preset first connection condition;

[0035] A confirmation instruction receiving unit is used to receive a connection confirmation instruction fed back by the second device based on the connection request, establish the cross-device control connection with the second device, and display the first control in the first operation interface.

[0036] In a possible implementation manner of the second aspect, the first device state includes: a screen state, a motion state, and a device placement posture; and the connection request sending unit includes:

[0037] The first device status confirmation unit is used to send a connection request to the second device if at least one of the following conditions is met: the screen status is in a bright screen status, the motion status is a static status, and the device placement posture is within a preset angle range.

[0038] In a possible implementation manner of the second aspect, sending the connection request includes:

[0039] a first Bluetooth signal sending unit, configured to generate a first Bluetooth signal carrying the connection request through a first Bluetooth module in the first device; the first Bluetooth signal is used for the second device to receive the first Bluetooth signal through a second Bluetooth module, and to determine a distance value between the second device and the first device through the first Bluetooth signal;

[0040] The confirmation instruction receiving unit comprises:

[0041] The second Bluetooth signal receiving unit is used to receive a second Bluetooth signal fed back by the second device; the second Bluetooth signal carries the connection confirmation instruction; the second Bluetooth signal is generated when the second device detects that the distance value is less than a preset distance threshold.

[0042] In a possible implementation manner of the second aspect, the interactive operation includes an interface sharing operation; the first operation interface is displayed in a first window; and the operation instruction generating unit includes:

[0043] a drag operation unit, configured to close the first window and generate an interface sharing instruction for the first operation interface if it is detected that the user initiates a drag operation of dragging the first window to the first control;

[0044] The sending of the operation instruction comprises:

[0045] The interface sharing instruction sending unit is used to send the interface sharing instruction to the second device to display the first operation interface in a third window on the second device.

[0046] In a possible implementation manner of the second aspect, the interaction area display unit includes:

[0047] A window layer changing unit is used to display the first operation interface through a first window and display the second operation interface through a second window if the first operation interface is the operation interface of a first application installed on the first device; the first layer where the first window is located is below the second layer where the second window is located.

[0048] In a possible implementation manner of the second aspect, the control device further includes:

[0049] A first click operation unit is configured to adjust the first layer to the second layer in response to a second click operation of the user on the first window; and / or

[0050] The second click operation unit is used to adjust the second layer to the first layer in response to a third click operation of the user on the second window.

[0051] In a possible implementation of the second aspect, the interactive operation includes a first sliding operation; and the operation instruction generation includes: a sliding instruction generation unit, configured to generate a sliding instruction corresponding to the first sliding operation in response to the first sliding operation initiated by the user on the second operation interface;

[0052] The sending of the operation instruction includes: a sliding instruction sending unit, which is used to send the sliding instruction to the second device to control the virtual cursor in the second device to move based on the sliding instruction.

[0053] In a possible implementation of the second aspect, the interactive operation includes a text input operation; the control device further includes: a text mode switching unit, configured to set the second operation interface to a text input mode in response to a fourth click operation initiated by a user on a second control in the second operation interface;

[0054] The operation instruction generating unit includes: a text input instruction generating unit, which is used to respond to a text input operation initiated by a user on the second operation interface, determine at least one character corresponding to the text input operation, and generate a text input instruction for the at least one character;

[0055] The operation instruction sending unit includes: a text input instruction sending unit, which is used to send the text input instruction to the second device, so as to input the at least one character in a text box corresponding to the second device.

[0056] In a possible implementation manner of the second aspect, the interactive operation includes a screenshot operation; the control device further includes: a screenshot mode setting unit, configured to set the second operation interface to a screenshot mode in response to a fifth click operation initiated by the user on a third control in the second operation interface;

[0057] The operation instruction generation includes: a first screenshot area determination unit, configured to determine, in response to a second sliding operation initiated by a user on the second operation interface, a first screenshot area corresponding to the second sliding operation; a screenshot instruction generation unit, configured to determine, based on the first screenshot area, a second screenshot area in a third operation interface of the second device, and generate a screenshot instruction for the second screenshot area;

[0058] The operation instruction sending includes: a screenshot instruction sending unit, used to send the screenshot instruction to the second device; and a screenshot image receiving unit, used to receive a screenshot image about the second screenshot area fed back by the second device based on the screenshot instruction.

[0059] In the third aspect, an embodiment of the present application provides a cross-device control method, applied to a second device, including: receiving an operation instruction sent by a first device; the operation instruction is generated based on an interactive operation initiated by a user in a cross-device interaction area displayed in a first operation interface of the first device; the cross-device interaction area includes a second operation interface and / or a first control; and responding to the interactive operation based on the operation instruction.

[0060] In a possible implementation of the third aspect, before receiving the operation instruction sent by the first device, it also includes: receiving a connection request sent by the first device; the connection request is sent when the first device state of the first device satisfies a preset first connection condition; if the second device state of the second device satisfies a preset second connection condition, a connection confirmation instruction is sent to the first device to establish a cross-device control connection with the first device; when the first device is in the state of the cross-device control connection, the first device will display the first control in the first operation interface; the first control is used to display the second operation interface in the first operation interface when the user initiates a first click operation.

[0061] In a possible implementation of the third aspect, if the second device state of the second device meets the preset second connection condition, a connection confirmation instruction is sent to the first device to establish a cross-device control connection with the first device, including: receiving a first Bluetooth signal sent by a first Bluetooth module of the first device through a second Bluetooth module; the first Bluetooth signal carries the connection request; in response to the connection request, determining a position relationship between the second device and the first device; if the position relationship meets the preset relative position condition, feeding back a second Bluetooth signal to the first device through the second Bluetooth module; the second Bluetooth signal carries the connection confirmation instruction.

[0062] In a possible implementation manner of the third aspect, the operation instruction includes an interface sharing instruction; the receiving the operation instruction sent by the first device includes: receiving the interface sharing instruction sent by the first device; the interface classification instruction is generated by the first device in response to a user initiating a drag operation of dragging the first window to the first control; the first window is a window displayed on the first operation interface;

[0063] The responding to the interactive operation based on the operation instruction includes: displaying the first operation interface through a third window in response to the interface sharing instruction.

[0064] In a possible implementation manner of the third aspect, the operation instruction includes a sliding instruction; the receiving the operation instruction sent by the first device includes: receiving the sliding instruction sent by the first device; the sliding instruction is generated by the first device in response to a first sliding operation initiated by a user on the second operation interface;

[0065] The responding to the interactive operation based on the operation instruction includes: in response to the sliding instruction, moving a virtual cursor displayed in the second device to a target position indicated by the sliding instruction.

[0066] In a possible implementation manner of the third aspect, the operation instruction includes a text input instruction; the receiving the operation instruction sent by the first device includes: receiving the text input instruction sent by the first device; the text input instruction is generated by the first device in response to a text input operation initiated by the user on the second operation interface; the text input instruction carries at least one character input by the user; the second operation interface is in a text input mode;

[0067] The responding to the interactive operation based on the operation instruction includes: responding to the text input instruction, inputting the at least one character in a currently selected text box in the second device.

[0068] In a possible implementation manner of the third aspect, the operation instruction includes a screenshot instruction; the receiving the operation instruction sent by the first device includes: receiving the screenshot instruction sent by the first device; the screenshot instruction is generated by the first device in response to a second sliding operation initiated by the user on the second operation interface; the second operation interface is in a screenshot mode;

[0069] The responding to the interactive operation based on the operation instruction includes: generating a second screenshot area in the second device in response to the second sliding operation; generating a screenshot image based on the image framed by the second screenshot area in a third operation interface in response to the screenshot instruction; the third operation interface is an operation interface displayed locally on the second device; and sending the screenshot image to the first device.

[0070] In a fourth aspect, an embodiment of the present application provides a cross-device control device, comprising: an operation instruction receiving unit, for receiving an operation instruction sent by a first device; the operation instruction is generated based on an interactive operation initiated by a user in a cross-device interaction area displayed in a first operation interface of the first device; the cross-device interaction area includes a second operation interface and / or a first control; an operation instruction responding unit, for responding to the interactive operation based on the operation instruction.

[0071] In a possible implementation manner of the fourth aspect, the control device further includes:

[0072] a connection request receiving unit, configured to receive a connection request sent by the first device; the connection request is sent when a first device state of the first device satisfies a preset first connection condition;

[0073] A confirmation instruction sending unit is used to send a connection confirmation instruction to the first device to establish a cross-device control connection with the first device if the second device state of the second device meets the preset second connection condition; when the first device is in the state of the cross-device control connection, the first device will display the first control in the first operation interface; the first control is used to display the second operation interface in the first operation interface when the user initiates a first click operation.

[0074] In a possible implementation manner of the fourth aspect, the confirmation instruction sending unit includes: a first Bluetooth signal receiving unit, configured to receive, through a second Bluetooth module, a first Bluetooth signal sent by a first Bluetooth module of the first device; the first Bluetooth signal carries the connection request;

[0075] a distance value determining unit, configured to determine a positional relationship between the second device and the first device in response to the connection request;

[0076] The second Bluetooth signal sending unit is used to feed back a second Bluetooth signal to the first device through the second Bluetooth module if the position relationship meets the preset relative position condition; the second Bluetooth signal carries the connection confirmation instruction.

[0077] In a possible implementation manner of the fourth aspect, the operation instruction includes an interface sharing instruction; the operation instruction receiving unit includes: a sharing instruction receiving unit, configured to receive the interface sharing instruction sent by the first device; the interface classification instruction is generated by the first device in response to a user initiating a drag operation of dragging the first window to the first control; the first window is a window displayed on the first operation interface;

[0078] The operation instruction response unit includes: a sharing instruction response unit, which is used to respond to the interface sharing instruction and display the first operation interface through a third window.

[0079] In a possible implementation manner of the fourth aspect, the operation instruction includes a sliding instruction; the operation instruction response unit includes: a sliding instruction receiving unit, configured to receive the sliding instruction sent by the first device; the sliding instruction is generated by the first device in response to a first sliding operation initiated by a user on the second operation interface;

[0080] The operation instruction response unit includes: a sliding instruction response unit, which is used to move a virtual cursor displayed in the second device to a target position indicated by the sliding instruction in response to the sliding instruction.

[0081] In a possible implementation manner of the fourth aspect, the operation instruction includes a text input instruction; the operation instruction receiving unit includes: a text input instruction receiving unit, configured to receive the text input instruction sent by the first device; the text input instruction is generated by the first device in response to a text input operation initiated by a user on the second operation interface; the text input instruction carries at least one character input by the user; and the second operation interface is in a text input mode;

[0082] The operation instruction receiving unit includes: an input instruction responding unit, which is used to input the at least one character in the currently selected text box in the second device in response to the text input instruction.

[0083] In a possible implementation manner of the fourth aspect, the operation instruction includes a screenshot instruction; the receiving the operation instruction sent by the first device includes: a screenshot instruction receiving unit, configured to receive the screenshot instruction sent by the first device; the screenshot instruction is generated by the first device in response to a second sliding operation initiated by a user on the second operation interface; and the second operation interface is in a screenshot mode;

[0084] The operation instruction response unit includes: a second screenshot area generation unit, used to generate a second screenshot area in the second device in response to the second sliding operation; a screenshot image generation unit, used to generate a screenshot image based on the image framed by the second screenshot area in a third operation interface in response to the screenshot instruction; the third operation interface is an operation interface displayed locally on the second device; and a screenshot image sending unit, used to send the screenshot image to the first device.

[0085] In a fifth aspect, an embodiment of the present application provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the cross-device control method described in any one of the first aspect or the third aspect when executing the computer program.

[0086] In a sixth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and wherein when the computer program is executed by a processor, the cross-device control method described in any one of the first aspect or the third aspect is implemented.

[0087] In a seventh aspect, an embodiment of the present application provides a computer program product. When the computer program product is run on an electronic device, the electronic device executes the cross-device control method described in any one of the first aspect or the third aspect.

[0088] In an eighth aspect, an embodiment of the present application provides a chip system, including a processor, the processor being coupled to a memory, the processor executing a computer program stored in the memory to implement a cross-device control method as described in any one of the first aspect or the third aspect.

[0089] It can be understood that the beneficial effects of the second to eighth aspects mentioned above can be found in the relevant description of the first aspect mentioned above, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0090] Figure 1 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application;

[0091] Figure 2 is a software structure block diagram of the electronic device of the embodiment of the present application;

[0092] Figure 3 This is a usage scenario of multi-device collaborative work provided by an embodiment of the present application;

[0093] Figure 4 It is a schematic diagram of the structure of a cross-device control system provided by an embodiment of the present application;

[0094] Figure 5 is a schematic diagram of the structure of a cross-device control system provided by another embodiment of the present application;

[0095] Figure 6 It is an implementation interaction flow chart of the device phase in the cross-device control method provided by an embodiment of the present application;

[0096] Figure 7 This is an operation diagram of enabling the cross-device control function provided by an embodiment of the present application;

[0097] Figure 8 This is a prompt diagram of prompt information provided by an embodiment of the present application;

[0098] Fig. 9 is a schematic diagram of a prompt for selecting to connect to a second device provided in an embodiment of the present application;

[0099] Fig.10 This is a schematic diagram of the operation of connection confirmation provided by an embodiment of the present application;

[0100] Fig.11 is a schematic diagram of a preset relative position relationship between a second device and a first device provided in an embodiment of the present application;

[0101] Fig.12 is a display schematic diagram of a first control provided by an embodiment of the present application;

[0102] Fig.13is a schematic diagram of a multi-window display provided by an embodiment of the present application;

[0103] Fig.14 This is an interaction flow chart for implementing the user interaction phase in the cross-device control method provided by an embodiment of the present application;

[0104] Fig.15 This is a response flow chart of an interactive operation provided by an embodiment of the present application;

[0105] Fig.16 This is an interactive flow chart of cursor control operation in a cross-device control process provided by an embodiment of the present application;

[0106] Fig.17 is a schematic diagram of a cursor control operation provided by an embodiment of the present application;

[0107] Fig.18 This is an interactive flow chart for a screenshot operation in a cross-device control process provided by an embodiment of the present application;

[0108] Fig.19 is a schematic diagram of a screenshot operation provided by an embodiment of the present application;

[0109] Fig. 20 is a schematic diagram of a drag operation on a screenshot image according to an embodiment of the present application;

[0110] Fig.21 This is an interactive flow chart for text input operations in a cross-device control process provided by an embodiment of the present application;

[0111] Fig. 22 is a schematic diagram of a text input process provided by an embodiment of the present application;

[0112] Fig.23 This is an interactive flow chart for recording operations in a cross-device control process provided by an embodiment of the present application;

[0113] Fig.24 is a schematic diagram of a second operation interface in a recording mode provided by an embodiment of the present application;

[0114] Fig.25 is a schematic diagram of a second operation interface in a recording mode provided by an embodiment of the present application;

[0115] Fig.26 is a schematic diagram of an interface sharing operation provided by an embodiment of the present application;

[0116] Fig. 27 This is a flowchart of the implementation of the cross-device control method provided by an embodiment of the present application on the first device side;

[0117] Fig.28is a structural block diagram of a cross-device control device provided in an embodiment of the present application;

[0118] Fig.29 This is a flowchart of the implementation of the cross-device control method provided by an embodiment of the present application on the second device side;

[0119] Fig.30 is a structural block diagram of a cross-device control device provided in an embodiment of the present application;

[0120] Fig.31 It is a structural block diagram of an electronic device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0121] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.

[0122] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, wholes, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or combinations thereof.

[0123] It should also be understood that the term “and / or” used in the specification and appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0124] As used in the specification and appended claims of this application, the term "if" can be interpreted as "when" or "uponce" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" can be interpreted as meaning "uponce it is determined" or "in response to determining" or "uponce [described condition or event] is detected" or "in response to detecting [described condition or event]", depending on the context.

[0125] In addition, in the description of the present application specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0126] References to "one embodiment" or "some embodiments" etc. described in the specification of this application mean that one or more embodiments of the present application include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0127] The electronic devices to which the cross-device control method provided in the embodiment of the present application can be applied may be mobile phones, tablet computers, wearable devices (such as sports bracelets and smart watches, etc.), augmented reality (AR) / virtual reality (VR) devices, laptop computers, ultra-mobile personal computers (UMPC), netbooks, personal digital assistants (PDA) and other electronic devices. In particular, the cross-device control method can be applied to electronic devices with touch screens, and the electronic devices with touch screens can display a first operation interface through the touch screen, and display a cross-device interaction area in the first operation interface, so that the user can simultaneously operate the electronic device (i.e., the first device) and another electronic device (i.e., the second device) through the touch screen. The embodiment of the present application does not impose any restrictions on the specific types of electronic devices.

[0128] For example, the above-mentioned electronic device can be a station (STAION, ST) in a WLAN, a cellular phone, a cordless phone, a Session Initiation Protocol (SIP) phone, a Wireless Local Loop (WLL) station, a personal digital processing (PDA) device, a handheld device with wireless communication function, a computing device or other processing device connected to a wireless modem, a computer, a laptop computer, a handheld communication device, a handheld computing device, and / or other devices for communicating on a wireless system and a next-generation communication system, for example, a mobile terminal in a 5G network or a mobile terminal in a future evolved Public Land Mobile Network (PLMN) network, etc.

[0129] Figure 1A schematic structural diagram of an electronic device 100 is shown.

[0130] The electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0131] It is to be understood that the structure illustrated in the embodiment of the present invention does not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may include more or fewer components than shown in the figure, or combine some components, or separate some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0132] The processor 110 may include one or more processing units, for example, the processor 110 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0133] The controller can generate operation control signals according to the instruction operation code and timing signal to complete the control of instruction fetching and execution.

[0134] The processor 110 may also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. The memory may store instructions or data that the processor 110 has just used or cyclically used. If the processor 110 needs to use the instruction or data again, it may be directly called from the memory. This avoids repeated access, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0135] In some embodiments, the processor 110 may include one or more interfaces. The interface may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0136] The I2C interface is a bidirectional synchronous serial bus, including a serial data line (SDA) and a serial clock line (SCL). In some embodiments, the processor 110 may include multiple groups of I2C buses. The processor 110 may be coupled to the touch sensor 180K, the charger, the flash, the camera 193, etc. through different I2C bus interfaces. For example: the processor 110 may be coupled to the touch sensor 180K through the I2C interface, so that the processor 110 communicates with the touch sensor 180K through the I2C bus interface, thereby realizing the touch function of the electronic device 100.

[0137] The I2S interface can be used for audio communication. In some embodiments, the processor 110 can include multiple I2S buses. The processor 110 can be coupled to the audio module 170 via the I2S bus to achieve communication between the processor 110 and the audio module 170. In some embodiments, the audio module 170 can transmit an audio signal to the wireless communication module 160 via the I2S interface to achieve the function of answering a call through a Bluetooth headset.

[0138] The PCM interface can also be used for audio communication, sampling, quantizing and encoding analog signals. In some embodiments, the audio module 170 and the wireless communication module 160 can be coupled via a PCM bus interface. In some embodiments, the audio module 170 can also transmit audio signals to the wireless communication module 160 via the PCM interface to realize the function of answering calls via a Bluetooth headset. Both the I2S interface and the PCM interface can be used for audio communication.

[0139] The UART interface is a universal serial data bus for asynchronous communication. The bus can be a bidirectional communication bus. It converts the data to be transmitted between serial communication and parallel communication. In some embodiments, the UART interface is generally used to connect the processor 110 and the wireless communication module 160. For example, the processor 110 communicates with the Bluetooth module in the wireless communication module 160 through the UART interface to implement the Bluetooth function. In some embodiments, the audio module 170 can transmit an audio signal to the wireless communication module 160 through the UART interface to implement the function of playing music through a Bluetooth headset.

[0140] The MIPI interface can be used to connect the processor 110 with peripheral devices such as the display screen 194 and the camera 193. The MIPI interface includes a camera serial interface (CSI), a display serial interface (DSI), etc. In some embodiments, the processor 110 and the camera 193 communicate via the CSI interface to implement the shooting function of the electronic device 100. The processor 110 and the display screen 194 communicate via the DSI interface to implement the display function of the electronic device 100.

[0141] The GPIO interface can be configured by software. The GPIO interface can be configured as a control signal or as a data signal. In some embodiments, the GPIO interface can be used to connect the processor 110 with the camera 193, the display 194, the wireless communication module 160, the audio module 170, the sensor module 180, etc. The GPIO interface can also be configured as an I2C interface, an I2S interface, a UART interface, a MIPI interface, etc.

[0142] The USB interface 130 is an interface that complies with USB standard specifications, and may be a Mini USB interface, a Micro USB interface, a USB Type C interface, etc. The USB interface 130 may be used to connect a charger to charge the electronic device 100, or may be used to transmit data between the electronic device 100 and a peripheral device. It may also be used to connect headphones to play audio through the headphones.

[0143] It is understandable that the interface connection relationship between the modules illustrated in the embodiment of the present invention is only a schematic illustration and does not constitute a structural limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may also adopt different interface connection methods in the above embodiments, or a combination of multiple interface connection methods.

[0144] The charging management module 140 is used to receive charging input from a charger. The charger may be a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 140 may receive charging input from a wired charger through the USB interface 130. In some wireless charging embodiments, the charging management module 140 may receive wireless charging input through a wireless charging coil of the electronic device 100. While the charging management module 140 is charging the battery 142, it may also power the electronic device through the power management module 141.

[0145] The power management module 141 is used to connect the battery 142, the charging management module 140 and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140, and supplies power to the processor 110, the internal memory 121, the display screen 194, the camera 193, and the wireless communication module 160. The power management module 141 can also be used to monitor parameters such as battery capacity, battery cycle number, battery health status (leakage, impedance), etc. In some other embodiments, the power management module 141 can also be set in the processor 110. In other embodiments, the power management module 141 and the charging management module 140 can also be set in the same device.

[0146] The wireless communication function of the electronic device 100 can be implemented through the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor and the baseband processor.

[0147] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in electronic device 100 can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve the utilization of antennas. For example, antenna 1 can be reused as a diversity antenna for a wireless local area network. In some other embodiments, the antenna can be used in combination with a tuning switch.

[0148] The mobile communication module 150 can provide solutions for wireless communications including 2G / 3G / 4G / 5G, etc., applied to the electronic device 100. The mobile communication module 150 may include at least one filter, a switch, a power amplifier, a low noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves from the antenna 1, and filter, amplify, and process the received electromagnetic waves, and transmit them to the modulation and demodulation processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modulation and demodulation processor, and convert it into electromagnetic waves for radiation through the antenna 1. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the processor 110. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the same device as at least some of the modules of the processor 110.

[0149] The modem processor may include a modulator and a demodulator. Among them, the modulator is used to modulate the low-frequency baseband signal to be sent into a medium-high frequency signal. The demodulator is used to demodulate the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After the low-frequency baseband signal is processed by the baseband processor, it is passed to the application processor. The application processor outputs a sound signal through an audio device (not limited to a speaker 170A, a receiver 170B, etc.), or displays an image or video through a display screen 194. In some embodiments, the modem processor may be an independent device. In other embodiments, the modem processor may be independent of the processor 110 and be set in the same device as the mobile communication module 150 or other functional modules.

[0150] The wireless communication module 160 can provide wireless communication solutions including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR), etc., which are applied to the electronic device 100. The wireless communication module 160 can be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via the antenna 2, modulates the frequency of the electromagnetic wave signal and performs filtering, and sends the processed signal to the processor 110. The wireless communication module 160 can also receive the signal to be sent from the processor 110, modulate the frequency of it, amplify it, and convert it into electromagnetic waves for radiation through the antenna 2.

[0151] In some embodiments, the antenna 1 of the electronic device 100 is coupled to the mobile communication module 150, and the antenna 2 is coupled to the wireless communication module 160, so that the electronic device 100 can communicate with the network and other devices through wireless communication technology. The wireless communication technology may include global system for mobile communications (GSM), general packet radio service (GPRS), code division multiple access (CDMA), wideband code division multiple access (WCDMA), time-division code division multiple access (TD-SCDMA), long term evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technology. The GNSS may include a global positioning system (GPS), a global navigation satellite system (GLONASS), a Beidou navigation satellite system (BDS), a quasi-zenith satellite system (QZSS) and / or a satellite based augmentation system (SBAS).

[0152] The electronic device 100 implements the display function through a GPU, a display screen 194, and an application processor. The GPU is a prompt microprocessor that connects the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs that execute program instructions to generate or change display information. It should be noted that the GPU can perform abnormal identification on the storage unit associated with the controlled display screen 194 through the cross-device control method provided in this embodiment. The GPU can transfer the image data to be displayed to the storage unit in the display screen 194 for storage for subsequent display.

[0153] The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode or an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), Miniled, MicroLed, Micro-oLed, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the electronic device 100 may include 1 or N display screens 194, where N is a positive integer greater than 1. The display screen 194 may include a touch panel and other input devices. The display screen 194 may be associated with one or more storage units, which are used to cache image data displayed on the display screen 194.

[0154] The electronic device 100 can realize the shooting function through ISP, camera 193, video codec, GPU, display screen 194 and application processor.

[0155] ISP is used to process the data fed back by camera 193. For example, when taking a photo, the shutter is opened, and the light is transmitted to the camera photosensitive element through the lens. The light signal is converted into an electrical signal, and the camera photosensitive element transmits the electrical signal to ISP for processing and converts it into an image visible to the naked eye. ISP can also perform algorithm optimization on the noise, brightness, and skin color of the image. ISP can also optimize the exposure, color temperature and other parameters of the shooting scene. In some embodiments, ISP can be set in camera 193.

[0156] The camera 193 is used to capture still images or videos. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, and then passes the electrical signal to the ISP to be converted into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV or other format. In some embodiments, the electronic device 100 may include 1 or N cameras 193, where N is a positive integer greater than 1.

[0157] The digital signal processor is used to process digital signals, and can process not only digital image signals but also other digital signals. For example, when the electronic device 100 is selecting a frequency point, the digital signal processor is used to perform Fourier transform on the frequency point energy.

[0158] Video codecs are used to compress or decompress digital videos. The electronic device 100 may support one or more video codecs. Thus, the electronic device 100 may play or record videos in a variety of coding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc.

[0159] NPU is a neural network (NN) computing processor. By drawing on the structure of biological neural networks, such as the transmission mode between neurons in the human brain, it can quickly process input information and can also continuously self-learn. Through NPU, applications such as intelligent cognition of electronic device 100 can be realized, such as image recognition, face recognition, voice recognition, text understanding, etc.

[0160] The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 100. The external memory card communicates with the processor 110 through the external memory interface 120 to implement a data storage function. For example, files such as music and videos are stored in the external memory card. Similarly, the cross-device control method in the embodiment of the present application can manage the storage space in the external memory card.

[0161] The internal memory 121 can be used to store computer executable program codes, which include instructions. The internal memory 121 may include a program storage area and a data storage area. Among them, the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc. The data storage area may store data created during the use of the electronic device 100 (such as audio data, a phone book, etc.), etc. In addition, the internal memory 121 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc. The processor 110 executes various functional applications and data processing of the electronic device 100 by running instructions stored in the internal memory 121, and / or instructions stored in a memory provided in the processor. Similarly, the cross-device control method in the embodiment of the present application can manage the storage space in the internal memory.

[0162] The electronic device 100 can implement audio functions such as music playing and recording through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the headphone jack 170D, and the application processor.

[0163] The audio module 170 is used to convert digital audio information into analog audio signal output, and is also used to convert analog audio input into digital audio signals. The audio module 170 can also be used to encode and decode audio signals. In some embodiments, the audio module 170 can be arranged in the processor 110, or some functional modules of the audio module 170 can be arranged in the processor 110.

[0164] The speaker 170A, also called a "speaker", is used to convert an audio electrical signal into a sound signal. The electronic device 100 can listen to music or listen to a hands-free call through the speaker 170A.

[0165] The receiver 170B, also called a "earpiece", is used to convert audio electrical signals into sound signals. When the electronic device 100 receives a call or voice message, the voice can be received by placing the receiver 170B close to the human ear.

[0166] Microphone 170C, also called "microphone" or "microphone", is used to convert sound signals into electrical signals. When making a call or sending a voice message, the user can speak by putting their mouth close to microphone 170C to input the sound signal into microphone 170C. The electronic device 100 can be provided with at least one microphone 170C. In other embodiments, the electronic device 100 can be provided with two microphones 170C, which can not only collect sound signals but also realize noise reduction function. In other embodiments, the electronic device 100 can also be provided with three, four or more microphones 170C to collect sound signals, reduce noise, identify the sound source, realize directional recording function, etc.

[0167] The earphone interface 170D is used to connect a wired earphone and can be a USB interface 130 or a 3.5 mm open mobile terminal platform (OMTP) standard interface or a cellular telecommunications industry association of the USA (CTIA) standard interface.

[0168] The pressure sensor 180A is used to sense the pressure signal and can convert the pressure signal into an electrical signal. In some embodiments, the pressure sensor 180A can be set on the display screen 194. For example, the electronic device can obtain the user's weight through the pressure sensor 180A. There are many types of pressure sensors 180A, such as resistive pressure sensors, inductive pressure sensors, capacitive pressure sensors, etc. The capacitive pressure sensor can be a parallel plate including at least two conductive materials. When a force acts on the pressure sensor 180A, the capacitance between the electrodes changes. The electronic device 100 determines the intensity of the pressure based on the change in capacitance. When a touch operation acts on the display screen 194, the electronic device 100 detects the touch operation intensity according to the pressure sensor 180A. The electronic device 100 can also calculate the touch position according to the detection signal of the pressure sensor 180A. In some embodiments, touch operations acting on the same touch position but with different touch operation intensities can correspond to different operation instructions. For example: when a touch operation with a touch operation intensity less than the first pressure threshold acts on the short message application icon, an instruction to view the short message is executed. When a touch operation with a touch operation intensity greater than or equal to the first pressure threshold acts on the short message application icon, an instruction to create a new short message is executed. The pressure sensor 180A can be used to receive an interactive operation initiated by a user on the cross-device interactive area in the first operation interface when the display screen 194 displays the first operation interface.

[0169] The gyro sensor 180B can be used to determine the motion posture of the electronic device 100. In some embodiments, the angular velocity of the electronic device 100 around three axes (i.e., x, y, and z axes) can be determined by the gyro sensor 180B. The gyro sensor 180B can be used for anti-shake shooting. For example, when the shutter is pressed, the gyro sensor 180B detects the angle of the electronic device 100 shaking, calculates the distance that the lens module needs to compensate based on the angle, and allows the lens to offset the shaking of the electronic device 100 through reverse movement to achieve anti-shake. The gyro sensor 180B can also be used for navigation and somatosensory game scenes. In this embodiment, the electronic device can collect the user's motion posture during the movement through the gyro sensor 180B, thereby generating corresponding motion data.

[0170] The air pressure sensor 180C is used to measure air pressure. In some embodiments, the electronic device 100 calculates the altitude through the air pressure value measured by the air pressure sensor 180C to assist positioning and navigation.

[0171] The magnetic sensor 180D includes a Hall sensor. The electronic device 100 can use the magnetic sensor 180D to detect the opening and closing of the flip leather case. In some embodiments, when the electronic device 100 is a flip phone, the electronic device 100 can detect the opening and closing of the flip cover according to the magnetic sensor 180D. Then, according to the detected opening and closing state of the leather case or the opening and closing state of the flip cover, the flip cover can be automatically unlocked.

[0172] The acceleration sensor 180E can detect the magnitude of the acceleration of the electronic device 100 in all directions (generally three axes). When the electronic device 100 is stationary, the magnitude and direction of gravity can be detected. It can also be used to identify the posture of the electronic device, and is applied to applications such as horizontal and vertical screen switching and pedometers. In this embodiment, the electronic device can collect the movement speed of the user during the movement through the gyroscope sensor 180B, thereby generating corresponding movement data.

[0173] The distance sensor 180F is used to measure the distance. The electronic device 100 can measure the distance by infrared or laser. In some embodiments, when shooting a scene, the electronic device 100 can use the distance sensor 180F to measure the distance to achieve fast focusing.

[0174] The proximity light sensor 180G may include, for example, a light emitting diode (LED) and a light detector, such as a photodiode. The light emitting diode may be an infrared light emitting diode. The electronic device 100 emits infrared light outward through the light emitting diode. The electronic device 100 uses a photodiode to detect infrared reflected light from nearby objects. When sufficient reflected light is detected, it can be determined that there is an object near the electronic device 100. When insufficient reflected light is detected, the electronic device 100 can determine that there is no object near the electronic device 100. The electronic device 100 can use the proximity light sensor 180G to detect that the user holds the electronic device 100 close to the ear to talk, so as to automatically turn off the screen to save power. The proximity light sensor 180G can also be used in leather case mode and pocket mode to automatically unlock and lock the screen.

[0175] The ambient light sensor 180L is used to sense the ambient light brightness. The electronic device 100 can adaptively adjust the brightness of the display screen 194 according to the perceived ambient light brightness. The ambient light sensor 180L can also be used to automatically adjust the white balance when taking pictures. The ambient light sensor 180L can also cooperate with the proximity light sensor 180G to detect whether the electronic device 100 is in a pocket to prevent accidental touches.

[0176] The fingerprint sensor 180H is used to collect fingerprints. The electronic device 100 can use the collected fingerprint characteristics to implement fingerprint unlocking, access application locks, fingerprint photography, fingerprint call answering, etc.

[0177] The temperature sensor 180J is used to detect temperature. In some embodiments, the electronic device 100 uses the temperature detected by the temperature sensor 180J to execute a temperature processing strategy. For example, when the temperature reported by the temperature sensor 180J exceeds a threshold, the electronic device 100 reduces the performance of a processor located near the temperature sensor 180J to reduce power consumption and implement thermal protection. In other embodiments, when the temperature is lower than another threshold, the electronic device 100 heats the battery 142 to avoid abnormal shutdown of the electronic device 100 due to low temperature. In other embodiments, when the temperature is lower than another threshold, the electronic device 100 performs a boost on the output voltage of the battery 142 to avoid abnormal shutdown caused by low temperature.

[0178] The touch sensor 180K is also called a "touch control device". The touch sensor 180K can be set on the display screen 194. The touch sensor 180K and the display screen 194 form a touch screen, also called a "touch control screen". The touch sensor 180K is used to detect touch operations acting on or near it. The touch sensor can pass the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through the display screen 194. In other embodiments, the touch sensor 180K can also be set on the surface of the electronic device 100, which is different from the position of the display screen 194.

[0179] The bone conduction sensor 180M can obtain a vibration signal. In some embodiments, the bone conduction sensor 180M can obtain a vibration signal of a vibrating bone block of the vocal part of the human body. The bone conduction sensor 180M can also contact the human pulse to receive a blood pressure beat signal. In some embodiments, the bone conduction sensor 180M can also be set in an earphone and combined into a bone conduction earphone. The audio module 170 can parse out a voice signal based on the vibration signal of the vibrating bone block of the vocal part obtained by the bone conduction sensor 180M to realize a voice function. The application processor can parse the heart rate information based on the blood pressure beat signal obtained by the bone conduction sensor 180M to realize a heart rate detection function.

[0180] The key 190 includes a power key, a volume key, etc. The key 190 may be a mechanical key or a touch key. The electronic device 100 may receive key input and generate key signal input related to user settings and function control of the electronic device 100.

[0181] Motor 191 can generate vibration prompts. Motor 191 can be used for incoming call vibration prompts, and can also be used for touch vibration feedback. For example, touch operations acting on different applications (such as taking pictures, audio playback, etc.) can correspond to different vibration feedback effects. For touch operations acting on different areas of the display screen 194, motor 191 can also correspond to different vibration feedback effects. Different application scenarios (for example: time reminders, receiving messages, alarm clocks, games, etc.) can also correspond to different vibration feedback effects. The touch vibration feedback effect can also support customization.

[0182] Indicator 192 may be an indicator light, which may be used to indicate charging status, power changes, messages, missed calls, notifications, etc.

[0183] The SIM card interface 195 is used to connect a SIM card. The SIM card can be connected to and separated from the electronic device 100 by inserting it into the SIM card interface 195 or pulling it out from the SIM card interface 195. The electronic device 100 can support 1 or N SIM card interfaces, where N is a positive integer greater than 1. The SIM card interface 195 can support Nano SIM cards, Micro SIM cards, SIM cards, and the like. Multiple cards can be inserted into the same SIM card interface 195 at the same time. The types of the multiple cards can be the same or different. The SIM card interface 195 can also be compatible with different types of SIM cards. The SIM card interface 195 can also be compatible with external memory cards. The electronic device 100 interacts with the network through the SIM card to implement functions such as calls and data communications. In some embodiments, the electronic device 100 uses an eSIM, i.e., an embedded SIM card. The eSIM card can be embedded in the electronic device 100 and cannot be separated from the electronic device 100.

[0184] The software system of the electronic device 100 may adopt a layered architecture, an event-driven architecture, a micro-core architecture, a micro-service architecture, or a cloud architecture. In the embodiment of the present invention, the Android system of the layered architecture is taken as an example to exemplify the software structure of the electronic device 100.

[0185] Figure 2 It is a software structure block diagram of the electronic device of the embodiment of the present application.

[0186] The layered architecture divides the software into several layers, each with clear roles and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into four layers, from top to bottom, namely, the application layer, the application framework layer, the system layer of the Android runtime (Android runtime), and the kernel layer.

[0187] The application layer can include a series of application packages.

[0188] like Figure 2 As shown, the application package may include camera, calendar, map, WLAN, Bluetooth, music, video, short message, mailbox, WeChat, WPS and other applications.

[0189] The application framework layer provides an application programming interface (API) and a programming framework for the applications in the application layer. The application framework layer includes some predefined functions.

[0190] like Figure 2 As shown, the application framework layer may include a window manager, a content provider, a view system, a phone manager, a resource manager, a notification manager, and the like.

[0191] The window manager is used to manage window programs. The window manager can obtain the display screen size, determine whether there is a status bar, lock the screen, capture the screen, etc.

[0192] Content providers are used to store and retrieve data and make it accessible to applications. The data may include videos, images, audio, calls made and received, browsing history and bookmarks, phone books, etc.

[0193] The view system includes visual controls, such as controls for displaying text, controls for displaying images, etc. The view system can be used to build applications. A display interface can be composed of one or more views. For example, a display interface including a text notification icon can include a view for displaying text and a view for displaying images.

[0194] The phone manager is used to provide communication functions for electronic devices, such as the management of call status (including answering, hanging up, etc.).

[0195] The resource manager provides various resources for applications, such as localized strings, icons, images, layout files, video files, and so on.

[0196] The notification manager enables applications to display notification information in the status bar. It can be used to convey notification-type messages and can disappear automatically after a short stay without user interaction. For example, the notification manager is used to notify download completion, message reminders, etc. The notification manager can also be a notification that appears in the system top status bar in the form of a chart or scroll bar text, such as notifications of applications running in the background, or a notification that appears on the screen in the form of a dialog window. For example, a text message is displayed in the status bar, a prompt sound is emitted, an electronic device vibrates, an indicator light flashes, etc.

[0197] Android Runtime includes core libraries and virtual machines. Android runtime is responsible for scheduling and management of the Android system.

[0198] The core library consists of two parts: one part is the function that needs to be called by the Java language, and the other part is the Android core library.

[0199] The application layer and the application framework layer run in a virtual machine. The virtual machine executes the Java files of the application layer and the application framework layer as binary files. The virtual machine is used to perform functions such as object life cycle management, stack management, thread management, security and exception management, and garbage collection.

[0200] The system layer may include multiple functional modules, such as surface manager, media library, 3D graphics processing library (such as OpenGL ES), 2D graphics engine (such as SGL), etc.

[0201] The surface manager is used to manage the display subsystem and provide the fusion of 2D and 3D layers for multiple applications.

[0202] The media library supports playback and recording of a variety of commonly used audio and video formats, as well as static image files, etc. The media library can support a variety of audio and video encoding formats, such as: MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc.

[0203] The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, synthesis and layer processing, etc.

[0204] A 2D graphics engine is a drawing engine for 2D drawings.

[0205] The kernel layer is the layer between hardware and software. The kernel layer contains at least display driver, camera driver, audio driver, and sensor driver.

[0206] The following is an illustrative description of the workflow of the software and hardware of the electronic device 100 in conjunction with capturing a photo scene.

[0207] When the touch sensor 180K receives a touch operation, the corresponding hardware interrupt is sent to the kernel layer. The kernel layer processes the touch operation into a raw input event (including touch coordinates, timestamp of the touch operation, and other information). The raw input event is stored in the kernel layer. The application framework layer obtains the raw input event from the kernel layer and identifies the control corresponding to the input event. For example, if the touch operation is a touch single-click operation and the control corresponding to the single-click operation is the control of the camera application icon, the camera application calls the interface of the application framework layer to start the camera application, and then starts the camera driver by calling the kernel layer to capture static images or videos through the camera 193.

[0208] With the continuous advancement of the electronic process, the number and types of electronic devices have increased exponentially. A user can hold multiple different electronic devices at the same time, for example, he can operate a smartphone, a laptop, a tablet computer, and a desktop computer at the same time. When a user needs to operate an electronic device, he needs to initiate an interactive operation through the interactive module corresponding to the electronic device. For example, the interactive module of a smartphone and a tablet computer is a touch screen. When the user operates the above two types of electronic devices, he needs to initiate a touch operation through the touch screen so that the above electronic devices respond according to the touch operation; and the interactive modules mainly configured for laptops and desktop computers are keyboards and mice. When users operate the above two types of electronic devices, they need to use their corresponding mice and keyboards to operate. It can be seen that different electronic devices have independent interactive modules, and users need to perform interactive operations on their associated interactive modules before they can generate corresponding operation instructions and respond.

[0209] However, as user usage requirements continue to increase, the number of scenarios where users use multiple electronic devices at the same time also increases. Figure 3 A usage scenario of multi-device collaborative work provided by an embodiment of the present application is shown. In this usage scenario, the user uses a laptop 11 and a tablet computer 12 at the same time. The user can watch a learning video through the laptop 11. In order to better watch the video content, the user displays the video screen through the display module of the laptop 11; at the same time, in the process of watching the learning video, in order to deepen the understanding of the knowledge points and record the relevant knowledge points, the user can take notes through the handwriting board application of the tablet computer 12. In this use process, if the user needs to control the laptop 11 to pause or rewind the learning video, it is necessary to use the keyboard or mouse 13 corresponding to the laptop computer, and the user often holds the stylus 14 of the tablet computer to take notes at any time during the video viewing process. It can be seen that the user needs to frequently switch back and forth between the mouse 13 and the stylus 14 to operate different electronic devices, thereby greatly reducing the operating efficiency when performing multi-device control, and also increasing the user's operating difficulty, affecting the user's experience and consistency of use.

[0210] It can be seen that with the existing device control technology, the interaction modules between different electronic devices are independent of each other. When a user uses multiple devices at the same time, the user needs to initiate an interactive operation on the interaction module corresponding to the electronic device in order to operate the specified electronic device. This requires frequent switching between different interaction modules, resulting in problems of great operational difficulty and low operational efficiency.

[0211] Therefore, in order to solve the problem that users need to frequently switch between interactive modules of different devices when using multiple devices at the same time, the present application provides a cross-device control method, which is specifically applied to the scenario of using multiple devices at the same time, involving at least two electronic devices, namely a first device and at least one second device. The electronic devices include but are not limited to: smart phones, laptops, tablet computers, desktop computers, smart TVs and other electronic devices.

[0212] For example, Figure 4 FIG. 1 shows a schematic diagram of the structure of a cross-device control system provided by an embodiment of the present application. Figure 4 As shown, the cross-device control system includes at least two electronic devices, namely device 1 and device 2, wherein in some implementation scenarios, the above-mentioned device 1 can be the first device in the embodiment of the present application, and the above-mentioned device 2 can be the second device in the embodiment of the present application; in some implementation scenarios, the above-mentioned device 1 can also be the second device in the embodiment of the present application, and the above-mentioned device 2 can also be the first device in the embodiment of the present application, therefore, the first device and the second device can be any type of device, which is not limited here.

[0213] Among them, a communication connection is established between device 1 and device 2, and the communication connection can be a wired communication connection or a wireless communication connection. Among them, if a wireless communication connection is established between device 1 and device 2, it can be a Bluetooth communication connection, a low-power Bluetooth communication connection, a WiFi communication connection, and a near-field communication NFC communication connection. After device 1 and device 2 establish a communication connection, a cross-device interaction area can be displayed in the first operation interface of device 1. Through this cross-device interaction area, according to the interactive operation initiated by the user, the generated operation instruction can be sent to device 2 through the communication connection established above, thereby achieving the purpose of cross-device control of device 2 to execute the corresponding operation instruction.

[0214] For example, Figure 5 FIG. 2 shows a schematic diagram of a cross-device control system provided by another embodiment of the present application. Figure 5 As shown, the control system of the cross or device may include more than two electronic devices, namely device 1, device 21, device 22 and device 23. Among them, device 1 can establish communication connections with multiple different other electronic devices (i.e., devices 21 to 22) at the same time, and the user can operate device 1 and multiple other electronic devices on device 1. Figure 5As shown in (a), when device 1 establishes a communication connection with multiple other electronic devices, the first operation interface can configure a cross-device interaction area for each electronic device. For example, the first operation interface of device 1 includes area 1, area 2 and area 3, which are used to control device 21, device 22 and device 23, respectively.

[0215] Of course, multiple different electronic devices can reuse the same cross-device interaction area, such as Figure 5 As shown in (b), the cross-device interaction area can be configured with multiple different controls, namely control 1 to control 3, corresponding to the above-mentioned devices 21, device 22 and device 23 respectively. When any control is lit, it means that the cross-device interaction control is used to control the corresponding other electronic devices. For example, when control 1 is lit, the cross-device interaction area is used to control device 21, and when control 2 is lit, the cross-device interaction area is used to control device 22.

[0216] Optionally, the cross-device interaction area proposed in the embodiment of the present application can be moved. The movement method can be active movement by the user, for example, the user drags the cross-device interaction area to a destination position; or it can be moved according to the relative position change of the second device and the first device, for example, if the second device is on the left side of the first device, the cross-device interaction area can be displayed on the left part of the first device interface; if the second device is moved to the right side of the first device, the cross-device interaction area can be automatically moved to the right part of the first device interface.

[0217] For the convenience of subsequent description, the following takes the cross-device control system including two electronic devices as an example. If there are two or more electronic devices, the description of this embodiment can be referred to for implementation. The cross-device control method can specifically include two stages, namely the device connection stage and the user interaction stage. The specific description is as follows:

[0218] Phase 1: Device connection phase

[0219] In this embodiment, before implementing cross-device control, it is necessary to establish a communication connection between the first device and the second device so as to transmit an operation instruction corresponding to the interactive operation initiated by the user through the communication connection. Figure 6 The implementation interaction flow chart of the device phase in the cross-device control method provided by an embodiment of the present application is shown. The cross-device control method is applied to a cross-device control system. The response system includes: at least two electronic devices, namely a first device and a second device. The specific implementation process of establishing a connection between the two electronic devices is described as follows:

[0220] In S601, the first device determines whether the first device state satisfies a preset first connection condition.

[0221] In this embodiment, the process of establishing a cross-device control connection between the first device and the second device may be automatically executed. When the cross-device control function is enabled on the first device, the first device may execute the operation of S601 when it is started.

[0222] For example, Figure 7 FIG. 1 shows an operation diagram of enabling the cross-device control function provided by an embodiment of the present application. Figure 7 As shown in (a) of FIG. 1 , a control setting icon 71 is displayed in the main interface of the first device. After the user clicks the above icon 71, the device setting interface is entered, such as Figure 7 The device setting interface includes an extended function control 72 and other related setting items. If the first device detects that the user clicks the extended function control 72, the extended function setting interface will be entered, such as Figure 7 As shown in (c) in the figure, a cross-device control function activation control 73 is configured in the setting interface of the extended function, and the user can click the activation control 73 to set whether the cross-device control function is activated on the first device.

[0223] In one possible implementation, when the above-mentioned cross-device control is in the turned-on state, the first device will automatically determine whether there is a controllable second device when detecting that the first connection condition is met. For example, another device that is closest to the first device (such as measuring the distance through a Bluetooth module) and located in front of the first device is identified as the second device that needs to be cross-device controlled, and the operation of S602 is performed. That is, the user does not need to perform any confirmation operation, but only needs to approach the second device and be in front of the second device, and the first device will automatically establish a cross-device control connection with the second device, thereby greatly improving the connection efficiency and reducing unnecessary operations.

[0224] In one possible implementation, when the cross-device control function of the first device is turned on, the setting interface may also include a control 74 for searching for connectable devices. The user can actively search for a second device in the scene that can be cross-device controlled by clicking the above control 74.

[0225] In this embodiment, when the first device needs to establish a cross-device control connection with the second device, it will obtain the local first device state. The first device state includes at least one of: screen state, motion state, and device placement posture. The screen state is used to determine the current screen on / off state of the first device; the motion state is used to determine whether the first device is currently in a stationary state or a moving state; the device placement posture is used to determine the placement posture of the first device, such as the angle with the horizontal plane, whether it is in a vertical screen placement or a horizontal screen placement. The first device can obtain the first connection condition corresponding to the first device state obtained, and determine whether the first device state meets the corresponding first connection condition. Exemplarily, the above-mentioned determination of whether the first device state meets the above-mentioned first connection condition can be divided into the following three cases:

[0226] Case 1: The first device state includes the screen state, and whether the first device state meets the first connection condition is: judging whether the screen state of the first device is in a bright screen state. If the screen state of the first device is in a bright screen state, it is judged that the first device state meets the first connection condition.

[0227] Case 2: The first device state includes a motion state, and whether the first device state meets the first connection condition is: judging whether the motion state of the first device is in a stationary state. If the motion state of the first device is in a stationary state, it is judged that the first device meets the first connection condition.

[0228] Case 3: The first device state includes the device placement posture. Whether the first device state satisfies the first connection condition is to determine whether the device placement posture of the first device is within a preset angle range. For example, whether the angle between the first device and the horizontal plane is less than a preset angle threshold. If the device placement posture of the first device is within the preset angle range, it is identified that the first device satisfies the first connection condition.

[0229] It should be noted that if any first device state in the first device does not satisfy the corresponding first connection condition, it means that the first device does not have the conditions to realize cross-device control. At this time, the first device will not send a request for establishing a cross-device control connection to the second device, but will periodically obtain the first device state of the first device based on a preset detection period, and perform the above-mentioned S601 operation until it is detected that the first device state of the first device satisfies the first connection condition.

[0230] In a possible implementation, if the user actively searches for a second device in the scene that can be controlled across devices by clicking the above-mentioned control 74, the first device can recognize that the state of the first device meets the first connection condition and execute the operation of S602.

[0231] In a possible implementation, if the user actively searches for a second device that can be cross-device controlled in the scene by clicking the control 74, the first device can obtain the first device state, and if the first device state does not meet the first connection condition, a prompt message can be generated to prompt the user to place the first device on the desktop in order to establish a cross-device control connection with the second device. Example diagram, Figure 8 FIG. 1 is a schematic diagram showing a prompt information provided by an embodiment of the present application. Figure 8 As shown in (a) of FIG. 1 , the user holds the first device 81 (i.e., a tablet computer) in a moving state, and its motion state is not a stationary state, that is, the state of the first device does not meet the first connection condition. At this time, a corresponding prompt message is generated on the tablet computer, prompting the user to place the device on the desktop so that the state of the first device meets the first connection condition, such as Figure 8 As shown in (b) in .

[0232] In an embodiment of the present application, the first device can obtain different types of first device states and match them with corresponding first connection conditions to ensure that the first device is in a suitable state before establishing the above-mentioned cross-device control connection, thereby improving the stability of the connection after the cross-device control connection is established.

[0233] In this embodiment, when the first device detects that the first connection condition is met, it can search for other electronic devices that can be connected in the scene, obtain device identifiers of the connectable electronic devices, and generate a corresponding controllable device list based on all device identifiers, such as Figure 7 As shown in (d), multiple other electronic devices that can establish a cross-device control connection with the first device can be displayed, such as a laptop computer 1, a desktop computer 2, a smart phone 3 and a smart TV 4. The user can select the electronic device to be connected from the multiple other electronic devices that can be connected. For example, if the user clicks on the above-mentioned laptop computer 1, the first device can generate a connection request based on the device identification of the electronic device selected by the user to connect, and send the connection request to the second device.

[0234] In a possible implementation, in addition to the user selecting the second device to be connected, the first device can also calculate the device distance between each other electronic device, for example, by measuring the distance between each other electronic device through Bluetooth signals, and select the closest other electronic device as the second device, and generate corresponding connection prompt information. For example, Fig. 9 FIG. 1 shows a schematic diagram of a prompt for selecting to connect to a second device provided by an embodiment of the present application. Fig. 9As shown in (a) of FIG. 1 , after the user clicks the active search control 74, if it is detected that there are other electronic devices that can be connected in the area where the first device is located, the other electronic device with the smallest distance value is selected as the second device, and prompt information about the second device is generated, such as Fig. 9 As shown in (a), "Whether to control the laptop computer 1 across devices" is configured with a corresponding consent control control 91 and a re-search control 92. If the user clicks the above control 91, a connection request for the laptop computer 1 is generated.

[0235] In a possible implementation, the first device may be set with a corresponding distance threshold. If it is detected that there are two or more electronic devices in the area where the first device is located, and the distance between the first device and the electronic device is less than the distance threshold, corresponding connection prompt information may be generated, such as Fig. 9 As shown in (b), the user can select one of the other electronic devices to be connected from the above connection prompt information, and the first device determines the second device based on the user's selection operation. For example, if the user selects laptop 1, the first device generates a connection request for laptop 1.

[0236] In a possible implementation, the first device also stores a list of connected devices, which records the device identifiers of other electronic devices that have established cross-device control connections with the first device. If the first device detects that any device in the above-mentioned connected device list exists in the area where the first device is located after the user clicks the active search control 74, it will identify it as a second device that needs to be connected, and send a connection request to the second device, so that the second device that has been controlled across devices can be automatically connected without user operation, thereby improving the user's operating efficiency and reducing unnecessary device selection operations.

[0237] In S602, if the first device detects that the first device state meets the first connection condition, the first device sends a connection request to the second device.

[0238] In this embodiment, when the first device detects that the state of the first device satisfies the first connection condition, the first device may send a connection request to the second device through a preset connection method. For example, if the first device and the second device have established a wired communication connection through a serial port, the generated connection request may be sent to the second device through the serial port connected thereto; if the first device and the second device have established a wireless communication connection through a wireless communication module (such as a Bluetooth module, a WIFI module, etc.), a wireless signal carrying a connection request may be generated through the wireless communication module, and the connection request may be sent to the second device through the wireless signal.

[0239] In one possible implementation, the first device is configured with a Bluetooth module, and a first Bluetooth signal carrying the above-mentioned connection request can be generated by the Bluetooth module, and the first Bluetooth signal can be broadcasted and sent by the Bluetooth module to establish a cross-device control connection with a second device in the area where the first device is located.

[0240] In S603, the second device determines whether the second device state satisfies the second connection condition.

[0241] In this embodiment, after receiving the above-mentioned connection request, the second device will obtain the corresponding second device status and determine whether the second device status meets the preset second connection condition. If the second device status meets the above-mentioned second connection condition, the operation of S604 is executed; conversely, if the second device status does not meet the above-mentioned second connection condition, the above-mentioned connection request is not responded. Optionally, the first device can be provided with a waiting timer. If the count value of the waiting timer is greater than the preset count threshold and no connection confirmation instruction is received from the second device, it is recognized that the second device does not establish a cross-device control connection with the first device, and a corresponding prompt message can be generated, such as a "connection failed" prompt message, to prompt the user to adjust the device status of the second device, or reselect other electronic devices that need to be controlled.

[0242] In a possible implementation, when the second device receives the connection request, it may generate corresponding connection confirmation information. For example, Fig.10 FIG. 2 shows a schematic diagram of the operation of connection confirmation provided by an embodiment of the present application. Fig.10 As shown, the second device is a laptop computer. When the laptop computer receives a connection request sent by other devices, it can generate corresponding connection confirmation information, such as "tablet computer applies to control this machine", and is configured with a control 101 for agreeing to control and a control 102 for rejecting control. If the second device detects that the user clicks the above control 101, it executes the operation of S603, that is, obtaining the second device state of the second device and determining whether the second device state meets the second connection condition; on the contrary, if the second device detects that the user clicks the above control 102, or does not receive the above control 101 within the preset countdown, it recognizes that the user does not perform cross-device control, and does not respond to the above connection request.

[0243] In a possible implementation, the second device may store an authorized control device list, which is used to record other electronic devices that have controlled the second device. If the first device is in the authorized control device list, it means that the first device has the authority to control the second device. In this case, when the second device receives the connection request, it can directly perform the operation of S603 without the user confirming the control operation.

[0244] In one possible implementation, if the state of the second device does not meet the second connection condition, the second device may generate a corresponding prompt message to prompt the user to adjust the device state of the second device to meet the connection requirements of cross-device control. The prompt content of the generated prompt message may be determined according to the device state content that does not meet the second connection condition. Exemplarily, if the distance value between the first device and the second device is greater than a preset distance threshold, a prompt message of "the distance is too far, and cross-device control is not possible" may be generated; if the resource occupancy rate of the second device is greater than a preset resource usage upper limit, a prompt message of "insufficient computing resources, and cross-device control is not possible" may be generated.

[0245] The second device status includes one or more combinations of: the usage status of the second device, the distance value between the second device and the first device, the position relationship between the second device and the first device, the resource occupancy rate of the second device, and the device control status of the second device. Exemplarily, the above judgment on whether the second device status satisfies the second connection condition can be divided into the following cases:

[0246] Case 1: The second device state includes a use state, and whether the second device state satisfies the second connection condition is: determining whether the use state of the second device is in use. If the use state of the second device is in use, it is identified that the second device state satisfies the second connection condition, a cross-device control connection can be established, and the operation of S604 is performed.

[0247] Case 2: The state of the second device includes the distance value between the second device and the first device. Whether the state of the second device satisfies the second connection condition is: judging whether the distance value between the second device and the first device is less than the preset distance threshold. If the distance value between the second device and the first device is less than the above distance threshold, it is identified that the state of the second device satisfies the second connection condition, and the operation of S605 is performed. Among them, the numerical value of the above distance threshold can be determined according to the size of the display module of the second device. Since the user needs to control the second device on the first device, when the user operates the first device, it is necessary to clearly see the content displayed on the second device. Therefore, if the size of the display module of the second device is larger, the visible distance is farther, and the corresponding distance threshold can be larger; conversely, if the size of the display module of the second device is smaller, the visible distance is shorter, and the corresponding distance threshold can be smaller, so that the distance threshold can be flexibly adjusted according to the size of the display module of the second device, which can improve the flexibility and accuracy of cross-device control judgment.

[0248] Case 3: If the state of the second device includes the positional relationship with the first device, whether the state of the second device satisfies the second connection condition is to determine whether the positional relationship between the second device and the first device satisfies a preset relative positional relationship. The relative positional relationship may be that the first device is located in the direction facing the front of the second device. For example, Fig.11 A schematic diagram showing a preset relative position relationship between a second device and a first device provided in an embodiment of the present application is shown. Fig.11 As shown in (a), since the user needs to control the second device on the first device, that is, the user needs to be able to view the display content of the second device when using the first device, the content display direction of the display module of the second device is defined as the front face of the device. If the first device is in the front face direction of the second device, it is recognized that the first device and the second device meet the preset relative position relationship; if the first device is located in other directions outside the front face of the second device, it is recognized that the first device and the second device do not meet the preset relative position relationship.

[0249] Optionally, the above relative position relationship may be the visible area range of the display module of the second device, such as Fig.11 As shown in (b), if the first device is located within the visible area corresponding to the second device, it is identified that the first device and the second device satisfy a preset relative position relationship; if the first device is located outside the visible area corresponding to the second device, it is identified that the first device and the second device do not satisfy the preset relative position relationship.

[0250] Case 4: The second device state includes the resource occupancy rate of the second device, and whether the second device state meets the second connection condition is: judging whether the resource occupancy rate is less than a preset resource occupancy threshold. If the resource occupancy rate of the second device is less than the resource occupancy threshold, it is identified that the second device state meets the second connection condition, and the operation of S604 is performed.

[0251] Case 5: The second device state includes the device control state of the second device, and whether the second device state meets the second connection condition is: judging whether the device control state is in a controlled state. If the device control state of the second device is in a controlled state, it means that the second device has been controlled by other devices. In order to avoid operation conflicts, it will be recognized that the second device state does not meet the preset second connection condition; conversely, if the device state of the second device is in an uncontrolled state, it means that the second device is not controlled by other devices, and the operation of S604 can be performed at this time.

[0252] It should be noted that if any second device state in the second device does not meet the corresponding second connection condition, it means that the second device does not have the conditions for realizing cross-device control. At this time, the second device will not perform the operation of S604.

[0253] In an embodiment of the present application, the second device can obtain different types of second device states and match them with corresponding second connection conditions to ensure that the second device is in a suitable state before establishing the above-mentioned cross-device control connection, thereby improving the stability of the connection after the cross-device control connection is established.

[0254] In S604, if the second device detects that the second device state meets the second connection condition, the second device sends a connection confirmation instruction to the first device to establish a cross-device control connection with the first device.

[0255] In this embodiment, when the second device detects that the conditions for establishing a cross-device control connection are met, it responds to the connection request and sends a connection confirmation instruction to the first device. After the first device receives the connection confirmation instruction feedback from the second device, it determines that the second device agrees to cross-device control, and thus a cross-device control connection is established between the first device and the second device.

[0256] In one possible implementation, if the first device sends a first Bluetooth signal carrying a connection request through a first Bluetooth module, after the second device receives the first Bluetooth request through a second Bluetooth module and confirms to establish a cross-device control connection with the first device, it can send a second Bluetooth signal carrying a connection confirmation instruction through the second Bluetooth module, and then establish the above-mentioned cross-device control connection based on the Bluetooth modules of the two devices.

[0257] In S605, the first device displays a first control in a first operation interface when in a cross-device control connection state.

[0258] In this embodiment, after the first device establishes a cross-device connection with the second device, the first device can display a first control in the first operation interface. The first control is used to respond to a cross-device interactive operation initiated by the user on the first device to the second device. It should be noted that if the first device disconnects the cross-device control connection with the second device, the first device will remove the first control displayed in the first operation interface; if the first device re-establishes the above-mentioned cross-device control connection with the second device, the above-mentioned first control will be displayed in the first operation interface again.

[0259] For example, Fig.12 FIG. 1 shows a display schematic diagram of a first control provided by an embodiment of the present application. Fig.12 As shown in (a) of FIG. 1 , when the first device is in a cross-device control connection state, a circular control 121 is generated in the first operation interface displayed by the first device. The circular control 121 is the first control mentioned above. The user can generate a second operation interface in the first operation interface by clicking the circular control mentioned above, such as Fig.12As shown in (b), the second operation interface is a virtual touch panel for controlling the second device, and the touch command for controlling the second device can be generated by the virtual touch panel. Among them, the second operation interface can include controls of multiple functional modes, such as controls 2 to 4. Among them, when the above controls 2 to 4 are in an unselected state, the second operation interface is set to a cursor control mode, and the virtual cursor on the second device can be controlled by the second operation interface to select the display object (such as files, applications, etc.) in the third operation interface of the second device. The user can change the operation mode of the second operation interface by clicking on the above controls 2 to 4.

[0260] Among them, when control 2 is in the selected state, the second operation interface is set to the screenshot mode, and the user can initiate a screenshot operation of the third operation interface displayed in the second device in the second operation interface; when control 3 is in the selected state, the second operation interface is set to the screen recording mode, and the user can start recording the video of the third operation interface displayed in the second device by clicking on control 3 in the above-mentioned second operation interface; when control 4 is in the selected state, the second operation interface is set to the text input mode, and the user can enter text characters in the second operation interface and fill the text characters entered by the user into the associated text input box in the second operation interface. The user can set the operation mode of the second operation interface according to actual usage requirements. The text input method can be various input methods such as physical keyboard input, virtual keyboard input, stylus input, finger input, etc., which are not limited in the embodiments of the present application.

[0261] It should be noted that the interaction module of the first device is a touch screen. In this case, the first device can determine the operation mode corresponding to the second operation interface according to the type of touch object interacting with the touch screen. For example, if the touch object is a fingertip, the first device sets the second operation interface to the cursor control mode; if the touch object is a stylus, the first device sets the second operation interface to the text input mode. Therefore, the user can change the touch mode of the second operation interface at any time by changing the switching object, and the operation mode can be switched without the user clicking the corresponding control, thereby improving the user's operation efficiency.

[0262] In one possible implementation, when the user clicks on the first control to call up the second operation interface, if the first operation interface of the first device is the operation interface of the first application installed on the first device, that is, the foreground of the first device is not in the main interface state, the operation interface of the first application will be displayed through the first window, and the second operation interface will be displayed through the second window.

[0263] For example, Fig.13 FIG. 1 is a schematic diagram showing a multi-window display provided by an embodiment of the present application. Fig.13 As shown in (a) of FIG. 1 , when the first device and the second device have established a cross-device control connection, the first operation interface of the first device may display a first control 131. At this time, the first operation interface is specifically an operation interface of a notepad application. If the first device detects that the user clicks on the first control 131, the second operation interface for initiating a cross-device interactive operation will be displayed through the second window 132, and the original operation interface of the notepad application can be displayed through the first window 133, as shown in FIG. Fig.13 In (b), the second layer where the second window is located is on the first layer where the first window is located, that is, when there is an overlap between the two windows, the second window will cover the first window. The display background of the first device can be a background image or the main interface of the first device, which can be set according to actual conditions.

[0264] In this embodiment, the layers where the first window and the second window are located can be adjusted according to the user's operation.

[0265] The first layer where the first window is located is under the second layer where the second window is located, such as Fig.13 In (b), the user initiates a second click operation on the first window, and the first device adjusts the layer relationship between the first layer and the second layer, and adjusts the first layer to the second layer, so that the first window is displayed in the foreground, such as Fig.13 (c) in .

[0266] The second layer where the second window is located is under the first layer where the first window is located, such as Fig.13 In (c), the user initiates a third click operation on the second window, and the first device adjusts the layer relationship between the first layer and the second layer, and adjusts the second layer to the first layer, so that the second window is displayed in the foreground, that is, from Fig.13 (c) in is adjusted to Fig.13 The display status of (b) in .

[0267] It should be noted that both the first window and the second window can be dragged, and the display positions of the two windows are adjusted according to the dragging operation of the user.

[0268] In an embodiment of the present application, after establishing a cross-device control connection between a first device and a second device, a first control is generated in a first operating interface of the first device, so that the second device can be controlled through the first control or a second operating interface awakened by the first control, thereby achieving the purpose of cross-device control and improving the convenience and efficiency of operations when multiple devices work together.

[0269] Phase 2: User interaction phase

[0270] In this embodiment, after the first device and the second device establish a cross-device control connection, the user can achieve cross-device control through the cross-device interaction area in the first operation interface, that is, the cross-device interaction area achieves the purpose of controlling the second device on the first device. Among them, the cross-device interaction area in the first operation interface includes: the first control displayed in the first operation interface when the cross-device control connection is established, and the second operation interface generated based on the first control. Fig.14 The following is an interactive flow chart of the user interaction phase in the cross-device control method provided by an embodiment of the present application. The specific implementation process is described in detail as follows:

[0271] In S1401, the first device generates an operation instruction corresponding to the interaction operation in response to an interaction operation initiated by a user in a cross-device interaction area; the cross-device interaction area includes a second operation interface and a first control.

[0272] In S1402, the first device sends an operation instruction to the second device.

[0273] In this embodiment, after the first device establishes a cross-device control connection with the second device, a cross-device interaction area may be displayed in the first operation interface of the first device. The cross-device interaction area includes two display states, the first state is a state in which it is displayed in the form of a control, that is, in the display state of the first control, the user can click the first control to activate the second operation interface in the first operation interface, in which the user can share the window displayed in the first operation interface to the third operation interface of the second device through the first control; the second state is a state in which it is displayed in the form of an interface, that is, in the display state of the second operation interface, the second operation interface can be used as a virtual touch panel of the second device, and the user can initiate a touch operation in the second operation interface to control the second device in the form of a touch operation.

[0274] In this embodiment, the user can initiate different types of interactive operations according to the different display states of the cross-device interactive area, including: window sharing operation, cursor control operation, screenshot operation, text input operation, text extraction operation, recording operation and other different types of interactive operations, which are not listed here one by one.

[0275] In this embodiment, the first device can respond to the interactive operation initiated by the user in the cross-device interactive area, and can determine the interaction type corresponding to the interactive operation according to the display state and operation mode of the cross-device interactive area, and generate the operation instruction corresponding to the interactive operation through the instruction conversion logic corresponding to the interaction type. Since the first device and the second device have established a cross-device control connection, the generated operation instruction can be transmitted through the control connection. It should be noted that the above-mentioned operation instruction is an operation instruction acting on the second device, and therefore, the instruction format corresponding to the operation instruction can be determined according to the operating system of the second device so that the second device can recognize the above-mentioned operation instruction.

[0276] In S1403, the second device responds to the interactive operation initiated by the user on the first device based on the operation instruction.

[0277] In this embodiment, after receiving the operation instruction sent by the first device, the second device can respond to the interactive operation initiated by the user based on the operation instruction.

[0278] For example, Fig.15 FIG. 1 shows a response flow chart of an interactive operation provided by an embodiment of the present application. Fig.15 , the user initiates a click operation in the second operation interface of the first device and generates a corresponding click instruction. After receiving the above click instruction, the second device will determine the display object indicated by the current cursor in the second operation interface (including controls, icons, etc. in the interface). If the cursor of the second device is on icon 1 at this time, the second device will perform a selection operation for the application corresponding to icon 151. If the user initiates a double-click operation in the second operation interface of the first device, the second device will perform an open operation for the application corresponding to icon 151. Depending on the different interactive operations initiated by the user, the second device can perform different actions according to the operation instruction.

[0279] It can be seen from the above that a cross-device control method provided by the embodiment of the present application can generate an operation instruction acting on other devices outside the first device, that is, the second device, by displaying a cross-device interaction area that can control other electronic devices in the first device, and by receiving an interactive operation initiated by a user in the cross-device interaction area, and send the operation instruction to the second device; after the second device receives the operation instruction, the operation instruction corresponding to the interactive operation can be executed in the second device to achieve the purpose of cross-device control. Compared with the existing device control technology, when the user operates the first device and the second device at the same time, there is no need to initiate an interactive operation on the interactive modules of different devices, but the user can control the first device in the first operation interface of the first device, and control the second device through the cross-device interaction area in the first operation interface, which improves the operation efficiency in the case of multi-device collaborative operation, reduces the user's operation difficulty, and then improves the user's experience.

[0280] Exemplarily, the interactive operation initiated by the user may include at least the following five types of operations: cursor control operation, screenshot operation, text input operation, recording operation and interface sharing operation. Among them, the cursor control operation, screenshot operation, text input operation and recording operation are implemented through the second operation interface displayed in the first operation interface; the interface sharing operation is implemented through the first control in the first operation interface.

[0281] The specific implementation process of the above operation types is as follows:

[0282] Type 1 cursor control operations:

[0283] Fig.16 FIG. 1 shows an interactive flow chart of a cursor control operation in a cross-device control process provided by an embodiment of the present application. Fig.16 As shown, in response to the cursor control operation, the interaction process between the first device and the second device includes S1601 to S1603, which are specifically described as follows:

[0284] In S1601, the first device generates a sliding instruction corresponding to the first sliding operation in response to a first sliding operation initiated by a user on a second operation interface.

[0285] In S1602, the first device sends a sliding instruction to the second device.

[0286] In this embodiment, the user can initiate a first sliding operation in the second operation interface, and the first sliding operation can be used to adjust the virtual cursor displayed in the second device. The first device can determine the sliding length and sliding direction of the first sliding operation, and then convert it into a corresponding sliding instruction based on the above two parameters, and send the generated sliding instruction to the second device. Among them, the sliding instruction can carry the sliding vector of the virtual cursor in the second device, so as to control the sliding cursor in the second device to move along the above sliding vector.

[0287] As described above, the second operation interface may include a plurality of controls for adjusting the operation mode, such as Fig.12 As shown in the control 2 to control 4, when the user needs to initiate a cursor control operation, the control 2 to control 4 are in a non-selected state. Fig.17 FIG. 1 is a schematic diagram showing a cursor control operation provided by an embodiment of the present application. Fig.17 As shown in (a), the user can initiate a sliding operation in the second operation interface of the first device, namely, track 171. The first device can generate a sliding instruction corresponding to track 171 and send the sliding instruction to the second device.

[0288] In S1603, in response to the sliding instruction, the second device moves the virtual cursor displayed in the second device to the target position indicated by the sliding instruction.

[0289] In this embodiment, after receiving the sliding instruction, the second device can control the virtual cursor to move to the corresponding target position according to the sliding vector carried in the sliding instruction, such as Fig.17 In (b), the virtual cursor is moved from position 172 to position 173 , and the specific moving distance and moving direction are determined according to trajectory 171 .

[0290] In an embodiment of the present application, a user can control a virtual cursor in a second device in a second operating interface of a first device without having to operate through a mouse or a touch panel of the second device, thereby improving the consistency and fluency of user operations and, in turn, improving control efficiency when controlling multiple devices.

[0291] Type 2: Screen capture operation

[0292] Fig.18 FIG. 1 shows an interactive flow chart of a screenshot operation in a cross-device control process provided by an embodiment of the present application. Fig.18 As shown, in response to the screenshot operation, the interaction process between the first device and the second device includes S1801 to S1807, which are specifically described as follows:

[0293] In S1801, the first device determines a first screenshot area corresponding to the second sliding operation in response to a second sliding operation initiated by a user on a second operation interface.

[0294] In this embodiment, in the second control of the second operation interface (such as Fig.12 When the control 2) in the second operation interface is in the selected state, the second operation interface is in the screenshot mode. In the screenshot mode, the first device will recognize the second sliding operation initiated by the user in the second operation interface as a screenshot operation initiated by the user, and thus determine the corresponding first screenshot area according to the second sliding operation of the user.

[0295] For example, Fig.19 FIG. 1 is a schematic diagram showing a screenshot operation provided by an embodiment of the present application. Fig.19 As shown in (a), the user can initiate a sliding operation for taking a screenshot when the control 2 in the second operation interface of the first device is in a selected state, that is, track 191. The first device can generate a corresponding first screenshot area according to the track 191. The aspect ratio of the first screenshot area is determined according to the sliding direction of the track 191, and the area of ​​the first screenshot area is determined according to the sliding length of the track 191. The first screenshot area can be displayed in the second operation interface, for example, in the form of a virtual frame, such as area 192, or it can not be displayed in the second operation interface, which can be determined according to actual conditions.

[0296] In S1802, the first device determines a second screenshot area in a third operation interface of the second device according to the first screenshot area, and generates a screenshot instruction for the second screenshot area.

[0297] In S1803, the second device generates a second screenshot area in the second device in response to the second sliding operation.

[0298] In this embodiment, since the interface size of the second operation interface is different from the interface size of the third operation interface in the second device, the area size of the first screenshot area needs to be proportionally adjusted according to the difference between the two interface sizes to determine the second screenshot area in the third operation interface.

[0299] In a possible implementation, the area parameters of the second screenshot area may be:

[0300]

[0301] Among them, Cutsize2 is the area parameter of the above-mentioned second screenshot area; Cutsize1 is the area parameter of the above-mentioned first screenshot area; MiroSize2 is the area size of the above-mentioned first screenshot area; MiroSize1 is the area size of the above-mentioned second screenshot area.

[0302] It should be noted that, when the user initiates the second sliding operation, the second device will adjust the second screenshot area displayed in the third operation interface in real time according to the second sliding operation, such as Fig.19 The area 193 in (a) can change in real time according to the change of the trajectory 191.

[0303] In this embodiment, after determining the second screenshot area, the first device may generate a screenshot instruction according to the coordinate information of the second screenshot area in the third operation interface.

[0304] In S1804, the first device sends the screenshot instruction to the second device.

[0305] In this embodiment, the first device can send the screenshot instruction to the second device through the cross-device control connection between the first device and the second device, so that the second device can capture the corresponding content from the third operation interface according to the screenshot instruction.

[0306] In S1805, the second device generates a screenshot image based on the image framed by the second screenshot area in the third operation interface.

[0307] In this embodiment, after receiving the screenshot instruction, the second device generates a corresponding second screenshot area in the third operation interface of the second device, and determines the display content in the second screenshot area in the third operation interface, that is, the image framed by the second screenshot area, and generates a corresponding screenshot image based on the above-determined display content, such as generating a corresponding screenshot image based on the display content of area 193, that is, image 194.

[0308] In S1806, the second device sends the screenshot image to the first device.

[0309] In S1807, the first device receives a screenshot image of the second screenshot area fed back by the second device based on the screenshot instruction.

[0310] In this embodiment, after obtaining the screenshot image, the second device can send the screenshot image to the first device through a cross-device control connection, and the first device can display the above screenshot image in a local operation interface (such as the first operation interface or the second operation interface) to operate the screenshot image locally on the first device.

[0311] In a possible implementation, after the first device receives the screenshot image, the second operation interface displayed in the first operation interface can be closed, that is, switched to the display state of the first control, such as Fig.19As shown in (b), after the second device sends the image 194 to the first device, the image 194 can be displayed in the first operation interface of the first device. At this time, the cross-device interaction area in the first device is displayed in the form of a control, such as control 195.

[0312] In a possible implementation, after the first device obtains the screenshot image, the user can drag the image 194 to any position in the first operation interface. Fig. 20 A schematic diagram showing a drag operation on a screenshot image according to an embodiment of the present application is shown. Fig. 20 As shown, if the placement position allows pasting of a custom image, the first device will paste the image 201 at the corresponding position after detecting that the user's dragging operation is completed, such as Fig. 20 In (a), the user drags it to position 202 and pastes the image; if the location where the image is placed does not allow pasting of the custom image, the first device can automatically store the image in the album. Optionally, if the location where the image is placed does not allow pasting of the custom image, the first device can also generate a prompt pop-up box to ask the user whether to save the screenshot image, such as Fig. 20 In (b), the first operation interface is an operation interface of a video playback application, which does not support the pasting operation of custom images. In this case, a corresponding prompt box 203 will be generated to prompt the user to store the image to the album or delete the image.

[0313] In one possible implementation, after the first device receives the screenshot image, it can also initiate a content extraction operation, for example, by long pressing to trigger a mode for extracting character information contained in the screenshot image, and in the mode for extracting character information, by sliding to select, determine the character content to be copied, thereby realizing text extraction of the image.

[0314] In an embodiment of the present application, by initiating a screenshot operation on the second device through the second operation interface of the first device, the content displayed in the second device can be directly obtained on the first device. There is no need for the user to initiate the screenshot in the second device and then send the screenshot content to the first device, which greatly reduces the operations required by the user, improves the operating efficiency, and also improves the efficiency of the first device in obtaining the screen content of the second device.

[0315] Type 3: Text input operation

[0316] Fig.21 FIG. 1 shows an interactive flow chart of a text input operation in a cross-device control process provided by an embodiment of the present application. Fig.21 As shown, in response to a text input operation, the interaction process between the first device and the second device includes S2101 to S2103, which are specifically described as follows:

[0317] In S2101, the first device determines at least one character corresponding to the text input operation in response to a text input operation initiated by the user in the second operation interface, and generates a text input instruction regarding the at least one character.

[0318] In this embodiment, in the fourth control (such as Fig.12 When the control 4) in the second operation interface is in the selected state, the second operation interface is in the text input mode. In the text input mode, the first device generates characters corresponding to the text input operation initiated by the user in the second operation interface. The text input operation includes: a sliding operation of drawing a text character trajectory and an input operation through an input module (such as a keyboard, a soft keyboard, etc.).

[0319] In a possible implementation, the above text input operation is a soft keyboard input operation, the first device can display a corresponding soft keyboard, and the user can achieve the purpose of character input by clicking on the controls in the soft keyboard.

[0320] In a possible implementation, the text input operation may be a sliding operation, and a text sliding track is generated based on the sliding operation initiated by the user, and the text sliding track is matched with each existing character template in the character library to determine the character input by the user. Among them, a text sliding track may correspond to one character or multiple characters, which may be determined according to the track shape input by the user, and is not limited here.

[0321] For example, Fig. 22 A schematic diagram of a text input process provided by an embodiment of the present application is shown. For example, Fig. 22 In (a), the user can draw a text sliding trajectory in the second operation interface of the first device, such as trajectory 221 corresponding to "Nice day". The first device can recognize according to the above text sliding trajectory, determine that the trajectory 221 contains 5 characters, namely the characters "Nice day", and generate a text input instruction based on the above recognized characters.

[0322] In a possible implementation, if the object selected by the virtual cursor in the second device is an inputtable text box, for example, the virtual cursor of the second device clicks on the text box 223, the second operation interface of the first device automatically enters the text input mode, that is, the user does not need to click on the text box 224. Fig.12 The control 4 in the text input mode can be automatically switched from the cursor control mode to the text input mode, and the control 4 is lit. If the user completes the input, the control 4 can be clicked to exit the text input mode and switch to the cursor control mode, thereby realizing fast switching between modes.

[0323] In a possible implementation, after receiving the text sliding track initiated by the user, the first device may determine a plurality of candidate characters and generate corresponding character selection boxes, such as Fig. 22 In the selection box (b) 222, the user can select a suitable character from a plurality of candidate characters as the character to be input this time.

[0324] In S2102, the first device sends a text input instruction to the second device.

[0325] In S2103, the second device inputs at least one character in a text box currently selected in the second device in response to the text input instruction.

[0326] In this embodiment, the first device may send a text input instruction carrying at least one character input by the user to the second device. After receiving the text input instruction, the second device will input the corresponding character into the currently selected text box, such as Fig. 22 As shown in (a) in the figure, in the text box 223, four characters "good day" are entered.

[0327] In an embodiment of the present application, the user can change the operation mode of the second operation interface to the text input mode, generate a corresponding text sliding trajectory in the second operation interface, and input corresponding characters in the second device, thereby eliminating the need for the user to switch input operations between multiple devices, thereby improving the continuity of operations.

[0328] Type 4: Recording Operation

[0329] Fig.23 FIG. 1 shows an interactive flow chart of a recording operation in a cross-device control process provided by an embodiment of the present application. Fig.23 As shown, in response to a text input operation, the interaction process between the first device and the second device includes S2301 to S2304, which are specifically described as follows:

[0330] In S2301, the first device sends a recording start instruction to the second device in response to a first click operation of the user clicking a recording start control on the second operation interface.

[0331] In S2302, the second device records the interface video of the third operation interface of the second device in response to the recording start instruction.

[0332] In this embodiment, in the third control (such as Fig.12 When the control 3) in the recording mode is selected, the second operation interface is in the recording mode. In the recording mode, the second operation interface may include two controls, for example, Fig.24 FIG. 1 is a schematic diagram showing a second operation interface in a recording mode provided by an embodiment of the present application, see Fig.24 As shown in (a) of FIG. 1 , the second operation interface includes two controls, namely, a control 241 for starting recording and a control 242 for pausing recording. If the user clicks the control 241, a recording start instruction will be generated and sent to the second device. At this time, the control 241 will be changed to a control for stopping recording. Fig.24 Control 243 shown in (b) .

[0333] In this embodiment, after receiving the recording start instruction, the second device will start recording the interface image data of the third operation interface and the audio data played by the third device, and generate a corresponding interface video. At this time, if the user needs to pause the recording, he can click the above-mentioned control 242 to pause the recording process. The first device will generate a pause recording instruction and send it to the second device, and the second device will stop recording the content of the third operation interface.

[0334] In S2303, the first device sends a recording completion instruction to the second device in response to the user's second click operation of clicking the recording completion control on the second operation interface.

[0335] In S2304, in response to the recording completion instruction, the second device sends the interface video recorded from the time when the recording start instruction is received to the time when the recording completion instruction is received to the first device.

[0336] In this embodiment, when the user does not need to record the screen changes of the second device, the user can click the recording stop control in the second operation interface of the first device, such as control 243, and the first device will send a recording completion instruction to the second device. In response to the above recording completion instruction, the second device will feed back the recorded interface video to the first device.

[0337] Type 5: Interface sharing operation

[0338] Fig.25 FIG. 1 shows an interactive flow chart of an interface sharing operation in a cross-device control process provided by an embodiment of the present application. Fig.25 As shown, in response to a text input operation, the interaction process between the first device and the second device includes S2501 to S2503, which are specifically described as follows:

[0339] In S2501, when the first device detects that the user drags the first window to the first control, the first device closes the first window and generates an interface sharing instruction for the first operation interface.

[0340] In this embodiment, the cross-device interaction area of ​​the first device is displayed in the form of a first control. In addition to being able to invoke the above-mentioned second operation interface by clicking, the first control can also receive an interface sharing operation initiated by the user dragging any window to its area. For example, Fig.26 A schematic diagram of an interface sharing operation provided by an embodiment of the present application is shown. Fig.26 As shown in (a) of FIG. 1 , when the first device detects that the user drags any window (such as the first window for displaying the first operation interface) to the first control, the first device closes the first window. Fig.26 In (b), after the first device closes the above-mentioned first window, an interface sharing instruction of the first operation interface displayed in the first window will be generated.

[0341] In S2502, the first device sends an interface sharing instruction to the second device.

[0342] In S2503, the second device displays the first operation interface through a third window in the third operation interface in response to the interface sharing instruction.

[0343] In this embodiment, the interface sharing instruction may carry interface data of the first operation interface. After receiving the interface data, the second device may regenerate the first operation interface locally on the second device through a third window. The user may view the content of the first operation interface in the third window of the second device. Fig.26 As shown in (b) in .

[0344] It should be noted that the application corresponding to the first operation interface can be running on the first device, that is, the first device will feedback the interface data of the application running in the first device to the second device through the cross-device control connection, so that the second device can update the first operation interface in the third device according to the feedback interface data.

[0345] The above embodiments illustrate the cross-device control method provided by the embodiments of the present application from the perspective of the two-end interaction between the first device and the second device, while this embodiment illustrates the cross-device control method provided by the embodiments of the present application from the perspective of the first device. For example, Fig. 27 The following is a flowchart of the implementation of the cross-device control method provided by an embodiment of the present application on the first device side, which is specifically described as follows:

[0346] In S2701, a cross-device interaction area is displayed in a first operation interface of a first device; the cross-device interaction area includes a second operation interface and / or a first control; the second operation interface is a virtual touch panel of the second device;

[0347] In S2702, in response to an interaction operation initiated by a user in a cross-device interaction area, an operation instruction corresponding to the interaction operation is generated;

[0348] In S2703, the operation instruction is sent to the second device to control the second device to execute the operation instruction corresponding to the interactive operation.

[0349] Optionally, a cross-device interaction area is displayed in a first operation interface of a first device, including: in response to a first click operation of a user on a first control in the first operation interface, a second operation interface is displayed in the first operation interface; the first control is displayed when the first device and the second device are in a state where a cross-device control connection has been established.

[0350] Optionally, in response to a user's first click operation on a first control in a first operation interface, before displaying the second operation interface in the first operation interface, it also includes: if the first device state of the first device meets a preset first connection condition, sending a connection request to the second device; receiving a connection confirmation instruction from the second device based on feedback of the connection request, establishing a cross-device control connection with the second device, and displaying the first control in the first operation interface.

[0351] Optionally, the first device state includes: screen state, motion state, and device placement posture; if the first device state of the first device meets a preset first connection condition, sending a connection request to the second device includes:

[0352] If at least one of the following conditions is met: the screen state is in a bright screen state, the motion state is in a stationary state, and the device placement posture is within a preset angle range, a connection request is sent to the second device.

[0353] Optionally, if the first device state of the first device meets a preset first connection condition, sending a connection request to the second device includes: generating a first Bluetooth signal carrying the connection request through a first Bluetooth module in the first device; the first Bluetooth signal is used for the second device to receive the first Bluetooth signal through the second Bluetooth module, and determining the distance value between the second device and the first device through the first Bluetooth signal;

[0354] Receiving a connection confirmation instruction fed back by the second device based on a connection request includes: receiving a second Bluetooth signal fed back by the second device; the second Bluetooth signal carries the connection confirmation instruction; the second Bluetooth signal is generated when the second device detects that the distance value is less than a preset distance threshold.

[0355] Optionally, the interactive operation includes an interface sharing operation; the first operation interface is displayed in the first window; in response to the interactive operation initiated by the user in the cross-device interactive area, an operation instruction corresponding to the interactive operation is generated, including: if it is detected that the user initiates a drag operation of dragging the first window to the first control, the first window is closed, and an interface sharing instruction for the first operation interface is generated;

[0356] Sending the operation instruction to the second device to control the second device to execute the operation instruction corresponding to the interactive operation includes: sending the interface sharing instruction to the second device to display the first operation interface in a third window on the second device.

[0357] Optionally, a cross-device interaction area is displayed in a first operation interface of a first device, including: if the first operation interface is an operation interface of a first application installed on the first device, the first operation interface is displayed through a first window, and the second operation interface is displayed through a second window; the first layer where the first window is located is under the second layer where the second window is located.

[0358] Optionally, if the first operation interface is the operation interface of the first application installed on the first device, after displaying the first operation interface through the first window and displaying the second operation interface through the second window, it also includes: in response to the user's second click operation on the first window, adjusting the first layer to the second layer; and / or in response to the user's third click operation on the second window, adjusting the second layer to the first layer.

[0359] Optionally, the interactive operation includes a first sliding operation; in response to the interactive operation initiated by the user in the cross-device interactive area, generating an operation instruction corresponding to the interactive operation includes: in response to the first sliding operation initiated by the user in the second operation interface, generating a sliding instruction corresponding to the first sliding operation;

[0360] Sending the operation instruction to the second device to control the second device to execute the operation instruction corresponding to the interactive operation includes: sending the sliding instruction to the second device to control the virtual cursor in the second device to move based on the sliding instruction.

[0361] Optionally, the interactive operation includes a text input operation;

[0362] Before generating an operation instruction corresponding to the interaction operation in response to the interaction operation initiated by the user in the cross-device interaction area, the method further includes: in response to a fourth click operation initiated by the user on the second control in the second operation interface, setting the second operation interface to a text input mode;

[0363] In response to an interactive operation initiated by a user in a cross-device interactive area, generating an operation instruction corresponding to the interactive operation, including: in response to a text input operation initiated by the user in a second operation interface, determining at least one character corresponding to the text input operation, and generating a text input instruction for the at least one character;

[0364] Sending an operation instruction to the second device to control the second device to execute the operation instruction corresponding to the interactive operation includes: sending a text input instruction to the second device to input at least one character in a text box corresponding to the second device.

[0365] Optionally, the interactive operation includes a screenshot operation; before generating an operation instruction corresponding to the interactive operation in response to the interactive operation initiated by the user in the cross-device interactive area, the method further includes: in response to a fifth click operation initiated by the user on a third control in the second operation interface, setting the second operation interface to a screenshot mode;

[0366] In response to an interactive operation initiated by a user in a cross-device interactive area, generating an operation instruction corresponding to the interactive operation, including: in response to a second sliding operation initiated by the user in a second operating interface, determining a first screenshot area corresponding to the second sliding operation; determining a second screenshot area in a third operating interface of the second device according to the first screenshot area, and generating a screenshot instruction for the second screenshot area;

[0367] Sending an operation instruction to the second device to control the second device to execute the operation instruction corresponding to the interactive operation includes: sending a screenshot instruction to the second device; and receiving a screenshot image of the second screenshot area fed back by the second device based on the screenshot instruction.

[0368] In the embodiment of the present application, a cross-device interaction area that can control other electronic devices is displayed in the first device, and an interaction operation initiated by the user in the cross-device interaction area is received to generate an operation instruction that acts on other devices outside the first device, that is, the second device, and send the operation instruction to the second device; after the second device receives the operation instruction, the operation instruction corresponding to the interaction operation can be executed in the second device to achieve the purpose of cross-device control. Compared with the existing device control technology, when the user operates the first device and the second device at the same time, there is no need to initiate an interaction operation on the interaction modules of different devices, but the user can control the first device in the first operation interface of the first device, and control the second device through the cross-device interaction area in the first operation interface, which improves the operation efficiency in the case of multi-device collaborative operation, reduces the user's operation difficulty, and then improves the user's experience.

[0369] Corresponding to the cross-device control method described in the above embodiment, Fig.28 A structural block diagram of a cross-device control apparatus provided in an embodiment of the present application is shown. For ease of explanation, only the parts related to the embodiment of the present application are shown.

[0370] Reference Fig.28 , the cross-device control device includes:

[0371] An interactive area display unit 281, configured to display a cross-device interactive area in a first operating interface of the first device; the cross-device interactive area includes a second operating interface and / or a first control; the second operating interface is a virtual touch panel of the second device;

[0372] An operation instruction generating unit 282, configured to generate an operation instruction corresponding to an interaction operation initiated by a user in the cross-device interaction area;

[0373] The operation instruction sending unit 283 is used to send the operation instruction to the second device to control the second device to execute the operation instruction corresponding to the interactive operation.

[0374] Optionally, the interactive area display unit:

[0375] The first click operation unit is used to display the second operation interface in the first operation interface in response to the user's first click operation on the first control in the first operation interface; the first control is displayed when the first device and the second device are in a state where a cross-device control connection has been established.

[0376] Optionally, the control device further includes:

[0377] a connection request sending unit, configured to send a connection request to the second device if the first device state of the first device satisfies a preset first connection condition;

[0378] A confirmation instruction receiving unit is used to receive a connection confirmation instruction fed back by the second device based on the connection request, establish the cross-device control connection with the second device, and display the first control in the first operation interface.

[0379] Optionally, the first device state includes: screen state, motion state and device placement posture; the connection request sending unit includes:

[0380] The first device status confirmation unit is used to send a connection request to the second device if at least one of the following conditions is met: the screen status is in a bright screen status, the motion status is a static status, and the device placement posture is within a preset angle range.

[0381] Optionally, the sending of the connection request includes:

[0382] a first Bluetooth signal sending unit, configured to generate a first Bluetooth signal carrying the connection request through a first Bluetooth module in the first device; the first Bluetooth signal is used for the second device to receive the first Bluetooth signal through a second Bluetooth module, and to determine a distance value between the second device and the first device through the first Bluetooth signal;

[0383] The confirmation instruction receiving unit comprises:

[0384] The second Bluetooth signal receiving unit is used to receive a second Bluetooth signal fed back by the second device; the second Bluetooth signal carries the connection confirmation instruction; the second Bluetooth signal is generated when the second device detects that the distance value is less than a preset distance threshold.

[0385] Optionally, the interactive operation includes an interface sharing operation; the first operation interface is displayed in a first window; and the operation instruction generating unit includes:

[0386] a drag operation unit, configured to close the first window and generate an interface sharing instruction for the first operation interface if it is detected that the user initiates a drag operation of dragging the first window to the first control;

[0387] The sending of the operation instruction comprises:

[0388] The interface sharing instruction sending unit is used to send the interface sharing instruction to the second device to display the first operation interface in a third window on the second device.

[0389] Optionally, the interactive area display unit includes:

[0390] A window layer changing unit is used to display the first operation interface through a first window and display the second operation interface through a second window if the first operation interface is the operation interface of a first application installed on the first device; the first layer where the first window is located is below the second layer where the second window is located.

[0391] Optionally, the control device further includes:

[0392] A first click operation unit is configured to adjust the first layer to the second layer in response to a second click operation of the user on the first window; and / or

[0393] The second click operation unit is used to adjust the second layer to the first layer in response to a third click operation of the user on the second window.

[0394] Optionally, the interactive operation includes a first sliding operation; and the operation instruction generation includes: a sliding instruction generation unit, configured to generate a sliding instruction corresponding to the first sliding operation in response to the first sliding operation initiated by the user on the second operation interface;

[0395] The sending of the operation instruction includes: a sliding instruction sending unit, which is used to send the sliding instruction to the second device to control the virtual cursor in the second device to move based on the sliding instruction.

[0396] Optionally, the interactive operation includes a text input operation; the control device further includes: a text mode switching unit, configured to set the second operation interface to a text input mode in response to a fourth click operation initiated by the user on the second control in the second operation interface;

[0397] The operation instruction generating unit includes: a text input instruction generating unit, which is used to respond to a text input operation initiated by a user on the second operation interface, determine at least one character corresponding to the text input operation, and generate a text input instruction for the at least one character;

[0398] The operation instruction sending unit includes: a text input instruction sending unit, which is used to send the text input instruction to the second device, so as to input the at least one character in a text box corresponding to the second device.

[0399] Optionally, the interactive operation includes a screenshot operation; the control device further includes: a screenshot mode setting unit, configured to set the second operation interface to a screenshot mode in response to a fifth click operation initiated by the user on the third control in the second operation interface;

[0400] The operation instruction generation includes: a first screenshot area determination unit, configured to determine, in response to a second sliding operation initiated by a user on the second operation interface, a first screenshot area corresponding to the second sliding operation; a screenshot instruction generation unit, configured to determine, based on the first screenshot area, a second screenshot area in a third operation interface of the second device, and generate a screenshot instruction for the second screenshot area;

[0401] The operation instruction sending includes: a screenshot instruction sending unit, used to send the screenshot instruction to the second device; and a screenshot image receiving unit, used to receive a screenshot image about the second screenshot area fed back by the second device based on the screenshot instruction.

[0402] Some of the aforementioned embodiments illustrate the cross-device control method provided by the embodiments of the present application from the perspective of two-end interaction, while this embodiment illustrates the cross-device control method provided by the embodiments of the present application from the perspective of the second device. For example, Fig.29 The following is a flowchart of a cross-device control method provided by an embodiment of the present application and implemented by a server, which is specifically described as follows:

[0403] In S2901, an operation instruction sent by a first device is received; the operation instruction is generated based on an interactive operation initiated by a user in a cross-device interactive area displayed in a first operation interface of the first device; the cross-device interactive area includes a second operation interface and / or a first control;

[0404] In S2902, respond to the interactive operation based on the operation instruction.

[0405] Optionally, before receiving the operation instruction sent by the first device, it also includes: receiving a connection request sent by the first device; the connection request is sent when the first device state of the first device satisfies a preset first connection condition; if the second device state of the second device satisfies a preset second connection condition, a connection confirmation instruction is sent to the first device to establish a cross-device control connection with the first device; when the first device is in the state of the cross-device control connection, the first device will display the first control in the first operation interface; the first control is used to display the second operation interface in the first operation interface when the user initiates a first click operation.

[0406] Optionally, if the second device state of the second device meets the preset second connection condition, a connection confirmation instruction is sent to the first device to establish a cross-device control connection with the first device, including: receiving a first Bluetooth signal sent by a first Bluetooth module of the first device through a second Bluetooth module; the first Bluetooth signal carries the connection request; in response to the connection request, determining the position relationship between the second device and the first device; if the position relationship meets the preset relative position condition, feeding back a second Bluetooth signal to the first device through the second Bluetooth module; the second Bluetooth signal carries the connection confirmation instruction.

[0407] Optionally, the operation instruction includes an interface sharing instruction; the receiving the operation instruction sent by the first device includes: receiving the interface sharing instruction sent by the first device; the interface classification instruction is generated by the first device in response to a user initiating a drag operation of dragging the first window to the first control; the first window is a window displayed on the first operation interface;

[0408] The responding to the interactive operation based on the operation instruction includes: displaying the first operation interface through a third window in response to the interface sharing instruction.

[0409] Optionally, the operation instruction includes a sliding instruction; the receiving the operation instruction sent by the first device includes: receiving the sliding instruction sent by the first device; the sliding instruction is generated by the first device in response to a first sliding operation initiated by the user on the second operation interface;

[0410] The responding to the interactive operation based on the operation instruction includes: in response to the sliding instruction, moving a virtual cursor displayed in the second device to a target position indicated by the sliding instruction.

[0411] Optionally, the operation instruction includes a text input instruction; the receiving the operation instruction sent by the first device includes: receiving the text input instruction sent by the first device; the text input instruction is generated by the first device in response to a text input operation initiated by the user on the second operation interface; the text input instruction carries at least one character input by the user; the second operation interface is in a text input mode;

[0412] The responding to the interactive operation based on the operation instruction includes: responding to the text input instruction, inputting the at least one character in a currently selected text box in the second device.

[0413] Optionally, the operation instruction includes a screenshot instruction; the receiving the operation instruction sent by the first device includes: receiving the screenshot instruction sent by the first device; the screenshot instruction is generated by the first device in response to a second sliding operation initiated by the user on the second operation interface; the second operation interface is in a screenshot mode;

[0414] The responding to the interactive operation based on the operation instruction includes: generating a second screenshot area in the second device in response to the second sliding operation; generating a screenshot image based on the image framed by the second screenshot area in a third operation interface in response to the screenshot instruction; the third operation interface is an operation interface displayed locally on the second device; and sending the screenshot image to the first device.

[0415] Corresponding to the cross-device control method described in the above embodiment, Fig.30 A structural block diagram of a cross-device control apparatus provided in an embodiment of the present application is shown. For ease of explanation, only the parts related to the embodiment of the present application are shown.

[0416] Reference Fig.30 , the cross-device control device includes:

[0417] An operation instruction receiving unit 301 is used to receive an operation instruction sent by a first device; the operation instruction is generated based on an interactive operation initiated by a user in a cross-device interactive area displayed in a first operation interface of the first device; the cross-device interactive area includes a second operation interface and / or a first control;

[0418] The operation instruction response unit 302 is used to respond to the interactive operation based on the operation instruction.

[0419] Optionally, the control device further includes:

[0420] a connection request receiving unit, configured to receive a connection request sent by the first device; the connection request is sent when a first device state of the first device satisfies a preset first connection condition;

[0421] A confirmation instruction sending unit is used to send a connection confirmation instruction to the first device to establish a cross-device control connection with the first device if the second device state of the second device meets the preset second connection condition; when the first device is in the state of the cross-device control connection, the first device will display the first control in the first operation interface; the first control is used to display the second operation interface in the first operation interface when the user initiates a first click operation.

[0422] Optionally, the confirmation instruction sending unit includes: a first Bluetooth signal receiving unit, configured to receive a first Bluetooth signal sent by a first Bluetooth module of the first device through a second Bluetooth module; the first Bluetooth signal carries the connection request;

[0423] a distance value determining unit, configured to determine a positional relationship between the second device and the first device in response to the connection request;

[0424] The second Bluetooth signal sending unit is used to feed back a second Bluetooth signal to the first device through the second Bluetooth module if the position relationship meets the preset relative position condition; the second Bluetooth signal carries the connection confirmation instruction.

[0425] Optionally, the operation instruction includes an interface sharing instruction; the operation instruction receiving unit includes: a sharing instruction receiving unit, configured to receive the interface sharing instruction sent by the first device; the interface classification instruction is generated by the first device in response to a user initiating a drag operation of dragging the first window to the first control; the first window is a window displayed on the first operation interface;

[0426] The operation instruction response unit includes: a sharing instruction response unit, which is used to respond to the interface sharing instruction and display the first operation interface through a third window.

[0427] Optionally, the operation instruction includes a sliding instruction; the operation instruction response unit includes: a sliding instruction receiving unit, configured to receive the sliding instruction sent by the first device; the sliding instruction is generated by the first device in response to a first sliding operation initiated by a user on the second operation interface;

[0428] The operation instruction response unit includes: a sliding instruction response unit, which is used to move a virtual cursor displayed in the second device to a target position indicated by the sliding instruction in response to the sliding instruction.

[0429] Optionally, the operation instruction includes a text input instruction; the operation instruction receiving unit includes: a text input instruction receiving unit, configured to receive the text input instruction sent by the first device; the text input instruction is generated by the first device in response to a text input operation initiated by a user on the second operation interface; the text input instruction carries at least one character input by the user; the second operation interface is in a text input mode;

[0430] The operation instruction receiving unit includes: an input instruction responding unit, which is used to input the at least one character in the currently selected text box in the second device in response to the text input instruction.

[0431] Optionally, the operation instruction includes a screenshot instruction; the receiving the operation instruction sent by the first device includes: a screenshot instruction receiving unit, configured to receive the screenshot instruction sent by the first device; the screenshot instruction is generated by the first device in response to a second sliding operation initiated by a user on the second operation interface; the second operation interface is in a screenshot mode;

[0432] The operation instruction response unit includes: a second screenshot area generation unit, used to generate a second screenshot area in the second device in response to the second sliding operation; a screenshot image generation unit, used to generate a screenshot image based on the image framed by the second screenshot area in a third operation interface in response to the screenshot instruction; the third operation interface is an operation interface displayed locally on the second device; and a screenshot image sending unit, used to send the screenshot image to the first device.

[0433] Fig.31 This is a schematic diagram of the structure of an electronic device provided by an embodiment of the present application. Fig.31 As shown, the electronic device 31 of this embodiment includes: at least one processor 310 ( Fig.31 Only one processor is shown in the figure), a memory 311, and a computer program 312 stored in the memory 311 and executable on the at least one processor 310, wherein the processor 310 implements the steps in any of the above-mentioned cross-device control method embodiments when executing the computer program 312.

[0434] The electronic device 31 may be a computing device such as a desktop computer, a notebook, a PDA, or a cloud server. The electronic device may include, but is not limited to, a processor 310 and a memory 311. Those skilled in the art will appreciate that Fig.31It is only an example of the electronic device 31 and does not constitute a limitation on the electronic device 31. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, it may also include input and output devices, network access devices, etc.

[0435] The processor 310 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0436] In some embodiments, the memory 311 may be an internal storage unit of the electronic device 31, such as a hard disk or memory of the electronic device 31. In other embodiments, the memory 311 may also be an external storage device of the electronic device 31, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the electronic device 31. Further, the memory 311 may also include both an internal storage unit of the electronic device 31 and an external storage device. The memory 311 is used to store an operating system, an application program, a boot loader (BootLoader), data, and other programs, such as the program code of the computer program, etc. The memory 311 may also be used to temporarily store data that has been output or is to be output.

[0437] It should be noted that the information interaction, execution process, etc. between the above-mentioned devices / units are based on the same concept as the method embodiment of the present application. Their specific functions and technical effects can be found in the method embodiment part and will not be repeated here.

[0438] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0439] An embodiment of the present application also provides an electronic device, which includes: at least one processor, a memory, and a computer program stored in the memory and executable on the at least one processor, wherein the processor implements the steps of any of the above-mentioned method embodiments when executing the computer program.

[0440] An embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in the above-mentioned method embodiments can be implemented.

[0441] An embodiment of the present application provides a computer program product. When the computer program product runs on a mobile terminal, the mobile terminal can implement the steps in the above-mentioned method embodiments when executing the computer program product.

[0442] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, which can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a computer-readable storage medium. When the computer program is executed by the processor, the steps of the above-mentioned various method embodiments can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may at least include: any entity or device that can carry the computer program code to the camera / electronic device, recording medium, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, RandomAccess Memory), electric carrier signal, telecommunication signal and software distribution medium. For example, a USB flash drive, a mobile hard disk, a magnetic disk or an optical disk. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electric carrier signals and telecommunication signals.

[0443] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0444] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0445] In the embodiments provided in the present application, it should be understood that the disclosed devices / network equipment and methods can be implemented in other ways. For example, the device / network equipment embodiments described above are merely schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0446] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0447] The embodiments described above are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A cross-device control method, applied to a first device, characterized in that: include: Displaying a cross-device interaction area in a first operation interface of the first device; the cross-device interaction area includes a second operation interface and / or a first control; The second operation interface is a virtual touch panel of the second device; In response to an interaction operation initiated by a user in the cross-device interaction area, generating an operation instruction corresponding to the interaction operation; The operation instruction is sent to the second device to control the second device to execute the operation instruction corresponding to the interactive operation.

2. The control method according to claim 1, characterized in that: The displaying the cross-device interaction area in the first operation interface of the first device includes: In response to a first click operation of a user on a first control in the first operation interface, the second operation interface is displayed in the first operation interface; the first control is displayed when the first device and the second device are in a state where a cross-device control connection has been established.

3. The control method according to claim 2, characterized in that: Before displaying the second operation interface in the first operation interface in response to the user's first click operation on the first control in the first operation interface, the method further includes: If the first device state of the first device meets a preset first connection condition, sending a connection request to the second device; Receive a connection confirmation instruction fed back by the second device based on the connection request, establish the cross-device control connection with the second device, and display the first control in the first operation interface.

4. The control method according to claim 3, characterized in that: The first device state includes: screen state, motion state and device placement posture; If the first device state of the first device satisfies a preset first connection condition, sending a connection request to the second device includes: If one or more of the following conditions is met: the screen state is in a bright screen state, the motion state is a stationary state, and the device placement posture is within a preset angle range, a connection request is sent to the second device.

5. The control method according to claim 3, characterized in that: If the first device state of the first device satisfies a preset first connection condition, sending a connection request to the second device includes: generating a first Bluetooth signal carrying the connection request through a first Bluetooth module in the first device; the first Bluetooth signal is used for the second device to receive the first Bluetooth signal through a second Bluetooth module, and to determine the distance value between the second device and the first device through the first Bluetooth signal; The receiving a connection confirmation instruction fed back by the second device based on the connection request includes: A second Bluetooth signal fed back by the second device is received; the second Bluetooth signal carries the connection confirmation instruction; the second Bluetooth signal is generated when the second device detects that the distance value is less than a preset distance threshold.

6. The control method according to any one of claims 1 to 5, characterized in that: The interactive operation includes an interface sharing operation; the first operation interface is displayed in a first window; The step of generating, in response to an interaction operation initiated by a user in the cross-device interaction area, an operation instruction corresponding to the interaction operation includes: If it is detected that the user initiates a drag operation of dragging the first window to the first control or the second operation interface, an interface sharing instruction for the first operation interface is generated; The sending the operation instruction to the second device to control the second device to execute the operation instruction corresponding to the interactive operation includes: The interface sharing instruction is sent to the second device to display the first operation interface on the second device.

7. The control method according to any one of claims 1 to 6, characterized in that: The displaying the cross-device interaction area in the first operation interface of the first device includes: If the first operation interface is the operation interface of the first application installed on the first device, the first operation interface is displayed through the first window, and the second operation interface is displayed through the second window; the first layer where the first window is located is under the second layer where the second window is located.

8. The control method according to claim 7, characterized in that: Also includes: In response to a second click operation on the first window by the user, adjusting the first layer to the second layer; and / or In response to a third click operation of the user on the second window, the second layer is adjusted to be on the first layer.

9. The control method according to any one of claims 1 to 8, characterized in that: The interactive operation includes a first sliding operation; The step of generating, in response to an interaction operation initiated by a user in the cross-device interaction area, an operation instruction corresponding to the interaction operation includes: In response to a first sliding operation initiated by a user on the second operation interface, generating a sliding instruction corresponding to the first sliding operation; The sending the operation instruction to the second device to control the second device to execute the operation instruction corresponding to the interactive operation includes: The sliding instruction is sent to the second device to control a virtual cursor in the second device to move based on the sliding instruction.

10. The control method according to any one of claims 1 to 9, characterized in that: The interactive operation includes a text input operation; The step of generating, in response to an interaction operation initiated by a user in the cross-device interaction area, an operation instruction corresponding to the interaction operation includes: In response to a text input operation initiated by a user on the second operation interface, determining at least one character corresponding to the text input operation, and generating a text input instruction for the at least one character; The sending the operation instruction to the second device to control the second device to execute the operation instruction corresponding to the interactive operation includes: The text input instruction is sent to the second device to input the at least one character in a text box corresponding to the second device.

11. The control method according to any one of claims 1 to 10, characterized in that: The interactive operation includes a screenshot operation; The step of generating, in response to an interaction operation initiated by a user in the cross-device interaction area, an operation instruction corresponding to the interaction operation includes: In response to a second sliding operation initiated by the user on the second operation interface, determining a first screenshot area corresponding to the second sliding operation; Determine a second screenshot area in a third operation interface of the second device according to the first screenshot area, and generate a screenshot instruction for the second screenshot area; The sending the operation instruction to the second device to control the second device to execute the operation instruction corresponding to the interactive operation includes: Sending the screenshot instruction to the second device; Receive a screenshot image of the second screenshot area fed back by the second device based on the screenshot instruction.

12. A cross-device control method, applied to a second device, characterized in that: include: receiving an operation instruction sent by the first device; The operation instruction is generated based on an interactive operation initiated by a user in a cross-device interactive area displayed in a first operation interface of a first device; the cross-device interactive area includes a second operation interface and / or a first control; The interactive operation is responded to based on the operation instruction.

13. The control method according to claim 12, characterized in that: Before receiving the operation instruction sent by the first device, the method further includes: receiving a connection request sent by the first device; the connection request is sent when a first device state of the first device satisfies a preset first connection condition; If the second device state of the second device meets the preset second connection condition, a connection confirmation instruction is sent to the first device to establish a cross-device control connection with the first device; when the first device is in the cross-device control connection state, the first device will display the first control in the first operation interface; the first control is used to display the second operation interface in the first operation interface when the user initiates a first click operation.

14. The control method according to claim 13, characterized in that: If the second device state of the second device meets the preset second connection condition, sending a connection confirmation instruction to the first device to establish a cross-device control connection with the first device includes: receiving, by the second Bluetooth module, a first Bluetooth signal sent by the first Bluetooth module of the first device; the first Bluetooth signal carrying the connection request; In response to the connection request, determining a positional relationship between the second device and the first device; If the position relationship meets the preset relative position condition, a second Bluetooth signal is fed back to the first device through the second Bluetooth module; the second Bluetooth signal carries the connection confirmation instruction.

15. The control method according to any one of claims 12 to 14, characterized in that: The operation instruction includes an interface sharing instruction; and the receiving the operation instruction sent by the first device includes: receiving the interface sharing instruction sent by the first device; the interface classification instruction is generated by the first device in response to a user initiating a drag operation of dragging the first window to the first control; the first window is a window displayed by the first operation interface; The responding to the interactive operation based on the operation instruction includes: In response to the interface sharing instruction, the first operation interface is displayed through a third window.

16. The control method according to any one of claims 12 to 15, characterized in that: The operation instruction includes a sliding instruction; and the receiving the operation instruction sent by the first device includes: receiving the sliding instruction sent by the first device; the sliding instruction is generated by the first device in response to a first sliding operation initiated by a user on the second operation interface; The responding to the interactive operation based on the operation instruction includes: In response to the sliding instruction, a virtual cursor displayed in the second device is moved to a target position indicated by the sliding instruction.

17. The control method according to any one of claims 12 to 16, characterized in that: The operation instruction includes a text input instruction; and the receiving the operation instruction sent by the first device includes: receiving the text input instruction sent by the first device; the text input instruction is generated by the first device in response to a text input operation initiated by a user on the second operation interface; the text input instruction carries at least one character input by the user; the second operation interface is in a text input mode; The responding to the interactive operation based on the operation instruction includes: In response to the text input instruction, the at least one character is input into a currently selected text box in the second device.

18. The control method according to any one of claims 12 to 17, characterized in that: The operation instruction includes a screenshot instruction; and the receiving the operation instruction sent by the first device includes: Receiving the screenshot instruction sent by the first device; the screenshot instruction is generated by the first device in response to a second sliding operation initiated by the user on the second operation interface; the second operation interface is in a screenshot mode; The responding to the interactive operation based on the operation instruction includes: In response to the second sliding operation, generating a second screenshot area in the second device; In response to the screenshot instruction, generating a screenshot image based on the image framed in the second screenshot area in a third operation interface; the third operation interface is an operation interface locally displayed by the second device; The screenshot image is sent to the first device.

19. An electronic device, characterized in that: The electronic device comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the steps of the method according to any one of claims 1 to 18.

20. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 18 are implemented.

Images