Information processing method and electronic device

Abstract

The present application provides a control method based on a touch screen, which is applied to a first electronic device, wherein the first electronic device includes a touch screen; the touch screen can be used to display an interface; the method includes: when the first electronic device establishes a projection connection with the second electronic device, a cursor is generated, and the cursor is displayed in the interface of the touch screen; projection content is generated based on the interface content in the touch screen; the projection content includes a cursor; the projection content is sent to the second electronic device; when receiving a target touch operation of the user on the touch screen, the target touch operation is responded to based on the display position of the cursor in the current interface; wherein the display position of the cursor in the current interface is used to indicate the response position for the target touch operation in the current interface; the response position is not related to the operation position of the target touch operation on the touch screen. Through the present invention, the first electronic device that is the source of the projection content can be transformed into a control device without adding additional equipment, so that the user can clearly perceive and control the touch screen of the first electronic device by watching the projection content of the second electronic device, and then operate the interface of the content source.

Description

Technical Field

[0001] The present application relates to the field of electronic technology, and in particular to an information processing method and an electronic device. Background Art

[0002] With the popularity of augmented reality (AR), virtual reality (VR) and mixed reality (MR), AR devices, VR devices and MR devices have been widely used in many scenarios such as work and entertainment. Different from traditional terminal devices, current AR devices, VR devices and MR devices use independent handles as the main interaction solution.

[0003] However, the handles of AR devices, VR devices and MR devices are large and heavy, the control systems are complex, and long-term use can easily cause fatigue.

[0004] In some demonstration scenarios, users need additional control devices such as laser pens to demonstrate, which also makes the control system highly dependent on third-party hardware. Summary of the invention

[0005] In a first aspect, an embodiment of the present application provides an information processing method, including:

[0006] The first electronic device establishes a connection with the second electronic device, wherein the second electronic device displays a cursor and interface content of the first electronic device, wherein the first electronic device includes a touch screen, and the second electronic device is an augmented reality (AR) device, a virtual reality (VR) device, or a mixed reality (MR) device;

[0007] Acquire a first operation on the first electronic device;

[0008] Based on a first operation on the first electronic device, a display position of the cursor on the second electronic device is controlled.

[0009] Optionally, in a design of the first aspect, the interface content is interface content of a front-end application of the first electronic device.

[0010] Optionally, in a design of the first aspect, the second electronic device further displays: a menu bar, and the menu bar does not belong to the interface content of the first electronic device.

[0011] Optionally, in a design of the first aspect, controlling a display position of the cursor on the second electronic device based on an operation on the first electronic device includes:

[0012] Acquiring position change information, i.e., posture change, of the first electronic device;

[0013] The cursor is moved on the display content of the second electronic device based on the posture change information.

[0014] Optionally, in a design of the first aspect, the posture change information includes a first rotation angle of the first electronic device on a horizontal plane and a second rotation angle of the first electronic device on a vertical plane, and moving the cursor on display content of the second electronic device based on the posture change information includes:

[0015] determining a horizontal displacement of the cursor according to the first rotation angle;

[0016] determining a vertical displacement of the cursor according to the second rotation angle;

[0017] The cursor is moved on the display content of the second electronic device according to the horizontal displacement and the vertical displacement.

[0018] Optionally, in a design of the first aspect, the posture change information includes a first displacement of the first electronic device in a first direction and a second displacement of the first electronic device in a second direction, the first direction is parallel to a left-right movement direction of a cursor displayed by the second electronic device, and the second direction is parallel to an up-and-down movement direction of a cursor displayed by the second electronic device, and moving the cursor on display content of the second electronic device based on the posture change information includes:

[0019] determining a horizontal displacement of the cursor according to the first displacement;

[0020] determining a vertical displacement of the cursor according to the second displacement;

[0021] The cursor is moved on the display content of the second electronic device according to the horizontal displacement and the vertical displacement.

[0022] Optionally, in a design of the first aspect, the method further includes:

[0023] When the cursor moves to the target object in the display content of the second electronic device, if a first touch operation on the display screen of the first electronic device is received;

[0024] Then, the target object in the display content of the second electronic device responds to the first touch operation.

[0025] Optionally, in a design of the first aspect, the operation position of the first touch operation corresponds to a second object in the interface content of the first electronic device, and the method further includes:

[0026] Shielding the second object from responding to the first touch operation.

[0027] Optionally, in a design of the first aspect, the first touch operation includes at least a click operation and a first sliding operation, and the target object includes at least an application and a function control.

[0028] Optionally, in a design of the first aspect, controlling the display position of the cursor on the second electronic device based on the operation on the first electronic device includes:

[0029] receiving a second sliding operation on the display screen of the first electronic device;

[0030] determining a displacement of the cursor according to the second sliding operation;

[0031] The cursor is moved in the display content of the second electronic device based on the displacement.

[0032] Optionally, in a design of the first aspect, the starting operation position corresponding to the second sliding operation corresponds to a third object in the interface content of the first electronic device, and the method further includes:

[0033] Shielding the third object from responding to the second touch operation.

[0034] Optionally, in a design of the first aspect, the method further includes:

[0035] When the cursor moves to the target object in the display content of the second electronic device, if a click operation on the display screen of the first electronic device is received;

[0036] Then, the target object in the display content of the second electronic device responds to the click operation.

[0037] Optionally, in a design of the first aspect, the operation position of the click operation corresponds to a fourth object in the interface content of the first electronic device, and the method further includes:

[0038] Shielding the fourth object from responding to the click operation.

[0039] Optionally, in a design of the first aspect, the second electronic device further displays: a ray, wherein the end point of the ray is the cursor.

[0040] Optionally, in a design of the first aspect, the method further includes:

[0041] Receiving a pressing operation on a physical button of the first electronic device;

[0042] Generate a corresponding second event according to the pressing operation;

[0043] The second event is executed on the first electronic device front-end application.

[0044] Optionally, in a design of the first aspect, the method further includes:

[0045] Receiving a second operation on the display screen of the first electronic device, where the second operation is a preset shortcut operation;

[0046] The first electronic device responds to the second operation.

[0047] Optionally, in a design of the first aspect, the operation position of the second operation corresponds to a fifth object in the interface content of the second electronic device, and the fifth object does not respond to the first operation.

[0048] Optionally, in a design of the first aspect, the preset shortcut operation at least includes:

[0049] Contacting a first preset area of ​​the display screen of the first electronic device, and sliding from the first preset area to a first preset direction;

[0050] The user touches a second preset area of ​​the display screen of the first electronic device, slides from the second preset area to a second preset direction, and the time of contacting the display screen of the first electronic device is greater than a preset time.

[0051] Optionally, in a design of the first aspect, the method further includes:

[0052] reducing the display brightness of the display screen of the first electronic device; or,

[0053] A screen-off operation is performed on the display screen of the first electronic device.

[0054] Optionally, in a design of the first aspect, controlling a display position of the cursor on the second electronic device based on an operation on the first electronic device includes:

[0055] Based on the first application being displayed on the second electronic device, obtaining position change information of the first electronic device, and moving the cursor on the display content of the second electronic device based on the position change information;

[0056] Based on displaying a second application on the second electronic device, receiving a second sliding operation on the display screen of the first electronic device, determining the displacement of the cursor according to the second sliding operation, and moving the cursor in the display content of the second electronic device based on the displacement, the first application and the second application are different applications.

[0057] In a second aspect, the present application provides an information processing method, which is applied to a first electronic device, where the first electronic device is connected to a second electronic device, where the first electronic device includes a touch screen, and the second electronic device is an augmented display AR device, a virtual reality VR device, or a mixed reality MR device, and the method includes:

[0058] When the handle is connected to the second electronic device, the handle or the first electronic device is used as a control device for the second electronic device;

[0059] When only the first electronic device is connected to the second electronic device, the first electronic device is used as a control device of the second electronic device.

[0060] Optionally, in a design of the second aspect, using the handle or the first electronic device as a control device for the second electronic device includes:

[0061] Using the handle as a control device for the second electronic device;

[0062] Receiving a first interaction mode switching instruction;

[0063] In response to the first interaction mode switching instruction, the control device of the second electronic device is switched from the handle to the first electronic device.

[0064] Optionally, in a design of the second aspect, receiving the first interaction mode switching instruction includes:

[0065] Receive a first interaction mode switching instruction sent by the handle, generate a corresponding first interaction mode switching instruction based on receiving a second operation on the first electronic device, or receive a first interaction mode switching instruction sent by the second electronic device.

[0066] Optionally, in a design of the second aspect, using the handle or the first electronic device as a control device for the second electronic device includes:

[0067] Using the first electronic device as a control device for the second electronic device;

[0068] receiving a second interaction mode switching instruction;

[0069] In response to the second interaction mode switching instruction, the control device of the second electronic device is switched from the first electronic device to the handle.

[0070] Optionally, in a design of the second aspect, receiving the second interaction mode switching instruction includes:

[0071] Receive a second interaction mode switching instruction sent by the handle, generate a corresponding second interaction mode switching instruction based on receiving a second operation on the first electronic device, or receive a second interaction mode switching instruction sent by the second electronic device.

[0072] Optionally, in a design of the second aspect, the second electronic device displays a cursor and interface content of the first electronic device, and the using the handle or the first electronic device as a control device for the second electronic device includes:

[0073] Based on the operation of the handle, controlling the display position of the cursor on the second electronic device;

[0074] Based on the operation on the first electronic device, the display position of the cursor on the second electronic device is controlled.

[0075] Optionally, in a design of the second aspect, the interface content is interface content of a front-end application of the first electronic device.

[0076] Optionally, in a design of the second aspect, controlling a display position of the cursor on the second electronic device based on an operation on the first electronic device includes:

[0077] Acquiring position change information of the first electronic device;

[0078] The cursor is moved on the display content of the second electronic device based on the posture change information.

[0079] Optionally, in a design of the second aspect, the method further includes:

[0080] When the cursor moves to the target object in the display content of the second electronic device, if a first touch operation on the display screen of the first electronic device is received;

[0081] Then, the target object in the display content of the second electronic device responds to the first touch operation.

[0082] Optionally, in a design of the second aspect, controlling the display position of the cursor on the second electronic device based on the operation on the first electronic device includes:

[0083] receiving a second sliding operation on the display screen of the first electronic device;

[0084] determining a displacement of the cursor according to the second sliding operation;

[0085] The cursor is moved in the display content of the second electronic device based on the displacement.

[0086] Optionally, in a design of the second aspect, the method further includes:

[0087] When the cursor moves to the target object in the display content of the second electronic device, if a click operation on the display screen of the first electronic device is received;

[0088] Then, the target object in the display content of the second electronic device responds to the click operation.

[0089] In a third aspect, the present application provides an electronic device, including:

[0090] A sending module, configured to establish a connection with a second electronic device, wherein the second electronic device displays a cursor and interface content of the first electronic device, wherein the first electronic device includes a touch screen, and the second electronic device is an augmented reality (AR) device, a virtual reality (VR) device, or a mixed reality (MR) device;

[0091] The processing module is used to obtain the operation on the first electronic device, and control the display position of the cursor on the second electronic device based on the operation on the first electronic device.

[0092] Optionally, in a design of the third aspect, the interface content is interface content of a front-end application of the first electronic device.

[0093] Optionally, in a design of the third aspect, the second electronic device further displays: a menu bar, and the menu bar does not belong to the interface content of the first electronic device.

[0094] Optionally, in a design of the third aspect, the processing module is specifically used to obtain posture change information of the first electronic device, and move the cursor on the display content of the second electronic device based on the posture change information.

[0095] Optionally, in a design of the third aspect, the posture change information includes a first rotation angle of the first electronic device on a horizontal plane and a second rotation angle of the first electronic device on a vertical plane, and the processing module is specifically used to:

[0096] determining a horizontal displacement of the cursor according to the first rotation angle;

[0097] determining a vertical displacement of the cursor according to the second rotation angle;

[0098] The cursor is moved on the display content of the second electronic device according to the horizontal displacement and the vertical displacement.

[0099] Optionally, in a design of the third aspect, the posture change information includes a first displacement of the first electronic device in a first direction and a second displacement of the first electronic device in a second direction, the first direction is parallel to a left-right movement direction of a cursor displayed by the second electronic device, and the second direction is parallel to an up-and-down movement direction of a cursor displayed by the second electronic device, and the processing module is specifically used to:

[0100] determining a horizontal displacement of the cursor according to the first displacement;

[0101] determining a vertical displacement of the cursor according to the second displacement;

[0102] The cursor is moved on the display content of the second electronic device according to the horizontal displacement and the vertical displacement.

[0103] Optionally, in a design of the third aspect, the processing module is further used to:

[0104] When the cursor moves to the target object in the display content of the second electronic device, if a first touch operation on the display screen of the first electronic device is received;

[0105] Then, the target object in the display content of the second electronic device responds to the first touch operation.

[0106] Optionally, in a design of the third aspect, the operation position of the first touch operation corresponds to a second object in the interface content of the first electronic device, and the processing module is further used to:

[0107] Shielding the second object from responding to the first touch operation.

[0108] Optionally, in a design of the third aspect, the first touch operation includes at least a click operation and a first sliding operation, and the target object includes at least an application and a function control.

[0109] Optionally, in a design of the third aspect, the processing module is specifically used to:

[0110] receiving a second sliding operation on the display screen of the first electronic device;

[0111] determining a displacement of the cursor according to the second sliding operation;

[0112] The cursor is moved in the display content of the second electronic device based on the displacement.

[0113] Optionally, in a design of the third aspect, the starting operation position corresponding to the second sliding operation corresponds to a third object in the interface content of the first electronic device, and the processing module is further used to:

[0114] Shielding the third object from responding to the second touch operation.

[0115] Optionally, in a design of the third aspect, the processing module is further used to:

[0116] When the cursor moves to the target object in the display content of the second electronic device, if a click operation on the display screen of the first electronic device is received;

[0117] Then, the target object in the display content of the second electronic device responds to the click operation.

[0118] Optionally, in a design of the third aspect, the operation position of the click operation corresponds to a fourth object in the interface content of the first electronic device, and the processing module is further used to:

[0119] Shielding the fourth object from responding to the click operation.

[0120] Optionally, in a design of the third aspect, the second electronic device further displays: a ray, wherein the end point of the ray is the cursor.

[0121] Optionally, in a design of the third aspect, the processing module is further used to:

[0122] Receiving a pressing operation on a physical button of the first electronic device;

[0123] Generate a corresponding second event according to the pressing operation;

[0124] The second event is executed on the first electronic device front-end application.

[0125] Optionally, in a design of the third aspect, the processing module is further used to receive a second operation on the display screen of the first electronic device, and the second operation is a preset shortcut operation, and enables the first electronic device to respond to the second operation.

[0126] Optionally, in a design of the third aspect, the operation position of the second operation corresponds to a fifth object in the interface content of the second electronic device, and the fifth object does not respond to the first operation.

[0127] Optionally, in a design of the third aspect, the preset shortcut operation at least includes:

[0128] Contacting a first preset area of ​​the display screen of the first electronic device, and sliding from the first preset area to a first preset direction;

[0129] The user touches a second preset area of ​​the display screen of the first electronic device, slides from the second preset area to a second preset direction, and the time of contacting the display screen of the first electronic device is greater than a preset time.

[0130] Optionally, in a design of the third aspect, the processing module is further used to reduce the display brightness of a display screen of the first electronic device; or,

[0131] A screen-off operation is performed on the display screen of the first electronic device.

[0132] Optionally, in a design of the third aspect, the processing module is specifically used to:

[0133] Based on the first application being displayed on the second electronic device, obtaining position change information of the first electronic device, and moving the cursor on the display content of the second electronic device based on the position change information;

[0134] Based on displaying a second application on the second electronic device, receiving a second sliding operation on the display screen of the first electronic device, determining the displacement of the cursor according to the second sliding operation, and moving the cursor in the display content of the second electronic device based on the displacement, the first application and the second application are different applications.

[0135] In a fourth aspect, the present application provides a first electronic device, wherein the first electronic device is connected to a second electronic device, wherein the first electronic device includes a touch screen, and the second electronic device is an augmented display (AR) device, a virtual reality (VR) device, or a mixed reality (MR) device, and wherein the first electronic device includes:

[0136] a processing module, configured to use the handle or the first electronic device as a control device for the second electronic device when the handle is connected to the second electronic device;

[0137] When only the first electronic device is connected to the second electronic device, the first electronic device is used as a control device of the second electronic device.

[0138] Optionally, in a design of the fourth aspect, the processing module is specifically used to:

[0139] Using the handle as a control device for the second electronic device;

[0140] Receiving a first interaction mode switching instruction;

[0141] In response to the first interaction mode switching instruction, the control device of the second electronic device is switched from the handle to the first electronic device.

[0142] Optionally, in a design of the fourth aspect, the processing module is specifically used to:

[0143] Receive a first interaction mode switching instruction sent by the handle, generate a corresponding first interaction mode switching instruction based on receiving a second operation on the first electronic device, or receive a first interaction mode switching instruction sent by the second electronic device.

[0144] Optionally, in a design of the fourth aspect, the processing module is specifically used to:

[0145] Using the first electronic device as a control device for the second electronic device;

[0146] The first electronic device further includes an acquisition module, which is used to receive a second interaction mode switching instruction;

[0147] The processing module is specifically configured to switch the control device of the second electronic device from the first electronic device to the handle in response to the second interaction mode switching instruction.

[0148] Optionally, in a design of the fourth aspect, the processing module is specifically used to:

[0149] Receive a second interaction mode switching instruction sent by the handle, generate a corresponding second interaction mode switching instruction based on receiving a second operation on the first electronic device, or receive a second interaction mode switching instruction sent by the second electronic device.

[0150] Optionally, in a design of the fourth aspect, the second electronic device displays a cursor and interface content of the first electronic device, and the processing module is specifically configured to:

[0151] Based on the operation of the handle, controlling the display position of the cursor on the second electronic device;

[0152] Based on the operation on the first electronic device, the display position of the cursor on the second electronic device is controlled.

[0153] Optionally, in a design of the fourth aspect, the interface content is interface content of a front-end application of the first electronic device.

[0154] Optionally, in a design of the fourth aspect, the processing module is specifically used to:

[0155] Acquiring position change information of the first electronic device;

[0156] The cursor is moved on the display content of the second electronic device based on the posture change information.

[0157] Optionally, in a design of the fourth aspect, the processing module is further used to:

[0158] When the cursor moves to the target object in the display content of the second electronic device, if a first touch operation on the display screen of the first electronic device is received;

[0159] Then, the target object in the display content of the second electronic device responds to the first touch operation.

[0160] Optionally, in a design of the fourth aspect, the processing module is further used to:

[0161] receiving a second sliding operation on the display screen of the first electronic device;

[0162] determining a displacement of the cursor according to the second sliding operation;

[0163] The cursor is moved in the display content of the second electronic device based on the displacement.

[0164] Optionally, in a design of the fourth aspect, the processing module is further used to:

[0165] When the cursor moves to the target object in the display content of the second electronic device, if a click operation on the display screen of the first electronic device is received;

[0166] Then, the target object in the display content of the second electronic device responds to the click operation.

[0167] In a fifth aspect, the present application provides an information processing method, the method comprising:

[0168] Displaying the interface content of the first electronic device in the second electronic device, wherein the first electronic device includes a touch screen and the second electronic device is an augmented reality (AR) device, a virtual reality (VR) device, or a mixed reality (MR) device;

[0169] Receive a first operation applied to a first display screen of the first electronic device; cause the interface content displayed by the second electronic device to respond to the first operation; wherein an operation position of the first operation corresponds to a first object in the interface content of the first electronic device; and the first object does not respond to the first operation.

[0170] Optionally, in a design of the fifth aspect, causing the interface content displayed by the second electronic device to respond to the first operation specifically includes:

[0171] A first position is determined in the interface content displayed by the first electronic device; and the interface content displayed by the second electronic device responds to the first operation based on the first position; wherein the first position is independent of the operation position of the first operation.

[0172] Optionally, in a design of the fifth aspect, the method further includes:

[0173] The cursor is projected onto the second electronic device so that the second electronic device displays the cursor.

[0174] Optionally, in a design of the fifth aspect, the position corresponding to the cursor in the interface content displayed by the first electronic device is the first position.

[0175] Optionally, in a design of the fifth aspect, determining the first position in the interface content displayed by the first electronic device includes:

[0176] Acquiring position change information of the first electronic device;

[0177] A first position of the cursor in the interface content displayed by the first electronic device is determined based on the posture change information.

[0178] Optionally, in a design of the fifth aspect, the posture change information includes a first rotation angle of the first electronic device on a horizontal plane and a second rotation angle of the first electronic device on a vertical plane, and determining a first position of the cursor in interface content displayed by the first electronic device based on the posture change information includes:

[0179] determining a horizontal displacement of the cursor according to the first rotation angle;

[0180] determining a vertical displacement of the cursor according to the second rotation angle;

[0181] A first position of the cursor in the interface content displayed by the first electronic device is determined according to the horizontal displacement and the vertical displacement.

[0182] Optionally, in a design of the fifth aspect, the posture change information includes a first displacement of the first electronic device in a first direction and a second displacement of the first electronic device in a second direction, the first direction is parallel to a left-right movement direction of a cursor displayed on the second electronic device, and the second direction is parallel to an up-and-down movement direction of a cursor displayed on the second electronic device, and determining a first position of the cursor in the interface content displayed by the first electronic device based on the posture change information includes:

[0183] determining a horizontal displacement of the cursor according to the first displacement;

[0184] determining a vertical displacement of the cursor according to the second displacement;

[0185] A first position of the cursor in the interface content displayed by the first electronic device is determined according to the horizontal displacement and the vertical displacement.

[0186] Optionally, in a design of the fifth aspect, the first operation includes at least a click operation and a first sliding operation, and the first object includes at least one of an application and a function control.

[0187] Optionally, in a design of the fifth aspect, determining the first position in the interface content displayed by the first electronic device includes:

[0188] receiving a second sliding operation on the display screen of the first electronic device;

[0189] determining a displacement of the cursor according to the second sliding operation;

[0190] A first position of the cursor in the interface content displayed by the first electronic device is determined according to the displacement of the cursor.

[0191] Optionally, in a design of the fifth aspect, the first operation includes at least a click operation, and the first object includes at least one of an application and a function control.

[0192] Optionally, in a design of the fifth aspect, the method further includes:

[0193] reducing the display brightness of the first display screen; or,

[0194] Perform a screen-off operation on the first display screen.

[0195] Optionally, in a design of the fifth aspect, the method further includes:

[0196] The menu bar is displayed on the second electronic device.

[0197] In a sixth aspect, the present application provides an electronic device, including:

[0198] A sending module, configured to display the interface content of the first electronic device in a second electronic device, wherein the first electronic device includes a touch screen, and the second electronic device is an augmented reality (AR) device, a virtual reality (VR) device, or a mixed reality (MR) device;

[0199] A processing module, used for receiving a first operation performed on a first display screen of the first electronic device; causing the interface content displayed by the second electronic device to respond to the first operation; wherein an operation position of the first operation corresponds to a first object in the interface content of the first electronic device; and the first object does not respond to the first operation.

[0200] Optionally, in a design of the sixth aspect, causing the interface content displayed by the second electronic device to respond to the first operation specifically includes:

[0201] A first position is determined in the interface content displayed by the first electronic device; and the interface content displayed by the second electronic device responds to the first operation based on the first position; wherein the first position is independent of the operation position of the first operation.

[0202] Optionally, in a design of the sixth aspect, the method further includes:

[0203] The cursor is projected onto the second electronic device so that the second electronic device displays the cursor.

[0204] Optionally, in a design of the sixth aspect, the position corresponding to the cursor in the interface content displayed by the first electronic device is the first position.

[0205] Optionally, in a design of the sixth aspect, determining the first position in the interface content displayed by the first electronic device includes:

[0206] Acquiring position change information of the first electronic device;

[0207] A first position of the cursor in the interface content displayed by the first electronic device is determined based on the posture change information.

[0208] Optionally, in a design of the sixth aspect, the posture change information includes a first rotation angle of the first electronic device on a horizontal plane and a second rotation angle of the first electronic device on a vertical plane, and determining a first position of the cursor in interface content displayed by the first electronic device based on the posture change information includes:

[0209] determining a horizontal displacement of the cursor according to the first rotation angle;

[0210] determining a vertical displacement of the cursor according to the second rotation angle;

[0211] A first position of the cursor in the interface content displayed by the first electronic device is determined according to the horizontal displacement and the vertical displacement.

[0212] Optionally, in a design of the sixth aspect, the posture change information includes a first displacement of the first electronic device in a first direction and a second displacement of the first electronic device in a second direction, the first direction is parallel to a left-right movement direction of a cursor displayed on the second electronic device, and the second direction is parallel to an up-and-down movement direction of a cursor displayed on the second electronic device, and determining a first position of the cursor in the interface content displayed by the first electronic device based on the posture change information includes:

[0213] determining a horizontal displacement of the cursor according to the first displacement;

[0214] determining a vertical displacement of the cursor according to the second displacement;

[0215] A first position of the cursor in the interface content displayed by the first electronic device is determined according to the horizontal displacement and the vertical displacement.

[0216] Optionally, in a design of the sixth aspect, the first operation includes at least a click operation and a first sliding operation, and the first object includes at least one of an application and a function control.

[0217] Optionally, in a design of the sixth aspect, determining the first position in the interface content displayed by the first electronic device includes:

[0218] receiving a second sliding operation on the display screen of the first electronic device;

[0219] determining a displacement of the cursor according to the second sliding operation;

[0220] A first position of the cursor in the interface content displayed by the first electronic device is determined according to the displacement of the cursor.

[0221] Optionally, in a design of the sixth aspect, the first operation includes at least a click operation, and the first object includes at least one of an application and a function control.

[0222] Optionally, in a design of the sixth aspect, the method further includes:

[0223] reducing the display brightness of the first display screen; or,

[0224] Perform a screen-off operation on the first display screen.

[0225] Optionally, in a design of the sixth aspect, the method further includes:

[0226] The menu bar is displayed on the second electronic device.

[0227] In the seventh aspect, the present application provides an electronic device, which is used as a first electronic device, and includes a processor, an input device, an output device and a memory, wherein the memory is used to store a computer program, and the computer program includes program instructions. When the processor runs the program instructions, the first electronic device executes the method described in any one of the first aspect, the second aspect or the fifth aspect above.

[0228] In an eighth aspect, the present application provides a screen projection system, comprising: a first electronic device as described in the first aspect, the second aspect or the fifth aspect above, and a second electronic device, wherein the first electronic device is connected to the second electronic device.

[0229] In the ninth aspect, an embodiment of the present application provides a computer-readable storage medium, characterized in that the computer-readable storage medium stores a computer program, and the computer program includes program instructions, which, when the program request is executed by the computer, enable the computer to execute the method described in any one of the first aspect, the second aspect or the fifth aspect above.

[0230] In a tenth aspect, the present application provides a computer program product comprising instructions, which, when executed on a computer, enables the computer to execute the method as described in any one of the first aspect, the second aspect or the fifth aspect above.

[0231] In an eleventh aspect, the present application provides a control method, which is applied to a first electronic device connected to a second electronic device, wherein the second electronic device includes an imaging system; the method includes:

[0232] Displaying interface contents of N applications running on the first electronic device in N display areas of the imaging system respectively; the first electronic device includes a first display screen, and N is an integer greater than 1;

[0233] Displaying a cursor in the imaging system, wherein the cursor is used to determine an operation object in the content displayed by the imaging system;

[0234] receiving a first sliding operation on the first display screen;

[0235] determining, according to the first sliding operation, a displacement of the cursor in the content displayed by the imaging system;

[0236] The starting position of the first sliding operation corresponds to a first object in the current interface content of the first electronic device, the first object does not respond to the first sliding operation, and the current interface content is the interface content of one application among the interface contents of the N applications.

[0237] Optionally, in a design of the eleventh aspect, the second electronic device includes a television, an AR device, a VR device, or a MR device.

[0238] Optionally, in a design of the eleventh aspect, the method further includes:

[0239] Determining a first operation object in the interface contents of N applications displayed by the imaging system according to the cursor, wherein the first object at least includes an application program and a function control;

[0240] A second operation is received, and the first operation object responds to the second operation.

[0241] Optionally, in a design of the eleventh aspect, the method further includes:

[0242] reducing the display brightness of the first display screen; or,

[0243] Perform a screen-off operation on the first display screen.

[0244] Optionally, in a design of the eleventh aspect, the method further includes:

[0245] A menu bar is displayed in the imaging system; the menu bar includes a functional control for adding or deleting interface content of an application in the imaging system.

[0246] In a twelfth aspect, the present application provides a first electronic device, characterized in that the first electronic device is connected to a second electronic device; the first electronic device includes a first display screen, and the second electronic device includes an imaging system; the first electronic device includes:

[0247] a sending module, configured to display interface contents of N applications running on the first electronic device in N display areas of the imaging system respectively; the first electronic device includes a first display screen, and N is an integer greater than 1; and further configured to display a cursor in the imaging system, the cursor being used to determine an operation object in the contents displayed by the imaging system;

[0248] a processing module, configured to receive a first sliding operation on the first display screen; and determine a displacement of the cursor in the content displayed by the imaging system according to the first sliding operation;

[0249] The starting position of the first sliding operation corresponds to a first object in the current interface content of the first electronic device, the first object does not respond to the first sliding operation, and the current interface content is the interface content of one application among the interface contents of the N applications.

[0250] Optionally, in a design of the twelfth aspect, the second electronic device includes a television, an AR device, a VR device, or an MR device.

[0251] Optionally, in a design of the twelfth aspect, the processing module is further specifically configured to:

[0252] Determining a first operation object in the interface contents of N applications displayed by the imaging system according to the cursor, wherein the first object at least includes an application program and a function control;

[0253] A second operation is received, and the first operation object responds to the second operation.

[0254] Optionally, in a design of the twelfth aspect, the processing module is further used to:

[0255] reducing the display brightness of the first display screen; or,

[0256] Perform a screen-off operation on the first display screen.

[0257] Optionally, in a design of the twelfth aspect, the sending module is further used to:

[0258] A menu bar is displayed in the imaging system; the menu bar includes a functional control for adding or deleting interface content of an application in the imaging system.

[0259] In a thirteenth aspect, the present application provides a first electronic device, characterized in that the first electronic device includes:

[0260] processors, input devices, output devices, and memory;

[0261] Among them, the input device is used to receive data and instructions; the output device is used to send data and instructions; the memory is used to store a computer program, and the computer program includes program instructions. When the processor runs the program instructions, the first electronic device executes the method described in the eleventh aspect.

[0262] In a fourteenth aspect, the present application provides a control method based on a touch screen, which is applied to a first electronic device, wherein the first electronic device includes a touch screen; the touch screen can be used to display an interface; the method includes: when the first electronic device establishes a projection connection with the second electronic device, a cursor is generated, and the cursor is displayed in the interface of the touch screen; projection content is generated based on the interface content in the touch screen; the projection content includes a cursor; the projection content is sent to the second electronic device; when receiving a target touch operation of the user on the touch screen, the target touch operation is responded to based on the display position of the cursor in the current interface; wherein the display position of the cursor in the current interface is used to indicate the response position for the target touch operation in the current interface; the response position is not related to the operation position of the target touch operation on the touch screen. The first electronic device includes a mobile phone or a tablet; the second electronic device includes a display, a TV, a projector, an AR device, a VR device, or an MR device.

[0263] In the fifteenth aspect, the present application provides a control device based on a touch screen, which is applied to a first electronic device, wherein the first electronic device includes a touch screen; the touch screen can be used to display an interface; the device includes: a generation module, which is used to generate a cursor when the first electronic device establishes a screen projection connection with a second electronic device, and the cursor is displayed in the interface of the touch screen; a screen projection module, which is used to generate screen projection content based on the interface content in the touch screen, and send the screen projection content to the second electronic device, and the screen projection content includes a cursor; a receiving module, which is used to receive a target touch operation of the user on the touch screen; a response module, which is used to respond to the target touch operation based on the display position of the cursor in the current interface; wherein the display position of the cursor in the current interface is used to indicate the response position in the current interface for the target touch operation; the response position is not related to the operation position of the target touch operation on the touch screen.

[0264] According to the fourteenth aspect or the fifteenth aspect, in a possible design, when the first electronic device does not establish a screen projection connection with the second electronic device, the touch screen interface does not include a cursor. The projection content includes part or all of the content in the interface, and the projection content is used to synchronize part or all of the content presented in the display system of the second electronic device with part or all of the content displayed on the touch screen. The projection content also includes the interface of the background application running on the first electronic device, a newly drawn menu bar, or at least one of the preset function controls. Both the first electronic device and the second electronic device can display the cursor.

[0265] According to the fourteenth aspect or the fifteenth aspect, in a possible design, the method of generating a cursor includes: obtaining the interface content of the front-end application corresponding to the current interface, and drawing the cursor in the interface content; or, drawing the cursor, and adding the cursor to the current interface based on the floating window interface; or, drawing a ray, and intersecting the ray with the current interface at an end point based on the floating window interface; the end point is the cursor. The method can be executed by a generation module. In subsequent operations, a predefined operation can also be received, and the predefined operation is used to activate or hide the display of the cursor.

[0266] According to the fourteenth aspect or the fifteenth aspect, in a possible design, the first event processing system is called, and the touch screen event processing system is shielded; the target touch operation is re-parsed into a target event, and the target event is injected into the first event processing system, so that the first electronic device responds to the target touch operation according to the first event processing system. Or; the first event processing system is called; a transparent target floating window is drawn on the current interface; the target floating window is used to shield the touch screen event processing system; the target event corresponding to the target touch operation is parsed through the target floating window, and the target event is injected into the first event processing system, so that the first electronic device responds to the target touch operation according to the first event processing system; wherein, the response of the touch screen event processing system to the target touch operation is related to the operation position of the target touch operation on the touch screen; the response of the first event processing system to the target touch operation is related to the position of the cursor, and is not related to the operation position of the target touch operation on the touch screen. The method can be executed by a calling module.

[0267] According to the fourteenth aspect or the fifteenth aspect, in a possible design, before calling the first event processing system, the method further includes: detecting that the first electronic device establishes a screen projection connection with the second electronic device; or detecting a first switching command input by a user; the first switching command is used to instruct the first electronic device to call the first event processing system and shield the touch screen event processing system. The method can be executed by a detection module.

[0268] According to the fourteenth aspect or the fifteenth aspect, in a possible design, the display brightness of the touch screen may be reduced, or the touch screen may be turned off, which may be performed by a brightness control module.

[0269] According to the fourteenth aspect or the fifteenth aspect, in a possible design, when the target touch operation is a sliding operation, the first electronic device changes the position of the cursor in the current interface according to the sliding operation, which may be performed by a response module.

[0270] According to the fourteenth aspect or the fifteenth aspect, in a possible design, when the target touch operation is a click operation, the first electronic device performs a confirmation operation on the object corresponding to the cursor in the current interface, which may be performed by a response module.

[0271] According to the fourteenth aspect or the fifteenth aspect, in a possible design, when the target touch operation is a long press followed by a slide operation, the first electronic device may slide or drag the object corresponding to the cursor in the interface, which may be executed by a response module.

[0272] According to the fourteenth aspect or the fifteenth aspect, in a possible design, when the target touch operation makes the movement trajectory of the cursor in the current interface match a preset shortcut operation command, or when the target touch operation matches a preset shortcut operation command, the first electronic device executes the shortcut operation command. The target touch operation includes: sliding from a first preset area of ​​the touch screen to a first preset direction; or sliding from a second preset area of ​​the touch screen to a second preset direction, and the time of contact with the touch screen is greater than a preset time; or the number of times the touch screen is clicked reaches a preset number; or the sliding trajectory of the target touch operation satisfies a preset pattern. It can be executed by a response module.

[0273] According to the fourteenth aspect or the fifteenth aspect, in a possible design, the method further includes: starting a multi-screen display mode; creating N virtual screens in the interface of the touch screen; determining N interfaces to be displayed; displaying the N interfaces to be displayed in the N virtual screens respectively; wherein the N interfaces to be displayed belong to M applications, M is an integer not greater than N, and N is an integer greater than 1; and the cursor can be located at any position in the N virtual screens. It can be performed by the multi-screen creation module.

[0274] According to the fourteenth aspect or the fifteenth aspect, in a possible design, the first electronic device may also obtain a second switching instruction; the second switching instruction is used to instruct the first electronic device to call a second event processing system and shield the first event processing system and the touch screen event processing system; call the second event processing system; wherein, when the posture of the first electronic device changes, the response of the second event processing system to the posture change of the first electronic device is to change the position of the cursor in the current interface. For example, the horizontal displacement of the cursor is determined according to the rotation angle in the left and right directions, and the vertical displacement of the cursor is determined according to the rotation angle in the up and down directions. The first event processing system may be a touchpad event processing system, and the second event processing system may be an air mouse event processing system. Any event processing system can be flexibly defined. It can be executed by a calling module.

[0275] The embodiment of the present application provides an information processing method, including: establishing a connection between a first electronic device and a second electronic device, wherein the second electronic device displays a cursor and the interface content of the first electronic device, wherein the first electronic device includes a touch screen, and the second electronic device is an augmented display AR device, a virtual reality VR device, or a mixed reality MR device; obtaining a first operation on the first electronic device; and based on the first operation on the first electronic device, controlling the display position of the cursor on the second electronic device. In the above manner, the first electronic device with a touch screen can be used as a control device for an AR device / VE device / MR device instead of a handle. BRIEF DESCRIPTION OF THE DRAWINGS

[0276] Figure 1 is a system architecture diagram of the screen projection system provided in an embodiment of the present application;

[0277] Figure 2 is a schematic structural diagram of a first electronic device provided in an embodiment of the present application;

[0278] Figure 3a is a software structure block diagram of the electronic device of the embodiment of the present application;

[0279] Figure 3b A schematic diagram of a flow chart of an information processing method provided in an embodiment of the present application;

[0280] Figure 4a A schematic diagram of an interface of a first electronic device provided in an embodiment of the present application;

[0281] Figure 4b A schematic diagram of an interface of a first electronic device provided in an embodiment of the present application;

[0282] Figure 4c A schematic diagram of an interface of a first electronic device provided in an embodiment of the present application;

[0283] Figure 5a A schematic diagram of an interface of a first electronic device provided in an embodiment of the present application;

[0284] Figure 5b A schematic diagram of an interface of a first electronic device provided in an embodiment of the present application;

[0285] Figure 5c A schematic diagram of an interface of a first electronic device provided in an embodiment of the present application;

[0286] Figure 6a A schematic diagram of an actual screen projection scenario;

[0287] Figure 6b A schematic diagram of an actual screen projection scenario;

[0288] Figure 6c This is a schematic diagram of an actual screen projection scenario;

[0289] Figure 6d This is a schematic diagram of an actual screen projection scenario;

[0290] Figure 7a This is a schematic diagram of an actual screen projection scenario;

[0291] Figure 7b A schematic diagram of a posture change of a first electronic device rotating in a horizontal direction;

[0292] Figure 7c A schematic diagram of the displacement of a cursor displayed by a second electronic device;

[0293] Figure 7d This is a schematic diagram of an actual screen projection scenario;

[0294] Figure 7e A schematic diagram of a posture change of a first electronic device rotating in a vertical direction;

[0295] Figure 7f A schematic diagram of the displacement of a cursor displayed by a second electronic device;

[0296] Figure 7g This is a schematic diagram of an actual screen projection scenario;

[0297] Figure 7h A schematic diagram of a user's sliding operation;

[0298] Figure 7i A schematic diagram of a user's sliding operation;

[0299] Figure 7j A schematic diagram of the displacement of a cursor displayed by a second electronic device;

[0300] Figure 8a A schematic diagram of screen projection content of a second electronic device provided in an embodiment of the present application;

[0301] Figure 8b A schematic diagram of screen projection content of a second electronic device provided in an embodiment of the present application;

[0302] Figure 9a A schematic diagram of screen projection content of a second electronic device provided in an embodiment of the present application;

[0303] Figure 9b A schematic diagram of screen projection content of a second electronic device provided in an embodiment of the present application;

[0304] Fig.9c A schematic diagram of screen projection content of a second electronic device provided in an embodiment of the present application;

[0305] Figure 9d A schematic diagram of a user operating a first electronic device provided in an embodiment of the present application;

[0306] Fig.10a A schematic diagram of screen projection content of a second electronic device provided in an embodiment of the present application;

[0307] Fig.10b A schematic diagram of screen projection content of a second electronic device provided in an embodiment of the present application;

[0308] Fig.10c A schematic diagram of an interaction of a first electronic device provided in an embodiment of the present application;

[0309] Fig.10d A schematic diagram of interface content of a first electronic device provided in an embodiment of the present application;

[0310] Fig.11a A schematic diagram of interaction between a first electronic device and a second electronic device;

[0311] Fig.11b A schematic diagram of interaction between a first electronic device and a second electronic device;

[0312] Fig.11c A schematic diagram of interaction between a first electronic device and a second electronic device;

[0313] Fig.11d A schematic diagram of interaction between a first electronic device and a second electronic device;

[0314] Fig.11e A schematic diagram of interaction between a first electronic device and a second electronic device;

[0315] Fig.11f A schematic diagram of interaction between a first electronic device and a second electronic device;

[0316] Figure 11g A schematic diagram of interaction between a first electronic device and a second electronic device;

[0317] Fig.12a A schematic diagram of an interface of a second electronic device provided in an embodiment of the present application;

[0318] Figure 12b A schematic diagram of an interface of a second electronic device provided in an embodiment of the present application;

[0319] Fig.12c A schematic diagram of an interface of a second electronic device provided in an embodiment of the present application;

[0320] Fig.12dA schematic diagram of a user operating a first electronic device provided in an embodiment of the present application;

[0321] Fig.12e A schematic diagram of an interface of a second electronic device provided in an embodiment of the present application;

[0322] Fig.12f A schematic diagram of an interface of a second electronic device provided in an embodiment of the present application;

[0323] Figure 12g A schematic diagram of a user operating a first electronic device provided in an embodiment of the present application;

[0324] Figure 12h A schematic diagram of an interface of a second electronic device provided in an embodiment of the present application;

[0325] Fig.12i A schematic diagram of an interface of a second electronic device provided in an embodiment of the present application;

[0326] Fig.12j A schematic diagram of an interface of a second electronic device provided in an embodiment of the present application;

[0327] Figure 12k A schematic diagram of an interface of a second electronic device provided in an embodiment of the present application;

[0328] Figure 12l A schematic diagram of a user operating a first electronic device provided in an embodiment of the present application;

[0329] Figure 12m A schematic diagram of a user operating a first electronic device provided in an embodiment of the present application;

[0330] Figure 12n A schematic diagram of an interface of a second electronic device provided in an embodiment of the present application;

[0331] Fig.12o A schematic diagram of a user operating a first electronic device provided in an embodiment of the present application;

[0332] Figure 12p A schematic diagram of an interface of a second electronic device provided in an embodiment of the present application;

[0333] Figure 12q A schematic diagram of a user operating a first electronic device provided in an embodiment of the present application;

[0334] Figure 12r A schematic diagram of an interface of a second electronic device provided in an embodiment of the present application;

[0335] Figure 12s A schematic diagram of a user operating a first electronic device provided in an embodiment of the present application;

[0336] Figure 12t A schematic diagram of an interface of a second electronic device provided in an embodiment of the present application;

[0337] Figure 12u A schematic diagram of a user operating a first electronic device provided in an embodiment of the present application;

[0338] Fig.13 A schematic diagram of a screen projection scenario provided in an embodiment of the present application;

[0339] Fig.14a A schematic diagram of screen projection content of a second electronic device provided in an embodiment of the present application;

[0340] Fig.14b A schematic diagram of screen projection content of a second electronic device provided in an embodiment of the present application;

[0341] Fig.14c A schematic diagram of screen projection content of a second electronic device provided in an embodiment of the present application;

[0342] Fig.14d A schematic diagram of a user operating a first electronic device provided in an embodiment of the present application;

[0343] Fig.14e A schematic diagram of screen projection content of a second electronic device provided in an embodiment of the present application;

[0344] Fig.14f A schematic diagram of a user operating a first electronic device provided in an embodiment of the present application;

[0345] Figure 14g A schematic diagram of screen projection content of a second electronic device provided in an embodiment of the present application;

[0346] Fig.15 A flowchart illustrating an information processing method provided in an embodiment of the present application;

[0347] Fig.16 A schematic diagram of an architecture of an embodiment of the present application;

[0348] Fig.17 A schematic diagram of an architecture of an embodiment of the present application;

[0349] Fig.18 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application;

[0350] Fig.19 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application;

[0351] Fig. 20A schematic diagram of interaction between a first electronic device and a second electronic device;

[0352] Fig.21a A schematic diagram showing a connection method of a first electronic device to a second electronic device;

[0353] Figure 21b A schematic diagram showing a connection method of a first electronic device to a second electronic device;

[0354] Fig.22a A schematic diagram showing a user using a handle as an interaction device;

[0355] Figure 22b A schematic diagram of a possible software structure of the first electronic device;

[0356] Fig.22c A schematic diagram of a virtual screen management method in a VR scenario;

[0357] Fig.22d A schematic diagram showing the processing of screen injection events by an Android system is shown;

[0358] Fig.23a A schematic diagram of interaction of a user using a first electronic device;

[0359] Figure 23b A schematic diagram of interaction of a user using a first electronic device;

[0360] Fig.23c A schematic diagram of interaction of a user using a first electronic device;

[0361] Fig.23d A schematic diagram of interaction of a user using a first electronic device;

[0362] Fig.24a A schematic diagram of an interaction of a user using a first electronic device provided in an embodiment of the present application;

[0363] Figure 24b A schematic diagram of an interaction of a user using a first electronic device provided in an embodiment of the present application;

[0364] Fig.24c A schematic diagram of an interaction of a user using a first electronic device provided in an embodiment of the present application;

[0365] Fig.25 A schematic diagram of the operation of a first electronic device provided in an embodiment of the present application;

[0366] Fig.26 A schematic diagram of an interaction of a user using a first electronic device provided in an embodiment of the present application;

[0367] Fig. 27 A schematic diagram of a system architecture provided for an embodiment of the present application;

[0368] Fig.28 A schematic diagram of a system architecture provided for an embodiment of the present application;

[0369] Fig.29 A flowchart illustrating an information processing method provided in an embodiment of the present application;

[0370] Fig.30 A schematic diagram of multi-screen control provided in an embodiment of the present application;

[0371] Fig.31 A flow chart of a control method provided in an embodiment of the present application;

[0372] Fig.32 This is a schematic diagram of an event processing process in an embodiment of the present application;

[0373] Fig.33 This is a schematic diagram of a control device in an embodiment of the present application. DETAILED DESCRIPTION

[0374] The embodiments of the present application provide an information processing method and an electronic device, and a user can use a mobile phone as a control device for an augmented display AR device, a virtual reality VR device, or a mixed reality MR device.

[0375] The embodiments of the present application are described below in conjunction with the accompanying drawings. Those skilled in the art will appreciate that, with the development of technology and the emergence of new scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0376] The terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and need not be used to describe a specific order or sequential order. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances, which is only to describe the distinction mode adopted by the objects of the same attributes when describing in the embodiments of the present application. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, so that the process, method, system, product or equipment comprising a series of units need not be limited to those units, but may include other units that are not clearly listed or inherent to these processes, methods, products or equipment.

[0377] The screen projection involved in the present application may include wired screen projection and wireless screen projection, wherein wired screen projection can establish a connection between electronic devices through a high definition multimedia interface (HDMI) and transmit media data through the HDMI transmission line; wireless screen projection can establish a connection between multiple electronic devices through, for example, the Miracast protocol and transmit media data through, for example, a wireless local area network (wireless fidelity, WIFI).

[0378] The screen projection system of the present application may include at least two electronic devices and a screen projection port, where the screen projection port may include a wired port and / or a wireless port. The wired port may be an HDMI port; the wireless port may be an application programming interface (API) or a hardware screen projection module. Figure 1 , Figure 1 : is a system architecture diagram of the screen projection system provided in the embodiment of the present application. Figure 1 As shown, the screen projection system includes a first electronic device 100 and a second electronic device 200. The first electronic device 100 may include a first wired port 101 and / or a first wireless port 102, and the second electronic device 200 may include a second wired port 201 and / or a second wireless port 202. The first wired port 101 and the first wireless port 102 may be integrated on the first electronic device 100, or may exist independently of the first electronic device 100; the second wired port 201 and the second wireless port 202 may be integrated on the second electronic device 200, or may exist independently of the second electronic device 200, which is not limited in the embodiments of the present application. The first electronic device 100 and the second electronic device 200 can establish a screen projection connection through the screen projection port (wired port or wireless port). Among them, the first electronic device 100 at least has the screen projection sending (Source) capability.

[0379] The first electronic device 100 may include an enhanced interactive service, which may acquire sensor information of the electronic device (for example, information from Figure 1 The sensor information includes but is not limited to the posture information of the first electronic device. The enhanced interaction service can also obtain the touch screen information of the electronic device (for example, it can be obtained from Figure 1 The touch screen information is obtained by the touch screen input in the first electronic device, and the touch screen information includes but is not limited to the touch information of the touch screen. How the enhanced interactive service of the first electronic device uses the acquired sensor information and touch screen information will be described in the following embodiments and will not be repeated here.

[0380] In one implementation, the enhanced interactive service can also obtain the interface content of the front-end application (foreground application) of the first electronic device, and draw other images (such as a cursor and a menu bar) on the interface content, and send the drawn interface content to the screen projection service. The screen projection service can generate the projection content based on the drawn interface content, and send the projection content to the second electronic device 200, so that the display screen of the second electronic device displays the projection content.

[0381] In another implementation, the enhanced interactive service can also add other images (such as a cursor and a menu bar) to the interface content of the front-end application of the first electronic device through the system's floating window interface. The screen projection service can obtain the interface content of the front-end application of the first electronic device with other images added, and generate the projection content based on the obtained interface content of the front-end application of the first electronic device with other images added, and send the projection content to the second electronic device 200 so that the display screen of the second electronic device displays the projection content.

[0382] The first electronic device 100 may include a screen projection service, which is used to implement a screen projection sending (source) capability. For example, the screen projection service can obtain the interface content of the first electronic device as projection data, or use the interface content drawn by the enhanced interactive service as projection content, and send the projection content to the second electronic device 200 via the first wireless port or the first wired port.

[0383] The second electronic device 100 may have, but is not limited to, a screen projection receiving (Sink) capability and an image display capability. The second electronic device 200 may include a screen projection service, which is used to implement the screen projection receiving (sink) capability, and the screen projection service can display the received screen projection content on the display screen of the second electronic device.

[0384] Examples of the first electronic device 100 include but are not limited to electronic devices running iOS, Android, Microsoft or other operating systems. Optionally, the first electronic device 100 may be a mobile phone, a tablet computer, a personal digital assistant (PDA) or a desktop computer. The second electronic device 200 may be a television, a tablet computer or a desktop computer.

[0385] It should be noted that the first electronic device may be an electronic device with a display function.

[0386] It should be noted that in some scenarios, if the second electronic device is a TV, a tablet computer or a desktop computer, the size of the display area of ​​its display screen may be larger than the size of the display area of ​​the display screen of the first electronic device.

[0387] For ease of understanding, the structure of the first electronic device 100 provided in the embodiment of the present application is described below by way of example. Figure 2 , Figure 2 It is a schematic diagram of the structure of a first electronic device provided in an embodiment of the present application.

[0388] like Figure 2 As shown, the first electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0389] It is to be understood that the structure illustrated in the embodiment of the present application does not constitute a specific limitation on the first electronic device 100. In other embodiments of the present application, the first electronic device 100 may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0390] The processor 110 may include one or more processing units, for example, the processor 110 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0391] The controller may be the nerve center and command center of the first electronic device 100. The controller may generate an operation control signal according to the instruction operation code and the timing signal to complete the control of fetching and executing instructions.

[0392] The processor 110 may also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. The memory may store instructions or data that the processor 110 has just used or cyclically used. If the processor 110 needs to use the instruction or data again, it may be directly called from the memory. This avoids repeated access, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0393] In some embodiments, the processor 110 may include one or more interfaces. The interface may include an inter-integrated circuit (IC) interface, an inter-integrated circuit sound (IS) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0394] It is understandable that the interface connection relationship between the modules illustrated in the embodiment of the present application is only a schematic illustration and does not constitute a structural limitation on the first electronic device 100. In other embodiments of the present application, the first electronic device 100 may also adopt different interface connection methods in the above embodiments, or a combination of multiple interface connection methods.

[0395] The charging management module 140 is used to receive charging input from a charger. The charger may be a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 140 may receive charging input from the wired charger through the USB interface 130 .

[0396] The power management module 141 is used to connect the battery 142, the charging management module 140 and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140 to power the processor 110, the internal memory 121, the external memory, the display screen 194, the camera 193, and the wireless communication module 160.

[0397] The wireless communication function of the first electronic device 100 can be implemented through the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor and the baseband processor.

[0398] In some feasible implementations, the first electronic device 100 can use a wireless communication function to communicate with other devices. For example, the first electronic device 100 can communicate with the second electronic device 200, the first electronic device 100 establishes a screen projection connection with the second electronic device 200, and the first electronic device 100 outputs the projection data to the second electronic device 200, etc. Among them, the projection data output by the first electronic device 100 can be audio and video data. The communication process between the first electronic device 100 and the second electronic device 200 can refer to the relevant description of the subsequent embodiments, which will not be repeated here.

[0399] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in the first electronic device 100 can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve the utilization of the antennas. For example, antenna 1 can be reused as a diversity antenna for a wireless local area network. In some other embodiments, the antenna can be used in combination with a tuning switch.

[0400] The mobile communication module 150 can provide solutions for wireless communications including 1G / 3G / 4G / 5G applied to the first electronic device 100. The mobile communication module 150 may include at least one filter, a switch, a power amplifier, a low noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves from the antenna 1, and filter, amplify, and process the received electromagnetic waves, and transmit them to the modulation and demodulation processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modulation and demodulation processor, and convert it into electromagnetic waves for radiation through the antenna 2. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the processor 110. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the same device as at least some of the modules of the processor 110.

[0401] The modem processor may include a modulator and a demodulator. Among them, the modulator is used to modulate the low-frequency baseband signal to be sent into a medium-high frequency signal. The demodulator is used to demodulate the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After the low-frequency baseband signal is processed by the baseband processor, it is passed to the application processor. The application processor outputs a sound signal through an audio device (not limited to a speaker 170A, a receiver 170B, etc.), or displays an image or video through a display screen 194. In some embodiments, the modem processor may be an independent device. In other embodiments, the modem processor may be independent of the processor 110 and be set in the same device as the mobile communication module 150 or other functional modules.

[0402] The wireless communication module 160 can provide wireless communication solutions including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR), etc., applied on the first electronic device 100. The wireless communication module 160 can be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via antenna 1, modulates and filters the electromagnetic wave signals, and sends the processed signals to the processor 110. The wireless communication module 160 can also receive the signal to be sent from the processor 110, modulate the frequency, amplify it, and convert it into electromagnetic waves for radiation through antenna 2.

[0403] In some embodiments, the antenna 1 of the first electronic device 100 is coupled to the mobile communication module 150, and the antenna 2 is coupled to the wireless communication module 160, so that the first electronic device 100 can communicate with the network and other devices through wireless communication technology. The wireless communication technology may include global system for mobile communications (GSM), general packet radio service (GPRS), code division multiple access (CDMA), wideband code division multiple access (WCDMA), time-division code division multiple access (TD-SCDMA), long term evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technology, etc. The GNSS may include a global positioning system (GPS), a global navigation satellite system (GLONASS), a Beidou navigation satellite system (BDS), a quasi-zenith satellite system (QZSS) and / or a satellite based augmentation system (SBAS).

[0404] The first electronic device 100 implements a display function through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, which connects the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs that execute program instructions to generate or change display information.

[0405] The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode or an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), Miniled, MicroLed, Micro-oLed, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the first electronic device 100 may include 1 or N display screens 194, where N is a positive integer greater than 1.

[0406] In some feasible implementations, the display screen 194 may be used to display various interfaces output by the system of the first electronic device 100. For various interfaces output by the first electronic device 100, reference may be made to the relevant descriptions of the subsequent embodiments.

[0407] The first electronic device 100 can implement a shooting function through an ISP, a camera 193, a video codec, a GPU, a display screen 194, and an application processor.

[0408] ISP is used to process the data fed back by camera 193. For example, when taking a photo, the shutter is opened, and the light is transmitted to the camera photosensitive element through the lens. The light signal is converted into an electrical signal, and the camera photosensitive element transmits the electrical signal to ISP for processing and converts it into an image visible to the naked eye. ISP can also perform algorithm optimization on the noise, brightness, and skin color of the image. ISP can also optimize the exposure, color temperature and other parameters of the shooting scene. In some embodiments, ISP can be set in camera 193.

[0409] The camera 193 is used to capture still images or videos. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, and then passes the electrical signal to the ISP to be converted into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV or other format. In some embodiments, the first electronic device 100 may include 1 or N cameras 193, where N is a positive integer greater than 1.

[0410] Digital signal processors are used to process digital signals. In addition to processing digital image signals, they can also process other digital signals.

[0411] The video codec is used to compress or decompress digital video. The first electronic device 100 may support one or more video codecs. In this way, the first electronic device 100 can play or record videos in multiple coding formats, such as: Moving Picture Experts Group (MPEG) 1, MPEG1, MPEG3, MPEG4, etc.

[0412] The NPU is a neural network (NN) computing processor that quickly processes input information by drawing on the structure of biological neural networks, such as the transmission mode between neurons in the human brain, and can also continuously self-learn. The NPU can realize applications such as intelligent cognition of the first electronic device 100, such as image recognition, face recognition, voice recognition, text understanding, etc.

[0413] The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the first electronic device 100. The external memory card communicates with the processor 110 through the external memory interface 120 to implement a data storage function, such as storing music, video and other files in the external memory card.

[0414] The internal memory 121 can be used to store computer executable program codes, which include instructions. The processor 110 executes various functional applications and data processing of the first electronic device 100 by running the instructions stored in the internal memory 121. The internal memory 121 may include a program storage area and a data storage area. Among them, the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc. The data storage area may store data created during the use of the first electronic device 100 (such as audio data, a phone book, etc.), etc. In addition, the internal memory 121 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc.

[0415] The first electronic device 100 can implement audio functions through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the headphone jack 170D, and the application processor, etc. For example, music playing, recording, etc. In some feasible implementations, the audio module 170 can be used to play the sound corresponding to the video. For example, when the display screen 194 displays the video playing screen, the audio module 170 outputs the sound of the video playing.

[0416] The audio module 170 is used to convert digital audio information into analog audio signal output, and is also used to convert analog audio input into digital audio signals.

[0417] The speaker 170A, also called a "horn", is used to convert audio electrical signals into sound signals.

[0418] The receiver 170B, also called a "handset", is used to convert audio electrical signals into sound signals.

[0419] Microphone 170C, also called "microphone" or "microphone", is used to convert sound signals into electrical signals.

[0420] The earphone interface 170D is used to connect a wired earphone and can be a USB interface 130 or a 3.5 mm open mobile terminal platform (OMTP) standard interface or a cellular telecommunications industry association of the USA (CTIA) standard interface.

[0421] The pressure sensor 180A is used to sense pressure signals and can convert pressure signals into electrical signals. In some embodiments, the pressure sensor 180A can be disposed on the display screen 194. The gyroscope sensor 180B can be used to determine the motion posture of the first electronic device 100. The air pressure sensor 180C is used to measure air pressure.

[0422] The acceleration sensor 180E can detect the magnitude of the acceleration of the first electronic device 100 in various directions (generally three axes). When the first electronic device 100 is stationary, the magnitude and direction of gravity can be detected. It can also be used to identify the posture of the electronic device and is applied to applications such as horizontal and vertical screen switching and pedometers.

[0423] The distance sensor 180F is used to measure the distance.

[0424] The ambient light sensor 180L is used to sense the brightness of ambient light.

[0425] The fingerprint sensor 180H is used to collect fingerprints.

[0426] The temperature sensor 180J is used to detect the temperature.

[0427] The touch sensor 180K is also called a "touch panel". The touch sensor 180K can be arranged on the display screen 194, and the touch sensor 180K and the display screen 194 form a touch screen, also called a "touch screen". The touch sensor 180K is used to detect touch operations acting on or near it. The touch sensor can pass the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through the display screen 194. In other embodiments, the touch sensor 180K can also be arranged on the surface of the first electronic device 100, which is different from the position of the display screen 194.

[0428] The key 190 includes a power key, a volume key, etc. The key 190 may be a mechanical key or a touch key. The first electronic device 100 may receive key input and generate key signal input related to user settings and function control of the first electronic device 100.

[0429] Motor 191 can generate vibration prompts.

[0430] Indicator 192 may be an indicator light, which may be used to indicate charging status, power changes, messages, missed calls, notifications, etc.

[0431] The SIM card interface 195 is used to connect a SIM card.

[0432] The above is a description of the structure of the first electronic device 100. Next, the software structure of the first electronic device is described. The software system of the first electronic device 100 can adopt a layered architecture, an event-driven architecture, a micro-core architecture, a micro-service architecture, or a cloud architecture. Examples of the first electronic device 100 include but are not limited to electronic devices equipped with iOS, Android, Microsoft or other operating systems. The embodiment of the present application takes the first electronic device 100 equipped with the Android system as an example to illustrate the software structure of the first electronic device 100.

[0433] Figure 3a It is a software structure block diagram of the first electronic device 100 according to an embodiment of the present application.

[0434] like Figure 3a As shown, the layered architecture divides the software into several layers, each with clear roles and division of labor. The layers communicate through software interfaces. In some embodiments, the Android system is divided into four layers, from top to bottom, namely, the application layer, the application framework layer, the Android runtime (Android runtime) and the system library, and the kernel layer. The application layer can include a series of application packages.

[0435] like Figure 3a As shown, the application package may include camera, gallery, calendar, call, map, navigation, WLAN, Bluetooth, music, video, short message and other applications.

[0436] The application framework layer provides an application programming interface (API) and a programming framework for the applications in the application layer. The application framework layer includes some predefined functions.

[0437] like Figure 3a As shown, the application framework layer may include a window manager, a content provider, a view system, a phone manager, a resource manager, a notification manager, and the like.

[0438] The window manager is used to manage window programs. The window manager can obtain the display screen size, determine whether there is a status bar, lock the screen, capture the screen, etc.

[0439] Content providers are used to store and retrieve data and make it accessible to applications. The data may include videos, images, audio, calls made and received, browsing history and bookmarks, phone books, etc.

[0440] The view system includes visual controls, such as controls for displaying text and controls for displaying images. The view system can be used to build applications. The interface content can be composed of one or more views. For example, the display interface including the SMS notification icon can include a view for displaying text and a view for displaying images.

[0441] The phone manager is used to provide communication functions of the electronic device 200, such as management of call status (including connecting, hanging up, etc.).

[0442] The resource manager provides various resources for applications, such as localized strings, icons, images, layout files, video files, and so on.

[0443] The notification manager enables applications to display notification information in the status bar. It can be used to convey notification-type messages and can disappear automatically after a short stay without user interaction. For example, the notification manager is used to notify download completion, message reminders, etc. The notification manager can also be a notification that appears in the system top status bar in the form of a chart or scroll bar text, such as notifications of applications running in the background, or a notification that appears on the screen in the form of a dialog window. For example, a text message is displayed in the status bar, a prompt sound is emitted, an electronic device vibrates, an indicator light flashes, etc.

[0444] Android runtime includes core libraries and virtual machines. Android runtime is responsible for scheduling and management of the Android system.

[0445] The core library consists of two parts: one part is the function that needs to be called by the Java language, and the other part is the Android core library.

[0446] The application layer and the application framework layer run in a virtual machine. The virtual machine executes the Java files of the application layer and the application framework layer as binary files. The virtual machine is used to perform functions such as object life cycle management, stack management, thread management, security and exception management, and garbage collection.

[0447] The system library may include multiple functional modules, such as surface manager, media libraries, 3D graphics processing library (such as OpenGL ES), 2D graphics engine (such as SGL), etc.

[0448] The surface manager is used to manage the display subsystem and provide the fusion of 2D and 3D layers for multiple applications.

[0449] The media library supports playback and recording of a variety of commonly used audio and video formats, as well as static image files, etc. The media library can support a variety of audio and video encoding formats, such as: MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc.

[0450] The graphics processing library is used to implement drawing, image rendering, compositing, and layer processing of 2D or 3D graphics.

[0451] A 2D graphics engine is a drawing engine for 2D drawings.

[0452] The kernel layer is the layer between hardware and software. The kernel layer contains at least display driver, camera driver, audio driver, and sensor driver.

[0453] It should be noted that the second electronic device may have Figure 2 and Figure 3a The whole or part of the structure shown is not limited here. For ease of understanding, the following embodiments of the present application will be based on Figure 2 and Figure 3a Taking the first electronic device with the shown structure as an example, a screen projection method provided in an embodiment of the present application is specifically described in combination with the accompanying drawings and application scenarios.

[0454] In the embodiment of the present application, the second electronic device 200 may be an electronic device such as a television, a tablet computer, or a desktop computer. Next, taking the second electronic device as a television as an example, the screen projection method provided in the embodiment of the present application is described.

[0455] First, a process of establishing a screen projection connection between the first electronic device 100 and the second electronic device 200 is introduced.

[0456] It should be noted that the number of second electronic devices can be one or more, that is, the first electronic device can establish a screen projection connection with one second electronic device, or can establish a screen projection connection with multiple second electronic devices at the same time.

[0457] Next, it is explained how the first electronic device establishes a screen projection connection with the second electronic device.

[0458] In an optional manner, the first electronic device can achieve a screen projection connection with the second electronic device by selecting the screen projection control provided by the system.

[0459] Optionally, refer to Figure 4a , Figure 4a A schematic diagram of an interface of a first electronic device provided in an embodiment of the present application, such as Figure 4a As shown in FIG. 1 , the user's finger can slide downward on the top area of ​​the main interface of the first electronic device 100. When the first electronic device 100 detects a sliding operation on the main interface, the following is displayed: Figure 4bThe notification management interface 40 shown. Figure 4b As shown, the notification management interface 40 includes a mobile data icon 402, a wireless network icon 403, and a wireless screen projection icon 401, etc. When the user clicks the wireless screen projection icon 401 on the notification management interface 40, the first electronic device 100 can start the wireless screen projection function.

[0460] Optionally, in an embodiment of the present application, if the wireless network is not turned on when the user clicks the wireless screen projection icon 401 on the notification management interface 40, the first electronic device 100 can prompt or automatically turn on the wireless network and prompt the user to select a wireless network connection. In an embodiment of the present application, the first electronic device 100 automatically searches for a screen projection device (indicating an electronic device with the ability to send / receive screen projection) connected to the WIFI through wireless fidelity (WIFI). The first electronic device 100 can display the following on the notification management interface 40: Figure 4c The search / selection box 404 shown includes the names of one or more screen projection devices searched for, so as to prompt the user to select a screen projection device from the searched screen projection devices to establish a screen projection connection. After the user selects the screen projection device to be projected, the first electronic device 100 can establish a screen projection connection with the screen projection device selected by the user.

[0461] In an optional manner, the first electronic device can also achieve a screen projection connection with the second electronic device by selecting a screen projection control in some applications (for example, a video application or an application for presentation).

[0462] Optionally, refer to Figure 5a , Figure 5a A schematic diagram of an interface of a first electronic device provided in an embodiment of the present application, such as Figure 5a As shown in , the first electronic device 100 may be installed with multiple applications, such as a video application (application, APP), etc. Among them, the video APP can be used to watch videos, live broadcasts, novels and / or comics, etc. The video APP at least has a screen casting and sending function. The video APP may be pre-installed when the first electronic device 100 leaves the factory, or it may be installed after the user downloads it. The video APP may be a video application developed by the manufacturer of the first electronic device 100, or it may be a video APP developed by a third-party manufacturer.

[0463] like Figure 5aAs shown in the figure, the interface of the first electronic device may also include: a status bar, and icons of multiple other applications, such as icons of social APPs, etc. Among them, the status bar may include a WiFi icon, signal strength, current remaining power and / or current time, etc. In some feasible implementations, the status bar may also include a Bluetooth icon, an alarm clock icon, etc. When the first electronic device 100 detects a click event of a user's finger or stylus on a certain application icon, in response to the click event, the first electronic device 100 starts the application and displays the user interface corresponding to the application icon. For example, when the first electronic device 100 detects that the user's finger touches the icon 501 of the video APP, the first electronic device 100 starts the video APP in response to the touch event and displays the main interface of the video APP. The user can select a video on the main interface of the video APP, and accordingly, the first electronic device 100 can receive the video selected by the user and display the video playback interface.

[0464] Optionally, in one embodiment, after the first electronic device 100 starts the video APP, the user can search or select the video he wants to play in the main interface of the video APP, and the first electronic device 100 searches the cloud platform for the video name entered by the user in the search bar, and displays the icon of the searched video. After the user selects the video he wants to play, he can click the video icon to enter the video playback interface. The video APP of the first electronic device 100 obtains the resource (such as video stream) of the video selected by the user from the cloud platform, and parses and outputs the video image of the video. Figure 5b , Figure 5b Schematic diagram of the video playback interface of the video APP of the first electronic device provided in the embodiment of the present application. For example, the video selected by the user is the first episode of "Lovely Myself", and the corresponding video playback interface 50 can be as follows: Figure 5b As shown. Exemplarily, the video playback interface 50 may include a video image 504, a screen projection control 502, a selection 506, a return control 503, and a full screen control 505, etc.

[0465] In addition to the above elements and controls, the video playback interface 40 may also include more content, such as a cache control, a comment control, etc. The user may touch the cache control on the video playback interface 50 with a finger to download the video 400; or the user may touch the comment control on the video playback interface 50 with a finger to view comments, post comments, etc. It is understandable that the description of the above interface is only an example. Different video apps or different videos may have different corresponding video playback interfaces, elements and controls contained in the full-screen playback interface, and the arrangement of each element and control, which is not limited here.

[0466] If the user wants to cast the video to the second electronic device 200 for playback, the user can click the casting control 502 on the video playback interface 50 with a finger or a stylus, and accordingly, the first electronic device 100 can receive the casting instruction generated by the user clicking the casting control 502. The first electronic device 100 can detect whether the first electronic device 100 has currently established a casting connection. When the first electronic device 100 detects that the casting connection is not currently established, the first electronic device 100 can search for a casting device, display the searched casting device, and prompt the user to select a casting device.

[0467] Exemplarily, when the first electronic device 100 detects that no screen projection connection is currently established, the first electronic device 100 searches for one or more screen projection devices connected to the wireless network through the wireless network. The screen projection device involved in the embodiment of the present application is an electronic device with screen projection sending (Source) / receiving (Sink) capability. When the first electronic device 100 searches for one or more screen projection devices, refer to Figure 5c , Figure 5c The following is a schematic diagram of the interface content of a screen projection device selection provided in an embodiment of the present application, such as Figure 5c As shown in FIG. 1 , the first electronic device 100 may display on the interface Figure 5c The search / selection box 507 shown includes the names of one or more screen projection devices searched for, so as to prompt the user to select a screen projection device from the searched screen projection devices to establish a screen projection connection. Exemplarily, the search / selection box 507 includes the names of the devices searched for that have the ability to receive screen projection: the TV in Office 1, the TV in Office 2, and my computer. After the user selects the device to which he wants to project the screen, the first electronic device 100 establishes a screen projection connection with the device selected by the user through the Miracast protocol, a wireless display standard in the wireless video display (Wi-Fi display, WFD) technology. Exemplarily, if the screen projection device selected by the user is the TV in Office 1, then the first electronic device 100 establishes a screen projection connection with the TV in Office 1. For the convenience of description, the following description refers to the screen projection device selected by the user as the second electronic device 200.

[0468] It should be noted that the interface content selected by the projection device may also include a refresh control 509 and an exit control 508, which is not limited in this application.

[0469] Optionally, in another embodiment, the user can also activate the wireless screen projection function of the first electronic device 100 by clicking other controls on the first electronic device 100, which is not limited in the present application.

[0470] Next, the implementation process of establishing a screen projection connection between the first electronic device and the second electronic device is explained.

[0471] Optionally, in one embodiment, the user can establish a wired screen projection connection between the first electronic device 100 and the second electronic device 200 via a high definition multimedia interface (HDMI).

[0472] Optionally, in another embodiment, the first electronic device 100 can establish a screen projection connection with the second electronic device 200 through a signal distribution terminal. In this embodiment, the signal distribution terminal may include a signal distributor or a signal distributor, etc., and this application does not limit this. During operation, the electronic device serving as the signal distribution terminal can run an application to receive the screen projection data sent by the first electronic device, and further distribute the screen projection data to the second electronic device to implement the screen projection function in this application. Among them, the application can be a screen projection application dedicated to the screen projection function, or it can be other applications that include the screen projection function.

[0473] It should be noted that currently commonly used electronic devices can use conventional processing methods to achieve the function of supporting wireless screen projection services; for example, configuring smart devices to support wireless screen projection related protocols, currently commonly used protocols include Miracast, DLNA (digital living network alliance), AirPlay protocol, etc., among which the conventional processing method is to install a wireless screen projection application that complies with the smart operating system on the electronic device, so that the smart device can support the wireless screen projection service. Of course, other methods can also be used to configure the electronic devices (the first electronic device and the second electronic device) to support the wireless screen projection service, and this is not specifically limited in this application.

[0474] Optionally, in this embodiment, after the first electronic device establishes a screen projection connection with the second electronic device, the first electronic device can send the projection data to the second electronic device, and the second electronic device can display the projection data corresponding to the projection data sent by the first electronic device.

[0475] Optionally, in this embodiment, after the first electronic device establishes a screen projection connection with the second electronic device, the first electronic device can indirectly send the projection data to the second electronic device: the first electronic device can send the projection data to the signal distribution end, and then the signal distribution end can further send the projection data to the second electronic device.

[0476] In fact, this application is only for illustrating that the projection data comes from the first electronic device, and is finally obtained and displayed by the second electronic device. However, this application does not limit how the projection data is transmitted from the first electronic device to the second electronic device.

[0477] The above describes the process of establishing a screen projection connection between the first electronic device 100 and the second electronic device 200. Next, it describes how the user interacts with the projection content displayed on the second electronic device during the screen projection process based on the operation of the first electronic device.

[0478] Reference Figure 3b , Figure 3b A flowchart of an information processing method provided in an embodiment of the present application, such as Figure 3b As shown in, the information processing method includes:

[0479] 301. A first electronic device generates screen projection content.

[0480] In an embodiment of the present application, the first electronic device can generate screen projection content after detecting that the first electronic device establishes a screen projection connection with the second electronic device.

[0481] In the embodiment of the present application, the projection content may include but is not limited to a cursor and interface content of the first electronic device.

[0482] Optionally, in an embodiment of the present application, the first electronic device may obtain the interface content of the front-end application of the first electronic device, and generate a cursor on the interface content to obtain the projection content.

[0483] The shape of the cursor may be a mouse shape or other shapes, which is not limited in this application. It should be noted that the cursor may be used to locate the operation position in the interface content, and the cursor may move on the display screen of the second electronic device in response to the user's operation on the first electronic device (changing the gesture information or sliding the display screen of the first electronic device). How the cursor moves on the display screen of the second electronic device based on the user's operation on the first electronic device will be described in the following embodiments and will not be repeated here.

[0484] Specifically, the first electronic device can obtain the interface content of the current front-end application. For example, the first electronic device can obtain the interface content of the current front-end application based on the screen recording interface provided by the system (such as the MediaProjection interface provided by Android), and draw a cursor on the obtained interface content. The first electronic device can use the above-drawn content as the projection content, and the projection service of the first electronic device can obtain the projection content and send the projection content (which can be encoded and / or the size of the content is transformed) to the second electronic device based on a certain projection protocol, so that the display screen of the second electronic device displays the projection content.

[0485] Optionally, in an embodiment of the present application, the first electronic device can also add the image information corresponding to the cursor to the interface content of the current front-end application of the first electronic device based on the floating window interface to generate projection data. At this time, the projection service of the first electronic device can obtain the above-mentioned projection data and send the projection data (which needs to undergo encoding operations and content size transformation) to the second electronic device based on a certain projection protocol, so that the display screen of the second electronic device displays the projection content corresponding to the projection data.

[0486] It should be noted that after the second electronic device establishes a connection with the first electronic device, the cursor may be displayed directly on the display screen of the second electronic device, or the cursor may not be displayed directly on the display screen of the second electronic device. Instead, the cursor may be displayed on the display screen of the second electronic device only after the user performs certain operations on the first electronic device.

[0487] Specifically, after establishing a connection with the first electronic device, the second electronic device can directly display a cursor in the center area of ​​the displayed projection content or in other preset areas.

[0488] In addition, after establishing a connection with the first electronic device, the second electronic device may not display the cursor in the center area of ​​the displayed projection content or other predetermined areas, but may detect the posture information of the first electronic device. If a change in the posture information of the first electronic device is detected, the cursor is displayed in the center area of ​​the displayed projection content or other predetermined areas. From the user's perspective, if the user does not see the cursor on the display screen of the second electronic device, the user can change the posture information of the first electronic device by waving the first electronic device, thereby triggering the display of the cursor on the second electronic device.

[0489] Optionally, in one embodiment, after the first electronic device and the second electronic device establish a screen projection connection, the cursor may not be displayed on the second electronic device first, and the user can activate the cursor display of the second electronic device by performing a touch operation on the display screen of the first electronic device. The above touch operation may be a click operation, a double-click operation, or a long press operation on a preset area on the display screen of the first electronic device, which is not limited in the present application.

[0490] For example, refer to Figure 6a , Figure 6a This is an illustration of a practical scenario of screen projection, such as Figure 6a As shown in FIG. 1 , at this time, the user holds the first electronic device 100, and the second electronic device 200 displays the projection content 60 sent by the first electronic device 100, wherein the projection content 60 includes the current interface content of the first electronic device 100 (for example, Figure 4aIt should be noted that the interface content of the first electronic device 100 may be, but is not limited to, the interface content of the front-end application.

[0491] Optionally, in another embodiment, based on Figure 5c As shown in the figure, after the user selects the screen projection device, refer to Figure 6d , Figure 6d This is an illustration of a practical scenario of screen projection, such as Figure 6d As shown in FIG. 1 , the second electronic device 200 displays the projection content 60 sent by the first electronic device 100, wherein the projection content 60 includes the current interface content of the first electronic device 100 ( Figure 5b The video playback interface shown in FIG. 1 ), in addition, the projection content 60 also includes: a cursor 601.

[0492] In an embodiment of the present application, after the first electronic device detects that a screen projection connection is established with the second electronic device, the display brightness of the display screen of the first electronic device can be reduced, or the display screen of the first electronic device can be turned off.

[0493] Optionally, in the embodiments of the present application, refer to Figure 6b , Figure 6b This is an illustration of a practical scenario of screen projection, such as Figure 6b As shown in FIG. 1 , the interface content of the first electronic device 100 held by the user is Figure 4a The interface shown (the main interface of the first electronic device), and after the first electronic device and the second electronic device establish a screen projection connection, such as Figure 6c As shown in FIG. 1 , at this time, the first electronic device turns off the screen. It should be noted that at this time, the first electronic device may not display Figure 4a Page, Figure 4a It is only the page that the first electronic device should present. The first electronic device now displays Figure 6c .

[0494] Optionally, in another embodiment, after the first electronic device and the second electronic device establish a screen projection connection, a control for whether to enter the screen-off state can be displayed on the first electronic device (or through other predetermined operations), and the user can make the first electronic device enter the screen-off state by clicking the control for entering the screen-off state.

[0495] Compared to the prior art, when projecting the screen, the first electronic device needs to keep the screen on. In the embodiment of the present application, after the first electronic device and the second electronic device establish a screen projection connection, the first electronic device can reduce the display brightness of the display screen, or directly enter the screen off state, thereby reducing the energy consumption of the first electronic device.

[0496] It should be noted that in an embodiment of the present application, after the first electronic device detects that a screen projection connection has been established with the second electronic device, the display brightness of the display screen of the first electronic device can be reduced through the system's brightness adjustment interface, or the display screen of the first electronic device can be turned off.

[0497] It should be noted that in the embodiment of the present application, the first electronic device is only in a screen-off state. At this time, the application in the first electronic device is still running, and the screen projection control in the first electronic device can still obtain the interface content of the current front-end application.

[0498] 302. The first electronic device sends the projected screen content to the second electronic device so that the display screen of the second electronic device displays the projected screen content, wherein the projected screen content includes a cursor and interface content of the first electronic device, and the cursor is used to locate an operation position in the interface content.

[0499] In an embodiment of the present application, after the first electronic device generates the projected screen content, it can send the projected screen content to the second electronic device so that the display screen of the second electronic device displays the projected screen content. The projected screen content includes a cursor and the interface content of the first electronic device, and the cursor is used to locate the operation position in the interface content.

[0500] The implementation method of step 302 can refer to the relevant description of the first electronic device sending the projection data to the second electronic device in the above embodiment, which will not be repeated here.

[0501] In the embodiment of the present application, in order to reduce the energy consumption of the first electronic device, after the first electronic device and the second electronic device establish a screen projection connection, the first electronic device enters a screen-off state. At this time, the display screen of the first electronic device is black, and the user cannot operate the objects on the first electronic device by operating on the interface content. For example, Figure 6d In the scenario shown, the user wants to click on the return control or drag the progress bar. However, because the interface content of the first electronic device is black, the user cannot locate the return control and progress bar in the first electronic device on the display screen of the first electronic device, and thus cannot perform the operation of clicking on the return control or dragging the progress bar. Next, the embodiment of the present application is discussed, when the first electronic device is in the off state, how the user operates the screen projection content displayed by the second electronic device.

[0502] First, it is introduced how the user adjusts the display position of the cursor on the display screen of the second electronic device.

[0503] In an embodiment of the present application, the cursor displayed on the second electronic device can be used to locate the operation position in the interface content. Therefore, the user can change the operation position by changing the display position of the cursor on the display screen of the second electronic device. The following describes how the user changes the display position of the cursor on the display screen of the second electronic device.

[0504] In an embodiment of the present application, the user can adjust the display position of the cursor on the display screen of the second electronic device by changing the posture of the first electronic device, or can adjust the display position of the cursor on the display screen of the second electronic device by performing a sliding operation on the display screen of the first electronic device.

[0505] 1. Adjusting the display position of the cursor on the display screen of the second electronic device by changing the posture of the first electronic device.

[0506] In an embodiment of the present application, the first electronic device can obtain the posture change information of the first electronic device, and move the cursor in the projected content displayed by the second electronic device based on the posture change information.

[0507] In the embodiment of the present application, a cursor 601 is displayed on the screen projection content of the second electronic device, and the cursor 601 can be displaced accordingly with the change of the posture of the first electronic device. Specifically, the user can change the posture of the first electronic device in the three-dimensional space by waving the first electronic device, and then the cursor 601 on the second electronic device can be displaced accordingly on the screen projection content of the second electronic device according to the change of the posture of the first electronic device in the three-dimensional space.

[0508] Optionally, refer to Figure 7a , Figure 7a This is an illustration of a practical scenario of screen projection, such as Figure 7a As shown in the figure, the user can change the posture (horizontal angle and / or vertical angle) of the first electronic device by waving the first electronic device, and accordingly, the cursor 601 displayed on the second electronic device 200 can be displaced accordingly in the horizontal direction and / or vertical direction.

[0509] Next, it is described how the cursor 601 displayed on the second electronic device 200 is displaced in the horizontal direction and the vertical direction.

[0510] 1) Horizontal direction:

[0511] In an embodiment of the present application, the first electronic device can determine the horizontal displacement of the cursor according to the second rotation angle, and move the cursor in the projected content displayed by the second electronic device according to the horizontal displacement.

[0512] Specifically, refer to Figure 7b , Figure 7b FIG. 1 is a diagram showing a posture change of a first electronic device rotating in a horizontal direction. Figure 7b As shown in FIG. 1 , the first electronic device rotates from posture 1 to posture 2 on the horizontal plane, and the rotation angle change component is θ1. In the embodiment of the present application, the first electronic device 100 can monitor the posture change in the three-dimensional space and obtain the spatial orientation parameter to obtain the spatial orientation parameter. Figure 7a Taking the posture change shown as an example, the first electronic device 100 can obtain a spatial orientation parameter (angle change component of horizontal rotation) as θ1.

[0513] In the embodiment of the present application, the first electronic device may obtain a mapping relationship between the angle change component of the horizontal rotation and the size L1 of the cursor 601 in the second electronic device in the horizontal displacement direction, wherein the mapping relationship may indicate that the larger the angle change component of the horizontal rotation of the first electronic device, the larger the size L1 of the horizontal displacement direction of the cursor 601 in the second electronic device may be. The first electronic device may determine the displacement size L1 of the cursor 601 in the screen projection content of the second electronic device based on the obtained angle change component of the horizontal rotation and the mapping relationship.

[0514] Exemplarily, the mapping relationship between the angle change component of the horizontal rotation obtained by the first electronic device and the horizontal displacement of the cursor 601 in the second electronic device is: for every 1° rotation of the first electronic device on the horizontal plane, the horizontal pixel coordinates of the cursor 601 in the projected content of the second electronic device change by 30 pixels.

[0515] For example, if the user waves the first electronic device so that the first electronic device rotates 15° on the horizontal plane, the horizontal pixel coordinates of the cursor 601 in the projection content of the second electronic device change by 450 pixels.

[0516] It should be noted that, during the process of establishing a projection connection between the first electronic device 100 and the second electronic device 200, the first electronic device 100 and the second electronic device 200 will exchange and negotiate performance parameters, that is, the first electronic device 100 can obtain the size parameters of the interface content on the second electronic device 200, etc. The first electronic device can adjust the above mapping relationship based on the size parameters of the interface content on the second electronic device 200. For example, the larger the horizontal size of the interface content on the second electronic device 200, the greater the change in the horizontal pixel coordinates of the cursor 601 in the projection content of the second electronic device when the first electronic device rotates in the horizontal direction with the same angular change component.

[0517] It should be noted that the first electronic device can rotate left or right (counterclockwise or clockwise from the perspective of the main interface of the first electronic device), and accordingly, the cursor 601 can be displaced left or right in the projection content of the second electronic device. For example, the first electronic device can rotate left, and accordingly, the cursor 601 can be displaced left in the projection content of the second electronic device. For another example, the first electronic device can rotate right, and accordingly, the cursor 601 can be displaced right in the projection content of the second electronic device.

[0518] Reference Figure 7c , Figure 7c The displacement diagram of a cursor displayed by a second electronic device is shown as follows: Figure 7c As shown in , when the first electronic device obtains that the angle change component of the horizontal rotation in the three-dimensional space is θ1, the first electronic device can determine that the horizontal pixel coordinate change of the cursor 601 in the screen projection content of the second electronic device is L1 according to the above mapping relationship, and send the information carrying the horizontal pixel coordinate change L1 to the second electronic device. Correspondingly, the second electronic device can change the display position of the cursor 601 based on the horizontal pixel coordinate change L1. At this time, since the first electronic device rotates to the left (counterclockwise) when rotating from posture 1 to posture 2, the cursor 601 is also displaced to the left on the screen projection content of the second electronic device, ensuring the consistency of the movement of the cursor 601 and the user operation.

[0519] 2) Vertical direction:

[0520] In an embodiment of the present application, the first electronic device can determine the vertical displacement of the cursor according to the second rotation angle, and move the cursor in the projected content displayed by the second electronic device according to the vertical displacement.

[0521] Similarly, refer to Figure 7d , Figure 7d This is an illustration of a practical scenario of screen projection, such as Figure 7d As shown in FIG. 1 , the user can change the posture (vertical angle) of the first electronic device by waving the first electronic device, and accordingly, the cursor 601 displayed on the second electronic device 200 can be displaced in the vertical direction accordingly.

[0522] It should be noted that the so-called change in the vertical angle of the first electronic device in the three-dimensional space may refer to the angle change component of the vertical rotation of the first electronic device in the three-dimensional space.

[0523] It should be noted that the so-called change in the vertical angle of the first electronic device in the three-dimensional space may refer to the angle change component of the vertical rotation of the first electronic device in the three-dimensional space. Figure 7e , Figure 7e The figure is a schematic diagram of a posture change of a first electronic device rotating in a vertical direction, such as Figure 7b As shown in FIG. 1 , the first electronic device rotates from posture 1 to posture 2 in the vertical direction, and the rotation angle change component is θ2. In the embodiment of the present application, the first electronic device 100 can monitor the posture change in the three-dimensional space and obtain the spatial orientation parameter to obtain the spatial orientation parameter. Figure 7a Taking the posture change shown as an example, the first electronic device 100 can obtain a spatial orientation parameter (angle change component of vertical rotation) as θ2.

[0524] In an embodiment of the present application, the first electronic device can obtain the mapping relationship between the angular change component of the vertical rotation and the vertical displacement size L2 of the cursor 601 in the second electronic device, and determine the displacement size L2 of the cursor 601 in the projected content of the second electronic device based on the obtained angular change component of the vertical rotation and the mapping relationship.

[0525] Regarding how the first electronic device determines the displacement size L2 of the cursor 601 in the projection content of the second electronic device, reference may be made to the description in the above embodiment, which will not be repeated here.

[0526] Reference Figure 7f , Figure 7f The displacement diagram of a cursor displayed by a second electronic device is shown as follows: Figure 7f As shown in , when the first electronic device obtains that the angle change of the vertical rotation in the three-dimensional space is θ2, the first electronic device can determine that the vertical pixel coordinate change of the cursor 601 in the screen projection content of the second electronic device is L2 according to the above mapping relationship, and send the information carrying the vertical pixel coordinate change L2 to the second electronic device. Correspondingly, the second electronic device can change the display position of the cursor 601 based on the vertical pixel coordinate change L2. At this time, since the first electronic device rotates upward when rotating from posture 1 to posture 2, the cursor 601 is also displaced upward on the screen projection content of the second electronic device, ensuring the consistency of the movement of the cursor 601 and the user operation.

[0527] Optionally, in another embodiment, the first electronic device may obtain the displacement size of the first electronic device in the first horizontal direction, and determine the horizontal displacement of the cursor on the interface content based on the translation size of the first electronic device in the horizontal direction. The first horizontal direction may be a horizontal direction parallel to the display screen of the second electronic device.

[0528] In an embodiment of the present application, the first electronic device can obtain the mapping relationship between the horizontal displacement size and the horizontal displacement of the cursor 601 in the second electronic device, and determine the horizontal displacement of the cursor 601 in the projected content of the second electronic device based on the obtained horizontal displacement size and the mapping relationship.

[0529] Optionally, in another embodiment, the first electronic device may obtain the displacement size of the first electronic device in the first vertical direction, and determine the vertical displacement of the cursor on the interface content based on the translation size of the first electronic device in the vertical direction. The first horizontal direction may be a vertical direction perpendicular to the display screen of the second electronic device.

[0530] In an embodiment of the present application, the first electronic device can obtain the mapping relationship between the vertical displacement size and the vertical displacement of the cursor 601 in the second electronic device, and determine the vertical displacement of the cursor 601 in the projected content of the second electronic device based on the obtained vertical displacement size and the mapping relationship.

[0531] It should be noted that, in practical applications, the first electronic device may monitor posture changes and obtain spatial orientation parameters through its built-in gravity acceleration sensor (such as a gyroscope), or may monitor posture changes and obtain spatial orientation parameters through infrared space detection technology or sound wave detection technology. This embodiment does not specifically limit the means by which the first electronic device obtains spatial orientation parameters.

[0532] 2. The user can adjust the position of the cursor 601 on the screen projection content of the second electronic device by performing a sliding operation on the display screen of the first electronic device.

[0533] In an embodiment of the present application, the first electronic device can receive a second sliding operation on the display screen of the first electronic device, determine the displacement of the cursor according to the second sliding operation, and move the cursor in the projected content displayed by the second electronic device based on the displacement.

[0534] It should be noted that the trajectory of the second sliding operation passes through one or more objects in the interface content of the first electronic device, wherein the above-mentioned "one or more objects" may be all objects passed by the trajectory of the second sliding operation.

[0535] Specifically, in the embodiment of the present application, a cursor 601 may be displayed on the screen projection content of the second electronic device, and the cursor 601 may be displaced accordingly based on the sliding operation of the user on the display screen (in the off state) of the first electronic device. Specifically, the user may slide on the display screen with a finger or a touch pen, and accordingly, the cursor 601 on the second electronic device may be displaced accordingly on the screen projection content of the second electronic device based on the sliding operation of the user.

[0536] Optionally, refer to Figure 7g , Figure 7g This is an illustration of a practical scenario of screen projection, such as Figure 7g As shown in the figure, the user can slide on the display screen with a finger or a touch pen, and accordingly, the cursor 601 on the second electronic device can be displaced accordingly on the projected content of the second electronic device based on the user's sliding operation.

[0537] In an embodiment of the present application, the first electronic device can obtain the mapping relationship between the user's sliding displacement L3 on the display screen of the first electronic device and the displacement size L3 of the cursor 601 in the second electronic device, and determine the displacement size L3 of the cursor 601 in the projected content of the second electronic device based on the obtained sliding displacement L3 on the display screen of the first electronic device and the mapping relationship.

[0538] Exemplarily, the mapping relationship between the sliding displacement L3 on the display screen of the first electronic device obtained by the first electronic device and the displacement size L3 of the cursor 601 in the second electronic device is: every time the user slides 1 pixel on the display screen of the first electronic device, the pixel coordinates of the cursor 601 in the projection content of the second electronic device change by 30 pixels. It should be noted that the above is only an example and does not constitute a limitation of the present application.

[0539] It should be noted that, during the process of establishing a projection connection between the first electronic device 100 and the second electronic device 200, the first electronic device 100 and the second electronic device 200 will exchange and negotiate performance parameters, that is, the first electronic device 100 can obtain the size parameters of the interface content on the second electronic device 200, etc. The first electronic device can adjust the above mapping relationship based on the size parameters of the interface content on the second electronic device 200. For example, the larger the horizontal size of the interface content on the second electronic device 200, the greater the pixel displacement (pixel coordinate change) of the cursor 601 in the projection content of the second electronic device when the user slides the same displacement on the display screen of the first electronic device.

[0540] It should be noted that the sliding displacement of the user on the display screen of the first electronic device may include displacement in two directions (x direction and y direction) perpendicular to each other. Figure 7h , Figure 7h is a user's sliding operation indication, correspondingly, Figure 7iAs shown in , the sliding displacement L3 of the user on the display screen of the first electronic device may include a displacement L5 in the first direction and a displacement L6 in the second direction, wherein the first direction is perpendicular to the central axis of the first electronic device, and the second direction is parallel to the central axis of the first electronic device. Correspondingly, the pixel displacement of the cursor 601 in the projection content of the second electronic device includes a displacement component in the horizontal direction and a displacement component in the vertical direction. At this time, the first electronic device can determine the horizontal displacement size and the vertical displacement size of the cursor 601 in the projection content of the second electronic device based on the above mapping relationship.

[0541] Reference Figure 7j , Figure 7j The displacement diagram of a cursor displayed by a second electronic device is shown as follows: Figure 7j As shown in , when the first electronic device obtains the sliding displacement L3 of the user on the display screen of the first electronic device, the first electronic device can determine that the displacement of the cursor 601 in the screen projection content of the second electronic device is L4 according to the above mapping relationship, and send the information carrying L4 to the second electronic device. Correspondingly, the second electronic device can change the display position of the cursor 601 based on L4. At this time, since the user slides diagonally upward to the right on the display screen of the first electronic device, the cursor 601 is also diagonally upward to the right on the screen projection content of the second electronic device, ensuring the consistency of the movement of the cursor 601 and the user operation.

[0542] In the embodiment of the present application, the operation position corresponding to the second sliding operation corresponds to a third object in the interface content of the first electronic device, and the first electronic device can shield the third object from responding to the second touch operation.

[0543] In the embodiment of the present application, the user can realize the movement of the cursor 601 on the display screen of the second electronic device based on the operation of the first electronic device (changing the posture or sliding on the display screen of the first electronic device) in the above manner.

[0544] Next, it is described how the user can perform operations such as clicking and sliding on the object where the cursor is located by operating the first electronic device.

[0545] In an embodiment of the present application, when the cursor moves to the target object in the projected content displayed by the second electronic device, if the first electronic device receives a first touch operation on the display screen of the first electronic device, the target object in the projected content displayed by the second electronic device responds to the first touch operation.

[0546] Exemplarily, the first touch operation may be a click operation or a slide operation.

[0547] In the embodiment of the present application, the operation position of the first touch operation corresponds to a second object in the interface content of the first electronic device, and the first electronic device can shield the second object from responding to the first touch operation.

[0548] Specifically, after receiving the first touch operation on the display screen of the first electronic device, the first electronic device can shield the interface content from responding to the operation position of the first touch operation, and determine the position of the cursor on the second display screen to determine the response position of the interface content to the first touch operation. For example, if the user performs a click operation on object A on the display screen of the first electronic device, and the cursor is located on object B in the second electronic device, the first electronic device can shield the response of object A to the click operation, and respond to the click operation on object B of the interface content.

[0549] The first touch operation may be, but is not limited to, a click operation or a slide operation. The following description will be made by taking the first touch operation as a click operation as an example.

[0550] In an embodiment of the present application, after the user moves the cursor 601 on the projected content displayed on the second electronic device to the area where the target object to be operated is located, the user can click on the display screen of the first electronic device, and accordingly, the click operation can be performed in the area where the cursor 601 is located.

[0551] Specifically, in one embodiment, after the cursor 601 moves to the object area to be operated on the projected content of the second electronic device, the first electronic device can obtain the specific pixel coordinate position of the cursor 601 in the projected content of the second electronic device, and determine the pixel coordinate position corresponding to the position in the interface content of the front-end application of the first electronic device based on the specific pixel coordinate position of the cursor 601 in the projected content of the second electronic device.

[0552] For example, if the user moves the cursor 601 in the second electronic device to the area of ​​the video APP icon, the first electronic device can determine the pixel coordinate position corresponding to the interface content of the cursor 601 in the first electronic device (the pixel coordinate position of the area of ​​the video APP icon).

[0553] The user performs a click operation on the display screen of the first electronic device. At this time, the first electronic device can shield the response of the interface content of the front-end application of the first electronic device to the operation position of the click operation. The first electronic device will perform an event corresponding to the click operation at the pixel coordinate position corresponding to the interface content of the cursor 601 in the first electronic device, which is equivalent to making the first electronic device perform the above-mentioned click operation at the pixel coordinate position corresponding to the cursor 601 in the first electronic device.

[0554] For example, if the user moves the cursor 601 in the second electronic device to the area of ​​the video APP icon and performs a click operation on the display screen of the first electronic device, the first electronic device can perform a click operation on the video APP in the current interface content and send the opened interface (screen projection data) to the second electronic device, and the second electronic device can display the received screen projection data.

[0555] Next, the description is given by taking the sliding operation as the first touch operation as an example.

[0556] In an embodiment of the present application, the first electronic device can generate a sliding event corresponding to the first sliding operation, and execute the sliding event on the target object in the interface content of the first electronic device, so that the target object in the projection content displayed by the second electronic device responds to the first sliding operation.

[0557] For example, if the user moves the cursor 601 in the second electronic device to the area of ​​the video APP icon and performs a sliding operation on the display screen of the first electronic device, the first electronic device can perform a drag operation on the video APP icon in the current interface content.

[0558] It should be noted that, in addition to the cursor 601 , a menu bar may also be displayed on the display screen of the second electronic device.

[0559] In the embodiment of the present application, the first electronic device can obtain the interface content of the front-end application of the first electronic device, and generate a cursor and a menu bar on the interface content to obtain the projection content; wherein the menu bar does not belong to the interface content of the first electronic device. In other words, the menu bar is not the original interface content on the first electronic device, but a newly added content.

[0560] Specifically, the first electronic device can add a menu bar to the interface content of the current front-end application of the first electronic device based on the floating window interface to generate projection content. At this time, the projection service of the first electronic device can obtain the above projection content and send the projection content (which needs to undergo encoding operations and content size transformation) to the second electronic device based on a certain projection protocol, so that the display screen of the second electronic device displays the projection content.

[0561] Optionally, the first electronic device may also obtain the interface content of the current front-end application. For example, the first electronic device may obtain the interface content of the current front-end application based on the screen recording interface provided by the system (such as the MediaProjection interface provided by Android), and draw a menu bar on the obtained interface content. The first electronic device may use the above-drawn content as the projection content, and the projection service of the first electronic device may obtain the projection content and send the projection content (which needs to be encoded and / or the size of the content is transformed) to the second electronic device based on a certain projection protocol, so that the display screen of the second electronic device displays the projection content.

[0562] In the embodiment of the present application, the screen projection content of the second electronic device may also include a menu bar that does not belong to the interface content of the first electronic device, for example, Figure 8a The functional control 800 shown in Figure 8a As shown in FIG. 1 , the user can change the posture of the first electronic device 100 to move the cursor 601 on the second electronic device to the area where the function control 800 is located, and perform a click operation 80. The first electronic device can respond to the click operation 80 by performing a click operation on the target object where the cursor 601 is located, and display the function selection area 801. Accordingly, referring to FIG. Figure 8b , Figure 8b The following is a schematic diagram of the screen projection content of a second electronic device provided in an embodiment of the present application, such as Figure 8b As shown in FIG. 8 , the second electronic device displays a function selection area 801, wherein the function selection area 801 may include a sensitivity adjustment control, a mouse size adjustment control, a sliding mode control 802, a brightness adjustment control, and a normal mode control.

[0563] For example, the user can click the sliding mode control 802 to implement Figure 7a and Figure 7d The gesture-based interaction model shown in Figure 7g The mode switching between the sliding interaction modes shown in FIG. Figure 8b As shown, the user can click the sliding mode control 802, and then the mode of changing the position of the cursor 601 based on adjusting the posture of the first electronic device can be switched to Figure 7g It should be noted that, in one embodiment, after switching to the sliding interaction mode, the sliding mode control 802 in the function selection area 801 can be replaced by a gesture mode control. If the user clicks the gesture mode control, the gesture mode control can be replaced by Figure 7g The slide interaction mode shown in the figure switches to Figure 7a and Figure 7d The gesture-based interaction mode shown in .

[0564] It should be noted that after the first electronic device and the second electronic device establish a screen projection connection, the default setting can be Figure 7a and Figure 7d The gesture-based interaction mode shown in , can also be used by default Figure 7g The sliding interaction mode shown in is not limited in this application.

[0565] Reference Figure 9a , Figure 9a The following is a schematic diagram of the screen projection content of a second electronic device provided in an embodiment of the present application, such as Figure 9a As shown in FIG. 1 , the user can click the sensitivity adjustment control 901, and accordingly, the second electronic device can display the sensitivity adjustment area 90. Figure 9b , Figure 9b The following is a schematic diagram of the screen projection content of a second electronic device provided in an embodiment of the present application, such as Figure 9b As shown in the figure, the second electronic device displays a sensitivity adjustment area 90, which may include a sliding control, and the user can adjust the sensitivity of the cursor 601 by dragging the sliding control. In addition, the sensitivity adjustment area 90 may also include a sensitivity size prompt. It should be noted that the arrangement of the above interface and controls is only a schematic, and this application does not limit it.

[0566] Optionally, in a mode of adjusting the position of the cursor 601 based on the posture of the first electronic device, the first electronic device can obtain the angle change of the horizontal or vertical rotation, and determine the displacement of the cursor 601 in the projection content of the second electronic device based on the mapping relationship between the angle change in the horizontal direction and the horizontal displacement of the cursor 601 in the second electronic device, and the mapping relationship between the angle change in the vertical direction and the vertical displacement of the cursor 601 in the second electronic device. The mapping relationship between the above-mentioned angle change in the horizontal direction and the horizontal displacement of the cursor 601 in the second electronic device, and the mapping relationship between the angle change in the vertical direction and the vertical displacement of the cursor 601 in the second electronic device can be adjusted through the above-mentioned sensitivity adjustment, so that when the first electronic device undergoes the same posture change, the displacement of the cursor 601 on the second electronic device changes.

[0567] For example, if the user feels that the cursor 601 moves too slowly when operating the cursor 601, the user can drag the cursor 601 to the right. Figure 9b The sliding control shown increases the sensitivity of the cursor 601. On the contrary, if the user feels that the cursor 601 moves too fast when operating the cursor 601, the user can drag the left Figure 9b The slide control shown reduces the sensitivity of cursor 601.

[0568] For example, when the sensitivity is 40, when the first electronic device rotates 1° on the horizontal plane, the horizontal pixel coordinates of the cursor 601 in the screen projection of the second electronic device change by 30 pixels, and when the first electronic device rotates 1° on the vertical plane, the vertical pixel coordinates of the cursor 601 in the screen projection of the second electronic device change by 50 pixels. Figure 9b When the sensitivity of the sliding control shown is adjusted to 58, the horizontal pixel coordinates of the cursor 601 in the projection content of the second electronic device change by 45 pixels for every 1° rotation of the first electronic device in the horizontal plane, and the vertical pixel coordinates of the cursor 601 in the projection content of the second electronic device change by 75 pixels for every 1° rotation of the first electronic device in the vertical plane. It should be noted that the description of the above mapping relationship is only an example and does not constitute a limitation of the present application.

[0569] Optionally, in one embodiment, after the first electronic device and the second electronic device establish a screen projection connection, the first electronic device can add the image data corresponding to the function control 800 to the current interface content through the floating window interface, so that the function control 800 is superimposed on the current interface content of the first electronic device. At this time, the screen projection service of the first electronic device can obtain the projection data of the interface content including the above-mentioned function control 800, and send the above-mentioned projection data to the second electronic device based on a certain screen projection protocol. Accordingly, the interface content including the function control 800 can be displayed on the second electronic device, and the first electronic device can obtain the user's operation (sliding / changing the posture of the first electronic device) to change the position of the cursor 601 in the current interface content. When the user controls the second electronic device, When the cursor 601 on the sub-device moves within the range of the function control 800, the first electronic device can determine that the position of the cursor 601 in the current interface content is within the range of the function control 800. If the user clicks on the display screen of the first electronic device, the first electronic device can add the image data corresponding to the function selection area 801 to the current interface content through the floating window interface, so that the function selection area 801 is superimposed on the current interface content of the first electronic device. At this time, the screen projection service of the first electronic device can obtain the projection data of the interface content including the above-mentioned function selection area 801, and send the above-mentioned projection data to the second electronic device based on a certain screen projection protocol. Accordingly, the interface content including the function selection area 801 can be displayed on the second electronic device.

[0570] by Figure 8bTaking the function selection area 801 shown as an example, the function selection area 801 may include a sensitivity adjustment control, a mouse size adjustment control, a sliding mode control 802, a brightness adjustment control, and a normal mode control. At this time, the first electronic device can obtain the position of each control in the function selection area 801 in the current interface content. The user operates the first electronic device to adjust the position of the cursor 601, and the first electronic device can obtain the position of the cursor 601. If the position of the cursor 601 is within the area of ​​a control in the function selection area 801, and the first electronic device detects the user's click operation on the display screen, the first electronic device can respond to the user's operation, for example, Figure 9a As shown, if the user clicks on the sensitivity control, in response to the user's click operation, the first electronic device can add the image data corresponding to the sensitivity adjustment area 90 to the current interface content through the floating window interface, so that the sensitivity adjustment area 90 is superimposed on the current interface content of the first electronic device, and the original function selection area 801 is removed (or the sensitivity adjustment area 90 is superimposed on the function selection area 801). At this time, the screen projection service of the first electronic device can obtain the projection data of the interface content including the above-mentioned sensitivity adjustment area 90, and send the above-mentioned projection data to the second electronic device based on a certain screen projection protocol. Accordingly, the second electronic device can display the following. Figure 9b The interface content including the sensitivity adjustment area 90 is shown in FIG. At this time, the user can slide on the display screen of the first electronic device and drag the slide button in the sensitivity adjustment area 90. Accordingly, the first electronic device responds to the user's sliding operation and adds the image data corresponding to the dragged slide button to the current interface content through the floating window interface. At the same time, based on the user's drag size, the relevant parameters of the sensitivity of the user's operation cursor 601 are modified. The above description is only an example and does not constitute a limitation of the present application.

[0571] It should be noted that the controls included in the above-mentioned function selection area 801 are only for illustration. In actual applications, a control for ending screen projection can also be set. The user can move the cursor in the second electronic device to the control for ending screen projection and click on the display screen of the first electronic device to trigger the first electronic device to end the screen projection connection with the second electronic device.

[0572] Optionally, the user may also trigger the first electronic device to end the screen projection connection with the second electronic device by other means, such as pressing the power button, etc., which is not limited here.

[0573] Optional, see Fig.9c , Fig.9c The following is a schematic diagram of the screen projection content of a second electronic device provided in an embodiment of the present application, such as Fig.9cAs shown in the figure, the user can click the normal mode control 902, and accordingly, the first electronic device can change the interaction mode with the user to the normal interaction mode. At this time, the display screen of the first electronic device can light up, and the user can operate normally on the display screen of the first electronic device.

[0574] Optional, see Figure 9d , Figure 9d An example of a user operating a first electronic device provided in an embodiment of the present application is as follows: Figure 9d As shown in the figure, the user can press the power button of the first electronic device, and accordingly, the first electronic device can change the interaction mode with the user to the normal interaction mode. At this time, the display screen of the first electronic device can light up, and the user can operate normally on the display screen of the first electronic device.

[0575] It should be noted that the above method of switching the first electronic device to the normal interaction mode is only an illustration and does not constitute a limitation to the present application.

[0576] Next, the switching of the first electronic device to the normal interaction mode is described in conjunction with a specific scenario.

[0577] Reference Fig.10a , Fig.10a The following is a schematic diagram of the screen projection content of a second electronic device provided in an embodiment of the present application, such as Fig.10a As shown in the figure, when the first electronic device receives a chat message sent by other electronic devices, a chat prompt window will be displayed on the second electronic device (the data in the prompt window is sent by the first electronic device, and the specific details can be referred to the above embodiment, which will not be repeated here).

[0578] like Fig.10a As shown in FIG. 6 , the user can move the cursor 601 into the area of ​​the prompt window and perform a click operation on the first electronic device. Accordingly, the second electronic device can display the following Fig.10b The chat interface shown in (the implementation of this process is that the first electronic device detects the user's click operation and performs the above-mentioned click event at the position of the cursor. The specific details can be referred to the above-mentioned embodiment and will not be repeated here), the chat interface includes an input keyboard. At this time, if the sentence is input by adjusting the position of the cursor displayed on the second electronic device in combination with the click operation on the first electronic device, the user experience will be very poor (because the display screen size of the second electronic device will be very large, and the user is not used to using the input keyboard through the above-mentioned interaction method).

[0579] Reference Fig.10c , Fig.10c An example of interaction of a first electronic device provided in an embodiment of the present application is as follows: Fig.10cAs shown in FIG, the user can press the power button of the first electronic device, and accordingly, the first electronic device can change the interaction mode with the user to the normal interaction mode. At this time, the display screen of the first electronic device can light up, and the user can operate normally on the display screen of the first electronic device. Fig.10d , Fig.10d The following is a schematic diagram of the interface content of a first electronic device provided in an embodiment of the present application, such as Fig.10d As shown in , the user can directly operate on the input keyboard displayed by the first electronic device. Optionally, after completing the input of the sentence, the power button of the first electronic device can be pressed again to switch the first electronic device to the screen-off and the above-mentioned interaction mode.

[0580] It should be noted that the above is only a schematic description of a scenario. In actual application, as long as the scenario requires operation through the display screen of the first electronic device, the user can switch to the normal mode of screen projection through the above method.

[0581] Optionally, after the first electronic device establishes a screen projection connection with the second electronic device, it can receive a call request from the other electronic device (for example, a voice call request or a video call request). Taking the call request as a voice call request as an example, the interface content corresponding to the call request of the first electronic device can be displayed on the second electronic device. The interface content includes a call receiving control and a call rejecting control.

[0582] Optionally, the user can operate the first electronic device to move the cursor displayed on the second electronic device to the receive call control and click on the display screen of the first electronic device, and then the user can make a call through the first electronic device.

[0583] Optionally, the user can switch the current interaction mode to a normal mobile phone interaction mode by pressing the power button of the first electronic device or other means (by clicking on the display screen of the first electronic device, moving the cursor displayed on the second electronic device to the receive call control, and clicking the receive call control on the display screen of the first electronic device, and then the user can make a call through the first electronic device.

[0584] Next, it is described how to combine the above sensitivity adjustment so that when the user waves the first electronic device, the front direction of the first electronic device can be as consistent as possible with the display position of the cursor on the second electronic device.

[0585] In the embodiment of the present application, after the first electronic device establishes a screen projection connection with the second electronic device, the first electronic device can set the cursor 601 to a preset position in the interface content, for example, the geometric center of the interface content. Fig.11a , Fig.11aThis is a schematic diagram of an interaction between a first electronic device and a second electronic device. After the first electronic device and the second electronic device establish a screen projection connection, the first electronic device can set the cursor 601 at the geometric center of the interface content, and correspondingly, the display content of the second electronic device can be as follows: Fig.11a As shown in , however, since when the first electronic device and the second electronic device establish a screen projection connection, the front end of the first electronic device may not be facing the center position of the second electronic device. At this time, the front end of the first electronic device is facing leftward. At this time, the user waves the first electronic device to the right, so that the display position of the cursor on the second electronic device moves to the right. At this time, when the user controls the cursor to move, the orientation of the first electronic device is biased to the left, which does not conform to the user's ideal operation method (the ideal user operation method is: the orientation of the front end of the first electronic device can be as consistent as possible with the display position of the cursor on the second electronic device). In order to solve the above technical problems, when the user waves the first electronic device to the right so that the front end of the first electronic device is near the center position of the second electronic device, as shown in . Fig.11b As shown in FIG. 1 , the cursor on the second electronic device is displayed at the border of the right end. At this time, the user can continue to wave the first electronic device to the right, such as Fig.11c As shown in FIG. 1 , the cursor can stay at the right edge of the second electronic device. After the user rotates the first electronic device to the right by a certain angle, the user can rotate the first electronic device to the left. Fig.11d As shown, the cursor on the second electronic device moves to the left, and when the front end of the first electronic device is toward the center of the second electronic device, the cursor is displayed on the second electronic device further to the left than before the adjustment. When the user moves the front end of the first electronic device toward the center of the second electronic device in the above manner, the cursor is adjusted to be located near the center of the second electronic device, and combined with the above sensitivity adjustment, when the user waves the first electronic device, the front end of the first electronic device can be oriented as closely as possible to the display position of the cursor on the second electronic device.

[0586] In an embodiment of the present application, the first electronic device can obtain the size of the display area of ​​the display screen of the second electronic device, and then the first electronic device can determine whether the cursor has moved to the boundary of the display area of ​​the second electronic device based on the size of the display area of ​​the display screen of the second electronic device and the pixel coordinate position of the cursor of the projection content displayed on the second electronic device. When the first electronic device determines whether the cursor has moved to the boundary of the display area of ​​the second electronic device, the display position of the cursor on the display screen of the second electronic device can stay at the boundary of the display area of ​​the second electronic device. It should be understood here that when the cursor moves to the left and right boundaries of the display area on the display screen of the second electronic device, although the cursor will not exceed the left and right boundaries of the display area on the display screen, the cursor can still move up and down. Similarly, when the cursor moves to the upper and lower boundaries of the display area on the display screen of the second electronic device, although the cursor will not exceed the upper and lower boundaries of the display area on the display screen, the cursor can still move left and right. Similarly, when the cursor moves to a corner point (the upper left corner, the lower left corner, the upper right corner or the lower right corner) of the display area on the display screen of the second electronic device, although the cursor will not exceed the boundary of the display area on the display screen, the cursor can still move in a certain direction. For example, when the cursor moves to the upper left corner point of the display area on the display screen of the second electronic device, the cursor can still move to the right, downward and to the lower right.

[0587] Optionally, in another embodiment, a quick adjustment mechanism for the cursor position can be set. For example, when the user touches the display screen of the first electronic device for more than a preset time, the first electronic device can initialize the position of the cursor in the interface content in response to the user's above operation.

[0588] For example, refer to Fig.11e , Fig.11e The following is a schematic diagram of interaction between a first electronic device and a second electronic device. Fig.11e As shown in FIG. 1 , after the first electronic device and the second electronic device establish a screen projection connection, since the front end of the first electronic device is facing the left side of the geometric center of the second electronic device, at this time, the user waves the first electronic device to the right and makes the front end of the first electronic device face the vicinity of the center position of the second electronic device, as shown in FIG. Fig.11f As shown in FIG. 1 , the cursor is displayed near the center of the second electronic device. At this time, the user can long press the display screen of the first electronic device for 5 seconds or more. Figure 11gAs shown in the figure, the first electronic device can adjust the display position of the cursor to the geometric center of the interface content in response to the user's long press of the display screen for 5S. When the user points the front end of the first electronic device toward the center of the second electronic device in the above manner, the cursor is adjusted to be located near the center of the second electronic device, and combined with the above sensitivity adjustment, when the user waves the first electronic device, the front end of the first electronic device can be oriented as closely as possible to the display position of the cursor on the second electronic device.

[0589] It should be noted that the above method of adjusting the cursor position is only an example and does not constitute a limitation to the present application.

[0590] Optionally, in one embodiment, if the user does not operate the first electronic device for a certain period of time after the first electronic device and the second electronic device establish a screen projection connection, the cursor displayed on the second electronic device may be hidden, and the user may activate the cursor display in the display screen of the second electronic device through the above-mentioned touch operation and / or by changing the posture of the first electronic device.

[0591] In the embodiment of the present application, the first electronic device may also receive a first operation on the display screen of the first electronic device, and the first operation is a preset shortcut operation, and the first electronic device may respond to the first operation.

[0592] In an embodiment of the present application, the first electronic device can detect a first operation on the display screen of the first electronic device and identify the first operation as a preset shortcut operation. At this time, the first electronic device can generate a first event corresponding to the first electronic device, and the first electronic device can also generate another event based on the position of the cursor and the first operation (referred to as the second operation here to distinguish it from the first operation). The first electronic device can determine the priority of the first operation and the second operation, and based on the priority of the first operation and the second operation, determine whether to execute the first operation or the second operation.

[0593] Optionally, in an embodiment of the present application, if the first electronic device recognizes that the first operation is a preset shortcut operation, at this time, the first electronic device can generate a first event corresponding to the first electronic device, and the first electronic device can also generate a second operation based on the position of the cursor and the first operation, and the execution priority of the first event is higher than the execution priority of the second event, then the first electronic device executes the above-mentioned first event. At this time, the operation position of the first operation corresponds to the fifth object in the interface content of the second electronic device, and since the execution priority of the second event is lower than the execution priority of the first event, the fifth object does not respond to the first operation.

[0594] Next, the screen projection method provided by this application is further described in combination with two application scenarios.

[0595] 1. Video playback

[0596] Reference Fig.12a , Fig.12a A schematic diagram of an interface of a second electronic device provided in an embodiment of the present application, such as Fig.12a As shown in FIG. 1 , the user can click on the video APP application by operating the first electronic device. After the user clicks on the video to be played, the second electronic device can display the following Figure 12b The video playback interface shown in .

[0597] Reference Fig.12a , Fig.12a A schematic diagram of an interface of a second electronic device provided in an embodiment of the present application, such as Fig.12a As shown in FIG. 1 , the user can click the full-screen control in the interface to play the video in full screen. In another embodiment, as shown in FIG. Fig.12d As shown in FIG. 1 , the user can rotate the first electronic device (from the vertical screen to the horizontal screen) to achieve full-screen playback of the video. The full-screen playback interface of the video can refer to Fig.12e The full-screen playback interface may include but is not limited to video images, pause / play controls, controls for switching to the next episode, etc.

[0598] In this embodiment of the present application, the user can click Fig.12e The video playback area in the corresponding reference Fig.12f , the screen projection content of the second electronic device may only include video images, or, if the user does not operate the first electronic device for a long time, the screen projection content of the second electronic device may only include video images (refer to Fig.12f ).

[0599] Reference Figure 12g , Figure 12g A schematic diagram of an operation of a first electronic device provided in an embodiment of the present application is shown in FIG. Figure 12g As shown in FIG. 1 , when the second electronic device displays a video playback interface, the user performs a horizontal sliding operation on the display screen of the first electronic device, and the video displayed by the second electronic device can be fast forwarded or rewound. The second electronic device displays as shown in FIG. Fig.12e The interface shown in FIG. 1 is displayed, and when the user slides to the right on the display screen of the first electronic device, as shown in FIG. Figure 12h As shown in FIG. 1 , the screen projection content of the second electronic device includes a fast-forward preview image and a fast-forward progress bar, such as Fig.12i As shown, the screen projection content of the second electronic device includes a fast-forward preview image and a fast-forward progress bar. After the user determines the fast-forward target position based on the preview image displayed on the second electronic device, the sliding operation on the display screen of the first electronic device can be ended. Fig.12jand Figure 12k As shown in the illustration, the video displayed by the second electronic device is fast-forwarded to the target position (10:03) determined by the user for fast-forwarding.

[0600] Reference Figure 12l , Figure 12l An operation diagram of a first electronic device provided in an embodiment of the present application is shown in FIG. Figure 12l As shown in FIG. 1 , the user can perform a vertical sliding operation on the right area of ​​the first electronic device display screen to adjust the volume of the currently playing video, or as shown in FIG. Figure 12m As shown in FIG. 1 , the user can adjust the volume of the currently playing video through the physical volume adjustment button 1201 of the first electronic device, such as Figure 12n As shown in Figure 12n The volume adjustment interface of the second electronic device is shown.

[0601] Reference Fig.12o , Fig.12o An operation diagram of a first electronic device provided in an embodiment of the present application is shown in FIG. Fig.12o As shown in FIG. 1 , the user can perform a vertical sliding operation on the left area of ​​the first electronic device display screen to adjust the display brightness of the currently playing video, such as Figure 12p As shown in Figure 12p The brightness adjustment interface of the second electronic device is shown.

[0602] Reference Figure 12q , Figure 12q An operation diagram of a first electronic device provided in an embodiment of the present application is shown in FIG. Figure 12q As shown in , the user can slide from the right border area to the center of the screen on the display screen of the first electronic device. At this time, the second electronic device can display the interface content of the previous level (that is, the first electronic device performs a return operation and returns to the previous interface content), such as Figure 12r As shown, at this time, the projection content of the second electronic device is the upper interface (video playback interface) of the full-screen video playback interface.

[0603] Reference Figure 12s , Figure 12s An operation diagram of a first electronic device provided in an embodiment of the present application is shown in FIG. Figure 12s As shown in FIG. 1 , the user can slide from the lower boundary area to the center of the screen on the display screen of the first electronic device. At this time, the second electronic device can display the main interface (that is, the first electronic device performs an operation of returning to the main interface), such as Figure 12t As shown, at this time, the projection content of the second electronic device is the main interface of the first electronic device.

[0604] In the embodiment of the present application, the first electronic device may also receive a first operation on the display screen of the first electronic device, and the first operation is a preset shortcut operation, and the first electronic device may respond to the first operation.

[0605] In an embodiment of the present application, the first electronic device can detect a first operation on the display screen of the first electronic device and identify the first operation as a preset shortcut operation. At this time, the first electronic device can generate a first event corresponding to the first electronic device, and the first electronic device can also generate another event based on the position of the cursor and the first operation (referred to as the second operation here to distinguish it from the first operation). The first electronic device can determine the priority of the first operation and the second operation, and based on the priority of the first operation and the second operation, determine whether to execute the first operation or the second operation.

[0606] Optionally, in an embodiment of the present application, if the first electronic device recognizes that the first operation is a preset shortcut operation, at this time, the first electronic device can generate a first event corresponding to the first electronic device, and the first electronic device can also generate a second operation based on the position of the cursor and the first operation, and the execution priority of the first event is higher than the execution priority of the second event, then the first electronic device executes the above-mentioned first event. At this time, the operation position of the first operation corresponds to the fifth object in the interface content of the second electronic device, and since the execution priority of the second event is lower than the execution priority of the first event, the fifth object does not respond to the first operation.

[0607] For example, refer to Figure 12u , Figure 12u This is a schematic diagram of the operation of a first electronic device provided in an embodiment of the present application. In addition to the single-click confirmation operation, the first electronic device can also support commonly used full-screen gesture operations (quick operations), which is consistent with the original way of using the first electronic device, reducing the user's learning cost. Figure 12u As shown in the figure, sliding from the left or right edge to the middle represents the return key, sliding from the bottom edge to the middle represents the home key, sliding from the bottom edge to the middle and staying represents the menu key, and the sliding operations up, down, left and right can be used to fast forward and rewind in the video scene.

[0608] Next, let's introduce the principle of implementing the above full-screen gesture operation:

[0609] 1. Back key

[0610] In an embodiment of the present application, sliding from the left edge or the right edge to the middle represents the return key. Specifically, when the first electronic device is in the screen-off state, if a sliding operation from the left edge or the right edge to the middle is detected, the sliding operation performed at the current cursor position can be intercepted, and the return event can be directly injected into the current front-end application. In response to the return event, the interface content of the system returns to the previous level, which is equivalent to the return operation performed by the mobile phone when the user slides from the left edge or the right edge to the middle in normal mode.

[0611] 2. Home button

[0612] In an embodiment of the present application, sliding from the bottom edge to the middle represents the home button. Specifically, when the first electronic device is in the screen-off state, if a sliding operation from the bottom edge to the middle is detected, the sliding operation performed at the current cursor position can be intercepted, and an event of returning to the main interface can be directly injected into the current front-end application. In response to the event of returning to the main interface, the system interface content returns to the main interface, which is equivalent to the user clicking the home button in normal mode.

[0613] 3. Menu key

[0614] In an embodiment of the present application, sliding from the bottom edge to the middle and staying there represents a menu key. Specifically, when the first electronic device is in the screen-off state, if an operation of sliding from the bottom edge to the middle and staying there is detected, the above operation can be intercepted at the current cursor position, and the event of displaying the menu is directly injected into the current front-end application. In response to the event of displaying the menu, the system's interface content displays a pull-up menu, which is equivalent to when in normal mode, the user clicks to slide from the bottom edge to the middle and stays there, and the phone will display a pull-up menu (the menu may include a list of currently running applications, or a list of historically running applications, etc., which is not limited in the present application).

[0615] In an embodiment of the present application, a first electronic device can receive a second touch operation on the display screen of the first electronic device, determine a corresponding third event according to the operation form of the second touch operation, wherein different operation forms correspond to different third events, and execute the third event on the front-end application of the first electronic device, wherein the operation form of the second touch operation includes at least one of the following operation forms: contacting a first preset area of ​​the display screen of the first electronic device and sliding from the first preset area to a first preset direction; contacting a second preset area of ​​the display screen of the first electronic device and sliding from the second preset area to a second preset direction, and the time of contacting the display screen of the first electronic device is greater than a preset time.

[0616] Specifically, after acquiring the first operation inputted by the touch screen, the enhanced interactive service can identify whether the first operation is a preset shortcut operation, and based on the first operation conforming to the preset shortcut operation, generate an event corresponding to the preset shortcut operation. At this time, the enhanced interactive service can directly inject the event into the front-end application instead of executing it at the location of the cursor (that is, the first electronic device can directly respond to the first operation). For details, please refer to Figure 12u The description of the corresponding embodiments will not be repeated here.

[0617] Optionally, in another implementation, the display screen can be divided into two areas (for example, an upper area and a lower area). If the user touches the display screen in the upper area of ​​the display screen, the cursor can be controlled to display the content of the interface. If the user touches the display screen in the lower area of ​​the display screen, the cursor can be controlled to display the content of the interface in a similar manner. Figure 12u The operation method in .

[0618] In addition, it should be noted that, in the above-mentioned video playback scenario, after the user selects a video to be played, the second electronic device can directly play the video in full screen.

[0619] Optionally, the first electronic device 100 may obtain the video 400 corresponding to the current video playback interface, and process the video 400 into a video stream. In some feasible implementations, the first electronic device 100 obtains the image data of the video, and compresses the image corresponding to each image frame in the image data of the video, and the image size of the compressed image corresponding to each image frame is the same as the image size on the second electronic device. Exemplarily, when the image size is in pixels, assuming that the image size on the second electronic device is 400×800 pixels, and the image size of the image corresponding to each image frame is 800×800 pixels, when the first electronic device compresses the image corresponding to each image frame, the image size of the image corresponding to each image frame can be compressed from 800×800 pixels to 400×800 pixels to obtain the compressed image corresponding to each image frame. The first electronic device may perform video compression encoding on the compressed images corresponding to multiple image frames to obtain a video stream. The multiple image frames may be image frames of the image data of the video at multiple consecutive time nodes, wherein the image data of the video has one image frame at one time node. In some other feasible implementations, after obtaining the image data of the video, the first electronic device may directly perform video compression encoding on multiple image frames of the image data at multiple consecutive time nodes to obtain a video stream.

[0620] In some feasible implementations, the first electronic device can obtain audio data of the video within a period of time, and the period of time is determined by the above-mentioned multiple continuous time nodes. The first electronic device can perform audio compression encoding on the audio data within this period of time, such as advanced audio coding (AAC). The first electronic device mixes the above-mentioned multiple image frames after video compression encoding with the audio data within this period of time after audio compression encoding into a video stream. The data format of the video stream is any data format that the second electronic device can receive, such as a video stream in MP4 (MPEG-4 part 14) format. The image and audio corresponding to the video stream are synchronously presented on the second electronic device.

[0621] The first electronic device can send a video stream to the second electronic device. Correspondingly, after receiving the video stream, the second electronic device processes the video stream into image and audio output. In some feasible implementations, the first electronic device can output the video stream as projection data to the second electronic device through the projection port using a video streaming protocol (such as real-time streaming protocol (RTSP)). The second electronic device performs video streaming protocol reception processing and video and audio decoding processing on the projection data (i.e., video stream), and then renders and outputs it. At this time, the second electronic device displays the image corresponding to the projection data in full screen and / or plays the audio corresponding to the projection data. For example, the screen width of the first electronic device 100 is S-W1, and the screen height is SH1; the screen width of the second electronic device 100 is S-W2, and the screen height is S-H2. When the second electronic device 200 displays the image corresponding to the projection data, it can adjust the aspect ratio of the image corresponding to the projection data to the same as the screen aspect ratio S-W2:S-H2 of the second electronic device 200 before displaying it. Assuming that the video 400 is played in full screen on the first electronic device 100 without black or white edges, and the screen aspect ratio S-W1:S-H1 of the first electronic device 100 is the same as the screen aspect ratio S-W2:S-H2 of the second electronic device 200, then the video playback screen displayed by the second electronic device 200 also has no black or white edges, and the first electronic device 100 and the second electronic device 200 display the same image and play the same audio.

[0622] In some feasible implementations, if the user moves the cursor on the second electronic device to the pause control and performs a click operation on the display screen of the first electronic device, the first electronic device responds to the above operation and pauses the playback of the video 400. At the same time, the first electronic device 100 will pause the transmission of the video stream of the video 400, and the second electronic device 200 will also pause the playback because there is no video stream transmitted. If the user moves the cursor on the second electronic device to the pause control and performs a click operation on the display screen of the first electronic device, the first electronic device responds to the above operation and continues to play from the current playback progress of the video 400, and the first electronic device 100 continues to transmit the video stream of the video 400, so the second electronic device 200 receives the video stream and continues to play. If the transmission of the video stream is segmented, the first electronic device 100 only transmits the video stream within a fixed time period each time it transmits the video stream. Exemplarily, if the playback time of the video 400 is 25 minutes and 34 seconds, the first electronic device 100 can transmit a 10-second long video stream each time, so the video 400 needs to be transmitted 154 times to be fully transmitted to the second electronic device 200.

[0623] The above introduces a description of an embodiment of the screen projection method provided by the present application in a video playback scenario. Next, an embodiment of the screen projection method provided by the present application in another application scenario is introduced.

[0624] 2. Demonstration Scene

[0625] Fig.13 A schematic diagram of a screen projection scenario provided in an embodiment of the present application. Fig.13 As shown, it is assumed that in a conference room of a certain enterprise, user A, user B, user C, user D, user E, user F, user G, user H and user I hold a meeting; wherein user A sends the projection data containing the meeting information to the second electronic device 200 through the first electronic device, and the second electronic device 200 displays the projection data. Generally speaking, the device screen of the first electronic device 100 is small, and it is difficult for user A to share the meeting information with other users through the first electronic device 100, while the second electronic device 200 (such as a TV) can display the projection data through a larger screen for other users to view.

[0626] For users B, C, D, and E who are close to the second electronic device 200, they can view the conference information shared by user A by watching the display content of the second electronic device 200, without having to carry and use additional electronic devices for assistance. However, due to the screen size limitation of the second electronic device 200 and the possible vision problems of some users such as myopia, users such as user F, user G, user H, and user I may not be able to see the display content of the second electronic device 200 clearly, affecting the normal progress of the meeting.

[0627] Therefore, the first electronic device 100 can simultaneously establish a screen projection connection with multiple second electronic devices 200. Based on the technical solution of the present application, user F, user G, user H, user I, etc. can respectively use their respective second electronic devices 200 to receive and display the projection data sent by the first electronic device 100, so that the corresponding users can view it. Among them, for each device in the multiple second electronic devices 200, the processing process in the present application is the same.

[0628] Reference Fig.14a , Fig.14a A schematic diagram of screen projection content of a second electronic device provided in an embodiment of the present application, such as Fig.14a As shown in FIG. 1 , the user can click on the PPT application by operating the first electronic device. After the user clicks on the PPT that he wants to present, the second electronic device can display the following Fig.14b The PPT presentation interface shown in .

[0629] like Fig.14a As shown in FIG. 1 , the PPT presentation interface may include a presentation control 1403, a current film presentation area 1404, and a film list area 1405. The current film presentation area 1404 is the film currently to be presented, and the film list area 1405 may include a list of films. The user may select a film to be presented in the film list area 1405. For example, the user may select film C by operating the first electronic device, and accordingly, the film displayed in the film presentation area 1404 is the selected film C.

[0630] like Fig.14b As shown in FIG. 1 , the user can click the presentation control 1403 in the PPT presentation interface to implement the film presentation. The film presentation interface can refer to FIG. Fig.14c . Reference Fig.14d , Fig.14d The operation diagram of a first electronic device provided in an embodiment of the present application is as follows: Fig.14d As shown in FIG. 1 , the user can perform a leftward activity operation on the display screen of the first electronic device, and at this time, the second electronic device can display Fig.14eThe demonstration interface shown in (where the demonstration film switches to the next one). Fig.14f , Fig.14f The operation diagram of a first electronic device provided in an embodiment of the present application is as follows: Fig.14f As shown in FIG. 1 , the user can provide a rotation of the first electronic device (from a vertical screen to a horizontal screen) to achieve a full-screen presentation of the film, such as Figure 14g As shown in Figure 14g A full-screen presentation interface of a film is shown in FIG.

[0631] Next, the specific process of the information processing method provided in the embodiment of the present application is introduced. Fig.15 , Fig.15 A flowchart of an information processing method provided in an embodiment of the present application is shown as follows: Fig.15 As shown in, the information processing method provided by the present application includes:

[0632] 1501. The first electronic device monitors the screen projection connection status.

[0633] In an embodiment of the present application, the first electronic device may install an enhanced interactive service, and the enhanced interactive service may monitor the screen projection connection status of the first electronic device.

[0634] 1502. The first electronic device detects that a screen projection connection is established with the second electronic device.

[0635] In the embodiment of the present application, the enhanced interactive service can detect that the first electronic device and the second electronic device have established a screen projection connection. For the method of establishing the screen projection between the first electronic device and the second electronic device, refer to Figures 4a to 5c The description of the corresponding embodiments will not be repeated here.

[0636] 1503. The first electronic device generates screen projection content.

[0637] In an embodiment of the present application, the enhanced interactive service of the first electronic device can obtain the interface content of the front-end application of the first electronic device, generate the cursor on the interface content, and obtain the projection content.

[0638] In an embodiment of the present application, the enhanced interactive service of the first electronic device can also generate the cursor and the menu bar on the interface content to obtain the projection content.

[0639] Specifically, refer to Fig.16 , Fig.16 A schematic diagram of an architecture of an embodiment of the present application is shown in FIG. Fig.16As shown in the figure, the enhanced interactive service can add a cursor and a menu bar to the interface content of the current front-end application of the first electronic device based on the floating window interface to generate projection data. At this time, the projection service of the first electronic device can obtain the above projection data and send the projection data (which needs to undergo encoding operations and content size transformation) to the second electronic device based on a certain projection protocol, so that the display screen of the second electronic device displays the projection content corresponding to the projection data.

[0640] Optionally, the first electronic device can obtain the posture change information of the first electronic device input by the sensor, and move the position of the cursor in the interface content based on the posture change information of the first electronic device. Regarding how the enhanced interactive service moves the cursor in the display screen of the second electronic device based on the posture change information of the first electronic device, you can refer to the description in the above embodiment and will not repeat it here.

[0641] Optionally, the first electronic device can receive a second sliding operation on the display screen of the first electronic device, determine the displacement of the cursor according to the second sliding operation, and move the cursor on the projected content based on the displacement. As for how the enhanced interactive service moves the cursor on the display screen of the second electronic device according to the second sliding operation, reference can be made to the description in the above embodiments, which will not be repeated here.

[0642] Optionally, the first electronic device may acquire a first touch operation inputted on the touch screen, generate a corresponding first event according to the first touch operation, and execute the first event at the target object of the interface content (the target object where the cursor is located at this time). For example, the first touch operation may be a click operation or a slide operation, and accordingly, the first event may be a click event or a slide event.

[0643] Optionally, the first electronic device may shield the response of the interface content to the operation position of the first touch operation. Specifically, after receiving the first touch operation on the display screen of the first electronic device, the first electronic device may shield the response of the interface content of the current front-end application to the operation position of the first touch operation, and instead determine the position of the cursor on the second display screen to determine the response position of the interface content to the first touch operation. For example, if the user performs a click operation on object A on the display screen of the first electronic device, and at this time, the cursor is located on object B in the second electronic device, at this time, the first electronic device may shield the response of object A to the click operation, and instead respond to the above-mentioned click operation on object B of the interface content.

[0644] Optionally, the first electronic device may receive a pressing operation on a physical button of the first electronic device, generate a corresponding second event according to the pressing operation, and execute the second event on the front-end application of the first electronic device. For example, the first electronic device may receive a pressing operation on a volume button of the first electronic device (for example, volume reduction), generate a corresponding volume reduction event according to the pressing operation, and execute the volume reduction event on the front-end application of the first electronic device, so that the volume played by the front-end application of the first electronic device is reduced.

[0645] Optionally, the first electronic device can receive a second touch operation on the display screen of the first electronic device, determine a corresponding third event according to the operation form of the second touch operation, wherein different operation forms correspond to different third events, and execute the third event on the front-end application of the first electronic device, wherein the operation form of the second touch operation includes at least one of the following operation forms: contacting a first preset area of ​​the display screen of the first electronic device and sliding from the first preset area to a first preset direction; contacting a second preset area of ​​the display screen of the first electronic device and sliding from the second preset area to a second preset direction, and the time of contacting the display screen of the first electronic device is greater than a preset time.

[0646] Specifically, after obtaining the touch operation input from the touch screen, the enhanced interactive service can identify whether the touch operation conforms to the preset operation form, and based on the touch operation conforming to the preset operation form, generate an event corresponding to the preset operation form. At this time, the enhanced interactive service can directly inject the event into the front-end application instead of executing it at the cursor location.

[0647] Optional, see Fig.17 , Fig.17 A schematic diagram of an architecture of an embodiment of the present application is shown in FIG. Fig.17 As shown in , the enhanced interactive service can obtain the interface content of the current front-end application. Optionally, the enhanced interactive service can obtain the interface content of the current front-end application based on the screen recording interface provided by the system (such as the MediaProjection interface provided by Android), and draw a cursor and menu bar on the obtained interface content. The enhanced interactive service can send the above-drawn content as projection data to the projection service of the first electronic device. At this time, the projection service of the first electronic device obtains the projection data and sends the projection data (which needs to undergo encoding operations and / or content size transformation) to the second electronic device based on a certain projection protocol.

[0648] 1504. The first electronic device sends the projection content to the second electronic device.

[0649] In an embodiment of the present application, after the first electronic device sends projection data to the second electronic device, the display screen of the second electronic device can display the projection content corresponding to the projection data, and the projection content includes a cursor and interface content of the first electronic device, and the cursor is used to locate the operation position in the interface content.

[0650] The implementation method of step 1504 can refer to the relevant description of the first electronic device sending the projection data to the second electronic device in the above embodiment, which will not be repeated here.

[0651] It should be noted that the above steps 1503 and 1504 can be implemented independently, rather than being executed after steps 1501 and 1502 are executed.

[0652] The embodiment of the present application provides an information processing method, including: generating screen projection content; sending the screen projection content to a second electronic device so that the display screen of the second electronic device displays the screen projection content, wherein the screen projection content includes a cursor and the interface content of the first electronic device, and the cursor is used to locate the operation position in the interface content. In the above manner, a cursor is added to the screen projection content so that the user can determine the position to be operated through the cursor displayed by the second electronic device, so that the user can determine the position to be operated based on the position of the cursor in the screen projection content displayed on the display screen of the second electronic device without looking at the interface content of the first electronic device, and then perform the operation at the position of the cursor.

[0653] The present application also provides an electronic device. Fig.18 , Fig.18 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application, the electronic device includes:

[0654] Processing module 1801, used to generate screen projection content;

[0655] The sending module 1802 is used to send the projected content to the second electronic device so that the display screen of the second electronic device displays the projected content. The projected content includes a cursor and the interface content of the first electronic device. The cursor is used to locate the operation position in the interface content.

[0656] Optionally, the processing module 1801 is specifically configured to:

[0657] The interface content of the front-end application of the first electronic device is obtained, and a cursor is generated on the interface content to obtain the projection content.

[0658] Optionally, the processing module 1801 is specifically configured to:

[0659] The interface content of the front-end application of the first electronic device is obtained, and a cursor and a menu bar are generated on the interface content to obtain the projection content; wherein the menu bar does not belong to the interface content of the first electronic device.

[0660] Optionally, the processing module 1801 is further used to detect that the first electronic device establishes a screen projection connection with the second electronic device.

[0661] Optionally, the processing module 1801 is further configured to obtain posture change information of the first electronic device;

[0662] Based on the posture change information, the cursor is moved in the projection content displayed by the second electronic device.

[0663] Optionally, the posture change information includes a first rotation angle of the first electronic device on a horizontal plane and a second rotation angle of the first electronic device on a vertical plane, and the processing module 1801 is specifically used to:

[0664] determining a horizontal displacement of the cursor according to the first rotation angle;

[0665] determining a vertical displacement of the cursor according to the second rotation angle;

[0666] The cursor is moved in the projection content displayed by the second electronic device according to the horizontal displacement and the vertical displacement.

[0667] Optionally, the posture change information includes a first displacement of the first electronic device in a first direction and a second displacement of the first electronic device in a second direction, the first direction is parallel to a left-right movement direction of a cursor displayed on a display screen of the second electronic device, and the second direction is parallel to an up-and-down movement direction of a cursor displayed on a display screen of the second electronic device, and the processing module 1801 is further used to:

[0668] determining a horizontal displacement of the cursor according to the first displacement;

[0669] determining a vertical displacement of the cursor according to the second displacement;

[0670] The cursor is moved in the projection content displayed by the second electronic device according to the horizontal displacement and the vertical displacement.

[0671] Optionally, the processing module 1801 is further used to:

[0672] When the cursor moves to the target object in the projection content displayed by the second electronic device, if a first touch operation on the display screen of the first electronic device is received;

[0673] Then, the target object in the projection content displayed by the second electronic device responds to the first touch operation.

[0674] Optionally, the processing module 1801 is further used to:

[0675] Shielding the second object from responding to the first touch operation.

[0676] Optionally, the first touch operation includes at least a click operation and a first sliding operation, and the target object includes at least an application and a function control.

[0677] Optionally, the processing module 1801 is further used to:

[0678] receiving a second sliding operation on the display screen of the first electronic device;

[0679] determining a displacement of the cursor according to the second sliding operation;

[0680] Based on the displacement, the cursor is moved in the projected content displayed by the second electronic device.

[0681] Optionally, the operation position corresponding to the second sliding operation corresponds to a third object in the interface content of the first electronic device, and the processing module 1801 is further used to:

[0682] Shielding the third object from responding to the second touch operation.

[0683] Optionally, the processing module 1801 is further used to:

[0684] When the cursor moves to the target object in the projection content displayed by the second electronic device, if a click operation on the display screen of the first electronic device is received;

[0685] Then, the target object in the projection content displayed by the second electronic device responds to the click operation.

[0686] Optionally, the operation position of the click operation corresponds to a fourth object in the interface content of the first electronic device, and the processing module 1801 is further configured to:

[0687] Shielding the fourth object from responding to the click operation.

[0688] Optionally, the processing module 1801 is further used to:

[0689] Receiving a pressing operation on a physical button of the first electronic device;

[0690] Generate a corresponding second event according to the pressing operation;

[0691] The second event is executed on the first electronic device front-end application.

[0692] Optionally, the processing module 1801 is further used to:

[0693] Receiving a first operation on the display screen of the first electronic device, where the first operation is a preset shortcut operation;

[0694] The first electronic device responds to the first operation.

[0695] Optionally, the operation position of the first operation corresponds to a fifth object in the interface content of the second electronic device, and the fifth object does not respond to the first operation.

[0696] Optionally, the preset shortcut operations include at least:

[0697] Contacting a first preset area of ​​the display screen of the first electronic device, and sliding from the first preset area to a first preset direction;

[0698] The user touches a second preset area of ​​the display screen of the first electronic device, slides from the second preset area to a second preset direction, and the time of contacting the display screen of the first electronic device is greater than a preset time.

[0699] Optionally, the processing module 1801 is further used to:

[0700] reducing the display brightness of the display screen of the first electronic device; or,

[0701] A screen-off operation is performed on the display screen of the first electronic device.

[0702] Optionally, a display area of ​​the display screen of the first electronic device is smaller than a display area of ​​the display screen of the second electronic device.

[0703] Next, an electronic device provided by an embodiment of the present application is introduced. Fig.19 , Fig.19 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. The electronic device 1900 can be specifically a mobile phone, a tablet, a smart wearable device, etc., which is not limited here. Specifically, the electronic device 1900 includes: a receiver 1901, a transmitter 1902, a processor 1903 and a memory 1904 (wherein the number of processors 1903 in the electronic device 1900 can be one or more, Fig.19In the example of FIG. 1903 , the processor 1903 may include an application processor 19031 and a communication processor 19032. In some embodiments of the present application, the receiver 1901, the transmitter 1902, the processor 1903 and the memory 1904 may be connected via a bus or other means.

[0704] The memory 1904 may include a read-only memory and a random access memory, and provides instructions and data to the processor 1903. A portion of the memory 1904 may also include a non-volatile random access memory (NVRAM). The memory 1904 stores processor and operation instructions, executable modules or data structures, or subsets thereof, or extended sets thereof, wherein the operation instructions may include various operation instructions for implementing various operations.

[0705] The processor 1903 controls the operation of the electronic device. In a specific application, the various components of the electronic device are coupled together through a bus system, wherein the bus system includes not only a data bus but also a power bus, a control bus, and a status signal bus, etc. However, for the sake of clarity, various buses are referred to as bus systems in the figure.

[0706] The method disclosed in the above embodiment of the present application can be applied to the processor 1903, or implemented by the processor 1903. The processor 1903 can be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit or software instructions in the processor 1903. The above processor 1903 can be a general processor, a digital signal processor (digital signal processing, DSP), a microprocessor or a microcontroller, and can further include an application specific integrated circuit (application specific integrated circuit, ASIC), a field programmable gate array (field-programmable gate array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The processor 1903 can implement or execute the methods, steps and logic block diagrams disclosed in the embodiments of the present application. The general processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in the embodiment of the present application can be directly embodied as a hardware decoding processor to execute, or the hardware and software modules in the decoding processor can be combined and executed. The software module may be located in a storage medium mature in the art, such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 1904, and the processor 1903 reads the information in the memory 1904 and completes the steps of the above method in combination with its hardware.

[0707] The receiver 1901 can be used to receive input digital or character information, and generate signal input related to the relevant settings and function control of the electronic device. The transmitter 1902 can be used to output digital or character information through the first interface; the transmitter 1902 can also be used to send instructions to the disk group through the first interface to modify the data in the disk group; the transmitter 1902 can also include a display device such as a display screen.

[0708] In an embodiment of the present application, in one case, the processor 1903 is used to execute the information processing method in the corresponding embodiment of the above embodiment.

[0709] Next, taking the second electronic device 200 as a head-mounted display (HMD) as an example, an embodiment provided by the present application is described, wherein the head-mounted display can be a VR device display, an AR device display or an MR device display.

[0710] In the embodiment of the present application, the user can immerse in an augmented reality environment, a virtual reality environment, or a mixed display environment by wearing a head mounted display (HMD), and the user can immerse in a 3D virtual environment and interact with the virtual environment through various types of inputs. For example, these inputs may include physical interactions, such as manipulation of the first electronic device 100 separated from the HMD, manipulation of the HMD itself (based on head movement), etc.

[0711] Reference Fig. 20 , Fig. 20 The following is a schematic diagram of interaction between a first electronic device and a second electronic device. Fig. 20 As shown in FIG. 1 , the user wearing the second electronic device 200 (HMD) is holding the first electronic device 100 as a control device for the second electronic device 200. It should be noted that although Fig. 20 In the example shown, only one first electronic device 100 is illustrated as a control device of the second electronic device 200, but two (or more) additional external devices may also be paired and / or interact with the HMD 100 in the virtual environment. During operation (after pairing), the first electronic device 100 (and / or other external devices) may communicate with the second electronic device 200 via, for example, a wired connection, or a wireless connection such as, for example, a WiFi or Bluetooth connection, or other communication modes available for the two devices.

[0712] Fig.21a A schematic diagram of connecting a first electronic device 100 to a second electronic device 200 using a cable 1600 is shown. The first electronic device 100 can be connected to the second electronic device 200 using one or more high-speed communication protocols (e.g., USB 2.0, USB 3.0, and USB 3.1). In some cases, the first electronic device 100 can be connected to the second electronic device 200 using an audio / video interface such as a high-definition multimedia interface (HDMI). In some cases, the first electronic device 100 can be connected to the second electronic device 200 using a DisplayPort alternate mode for a USB Type-C standard interface. The DisplayPort alternate mode may include a high-speed USB communication interface and a DisplayPort function.

[0713] The cable 2100 may include a suitable connector inserted into the second electronic device 200 and the first electronic device 100 at either end. For example, the cable may include a universal serial bus (USB) connector at both ends. The USB connector may be an identical USB type connector, or each USB connector may be a different type of USB connector. Various types of USB connectors may include, but are not limited to, USB A type connector, USB B type connector, micro-USB A connector, micro-USB B connector, micro-USB AB connector, USB five-pin Mini-b connector, USB four-pin Mini-b connector, USB 3.0A type connector, USB 3.0B type connector, USB 3.0Micro B connector, and USB C type connector, etc.

[0714] Figure 21b is shown in the case where no cables are required (e.g., no Fig.21a 102) is a schematic diagram of connecting a first electronic device 100 to a second electronic device 100 using a wireless connection 1601. The first electronic device 100 can be connected to the second electronic device 200 using a wireless connection 1601 by implementing one or more high-speed communication protocols (e.g., WiFi, Bluetooth, or Bluetooth Low Energy (LE)).

[0715] It should be noted that the second electronic device 200 can also be connected to other control devices, such as a handle, etc. at the same time.

[0716] like Fig.22a As shown, at this time, the user uses the handle as an interactive device. However, in some scenarios, the portability of the handle is poor. The independent handle needs to use batteries and is wirelessly connected to the first electronic device 100, which requires additional power consumption. In addition, the handles of some AR / VR devices are large and heavy, and long-term use can easily cause user fatigue.

[0717] The software system of the electronic device 100 may adopt a layered architecture, an event-driven architecture, a micro-core architecture, a micro-service architecture, or a cloud architecture. The embodiment of the present application takes the Android system of the layered architecture as an example to exemplify the software structure of the electronic device 100.

[0718] Figure 22b It is a software structure block diagram of the first electronic device 100 according to an embodiment of the present application.

[0719] The layered architecture divides the software into several layers, each with clear roles and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into four layers, from top to bottom: the application layer, the application framework layer, the Android runtime and system library, and the kernel layer.

[0720] The application layer can include a series of application packages.

[0721] like Figure 22b As shown, the application package may include applications such as phone, camera, gallery, calendar, call, map, navigation, WLAN, Bluetooth, music, video, short message, VR glasses application, etc. Among them, the VR glasses application includes a 3D background drawing module, a handle event management module, an application icon loading module, a virtual screen management module, and a virtual screen content acquisition module.

[0722] Among them, the 3D background drawing module is used to complete the drawing of the background picture displayed in the 3D virtual environment, so that the user can get the feeling as if he is in a real scene.

[0723] The handle event management module is used to process events from the handle so that the user can touch the controls in the virtual display interface by operating the handle.

[0724] The application icon loading module is used to load and display icons of several applications on electronic devices (such as WeChat, Weibo, Douyin, etc.) in the virtual environment of VR glasses.

[0725] The virtual screen management module is used to create a virtual screen when the user clicks the application icon to start the application, and to destroy the virtual screen when the user closes the application.

[0726] The virtual screen content acquisition module is used to acquire the content in the application when the user clicks to start the application, and render the content in the application through distortion to display it in the virtual environment.

[0727] The application framework layer provides an application programming interface (API) and a programming framework for the applications in the application layer. The application framework layer includes some predefined functions.

[0728] like Figure 22b As shown, the application framework layer may include a window manager, a content provider, a view system, a phone manager, a resource manager, a notification manager, and the like.

[0729] The window manager is used to manage window programs. The window manager can obtain the display screen size, determine whether there is a status bar, lock the screen, capture the screen, etc.

[0730] Content providers are used to store and retrieve data and make it accessible to applications. The data may include videos, images, audio, calls made and received, browsing history and bookmarks, phone books, etc.

[0731] The view system includes visual controls, such as controls for displaying text, controls for displaying images, etc. The view system can be used to build applications. A display interface can be composed of one or more views. For example, a display interface including a text notification icon can include a view for displaying text and a view for displaying images.

[0732] The phone manager is used to provide communication functions of the electronic device 100, such as management of call status (including connecting, hanging up, etc.).

[0733] The resource manager provides various resources for applications, such as localized strings, icons, images, layout files, video files, and so on.

[0734] The notification manager enables applications to display notification information in the status bar. It can be used to convey notification-type messages and can disappear automatically after a short stay without user interaction. For example, the notification manager is used to notify download completion, message reminders, etc. The notification manager can also be a notification that appears in the system top status bar in the form of a chart or scroll bar text, such as notifications of applications running in the background, or a notification that appears on the screen in the form of a dialog window. For example, a text message is displayed in the status bar, a prompt sound is emitted, an electronic device vibrates, an indicator light flashes, etc.

[0735] In an embodiment of the present application, the activity manager service (Activity Manager Service, AMS), the window management service (Window Manager Service, WMS) and the download management service (Download Manager Service, DMS) in the application framework layer may also include an application keep-alive module, an event injection module, and a virtual screen management module.

[0736] Among them, the application keep-alive module is used to control the electronic device to enter the VR multi-screen display mode after the application with the multi-screen display mode function is started. In this mode, the electronic device can run multiple applications at the same time and support each application to be active at the same time.

[0737] The event injection module is used to obtain the events corresponding to the user's operations in the multi-screen display mode, and distribute the events to the virtual screen corresponding to the application.

[0738] The virtual screen management module is used to provide the electronic device with the ability to create and destroy a virtual screen.

[0739] Android Runtime includes core libraries and virtual machines. Android runtime is responsible for scheduling and management of the Android system.

[0740] The core library consists of two parts: one part is the function that needs to be called by the Java language, and the other part is the Android core library.

[0741] The application layer and the application framework layer run in a virtual machine. The virtual machine executes the Java files of the application layer and the application framework layer as binary files. The virtual machine is used to perform functions such as object life cycle management, stack management, thread management, security and exception management, and garbage collection.

[0742] The system library may include multiple functional modules, such as surface manager, media library, 3D graphics processing library (such as OpenGL ES), 2D graphics engine (such as SGL), etc.

[0743] The surface manager is used to manage the display subsystem and provide the fusion of 2D and 3D layers for multiple applications.

[0744] The media library supports playback and recording of a variety of commonly used audio and video formats, as well as static image files, etc. The media library can support a variety of audio and video encoding formats, such as: MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc.

[0745] The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, compositing, and layer processing.

[0746] A 2D graphics engine is a drawing engine for 2D drawings.

[0747] The kernel layer is the layer between hardware and software. The kernel layer contains at least display driver, camera driver, audio driver, and sensor driver.

[0748] like Fig.22cAs shown, in the embodiment of the present application, the VR scene display adapter module (HwVRDisplayAdaper) implements the creation and destruction of virtual screens in multi-screen mode, realizes the management of virtual screens, and opens the creation of virtual screen interfaces (createVRDisplay{}) and destruction of virtual screen interfaces (destroyVRDisplay{}) for other services (such as the display management module (DisplayManager), the display management global module (DisplayManagerGlobal), and the display management service module (DisplayManagerService), etc.). The display management module, the display management global module, and the display management service module complete the function of creating a virtual screen when entering the VR multi-screen display mode by calling the creation virtual screen interface layer by layer. In addition, the display management module, the display management global module, and the display management service module complete the function of destroying the virtual screen when exiting the VR multi-screen display mode by calling the creation virtual screen interface layer by layer. Fig.22c In the example, the display management service module will register the callback required to create a virtual screen when the electronic device is powered on and initialized, that is, the display management service module first calls the interface of the VR scene display adapter module (registerVRDisplayAdapterLoked{}), and then calls the registration interface (registerLocked{}) to complete the registration and creation of the virtual screen.

[0749] like Fig.22d As shown, in the embodiment of the present application, Android supports the specified screen injection event, and customizes the interface to open the screen injection event capability to other services. Fig.22d In the example, the input management module (IputManager) calls the injectInputEvent(event, mode, displayid) interface, the injectInputEventToDisplay(event, mode, displayid) interface, and the injectInputEventInternal(event, displayId, mode)) interface in sequence to inject events into the specified virtual screen.

[0750] Reference Fig.23a , Fig.23a This is an interactive diagram of a user using a first electronic device, where the user operates a VR / AR application on the first electronic device based on a handle, such as Fig.23aAs shown in the figure, the display interface of the second electronic device includes the current display image 2300 of the first electronic device, a function selection area 2302 and an interaction mode display area 2301, wherein the interaction mode display area 2301 indicates that the current interaction mode is a handle, and the user can operate the handle to operate the current interface.

[0751] Reference Figure 23b , Figure 23b It is a schematic diagram of an interaction of a user using a first electronic device, such as Figure 23b As shown in FIG. 2 , the user can slide the touch screen 2303 on the handle to click the interactive mode display area 2301 to switch the interactive mode based on the handle to the interactive mode based on the first electronic device (such as Fig.23d interface shown).

[0752] In an embodiment of the present application, when the handle establishes a connection with the second electronic device, the handle or the first electronic device can be used as a control device for the second electronic device; when only the first electronic device establishes a connection with the second electronic device, the first electronic device can be used as a control device for the second electronic device.

[0753] In an embodiment of the present application, the handle can be used as a control device of the second electronic device, and based on receiving a first interaction mode switching instruction and in response to the first interaction mode switching instruction, the control device of the second electronic device is switched from the handle to the first electronic device.

[0754] In an embodiment of the present application, the first electronic device can receive a first interaction mode switching instruction sent by the handle, generate a corresponding first interaction mode switching instruction based on receiving a second operation on the first electronic device, or receive a first interaction mode switching instruction sent by the second electronic device.

[0755] Optionally, in one embodiment, a corresponding physical button may be provided on the handle to implement the function of switching to an interactive mode based on the first electronic device. In the embodiment of the present application, the user may press the above-mentioned physical button to switch to an interactive mode based on the first electronic device (switching the control device of the second electronic device from the handle to the first electronic device).

[0756] Optionally, in another embodiment, the user can also switch to the interaction mode based on the first electronic device by pressing a physical button on the first electronic device (for example, Fig.23c In the embodiment of the present application, the user can click the power button to switch to the interaction mode based on the first electronic device.

[0757] Optionally, in another embodiment, the interaction mode based on the first electronic device can be switched by pressing a physical button on the second electronic device. In the embodiment of the present application, the user can press the above physical button to switch to the interaction mode based on the first electronic device.

[0758] Optionally, in another embodiment, if it is detected that the handle is not connected to the second electronic device, the interaction mode based on the first electronic device can be directly used.

[0759] Similarly, the first electronic device can switch to the handle-based interaction mode in the same manner.

[0760] In the handle-based interaction mode, the user can control the display position of the cursor on the second electronic device based on the operation of the handle.

[0761] In the interaction mode based on the first electronic device, the user can control the display position of the cursor on the second electronic device based on the operation on the first electronic device.

[0762] Regarding how the user controls the display position of the cursor on the second electronic device based on the operation on the first electronic device, reference may be made to the description in the above embodiment, which will not be repeated here.

[0763] Optionally, in one embodiment, different from the above-mentioned method of indicating the user's operation on the object by displaying a cursor, in an embodiment of the present application, a ray radiating from the bottom side of the screen (or from the mobile phone image displayed by the second electronic device) to the currently displayed content can be displayed in the second electronic device, and the ray includes an end point, which can indicate the user's currently selected position (equivalent to the above-mentioned cursor).

[0764] At this time, the user can adjust the direction of the ray by adjusting the posture of the first electronic device, so that the user can adjust the position of the end point of the ray to the target object that the user wants to operate. Fig.24a , Fig.24a An example of an interaction of a user using a first electronic device provided in an embodiment of the present application is as follows: Fig.24a As shown, if the user wants to operate the position corresponding to area A, the user can adjust the direction of the ray by adjusting the posture of the first electronic device so that the user can adjust the position of the end point of the ray to the target position (area A) that the user wants to operate.

[0765] Reference Figure 24b , Figure 24b An example of an interaction of a user using a first electronic device provided in an embodiment of the present application is as follows: Figure 24bAs shown, the user can rotate the first electronic device on a horizontal plane, and accordingly, the direction of the ray displayed by the second electronic device will change accordingly, and the end point position of the ray will be displaced in the horizontal direction.

[0766] Reference Fig.24c , Fig.24c An example of an interaction of a user using a first electronic device provided in an embodiment of the present application is as follows: Fig.24c As shown, the user can rotate the first electronic device in the vertical plane, and accordingly, the end point position of the ray displayed by the second electronic device will be displaced in the vertical direction.

[0767] In the embodiment of the present application, after the user adjusts the position of the end point of the ray to the target position to be operated, the user can click on the display screen of the first electronic device, and accordingly, the click operation can be performed at the position of the end point of the ray.

[0768] Specifically, in one embodiment, after the user adjusts the position of the end point of the ray to the target position he wants to operate, the first electronic device can obtain the specific pixel coordinate position of the end point of the ray in the display interface of the second electronic device, and determine the pixel coordinate position corresponding to the display interface of the first electronic device based on the specific pixel coordinate position of the end point of the ray in the display interface of the second electronic device.

[0769] The user performs a click operation on the display screen of the first electronic device. At this time, the click operation will not be responded to by the foreground application (the display interface of the first electronic device at this time). The first electronic device will inject the event corresponding to the click operation into the pixel coordinate position corresponding to the end point of the ray in the display interface of the first electronic device, which is equivalent to making the first electronic device perform the above-mentioned click operation at the pixel coordinate position corresponding to the end point of the ray in the first electronic device.

[0770] Optionally, in another embodiment, the user performs a sliding operation on the display screen of the first electronic device. At this time, the sliding operation will not be responded to by the foreground application (the display interface of the first electronic device at this time), and the first electronic device will inject the event corresponding to the sliding operation into the pixel coordinate position corresponding to the end point of the ray in the display interface of the first electronic device, which is equivalent to making the first electronic device perform the above sliding operation at the pixel coordinate position corresponding to the end point of the ray in the first electronic device. For more detailed descriptions, please refer to the above embodiments, which will not be repeated here.

[0771] Reference Fig.25 , Fig.25This is a schematic diagram of the operation of a first electronic device provided by an embodiment of the present application. In addition to the single-click confirmation operation, the first electronic device can also support commonly used full-screen gesture operations, which is consistent with the original way of using the handle, reducing the user's learning cost. Fig.25 As shown in the figure, sliding from the left or right edge to the middle represents the return button (corresponding to the return button in the handle), sliding from the bottom edge to the middle represents the home button (corresponding to the home button in the handle), sliding up, down, left and right can be used to fast forward and rewind in the video scene (corresponding to the touch screen sliding in the handle), the volume button adjusts the volume (corresponding to the volume adjustment button in the handle), sliding from the bottom edge to the middle and staying to return the perspective to the center (corresponding to the long press operation of the home button in the handle to return the perspective to the center), pressing the volume down button and the power button at the same time to take a screenshot, etc.

[0772] The principle of implementing the above-mentioned full-screen gesture operation can be referred to the description of the embodiment in the screen projection scenario, which will not be repeated here.

[0773] Reference Fig.26 , Fig.26 An example of an interaction of a user using a first electronic device provided in an embodiment of the present application is as follows: Fig.26 As shown, the user can slide on the display screen of the first electronic device, and accordingly, the pointer displayed on the second electronic device is displaced. Regarding how the user slides on the display screen of the first electronic device to achieve the displacement of the pointer displayed on the second electronic device, please refer to the description in the above embodiment, which will not be repeated here.

[0774] Reference Fig. 27 , Fig. 27 A schematic diagram of a system architecture provided for this application example, such as Fig. 27 As shown in FIG, taking the first electronic device as a mobile phone as an example, the system architecture includes: an AR device / VR device / MR device, a mobile phone, and an independent handle. The independent handle can be connected to the AR device / VR device / MR device, and the mobile phone can be connected to the AR device / VR device / MR device.

[0775] In one implementation, the user can interact with the first electronic device and control the display content of the AR device / VR device / MR device by operating the independent handle.

[0776] Specifically, the independent handle can obtain its own posture information or sliding information on the touchpad, and send the obtained posture information or sliding information to the mobile phone through the AR device / VR device / MR device. The mobile phone can process the above posture information or sliding information based on the independent handle interaction method, and move the pointer in the interface content based on the posture information or sliding information. In addition, the independent handle can obtain selection instructions (for example, through physical buttons on the independent handle) and send the selection instructions to the mobile phone. The mobile phone can process the above selection instructions based on the independent handle interaction method.

[0777] In one implementation, the user can interact with the first electronic device and control the display content of the AR device / VR device / MR device by operating the mobile phone.

[0778] Specifically, the mobile phone can obtain its own posture information or sliding information on the display screen. The mobile phone can process the above posture information or sliding information based on the mobile phone interaction method, and move the pointer in the interface content based on the posture information or sliding information. In addition, the independent handle can obtain touch operations, generate corresponding events, execute the generated events, and execute the events in the current front-end application.

[0779] Reference Fig.28 , Fig.28 A schematic diagram of a system architecture provided in an embodiment of the present application is shown as follows: Fig.28 As shown in the figure, the system architecture includes a system (for example, an Android system), enhanced interactive services, screen-off and screen-on services, and AR / VR / MR services.

[0780] Among them, the system can send information input by the sensor (such as posture information) to the enhanced interaction service, the system can send information input by the touch screen (such as touch events) to the AR / VR / MR service, the AR / VR / MR service can send information input by the touch screen to the enhanced interaction service, the enhanced interaction service can send instructions for turning off or on the screen to the screen on / off service, and the screen on / off service can send instructions for turning off or on the screen to the system to realize turning off or on the screen of the first electronic device.

[0781] The enhanced interaction service can process the received information (information input by the sensor or the touch screen), draw the pointer, and execute corresponding events for the AR / VR / MR application (for example, it can be based on the position of the pointer).

[0782] The present application also provides an electronic device, the electronic device comprising:

[0783] A sending module, configured to establish a connection with a second electronic device, wherein the second electronic device displays a cursor and interface content of the first electronic device, wherein the first electronic device includes a touch screen, and the second electronic device is an augmented reality (AR) device, a virtual reality (VR) device, or a mixed reality (MR) device;

[0784] The processing module is used to obtain the operation on the first electronic device, and control the display position of the cursor on the second electronic device based on the operation on the first electronic device.

[0785] Optionally, the interface content is the interface content of the front-end application of the first electronic device.

[0786] Optionally, the second electronic device further displays: a menu bar, and the menu bar does not belong to the interface content of the first electronic device.

[0787] Optionally, the processing module is specifically configured to obtain position change information of the first electronic device, and move the cursor on display content of the second electronic device based on the position change information.

[0788] Optionally, the posture change information includes a first rotation angle of the first electronic device on a horizontal plane and a second rotation angle of the first electronic device on a vertical plane, and the processing module is specifically used to:

[0789] determining a horizontal displacement of the cursor according to the first rotation angle;

[0790] determining a vertical displacement of the cursor according to the second rotation angle;

[0791] The cursor is moved on the display content of the second electronic device according to the horizontal displacement and the vertical displacement.

[0792] Optionally, the posture change information includes a first displacement of the first electronic device in a first direction and a second displacement of the first electronic device in a second direction, the first direction is parallel to a left-right movement direction of a cursor displayed by the second electronic device, and the second direction is parallel to an up-and-down movement direction of a cursor displayed by the second electronic device, and the processing module is specifically used to:

[0793] determining a horizontal displacement of the cursor according to the first displacement;

[0794] determining a vertical displacement of the cursor according to the second displacement;

[0795] The cursor is moved on the display content of the second electronic device according to the horizontal displacement and the vertical displacement.

[0796] Optionally, the processing module is further used to:

[0797] When the cursor moves to the target object in the display content of the second electronic device, if a first touch operation on the display screen of the first electronic device is received;

[0798] Then, the target object in the display content of the second electronic device responds to the first touch operation.

[0799] Optionally, the operation position of the first touch operation corresponds to a second object in the interface content of the first electronic device, and the processing module is further used to:

[0800] Shielding the second object from responding to the first touch operation.

[0801] Optionally, the first touch operation includes at least a click operation and a first sliding operation, and the target object includes at least an application and a function control.

[0802] Optionally, the processing module is specifically used to:

[0803] receiving a second sliding operation on the display screen of the first electronic device;

[0804] determining a displacement of the cursor according to the second sliding operation;

[0805] The cursor is moved in the display content of the second electronic device based on the displacement.

[0806] Optionally, the starting operation position corresponding to the second sliding operation corresponds to a third object in the interface content of the first electronic device, and the processing module is further used to:

[0807] Shielding the third object from responding to the second touch operation.

[0808] Optionally, the processing module is further used to:

[0809] When the cursor moves to the target object in the display content of the second electronic device, if a click operation on the display screen of the first electronic device is received;

[0810] Then, the target object in the display content of the second electronic device responds to the click operation.

[0811] Optionally, the operation position of the click operation corresponds to a fourth object in the interface content of the first electronic device, and the processing module is further used to:

[0812] Shielding the fourth object from responding to the click operation.

[0813] Optionally, the second electronic device further displays: a ray, the end point of the ray being the cursor.

[0814] Optionally, the processing module is further used to:

[0815] Receiving a pressing operation on a physical button of the first electronic device;

[0816] Generate a corresponding second event according to the pressing operation;

[0817] The second event is executed on the first electronic device front-end application.

[0818] Optionally, the processing module is further used to receive a second operation on the display screen of the first electronic device, where the second operation is a preset shortcut operation, and enable the first electronic device to respond to the second operation.

[0819] Optionally, the operation position of the second operation corresponds to a fifth object in the interface content of the second electronic device, and the fifth object does not respond to the first operation.

[0820] Optionally, the preset shortcut operations include at least:

[0821] Contacting a first preset area of ​​the display screen of the first electronic device, and sliding from the first preset area to a first preset direction;

[0822] The user touches a second preset area of ​​the display screen of the first electronic device, slides from the second preset area to a second preset direction, and the time of contacting the display screen of the first electronic device is greater than a preset time.

[0823] Optionally, the processing module is further used to reduce the display brightness of the display screen of the first electronic device; or,

[0824] A screen-off operation is performed on the display screen of the first electronic device.

[0825] Optionally, the processing module is specifically used to:

[0826] Based on the first application being displayed on the second electronic device, obtaining position change information of the first electronic device, and moving the cursor on the display content of the second electronic device based on the position change information;

[0827] Based on displaying a second application on the second electronic device, receiving a second sliding operation on the display screen of the first electronic device, determining the displacement of the cursor according to the second sliding operation, and moving the cursor in the display content of the second electronic device based on the displacement, the first application and the second application are different applications.

[0828] The embodiment of the present application further provides a first electronic device, the first electronic device is connected to a second electronic device, the first electronic device includes a touch screen, the second electronic device is an augmented reality (AR) device, a virtual reality (VR) device, or a mixed reality (MR) device, and the first electronic device includes:

[0829] a processing module, configured to use the handle or the first electronic device as a control device for the second electronic device when the handle is connected to the second electronic device;

[0830] When only the first electronic device is connected to the second electronic device, the first electronic device is used as a control device of the second electronic device.

[0831] Optionally, the processing module is specifically used to:

[0832] Using the handle as a control device for the second electronic device;

[0833] Receiving a first interaction mode switching instruction;

[0834] In response to the first interaction mode switching instruction, the control device of the second electronic device is switched from the handle to the first electronic device.

[0835] Optionally, the processing module is specifically used to:

[0836] Receive a first interaction mode switching instruction sent by the handle, generate a corresponding first interaction mode switching instruction based on receiving a second operation on the first electronic device, or receive a first interaction mode switching instruction sent by the second electronic device.

[0837] Optionally, the processing module is specifically used to:

[0838] Using the first electronic device as a control device for the second electronic device;

[0839] The first electronic device further includes an acquisition module, which is used to receive a second interaction mode switching instruction;

[0840] The processing module is specifically configured to switch the control device of the second electronic device from the first electronic device to the handle in response to the second interaction mode switching instruction.

[0841] Optionally, the processing module is specifically used to:

[0842] Receive a second interaction mode switching instruction sent by the handle, generate a corresponding second interaction mode switching instruction based on receiving a second operation on the first electronic device, or receive a second interaction mode switching instruction sent by the second electronic device.

[0843] Optionally, the second electronic device displays a cursor and interface content of the first electronic device, and the processing module is specifically configured to:

[0844] Based on the operation of the handle, controlling the display position of the cursor on the second electronic device;

[0845] Based on the operation on the first electronic device, the display position of the cursor on the second electronic device is controlled.

[0846] Optionally, the interface content is the interface content of the front-end application of the first electronic device.

[0847] Optionally, the processing module is specifically used to:

[0848] Acquiring position change information of the first electronic device;

[0849] The cursor is moved on the display content of the second electronic device based on the posture change information.

[0850] Optionally, the processing module is further used to:

[0851] When the cursor moves to the target object in the display content of the second electronic device, if a first touch operation on the display screen of the first electronic device is received;

[0852] Then, the target object in the display content of the second electronic device responds to the first touch operation.

[0853] Optionally, the processing module is further used to:

[0854] receiving a second sliding operation on the display screen of the first electronic device;

[0855] determining a displacement of the cursor according to the second sliding operation;

[0856] The cursor is moved in the display content of the second electronic device based on the displacement.

[0857] Optionally, the processing module is further used to:

[0858] When the cursor moves to the target object in the display content of the second electronic device, if a click operation on the display screen of the first electronic device is received;

[0859] Then, the target object in the display content of the second electronic device responds to the click operation.

[0860] The present application also provides an electronic device, including:

[0861] A sending module, configured to display the interface content of the first electronic device in a second electronic device, wherein the first electronic device includes a touch screen, and the second electronic device is an augmented reality (AR) device, a virtual reality (VR) device, or a mixed reality (MR) device;

[0862] A processing module, used for receiving a first operation performed on a first display screen of the first electronic device; causing the interface content displayed by the second electronic device to respond to the first operation; wherein an operation position of the first operation corresponds to a first object in the interface content of the first electronic device; and the first object does not respond to the first operation.

[0863] Optionally, causing the interface content displayed by the second electronic device to respond to the first operation specifically includes:

[0864] A first position is determined in the interface content displayed by the first electronic device; and the interface content displayed by the second electronic device responds to the first operation based on the first position; wherein the first position is independent of the operation position of the first operation.

[0865] Optionally, the electronic device further includes:

[0866] The cursor is projected onto the second electronic device so that the second electronic device displays the cursor.

[0867] Optionally, the position corresponding to the cursor in the interface content displayed by the first electronic device is the first position.

[0868] Optionally, determining the first position in the interface content displayed by the first electronic device includes:

[0869] Acquiring position change information of the first electronic device;

[0870] A first position of the cursor in the interface content displayed by the first electronic device is determined based on the posture change information.

[0871] Optionally, the posture change information includes a first rotation angle of the first electronic device on a horizontal plane and a second rotation angle of the first electronic device on a vertical plane, and determining a first position of the cursor in interface content displayed by the first electronic device based on the posture change information includes:

[0872] determining a horizontal displacement of the cursor according to the first rotation angle;

[0873] determining a vertical displacement of the cursor according to the second rotation angle;

[0874] A first position of the cursor in the interface content displayed by the first electronic device is determined according to the horizontal displacement and the vertical displacement.

[0875] Optionally, the posture change information includes a first displacement of the first electronic device in a first direction and a second displacement of the first electronic device in a second direction, the first direction is parallel to a left-right movement direction of a cursor displayed on the second electronic device, and the second direction is parallel to an up-and-down movement direction of a cursor displayed on the second electronic device, and determining a first position of the cursor in the interface content displayed by the first electronic device based on the posture change information includes:

[0876] determining a horizontal displacement of the cursor according to the first displacement;

[0877] determining a vertical displacement of the cursor according to the second displacement;

[0878] A first position of the cursor in the interface content displayed by the first electronic device is determined according to the horizontal displacement and the vertical displacement.

[0879] Optionally, the first operation includes at least a click operation and a first sliding operation, and the first object includes at least one of an application and a function control.

[0880] Optionally, determining the first position in the interface content displayed by the first electronic device includes:

[0881] receiving a second sliding operation on the display screen of the first electronic device;

[0882] determining a displacement of the cursor according to the second sliding operation;

[0883] A first position of the cursor in the interface content displayed by the first electronic device is determined according to the displacement of the cursor.

[0884] Optionally, the first operation includes at least a click operation, and the first object includes at least one of an application and a function control.

[0885] Optionally, the electronic device further includes:

[0886] reducing the display brightness of the first display screen; or,

[0887] Perform a screen-off operation on the first display screen.

[0888] Optionally, the electronic device further includes:

[0889] The menu bar is displayed on the second electronic device.

[0890] Reference Fig.29 , Fig.29 A schematic flow chart of an operation method for screen projection provided in an embodiment of the present application, wherein a first electronic device has a first display screen, and a second electronic device has a second display screen; Fig.29 As shown in the figure, the operation method for screen projection provided by the embodiment of the present application includes:

[0891] 2901. The first electronic device projects the interface content of the first electronic device to a second electronic device, so that the second display screen of the second electronic device displays the interface content.

[0892] In the embodiment of the present application, the first electronic device may install an enhanced interactive service, and the enhanced interactive service may monitor the screen projection connection status of the first electronic device. The enhanced interactive service may detect that the first electronic device has established a screen projection connection with the second electronic device. For the method of establishing the screen projection between the first electronic device and the second electronic device, please refer to Figures 4a to 5c The description of the corresponding embodiments will not be repeated here.

[0893] In an embodiment of the present application, after detecting that a first electronic device establishes a screen projection connection with a second electronic device, the first electronic device projects the interface content displayed on the first display screen to the second electronic device, so that the second display screen displays the interface content.

[0894] Optionally, the first electronic device may generate a cursor and project the cursor on the second electronic device so that the second display screen displays the cursor. For how the first electronic device generates a cursor, reference may be made to the corresponding embodiment described in step 301, which will not be repeated here.

[0895] Optionally, the first electronic device may generate a menu bar and project the menu bar on the second electronic device so that the second display screen displays the menu bar. For how the first electronic device generates a menu bar, reference may be made to the corresponding embodiment described in step 301, which will not be repeated here.

[0896] Optionally, the first electronic device may obtain the posture change information of the first electronic device, and move the cursor on the content displayed on the second display screen based on the posture change information. Regarding how to move the cursor on the display screen of the second electronic device based on the posture change information of the first electronic device, reference may be made to the description in the above embodiment, which will not be repeated here.

[0897] Optionally, the first electronic device can determine a first position in the interface content displayed by the first electronic device, so that the interface content displayed by the second display screen responds to the first operation based on the first position; wherein the first position is independent of the operation position of the first operation.

[0898] Optionally, the first electronic device may obtain the position change information of the first electronic device, and determine the first position of the cursor in the interface content displayed by the first electronic device based on the position change information. In this case, the position corresponding to the cursor in the interface content displayed by the first electronic device is the first position.

[0899] Optionally, the first electronic device may also receive a second sliding operation on the first display screen, determine the displacement of the cursor according to the second sliding operation, and move the cursor on the content displayed on the second display screen based on the displacement of the cursor. Regarding how to move the cursor on the display screen of the second electronic device according to the second sliding operation, reference may be made to the description in the above embodiment, which will not be repeated here.

[0900] Optionally, the first electronic device may receive a second sliding operation on the display screen of the first electronic device, determine the displacement of the cursor according to the second sliding operation, and determine the first position of the cursor in the interface content displayed by the first electronic device according to the displacement of the cursor.

[0901] In the embodiment of the present application, after the first position is determined in the interface content displayed by the first electronic device, the interface content displayed by the second display screen can respond to the first operation based on the first position. Regarding how the first position responds to the first operation, reference can be made to the description of step 302 in the above embodiment, which will not be repeated here.

[0902] Optionally, after detecting that the first electronic device establishes a screen projection connection with the second electronic device, the first electronic device may also reduce the display brightness of the first display screen; or, perform a screen turning off operation on the first display screen.

[0903] 2902. The first electronic device receives a first operation applied to a first display screen of the first electronic device; causes the interface content displayed on the second display screen to respond to the first operation; wherein an operation position of the first operation corresponds to a first object in the interface content of the first electronic device; and the first object does not respond to the first operation.

[0904] In the embodiment of the present application, the first object may include at least one of an application and a function control.

[0905] In this embodiment, the first electronic device can shield the response of the content of the current application to the operation position of the first operation, that is, the operation position of the first operation corresponds to a first object in the interface content of the first electronic device, and the first object does not respond to the first operation.

[0906] Specifically, after receiving the first operation on the display screen of the first electronic device, the first electronic device can shield the response of the interface content to the operation position of the first operation, and determine the position of the cursor on the second display screen to determine the response position (first position) of the interface content to the first operation. For example, if the user performs a click operation on object A of the first display screen of the first electronic device, and the cursor is located on object B in the second electronic device, the first electronic device can shield the response of object A to the click operation, and respond to the above click operation on object B of the interface content. In other words, the first position has nothing to do with the operation position of the first operation, but is only related to the position of the cursor on the second display screen of the second electronic device.

[0907] In an embodiment of the present application, the first electronic device responds to a first operation at the first position of the content displayed on the first display screen, and the second display screen of the second electronic device also synchronously displays the content displayed on the first display screen updated after responding to the first operation at the first position, that is, the content displayed on the second display screen responds to the first operation.

[0908] In an embodiment of the present application, the interface content of a first electronic device is projected onto a second electronic device, so that the second display screen of the second electronic device displays the interface content; a first operation acting on the first display screen of the first electronic device is received; so that the interface content displayed on the second display screen responds to the first operation; wherein the operation position of the first operation corresponds to a first object in the interface content of the first electronic device; and the first object does not respond to the first operation. In the above manner, after the first electronic device and the second electronic device establish a projection connection, the content in the first display screen does not respond to the first object corresponding to the operation position where the first operation is located, and the content displayed on the second display screen responds to the first operation, so that the user does not need to look at the content of the first display screen of the first electronic device, but can operate the content of the first display screen based on the content displayed on the second display screen.

[0909] The embodiment of the present application further provides a first electronic device, the first electronic device comprising:

[0910] A processing module, configured to project the interface content of the first electronic device to a second electronic device, so that a second display screen of the second electronic device displays the interface content;

[0911] The processing module is also used to receive a first operation performed on a first display screen of the first electronic device; so that the interface content displayed on the second display screen responds to the first operation; wherein the operation position of the first operation corresponds to a first object in the interface content of the first electronic device; and the first object does not respond to the first operation.

[0912] Optionally, the processing module is specifically used to:

[0913] A first position is determined in the interface content displayed by the first electronic device; and the interface content displayed by the second display screen responds to the first operation based on the first position; wherein the first position is independent of the operation position of the first operation.

[0914] Optionally, the processing module is further used to:

[0915] The cursor is projected onto the second electronic device so that the second display screen displays the cursor.

[0916] Optionally, the position corresponding to the cursor in the interface content displayed by the first electronic device is the first position.

[0917] Optionally, the processing module is specifically used to:

[0918] Acquiring position change information of the first electronic device;

[0919] A first position of the cursor in the interface content displayed by the first electronic device is determined based on the posture change information.

[0920] Optionally, the posture change information includes a first rotation angle of the first electronic device on a horizontal plane and a second rotation angle of the first electronic device on a vertical plane, and the processing module is specifically used to:

[0921] determining a horizontal displacement of the cursor according to the first rotation angle;

[0922] determining a vertical displacement of the cursor according to the second rotation angle;

[0923] A first position of the cursor in the interface content displayed by the first electronic device is determined according to the horizontal displacement and the vertical displacement.

[0924] Optionally, the posture change information includes a first displacement of the first electronic device in a first direction and a second displacement of the first electronic device in a second direction, the first direction is parallel to a left-right movement direction of a cursor displayed on a display screen of the second electronic device, and the second direction is parallel to an up-and-down movement direction of a cursor displayed on a display screen of the second electronic device, and the processing module is specifically used to:

[0925] determining a horizontal displacement of the cursor according to the first displacement;

[0926] determining a vertical displacement of the cursor according to the second displacement;

[0927] A first position of the cursor in the interface content displayed by the first electronic device is determined according to the horizontal displacement and the vertical displacement.

[0928] Optionally, the first operation includes at least a click operation and a first sliding operation, and the first object includes at least one of an application and a function control.

[0929] Optionally, the processing module is specifically used to:

[0930] receiving a second sliding operation on the display screen of the first electronic device;

[0931] determining a displacement of the cursor according to the second sliding operation;

[0932] A first position of the cursor in the interface content displayed by the first electronic device is determined according to the displacement of the cursor.

[0933] Optionally, the first operation includes at least a click operation, and the first object includes at least one of an application and a function control.

[0934] Optionally, the first operation is a preset shortcut operation, and the processing module is further used to:

[0935] Causes the first electronic device to respond to the first operation.

[0936] Optionally, the operation position of the first operation corresponds to a third object in the interface content of the second electronic device, and the third object does not respond to the first operation.

[0937] Optionally, the preset shortcut operations include at least:

[0938] Contacting a first preset area of ​​the display screen of the first electronic device, and sliding from the first preset area to a first preset direction;

[0939] The user touches a second preset area of ​​the display screen of the first electronic device, slides from the second preset area to a second preset direction, and the time of contacting the display screen of the first electronic device is greater than a preset time.

[0940] Optionally, the processing module is further used to:

[0941] reducing the display brightness of the first display screen; or,

[0942] Perform a screen-off operation on the first display screen.

[0943] Optionally, the processing module is further used to:

[0944] Generate a menu bar;

[0945] Project the menu bar onto the second electronic device so that the second display screen displays the menu bar.

[0946] Optionally, the processing module is further used to:

[0947] It is detected that the first electronic device establishes a screen projection connection with the second electronic device.

[0948] The embodiment of the present application also provides a method for controlling multi-application display.

[0949] Specifically, the second electronic device (such as a large screen, a TV, an AR, VR, or MR device, etc.) can display multiple display interfaces of the first electronic device on the imaging system therein (such as a display screen, glasses, and other imaging systems).

[0950] Optionally, the first electronic device can generate a menu bar when establishing content transmission with the second electronic device. This menu bar can be displayed in the imaging system of the second electronic device. The user can add a new independent display interface of the first electronic device (such as a mobile phone interface) to the imaging system of the second electronic device by clicking the "Add" button in the menu bar. Fig.30 As shown;

[0951] Optionally, each time the user clicks on an add button, an independent mobile phone interface can be added to the imaging system. The state of the newly added interface can be the home interface of the mobile phone, the current display interface of the mobile phone, a pre-set default interface, an interface of a preset application, a copy interface of the mobile phone interface currently displayed by the imaging system, or a random interface. This is only an example and is not limited in the present invention. Since multiple independent interfaces of the mobile phone (such as the first interface and the second interface) can be presented in the imaging system, the content on the large screen will become richer.

[0952] Optionally, the user may also select to run a new application in the imaging system, and a new independent application interface will be generated in the imaging system of the second electronic device.

[0953] Optionally, by clicking a title in the app, the user can open a new independent interface of the second level of the app extension, such as a chat with a specific contact in WeChat, or a specific article in a news app.

[0954] Optionally, when the user establishes a content connection and transmission between the first electronic device and the second electronic device, the first electronic device has already run N applications, and the interface contents of the N applications can be correspondingly displayed in N display areas in the imaging system.

[0955] Optionally, the above-mentioned multiple interface contents may be distributed without overlap, or may be distributed in a stacked manner (keeping the display interface of the currently running front-end application displayed at the front-end), which is not limited in the present application.

[0956] When the first electronic device establishes content transmission with the second electronic device, the first electronic device can generate a cursor (such as the method of generating a cursor as described above), which can be displayed in the imaging system through data transmission. The moving range of the cursor can be adapted to the entire range of the imaging system. Specifically, the first electronic device can first obtain parameter information of the imaging system, such as the size of the large-screen display screen, the field of view parameters of the ARVR device, etc.; based on the parameter information, the moving range and mapping rules of the cursor are determined, which are no longer limited to a single first electronic device display interface, but can be operated across multiple different first electronic device display interfaces, and can reach the display boundary of the entire imaging system. In this way, when multiple first electronic device display interfaces are generated in the imaging system, the operating range of the cursor in the imaging system can cover the content in multiple first electronic device display interfaces, and can move freely in multiple first electronic device display interfaces. The cursor is used to determine the operation object in the content of the entire display range of the imaging system. The operation object can be an object in the display interface of the first electronic device, or it can be other objects in the imaging system that do not belong to the display interface of the first electronic device.

[0957] Optionally, the user can control the position of the cursor in the imaging system by adjusting the posture of the first electronic device as mentioned above (which can also be understood as posture in this embodiment) or sliding the screen of the first electronic device, and then determine the operation object in the content displayed in the imaging system according to the user's operation. Among them, the starting position of the sliding operation corresponds to the first object in the interface content of the first electronic device; the first object does not respond to the sliding operation. It should be understood that the sliding operation is a random operation that can be performed multiple times. The longer the trajectory of the first sliding operation, the larger the movement range of the corresponding cursor in the imaging system.

[0958] Optionally, when the cursor is located on an object (including but not limited to an application icon, a function key, an option of unlimited type, or a preset position (such as a blank space), etc.) in the imaging system, it indicates that the object has been confirmed as an operation object. At this time, the user inputs an operation representing "confirmation" on the terminal, such as a key, touch, click, or special slide, etc., and the operation object will respond to the "confirmation" command. For example, entering an application, enabling a function key, selecting an option, generating a shortcut, etc., the present invention does not list them all. It should be understood that at this time, if the "confirmation" operation is an operation on the touch screen, and the touch position of the "confirmation" operation corresponds to the second object in the interface content of the first electronic device; then the second object does not respond to the "confirmation" operation.

[0959] In a specific implementation process, the content of the first electronic device interface may be adapted to the content in the imaging system.

[0960] Optionally, the current interface content of the first electronic device may be "single screen". For example, if the cursor in the display interface of the imaging system is located at the first application interface, then the interface content of the first electronic device is also synchronized to the first application interface; if the cursor in the imaging system switches from the first application interface to the second application interface, then the interface content of the first electronic device is also switched from the first application interface to the second application interface.

[0961] Optionally, in the specific implementation process, the content of the interface of the first electronic device can be "multi-screen", for example, the display interface of the first electronic device corresponds to all display interfaces in the second electronic device. Specifically, if the display interface in the imaging system includes the first application interface and the second application interface, then the interface content of the first electronic device is also synchronized to the first application interface and the second application interface; if the display interface in the imaging system includes the first application interface, the second application interface and the third application interface, then the interface content of the first electronic device is also synchronized to the first application interface, the second application interface and the third application interface.

[0962] In order to save power consumption, when the first electronic device transmits content for display to the imaging system, the display brightness of the first electronic device may be reduced; or, a screen-off operation may be performed on the first electronic device.

[0963] In combination with the aforementioned embodiments, the present application proposes a novel control method based on a touch screen. The method can be applied to screen projection scenarios, which may include data content transmission and projection from smart terminal devices such as mobile phones, tablet computers, and notebooks to large display terminals such as televisions and large screens; it may also include data content transmission and projection from the above-mentioned smart terminal devices to imaging devices such as AR, VR, and MR...

Claims

1. A control method based on a touch screen, characterized in that: The method is applied to a first electronic device, wherein the first electronic device includes a touch screen; the touch screen can be used to display an interface; the method includes: When the first electronic device establishes a screen projection connection with the second electronic device, a cursor is generated, and the cursor is displayed on the interface of the touch screen; Generate projection content based on the interface content in the touch screen; the projection content includes the cursor; The projection content is sent to the second electronic device, and all or part of the changes in the interface content in the first electronic device are displayed in real time in the second electronic device following the projection content, and the movement of the cursor in the first electronic device is synchronously displayed in the second electronic device; When receiving a target touch operation performed by a user on the touch screen of the first electronic device, the target touch operation is responded to in the first electronic device based on the display position of the cursor in the current interface; wherein the display position of the cursor in the current interface is used to indicate the response position to the target touch operation in the current interface; and the response position is not related to the operation position of the target touch operation on the touch screen.

2. The method according to claim 1, characterized in that: When the first electronic device has not established a screen projection connection with the second electronic device, the touch screen interface does not include the cursor.

3. The method according to claim 1 or 2, characterized in that: The projected screen content includes part or all of the content in the interface, and the projected screen content is used to synchronize part or all of the content presented in the display system of the second electronic device with part or all of the content displayed on the touch screen.

4. The method according to claim 1 or 2, characterized in that: Generating a cursor comprises: Obtaining the interface content of the front-end application corresponding to the current interface, and drawing a cursor in the interface content; or, Draw a cursor, and add the cursor to the current interface based on a floating window interface; or, A ray is drawn, and based on the floating window interface, the ray is intersected with the current interface at an end point; the end point is the cursor.

5. The method according to claim 1 or 2, characterized in that: Before generating the cursor, the method further includes: calling the first event processing system and shielding the touch screen event processing system; The responding to the target touch operation in the first electronic device based on the display position of the cursor in the current interface includes: Re-analyzing the target touch operation into a target event, and injecting the target event into the first event processing system, so that the first electronic device responds to the target touch operation according to the first event processing system; Among them, the response of the touch screen event processing system to the target touch operation is related to the operation position of the target touch operation on the touch screen; the response of the first event processing system to the target touch operation is related to the position of the cursor, and is not related to the operation position of the target touch operation on the touch screen.

6. The method according to claim 1 or 2, characterized in that: Before generating the cursor, the method further includes: calling a first event processing system; Drawing a transparent target floating window on the current interface; the target floating window is used to shield the touch screen event processing system; The responding to the target touch operation in the first electronic device based on the display position of the cursor in the current interface includes: parsing a target event corresponding to the target touch operation through the target floating window, and injecting the target event into the first event processing system, so that the first electronic device responds to the target touch operation according to the first event processing system; Among them, the response of the touch screen event processing system to the target touch operation is related to the operation position of the target touch operation on the touch screen; the response of the first event processing system to the target touch operation is related to the position of the cursor, and is not related to the operation position of the target touch operation on the touch screen.

7. The method according to claim 5, characterized in that Before calling the first event processing system, the method further includes: detecting that the first electronic device establishes a screen projection connection with the second electronic device; or, A first switching command input by a user is detected; the first switching command is used to instruct the first electronic device to call the first event processing system and shield the touch screen event processing system.

8. The method according to any one of claims 1 to 2 and 7, characterized in that: The method further comprises: Reduce the display brightness of the touch screen, or turn off the touch screen.

9. The method according to any one of claims 1 to 2 and 7, characterized in that: When the target touch operation is a sliding operation, The responding to the target touch operation in the first electronic device based on the display position of the cursor in the current interface includes: The position of the cursor in the current interface is changed according to the sliding operation.

10. The method according to any one of claims 1 to 2 and 7, characterized in that: When the target touch operation is a click operation, The responding to the target touch operation in the first electronic device based on the display position of the cursor in the current interface includes: A confirmation operation is performed on the object corresponding to the cursor in the current interface.

11. The method according to any one of claims 1 to 2 and 7, characterized in that: When the target touch operation is a long press followed by a slide operation, The responding to the target touch operation in the first electronic device based on the display position of the cursor in the current interface includes: Slide or drag the cursor to the object corresponding to the interface.

12. The method according to any one of claims 1 to 2 and 7, characterized in that: When the target touch operation causes the movement trajectory of the cursor in the current interface to match a preset shortcut operation command, or when the target touch operation matches a preset shortcut operation command, The responding to the target touch operation in the first electronic device based on the display position of the cursor in the current interface includes: Execute the shortcut operation command.

13. The method according to claim 12, characterized in that: The target touch operation includes: Slide from a first preset area of ​​the touch screen to a first preset direction; or, Slide from a second preset area of ​​the touch screen to a second preset direction, and the time of contact with the touch screen is greater than a preset time; or, The number of times the touch screen is clicked reaches a preset number; or, The sliding track of the target touch operation satisfies a preset pattern.

14. The method according to any one of claims 1 to 2, 7 and 13, characterized in that: After the first electronic device establishes a screen projection connection with the second electronic device, the method further includes: Start multi-screen display mode; Creating N virtual screens in the interface of the touch screen; Determine N interfaces to be displayed; The N interfaces to be displayed are displayed respectively in the N virtual screens; wherein the N interfaces to be displayed belong to M applications, M is an integer not greater than N, and N is an integer greater than 1; the cursor can be located at any position in the N virtual screens.

15. The method according to any one of claims 1 to 2, 7 and 13, characterized in that: The projection content also includes an interface of a background application running on the first electronic device, a newly drawn menu bar, or at least one of preset function controls.

16. The method according to any one of claims 1 to 2, 7 and 13, characterized in that: The first electronic device includes a mobile phone or a tablet; the second electronic device includes a display, a television, a projector, an AR device, a VR device or an MR device.

17. The method according to claim 5, characterized in that: The first electronic device further includes a second event processing system; and the method further includes: Acquire a second switching instruction; the second switching instruction is used to instruct the first electronic device to call a second event processing system and shield the first event processing system and the touch screen event processing system; The second event processing system is called; wherein, when the posture of the first electronic device changes, the response of the second event processing system to the posture change of the first electronic device is to change the position of the cursor in the current interface.

18. The method according to any one of claims 1-2, 7, 13, and 17, characterized in that: Before the cursor is displayed in the interface of the touch screen, the method further includes: receiving a predefined operation, where the predefined operation is used to activate or hide the display of the cursor.

19. A control device based on a touch screen, characterized in that: The device is applied to a first electronic device, wherein the first electronic device includes a touch screen; the touch screen can be used to display an interface; the device includes: A generating module, configured to generate a cursor when the first electronic device establishes a screen projection connection with the second electronic device, wherein the cursor is displayed in the interface of the touch screen; a screen projection module, configured to generate screen projection content based on the interface content in the touch screen, wherein the screen projection content includes the cursor; and to send the screen projection content to the second electronic device, so that all or part of the changes in the interface content in the first electronic device are displayed in real time in the second electronic device following the screen projection content, and the movement of the cursor in the first electronic device is synchronously displayed in the second electronic device; A receiving module, configured to receive a target touch operation performed by a user on the touch screen of the first electronic device; A response module, configured to respond to the target touch operation in the first electronic device based on the display position of the cursor in the current interface; The display position of the cursor in the current interface is used to indicate the response position of the target touch operation in the current interface; the response position is irrelevant to the operation position of the target touch operation on the touch screen.

20. The device according to claim 19, characterized in that When the first electronic device has not established a screen projection connection with the second electronic device, the touch screen interface does not include the cursor.

21. The device according to claim 19 or 20, characterized in that The projected screen content includes part or all of the content in the interface, and the projected screen content is used to synchronize part or all of the content presented in the display system of the second electronic device with part or all of the content displayed on the touch screen.

22. The device according to claim 19 or 20, characterized in that The generation module is specifically used for: Obtaining the interface content of the front-end application corresponding to the current interface, and drawing a cursor in the interface content; or, Draw a cursor, and add the cursor to the current interface based on a floating window interface; or, Draw a ray, and intersect the ray with the current interface at an end point based on the floating window interface; The end point is the cursor.

23. The device according to claim 19 or 20, characterized in that The device also includes: A calling module, used for calling the first event processing system and shielding the touch screen event processing system before the generating module generates the cursor; The response module is specifically used to re-parse the target touch operation into a target event, and inject the target event into the first event processing system, so that the first electronic device responds to the target touch operation according to the first event processing system; Among them, the response of the touch screen event processing system to the target touch operation is related to the operation position of the target touch operation on the touch screen; the response of the first event processing system to the target touch operation is related to the position of the cursor, and is not related to the operation position of the target touch operation on the touch screen.

24. The device according to claim 19 or 20, characterized in that The device also includes: The calling module is used to call the first event processing system before the generating module generates the cursor; and draw a transparent target floating window on the front-end application corresponding to the current interface; the target floating window is used to shield the touch screen event processing system; The response module is specifically used to parse the target event corresponding to the target touch operation through the target floating window, and inject the target event into the first event processing system, so that the first electronic device responds to the target touch operation according to the first event processing system; Among them, the response of the touch screen event processing system to the target touch operation is related to the operation position of the target touch operation on the touch screen; the response of the first event processing system to the target touch operation is related to the position of the cursor, and is not related to the operation position of the target touch operation on the touch screen.

25. The device as claimed in claim 23, characterized in that The device further includes a detection module, which is used for, before the calling module calls the first event processing system, Detecting that the first electronic device establishes a screen projection connection with the second electronic device; or, A first switching command input by a user is detected; the first switching command is used to instruct the first electronic device to call the first event processing system and shield the touch screen event processing system.

26. The device according to any one of claims 19 to 20 and 25, characterized in that: The device also includes: The brightness control module is used to reduce the display brightness of the touch screen or turn off the touch screen.

27. The device according to any one of claims 19 to 20 and 25, characterized in that: When the target touch operation is a sliding operation, the response module is used to change the position of the cursor in the current interface according to the sliding operation; or, When the target touch operation is a click operation, the response module is used to perform a confirmation operation on the object corresponding to the cursor in the current interface; or, When the target touch operation is a long press followed by a slide operation, the response module is used to slide or drag the object corresponding to the cursor in the interface; or, When the target touch operation causes the movement trajectory of the cursor in the current interface to match a preset shortcut operation command, or when the target touch operation matches a preset shortcut operation command, the response module is used to execute the shortcut operation command.

28. The device according to claim 27, characterized in that The target touch operation matching the preset shortcut operation command includes: Slide from a first preset area of ​​the touch screen to a first preset direction; or, Slide from a second preset area of ​​the touch screen to a second preset direction, and the time of contact with the touch screen is greater than a preset time; or, The number of times the touch screen is clicked reaches a preset number; or, The sliding track of the target touch operation satisfies a preset pattern.

29. The device according to any one of claims 19 to 20, 25 and 28, characterized in that: The device also includes a multi-screen creation module. Used to start multi-screen display mode; Creating N virtual screens in the interface of the touch screen; Determine N interfaces to be displayed; The N interfaces to be displayed are displayed respectively in the N virtual screens; wherein the N interfaces to be displayed belong to M applications, M is an integer not greater than N, and N is an integer greater than 1; the cursor can be located at any position in the N virtual screens.

30. The device according to any one of claims 19 to 20, 25 and 28, characterized in that: The projection content also includes an interface of a background application running on the first electronic device, a newly drawn menu bar, or at least one of preset function controls.

31. The device according to any one of claims 19 to 20, 25 to 28, characterized in that: The first electronic device includes a mobile phone or a tablet; the second electronic device includes a display, a television, a projector, an AR device, a VR device or an MR device.

32. The device according to claim 23, characterized in that The first electronic device also includes a second event processing system; The calling module is also used for: Acquire a second switching instruction; the second switching instruction is used to instruct the first electronic device to call a second event processing system and shield the first event processing system and the touch screen event processing system; The second event processing system is called; wherein, when the posture of the first electronic device changes, the response of the second event processing system to the posture change of the first electronic device is to change the position of the cursor in the current interface.

33. The device according to any one of claims 19 to 20, 25, 28 and 32, characterized in that: The device further comprises an activation module, which is used to receive a predefined operation, where the predefined operation is used to activate or hide the display of the cursor.

34. A terminal device, characterized in that: The terminal device includes a memory, a processor, a touch screen, and a bus; the memory, the touch screen, and the processor are connected via the bus; The memory is used to store computer programs and instructions; The touch screen is used to receive touch operations and also to display an interface; The processor is used to call the computer program and instructions stored in the memory, and can execute the method as claimed in any one of claims 1 to 18.

35. The terminal device as described in claim 34 further includes an antenna system, and the antenna system, under the control of the processor, receives and sends wireless communication signals to achieve wireless communication with a mobile communication network; the mobile communication network includes one or more of the following: GSM network, CDMA network, 3G network, 4G network, 5G network, FDMA, TDMA, PDC, TACS, AMPS, WCDMA, TDSCDMA, WIFI and LTE network.

36. A control method based on a touch screen, characterized in that: The method is applied to a first electronic device, wherein the first electronic device includes a touch screen; the touch screen can be used to display an interface; the method includes: When the first electronic device establishes a screen projection connection with the second electronic device, switching the touch screen event processing system of the first electronic device to a touch pad event processing system; generating a cursor, wherein the cursor is displayed in an interface of the touch screen; Generate projection content based on the interface content in the touch screen, wherein the projection content includes the cursor; The projection content is sent to the second electronic device, and all or part of the changes in the interface content in the first electronic device are displayed in real time in the second electronic device following the projection content, and the movement of the cursor in the first electronic device is synchronously displayed in the second electronic device; When receiving a target touch operation performed by a user on the touch screen of the first electronic device, the target touch operation is responded to in the first electronic device based on the touchpad event processing system; wherein the response to the target touch operation based on the touchpad event processing system is related to the position of the cursor and is irrelevant to the operation position of the target touch operation on the touch screen; the touch screen event processing system does not include the cursor.

37. The method of claim 36, wherein: Generating a cursor comprises: Obtain the interface content of the front-end application corresponding to the current interface, and draw a cursor in the interface content; or, draw a cursor and add the cursor to the current interface based on a floating window interface; or, A ray is drawn, and based on the floating window interface, the ray is intersected with the current interface at an end point; the end point is the cursor.

38. The method according to claim 36 or 37, characterized in that Before switching the touch screen event processing system of the first electronic device to the touch pad event processing system, the method further includes: detecting that the first electronic device establishes a screen projection connection with the second electronic device; or, A first switching command input by a user is detected; the first switching command is used to instruct the first electronic device to call the touchpad event processing system and shield the touchscreen event processing system.

39. The method of claim 36 or 37, wherein: The method further comprises: Reduce the display brightness of the touch screen, or turn off the touch screen.

40. The method of claim 36 or 37, wherein: When the target touch operation is a sliding operation, The responding to the target touch operation in the first electronic device based on the display position of the cursor in the current interface includes: Changing the position of the cursor in the current interface according to the sliding operation; When the target touch operation is a click operation, The responding to the target touch operation in the first electronic device based on the display position of the cursor in the current interface includes: A confirmation operation is performed on the object corresponding to the cursor in the current interface.

41. The method of claim 36 or 37, wherein: When the target touch operation is a long press followed by a slide operation, The responding to the target touch operation in the first electronic device based on the display position of the cursor in the current interface includes: Slide or drag the cursor to the object corresponding to the interface.

42. The method of claim 36 or 37, wherein: When the target touch operation causes the movement trajectory of the cursor in the current interface to match a preset shortcut operation command, or when the target touch operation matches a preset shortcut operation command, The responding to the target touch operation in the first electronic device based on the display position of the cursor in the current interface includes: Execute the shortcut operation command; wherein, The target touch operation includes: Slide from a first preset area of ​​the touch screen to a first preset direction; or, Slide from a second preset area of ​​the touch screen to a second preset direction, and the time of contact with the touch screen is greater than a preset time; or, The number of times the touch screen is clicked reaches a preset number; or, The sliding track of the target touch operation satisfies a preset pattern.

43. The method of claim 36 or 37, wherein: After the first electronic device establishes a screen projection connection with the second electronic device, the method further includes: Start multi-screen display mode; Creating N virtual screens in the interface of the touch screen; Determine N interfaces to be displayed; The N interfaces to be displayed are displayed respectively in the N virtual screens; wherein the N interfaces to be displayed belong to M applications, M is an integer not greater than N, and N is an integer greater than 1; the cursor can be located at any position in the N virtual screens.

44. The method of claim 37, wherein: Get a second switching instruction; the second switching instruction is used to instruct the first electronic device to switch the touchpad event processing system to the air mouse event processing system; wherein, when the posture of the first electronic device changes, the response of the air mouse event processing system to the posture change of the first electronic device is to change the position of the cursor in the current interface.

45. The method according to any one of claims 36 to 37 and 44, characterized in that: The first electronic device includes a mobile phone or a tablet; the second electronic device includes a display, a television, a projector, an AR device, a VR device or an MR device.

46. ​​A control device based on a touch screen, characterized in that: The device is applied to a first electronic device, wherein the first electronic device includes a touch screen; the touch screen can be used to display an interface; the device includes: A switching module, used for switching the touch screen event processing system of the first electronic device to the touch pad event processing system when the first electronic device establishes a screen projection connection with the second electronic device; A generating module, used for generating a cursor, wherein the cursor is displayed in the interface of the touch screen; A screen projection module, configured to generate screen projection content based on the interface content in the touch screen, wherein the screen projection content includes the cursor; and send the screen projection content to the second electronic device, so that all or part of the changes in the interface content in the first electronic device are displayed in real time in the second electronic device following the screen projection content, and the movement of the cursor in the first electronic device is synchronously displayed in the second electronic device; a response module, configured to respond to the target touch operation in the first electronic device based on the touch panel event processing system when receiving a target touch operation of the user on the touch screen of the first electronic device; The response of the touchpad event processing system to the target touch operation is related to the position of the cursor and is irrelevant to the operation position of the target touch operation on the touch screen; and the touch screen event processing system does not include the cursor.

47. The device according to claim 46, characterized in that The generation module is specifically used for: Obtain the interface content of the front-end application corresponding to the current interface, and draw a cursor in the interface content; or, draw a cursor and add the cursor to the current interface based on a floating window interface; or, Draw a ray, and intersect the ray with the current interface at an end point based on the floating window interface; The end point is the cursor.

48. The device according to claim 46 or 47, characterized in that The device also includes a detection module, which is used to: Before the switching module switches the touch screen event processing system of the first electronic device to the touch pad event processing system, it is detected that the first electronic device establishes a screen projection connection with the second electronic device; or, A first switching command input by a user is detected; the first switching command is used to instruct the first electronic device to call the touchpad event processing system and shield the touchscreen event processing system.

49. The device according to claim 46 or 47, characterized in that The device also includes a brightness control module, which is used to reduce the display brightness of the touch screen or turn off the touch screen.

50. The device according to claim 47, characterized in that When the target touch operation is a sliding operation, the response module is used to change the position of the cursor in the current interface according to the sliding operation; or, When the target touch operation is a click operation, the response module is used to perform a confirmation operation on the object corresponding to the cursor in the current interface; or, When the target touch operation is a long press followed by a slide operation, the response module is used to slide or drag the object corresponding to the cursor in the interface; or, When the target touch operation causes the movement trajectory of the cursor in the current interface to match a preset shortcut operation command, or when the target touch operation matches a preset shortcut operation command, the response module is used to execute the shortcut operation command.

51. The device according to any one of claims 46 to 47 and 50, characterized in that: The device also includes a multi-screen creation module. Used for: Start multi-screen display mode; Creating N virtual screens in the interface of the touch screen; Determine N interfaces to be displayed; The N interfaces to be displayed are displayed respectively in the N virtual screens; wherein the N interfaces to be displayed belong to M applications, M is an integer not greater than N, and N is an integer greater than 1; the cursor can be located at any position in the N virtual screens.

52. The device according to claim 47 or 50, characterized in that The switching module is also used for: Get a second switching instruction; the second switching instruction is used to instruct the first electronic device to switch the touchpad event processing system to the air mouse event processing system; wherein, when the posture of the first electronic device changes, the response of the air mouse event processing system to the posture change of the first electronic device is to change the position of the cursor in the current interface.

53. The device according to any one of claims 46 to 47 and 50, characterized in that: The first electronic device includes a mobile phone or a tablet; the second electronic device includes a display, a television, a projector, an AR device, a VR device or an MR device.

54. A terminal device, characterized in that: The terminal device includes a memory, a processor, a touch screen, and a bus; the memory, the touch screen, and the processor are connected via the bus; The memory is used to store computer programs and instructions; The touch screen is used to receive touch operations and also to display an interface; The processor is used to call the computer program and instructions stored in the memory, and can execute the method as claimed in any one of claims 36 to 45.

55. The terminal device as described in claim 54 further comprises an antenna system, and the antenna system, under the control of the processor, receives and sends wireless communication signals to achieve wireless communication with a mobile communication network; the mobile communication network comprises one or more of the following: GSM network, CDMA network, 3G network, 4G network, 5G network, FDMA, TDMA, PDC, TACS, AMPS, WCDMA, TDSCDMA, WIFI and LTE network.

56. A screen projection system, characterized in that: include: A terminal device as described in claim 34, 35, 54 or 55, and a second electronic device; the terminal device and the second electronic device are connected by screen projection.

57. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, wherein the computer program includes program instructions, and when the program request is executed by a computer, the computer is caused to execute the method according to any one of claims 1 to 18 or 36 to 45.

Images