Control method, first equipment and second equipment

Abstract

The invention provides a control method, first equipment and second equipment, the method is applied to the first equipment, and the method comprises the following steps: displaying a target application interface to a display screen; determining that the target condition is met, and sending a first control interface of the target application interface to the second device, the first control interface being different from the target application interface; and receiving first operation information which is sent by the second equipment and is used for operating the first control interface, and manipulating display contents in the target application interface based on the first operation information.

Description

Technical Field

[0001] The present application relates to the field of data processing and data transmission, and in particular to a control method, a first device and a second device. Background Art

[0002] With the development of intelligent devices and device interconnection technology, the technology of controlling each other between different devices has become more and more popular. Users are used to using small devices to control large devices, such as using mobile phones to control computers. Previously, most methods of controlling Windows systems were remote desktop methods, such as projecting the entire Windows system interface to a mobile phone or tablet through QQ Remote Desktop, Sunflower and other software, and then using the mobile phone or tablet to control the Windows system. However, this method is too cumbersome in a near-field environment, and the amount of data transmitted is large. In a poor network environment, it will frequently freeze or even fail. Summary of the invention

[0003] The present application provides a control method, a first device and a second device.

[0004] On the one hand, an embodiment of the present application provides a control method, which is applied to a first device, and the method includes:

[0005] Display the target application interface on the display screen;

[0006] Determining that the target condition is met, sending a first control interface of the target application interface to the second device, where the first control interface is different from the target application interface;

[0007] Receive first operation information for operating the first control interface sent by the second device, and manipulate display content in the target application interface based on the first operation information.

[0008] Among them, the control information included in the first control interface is the same as the control information included in the second control interface in the target application interface, or the control information included in the first control interface is a subset of the control information included in the second control interface in the target application interface, and the appearance of the first control included in the first control interface is the same as or similar to the second control included in the second control interface.

[0009] Among them, meeting the target conditions includes one of the following:

[0010] receiving a request control instruction for the target application sent by the second device;

[0011] or, receiving a request control instruction for the target application sent by the second device, and the display screen size of the second device is smaller than a preset threshold;

[0012] or, receiving a screen projection control instruction for a control interface of the target application initiated by the first device;

[0013] Or, an application interface projection control instruction for the target application initiated by the first device is received, and the display screen size of the second device is smaller than a preset threshold.

[0014] The method further includes: responding to the control instruction, obtaining at least one control information of the target application, and drawing the first control interface based on the at least one control information of the target application, wherein obtaining the at least one control information of the target application includes one of the following methods:

[0015] Matching the application information of the target application with a plurality of mapping data in a database; if the application information and the database are successfully matched, determining at least one control information corresponding to the successfully matched mapping data as the control information of the target application;

[0016] Or, determining a target storage area of ​​the target application based on the application data; acquiring at least one control information of the target application from the target storage area;

[0017] Or, the target application is identified based on a pre-trained model to obtain at least one control information of the target application, and the pre-trained model is trained based on an application interface training set.

[0018] Wherein, drawing display content based on the control information includes:

[0019] If the control type of the control information belongs to the first type;

[0020] Determine a corresponding control component based on the control type of the control information;

[0021] The first control interface is obtained by drawing a corresponding control in a target area of ​​the virtual screen based on the control component, the target area is determined based on the control position of the control information, and the virtual screen is created based on the screen size of the second device.

[0022] Wherein, drawing display content based on the control information includes:

[0023] If the control type of the control information belongs to the second type, wherein the control information of the second type further includes optional item data;

[0024] Determine a corresponding control component based on the control type of the control information;

[0025] Loading corresponding optional data into the control component;

[0026] Based on the control component after loading the optional item data, the corresponding control is drawn in the target area of ​​the virtual screen to obtain the first control interface, the target area is determined based on the control position of the control information, and the virtual screen is created based on the screen size of the second device.

[0027] Among them, it also includes:

[0028] Associate the first control interface with the second control interface in the application interface, and in response to receiving first operation information sent by the second device for operating the first control in the first control interface, map the first operation information to second operation information of the second control in the second control interface, so as to manipulate the display content in the target application interface through the second control.

[0029] Another aspect of the present application provides a control method, which is applied to a second device, and the method includes:

[0030] Receiving a first control interface of a target application interface sent by a first device, where the first control interface is different from the target application interface;

[0031] Displaying the first control interface;

[0032] First operation information for operating the first control interface is sent to the first device, so that the second device manipulates the display content in the target application interface based on the first operation information.

[0033] In another aspect, the present application provides a first device, including: a display screen, a processor, and a communication module;

[0034] The display screen displays the target application interface;

[0035] The processor determines that the target condition is met, controls the communication module to send the first control interface of the target application interface to the second device, and the first control interface is different from the target application interface; and controls the communication module to receive the first operation information for operating the first control interface sent by the second device, and manipulates the display content in the target application interface based on the first operation information.

[0036] In another aspect, the present application provides a second device, including: a receiver, a processor, a display screen, and a transmitter;

[0037] The receiver receives a first control interface of a target application interface sent by a first device, where the first control interface is different from the target application interface;

[0038] The processor controls the display screen to display the first control interface,

[0039] The transmitter sends first operation information for operating the first control interface to the second device, so that the second device manipulates the display content in the target application interface based on the first operation information.

[0040] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present application, nor is it intended to limit the scope of the present application. Other features of the present application will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] By reading the detailed description below with reference to the accompanying drawings, the above and other purposes, features and advantages of the exemplary embodiments of the present application will become readily understood. In the accompanying drawings, several embodiments of the present application are shown in an exemplary and non-limiting manner, wherein:

[0042] In the drawings, the same or corresponding reference numerals represent the same or corresponding parts.

[0043] Figure 1 A flow chart of a control method according to an embodiment of the present application is shown;

[0044] Figure 2 A schematic diagram of an application interface according to an embodiment of the present application is shown;

[0045] Figure 3 A schematic diagram of a first control interface according to an embodiment of the present application is shown;

[0046] Figure 4 A flow chart showing a control method according to another embodiment of the present application is shown;

[0047] Figure 5 A schematic diagram showing a mapping relationship between a control type and a control component according to an embodiment of the present application;

[0048] Figure 6 A flow chart showing a control method according to another embodiment of the present application is shown;

[0049] Figure 7 A schematic diagram showing how to manipulate the display content of a target application interface by operating a first control interface according to an embodiment of the present application is shown;

[0050] Figure 8 A flow chart showing a control method according to another embodiment of the present application is shown;

[0051] Fig. 9 A schematic diagram of the structure of an electronic device according to an embodiment of the present application is shown;

[0052] Fig.10A schematic diagram of the structure of an electronic device according to another embodiment of the present application is shown;

[0053] Fig.11 A schematic structural diagram of a control device according to an embodiment of the present application is shown;

[0054] Fig.12 A schematic structural diagram of a control device according to another embodiment of the present application is shown. DETAILED DESCRIPTION

[0055] In order to make the purpose, features, and advantages of the present application more obvious and easy to understand, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present application.

[0056] In order to simplify the operation steps of remote control, reduce the amount of data to be transmitted, and improve stability in a poor network environment, an embodiment of the present application provides a control method, which is applied to a first device, such as Figure 1 As shown, the method includes:

[0057] Step 101, displaying the target application interface on a display screen.

[0058] In this embodiment, the first device is a controlled end, and the second device is a controlling end.

[0059] The first device displays the interface of the target application on the display screen of the first device.

[0060] The target application is an application in the first device that the second device needs to control, and can be determined by a control instruction sent by the second device.

[0061] Step 102: Determine that the target condition is met, and send a first control interface of the target application interface to the second device, where the first control interface is different from the target application interface.

[0062] After determining that the target condition is met, the first device obtains the control information of the target application, and draws the first control interface of the target application interface based on the control information, and then sends it to the second device. The first control interface is different from the target application interface. The first control interface only has the controls of the target application interface, and the control appearance is a simplified version of the control appearance of the target application interface.

[0063] Step 103: Receive first operation information for operating the first control interface sent by the second device, and manipulate display content in the target application interface based on the first operation information.

[0064] After receiving the first control interface, the second device displays the first control interface on the display screen of the second device. The user can operate the controls in the first control interface on the second device to generate corresponding first operation information. The second device sends the first operation information for operating the first control interface to the first device. After receiving the first operation information, the second device manipulates the display content in the target application interface based on the first operation information. The first operation information is used to change or control the display content of the target application, such as play, pause, drag the playback progress, switch the playback speed, switch the resolution, zoom the playback content, etc.

[0065] For example, the target application is a playback application, and the user switches the resolution from 720P to 1080P in the first control interface of the second device, generating corresponding first operation information. The second device sends the first operation information to the first device, and after receiving the first operation information, the first device operates the target application to switch the resolution from 720P to 1080P.

[0066] For another example, the target application is a game application, and the user clicks the forward control in the first control interface of the second device, generating corresponding first operation information. The second device sends the first operation information to the first device, and after receiving the first operation information, the first device operates the virtual character in the target application to move forward.

[0067] In the above scheme, by receiving the control instruction sent by the second device, the first device obtains the control information of the target application, draws the first control interface of the target application interface based on the control information, and then sends it to the second device. Since the first control interface only has the controls of the target application interface, and the control appearance is a simplified version of the control appearance of the target application interface, the amount of data of the first control interface that needs to be transmitted to the second device is relatively small, which significantly improves the stability in poor network environments. After the second device receives the first control interface and displays it on the display screen of the second device, the user can generate corresponding first operation information by operating the first control interface, and send it to the first device, so that the target application in the first device can be remotely controlled by operating in the second device, which simplifies the operation steps of remote control and improves the user experience.

[0068] In an example of the present application, a control method is also provided, wherein the control information included in the first control interface is the same as the control information included in the second control interface in the target application interface, or the control information included in the first control interface is a subset of the control information included in the second control interface in the target application interface, wherein the appearance of the first control included in the first control interface is the same as or similar to the second control included in the second control interface.

[0069] The control information included in the first control interface is the same as the control information included in the second control interface in the target application interface, or the control information included in the first control interface is a subset of the control information included in the second control interface in the target application interface.

[0070] For example, the target application is a playback application, and the target application interface includes five controls: playback, progress bar, next episode, resolution, and speed. The first control interface also includes five controls: playback, progress bar, next episode, resolution, and speed.

[0071] For another example, the target application is a social application, and the target application interface includes an input box, send, emoticon package, send file, screenshot Figure 5 The first control interface only includes two controls: input box and send.

[0072] The appearance of a first control included in the first control interface is the same as or similar to the appearance of a second control included in the second control interface.

[0073] For example, Figure 2 and Figure 3 As shown, Figure 2 The application interface for the playback application. Figure 3 It is the first control interface corresponding to the playback application. Figure 3 It includes 5 controls: play, next level, progress bar, resolution and speed. Figure 2 The corresponding control in the Figure 2 A simplified version of the corresponding appearance in .

[0074] In the above scheme, a first control interface including corresponding controls is drawn based on the control information of at least one control in the target application interface, so that the user can control the display content of the target application in the first device by operating the first control interface in the second device. The appearance of the first control included in the first control interface is the same or similar to that of the second control included in the second control interface. Firstly, the user can clearly and intuitively find the control required for the operation, and secondly, the data volume of the first control interface can be reduced by simplifying the control appearance. That is, the user's experience is improved, and the stability in a poor network environment is also improved.

[0075] An embodiment of the present application further provides a control method, which satisfies the target condition and includes one of the following:

[0076] 1. Receive a request control instruction for the target application sent by the second device.

[0077] The user initiates a request control instruction on the second device to control the target application in the first device, and the second device sends the request control instruction to the first device. After receiving the request control instruction sent by the second device, the first device obtains the control information of the target application, and draws the first control interface of the target application interface based on the control information, and then sends it to the second device.

[0078] Second, a request control instruction for the target application sent by the second device is received, and the display screen size of the second device is smaller than a preset threshold.

[0079] The user initiates a request control instruction on the second device to control the target application in the first device, and the second device sends the request control instruction to the first device. After receiving the request control instruction sent by the second device, the first device also needs to determine whether the display screen size of the second device is smaller than a preset threshold. If the display screen size of the second device is smaller than the preset threshold, the control information of the target application is obtained, and the first control interface of the target application interface is drawn based on the control information, and then sent to the second device.

[0080] 3. Receive a screen projection control instruction for the control interface of the target application initiated by the first device.

[0081] The user initiates a screen projection control command on the first device to project the control interface of the target application to the second device. After the first device detects that the user has initiated the screen projection control command, it obtains the control information of the target application, draws the first control interface of the target application interface based on the control information, and then sends it to the second device.

[0082] Fourth, an application interface projection control instruction for the target application initiated by the first device is received, and the display screen size of the second device is smaller than a preset threshold.

[0083] The user initiates a screen projection control command on the first device to project the control interface of the target application to the second device. After the first device detects that the user has initiated the screen projection control command, it is also necessary to obtain the display screen size of the second device and determine whether the display screen size of the second device is smaller than a preset threshold. If the display screen size of the second device is smaller than the preset threshold, the control information of the target application is obtained, and the first control interface of the target application interface is drawn based on the control information, and then sent to the second device.

[0084] The preset size is generally set based on the type of the second device. For example, if the second device is a mobile phone, the preset size can be set to 200 mm*100 mm*10 mm. For another example, if the second device is a tablet, the preset size can be set to 350 mm*250 mm*10 mm.

[0085] In the above scheme, the user can initiate a request control instruction to control the target application in the first device on the second device. It is also possible to initiate a screen projection control instruction on the first device to project the control interface of the target application to the second device. Both initiation methods can enable the first device to project the control interface of the target application to the second device, and the second device can control the target application in the first device based on the user's operation on the control interface. While simplifying the operating steps, it allows users to initiate control instructions in either device, further improving the user experience. At the same time, it can also determine whether the screen size of the second device is less than the preset threshold. When the screen size of the second device is less than the preset threshold, the control interface of the target application is sent to the second device to prevent the display content projected to the second device from being deformed due to the large screen size difference, thereby improving the stability and usability of remote control.

[0086] An embodiment of the present application further provides a control method, further comprising: responding to the control instruction, acquiring at least one control information of the target application, and drawing the first control interface based on the at least one control information of the target application.

[0087] After receiving or detecting the control instruction, the first device obtains at least one control information of the target application specified by the control instruction, and draws a first control interface based on the control information.

[0088] Wherein, obtaining at least one control information of the target application includes one of the following methods:

[0089] Matching the application information of the target application with multiple mapping data in the database; if the application information and the database are successfully matched, determining at least one control information corresponding to the successfully matched mapping data as the control information of the target application.

[0090] A target storage area of ​​the target application is determined based on the application data; and at least one control information of the target application is acquired from the target storage area.

[0091] The target application is identified based on a pre-trained model to obtain at least one control information of the target application, wherein the pre-trained model is obtained by training based on an application interface training set.

[0092] In this embodiment, at least one control information of the target application can be obtained in the following three ways:

[0093] The first method is to obtain control information of a preset number of applications that are used most frequently in the first device in advance, and then form the application information and control information of the applications into corresponding mapping data, and store the mapping data corresponding to these applications in a database.

[0094] When it is necessary to obtain the control information of the target application, the application information of the target application is matched with the plurality of mapping data stored in the database. If the match is successful, at least one control information corresponding to the successfully matched mapping data is determined as the control information of the target application.

[0095] In the second method, when the first device runs the target application, the data of the target application is stored in the memory, and this data includes the control information of the target application. Therefore, when the control information of the target application needs to be obtained, the target storage area of ​​the target application in the memory is found based on the application information of the target application, and then at least one control information of the target application is obtained.

[0096] For example, use inspect (an application used for development and debugging of automation and accessibility technologies) to search for data in a target storage area in memory, obtain the value corresponding to at least one AutomationID (a unique identifier used to identify control elements in an application) in the data, and obtain at least one control information of the target application.

[0097] The third method is to pre-train the image recognition model using a training set, where the training set includes multiple application interfaces and at least one control information corresponding to the application interface. After the training is completed, a pre-trained model is obtained. When it is necessary to obtain the control information of the target application, the pre-trained model is used to identify the target application to obtain at least one control information of the target application.

[0098] It should be pointed out that the above three methods can be used as parallel solutions, or they can be combined arbitrarily to form a new solution.

[0099] For example, when it is necessary to obtain the control information of the target application, the application information of the target application is first matched with the plurality of mapping data in the database. If the match is unsuccessful, the control information of the target application is then obtained from the target storage area in the memory.

[0100] For another example, when it is necessary to obtain the control information of the target application, the application information of the target application is first matched with the multiple mapping data in the database. If the match is unsuccessful, the pre-trained model is used to identify the target application and obtain the control information of the target application.

[0101] In the above scheme, by matching the application information of the target application with the multiple mapping data stored in the database, if the match is successful, at least one control information corresponding to the successfully matched mapping data is determined as the control information of the target application, which can effectively improve the efficiency of remote control. If the match is unsuccessful, the control information of the target application is obtained from the target storage area in the memory or the target application is identified by using a pre-trained model to obtain the control information of the target application, which significantly improves the stability and availability of remote control.

[0102] An embodiment of the present application also provides a control method, such as Figure 4 As shown, the drawing of display content based on the control information includes:

[0103] Step 201: If the control type of the control information belongs to the first type.

[0104] In this embodiment, the first type is set as a control that does not require the user to select an option, such as a play, next episode, progress bar, etc. control.

[0105] Step 202: Determine a corresponding control component based on the control type of the control information.

[0106] In this embodiment, a control component corresponding to a control type is preset, and the control component can be appearance data or code. Figure 5 As shown, if the control type is play, the corresponding control component is set to a control component including a play button. If the control type is a progress bar, the corresponding control component is set to a control component including a progress bar and a drag button.

[0107] Step 203, drawing a corresponding control in a target area of ​​the virtual screen based on the control component to obtain the first control interface, the target area is determined based on the control position of the control information, and the virtual screen is created based on the screen size of the second device.

[0108] A virtual screen is created based on the screen size of the second device, and then a corresponding control is drawn based on the target area of ​​the control component in the virtual screen. The control information also includes the control position, and the target area of ​​the control component in the virtual screen can be determined based on the control position and the size of the control component. After drawing the control component corresponding to the control information, a first control interface of the target application is obtained.

[0109] In the above scheme, although the appearance of the control component needs to be simplified in order to reduce the amount of data in the first control interface, in order to facilitate the user to understand the function of the control, the appearance of the control component is set to be the same as or similar to the corresponding component in the target application. Then, the first control interface drawn based on these control components enables the user to intuitively understand the function of each control, which significantly improves the user experience.

[0110] An embodiment of the present application also provides a control method, such as Figure 6 As shown, the drawing of display content based on the control information includes:

[0111] Step 301: If the control type of the control information belongs to the second type, wherein the second type of control information also includes optional item data.

[0112] In this embodiment, the first type is set as a control that requires the user to select an option, such as a resolution, speed, or selection.

[0113] Step 302: Determine a corresponding control component based on the control type of the control information.

[0114] Likewise, in this embodiment, a control component corresponding to a control type is preset, and the control component may be appearance data or code.

[0115] Step 303: Load corresponding optional item data into the control component.

[0116] The control type belongs to the second type of control information, and the optional data needs to be loaded.

[0117] For example, if the control type is resolution, the corresponding optional data is 480P, 720P, 1080P, Blu-ray, etc. These optional data need to be loaded into the resolution control component for users to choose.

[0118] Step 304, based on the control component after loading the optional item data, draw the corresponding control in the target area of ​​the virtual screen to obtain the first control interface, the target area is determined based on the control position of the control information, and the virtual screen is created based on the screen size of the second device.

[0119] Similarly, a virtual screen is created based on the screen size of the second device, and then corresponding controls are drawn based on the target area of ​​the control component in the virtual screen. The control information also includes the control position, and the target area of ​​the control component in the virtual screen can be determined based on the control position and the size of the control component. After drawing the control component corresponding to the control information, the first control interface of the target application is obtained.

[0120] In the above scheme, in order to facilitate users to understand the functions of the controls, the appearance of the control components is set to be the same or similar to the corresponding components in the target application, and the same optional data as in the target application is loaded. Then, the first control interface drawn based on these control components enables users to intuitively understand the functions of each control, which significantly improves the user experience.

[0121] An embodiment of the present application further provides a control method, further comprising:

[0122] Associate the first control interface with the second control interface in the application interface, and in response to receiving first operation information sent by the second device for operating the first control in the first control interface, map the first operation information to second operation information of the second control in the second control interface, so as to manipulate the display content in the target application interface through the second control.

[0123] For example, Figure 7 As shown, Figure 7 The left side of the middle is the first control interface of the second device. Figure 7 The second control interface in the application interface of the first device on the right is associated with the second control interface. The user clicks the resolution control in the first control interface, and the optional data of the resolution control pops up. The second device sends the first operation information of clicking the resolution control to the first device. After receiving the first operation information of clicking the resolution control, the first device maps the first operation information to the second operation information of operating the second control in the second control interface. The mapped second operation information is clicking the resolution control, and the resolution control in the target application interface is manipulated through the second operation information to pop up the optional data.

[0124] In the above scheme, by associating the first control interface with the second control interface in the application interface, the first operation information of operating the first control in the first control interface sent by the second device is received, and it is mapped to the second operation information of operating the second control in the second control interface, and finally the display content in the target application interface is manipulated by the second control, so that the display content of the target application in the first device is remotely controlled by the first control interface in the second device.

[0125] An embodiment of the present application also provides a control method, which is applied to a second device, such as Figure 8 As shown, the method includes:

[0126] Step 401: Receive a first control interface of a target application interface sent by a first device, where the first control interface is different from the target application interface.

[0127] The second device receives the first control interface of the target application interface sent by the first device. Similarly, the first control interface is different from the target application interface, the first control interface only has controls of the target application interface, and the control appearance is a simplified version of the control appearance of the target application interface.

[0128] Step 402: display the first control interface.

[0129] The second device displays the received first control interface.

[0130] Step 403: Send first operation information for operating the first control interface to the first device, so that the second device manipulates the display content in the target application interface based on the first operation information.

[0131] The user can operate the control in the first control interface of the second device, generate first operation information for operating the first control interface, and send it to the first device. After receiving the first operation information, the first device manipulates the display content in the target application interface based on the first operation information.

[0132] In the above scheme, the second device receives the first control interface of the target application interface sent by the first device and displays the first control interface. This allows the user to operate the first control interface in the second device, thereby generating first operation information for operating the first control interface and then sending it to the first device. After receiving the first operation information, the first device manipulates the content in the target application interface based on the first operation information. This enables the second device to remotely control the target application in the first device.

[0133] An embodiment of the present application further provides a control method, which includes one of the following steps before receiving a first control interface of a target application interface sent by a first device:

[0134] 1. Send a request control instruction for the target application to the first device.

[0135] The user initiates a request control instruction on the second device to control the target application in the first device, and the second device sends the request control instruction to the first device. After receiving the request control instruction sent by the second device, the first device obtains the control information of the target application, and draws the first control interface of the target application interface based on the control information, and then sends it to the second device.

[0136] Second, determine that the display screen size of the second device is smaller than a preset threshold, and send a request control instruction for the target application to the first device.

[0137] The user initiates a request control instruction on the second device to control the target application in the first device, and the second device needs to determine whether the display screen size of the second device is smaller than a preset threshold. If the display screen size of the second device is smaller than the preset threshold, the request control instruction is sent to the first device. After receiving the request control instruction sent by the second device, the first device obtains the control information of the target application, draws the first control interface of the target application interface based on the control information, and then sends it to the second device.

[0138] It should be noted that determining whether the display screen size of the second device is smaller than a preset threshold may be performed on the second device side or on the first device side.

[0139] 3. Receive a screen projection control instruction for the control interface of the target application sent by the first device.

[0140] The user initiates a screen projection control command on the first device to project the control interface of the target application to the second device. After the first device detects that the user has initiated the screen projection control command, it first sends the screen projection control command to the second device. After receiving the screen projection control command, the second device needs to confirm whether it agrees to the screen projection on the first device. After the user of the second device agrees, a notification of consent to the screen projection is sent to the first device. After receiving the notification of consent to the screen projection from the second device, the first device obtains the control information of the target application, draws the first control interface of the target application interface based on the control information, and then sends it to the second device.

[0141] Fourth, receiving a screen projection control instruction for the control interface of the target application sent by the first device when it is determined that the display screen size of the second device is smaller than a preset threshold.

[0142] The user initiates a screen projection control command on the first device to project the control interface of the target application to the second device. After the first device detects that the user has initiated the screen projection control command, it first sends the screen projection control command to the second device. After receiving the screen projection control command, the second device needs to confirm whether it agrees to the screen projection on the first device side. After the user of the second device agrees, it sends a notification of consent to the screen projection to the first device side. After receiving the notification of consent to the screen projection on the second device side, the first device side also needs to obtain the display screen size of the second device and determine whether the display screen size of the second device is less than the preset threshold. When the display screen size of the second device is less than the preset threshold, the control information of the target application is obtained, and the first control interface of the target application interface is drawn based on the control information, and then sent to the second device.

[0143] In the above scheme, the user can initiate a request control instruction to control the target application in the first device on the second device. It is also possible to initiate a screen projection control instruction to project the control interface of the target application to the second device on the first device. Both initiation methods can enable the first device to project the control interface of the target application to the second device, and the second device can control the target application in the first device based on the user's operation on the control interface. While simplifying the operation steps, the user can initiate control instructions in any end device, further improving the user experience. At the same time, it can also determine whether the screen size of the second device is less than the preset threshold. When the screen size of the second device is less than the preset threshold, the control interface of the target application is sent to the second device to prevent the display content of the screen projection to the second device from being deformed due to the large screen size difference, thereby improving the stability and usability of remote control. And when the user initiates a screen projection control instruction to project the control interface of the target application to the second device on the first device side, the second device side is first approved before the projection is performed to prevent the user from projecting to the wrong second device.

[0144] An embodiment of the present application provides a first device, such as Fig. 9 As shown, the first device 500 includes: a display screen 501, a processor 502 and a communication module 503;

[0145] The display screen 501 displays the target application interface;

[0146] The processor 502 determines that the target condition is met, controls the communication module 503 to send the first control interface of the target application interface to the second device, where the first control interface is different from the target application interface; and controls the communication module 503 to receive the first operation information for operating the first control interface sent by the second device, and manipulates the display content in the target application interface based on the first operation information.

[0147] In which, the processor 502 controls the communication module 503 to send the first control interface of the target application interface to the second device, the control information included in the first control interface is the same as the control information included in the second control interface in the target application interface, or the control information included in the first control interface belongs to a subset of the control information included in the second control interface in the target application interface, and the appearance of the first control included in the first control interface is the same as or similar to the second control included in the second control interface.

[0148] Among them, the processor 502 controls the communication module 503 to receive a request control instruction for the target application sent by the second device; or, receives a request control instruction for the target application sent by the second device, and the display screen size of the second device is smaller than a preset threshold; or, receives a control interface screen projection control instruction initiated by the first device for the target application; or, receives an application interface screen projection control instruction initiated by the first device for the target application, and the display screen size of the second device is smaller than a preset threshold.

[0149] The processor 502 responds to the control instruction, obtains at least one control information of the target application, and draws the first control interface based on the at least one control information of the target application, wherein obtaining the at least one control information of the target application includes one of the following methods:

[0150] The processor 502 matches the application information of the target application with multiple mapping data in the database; when the application information and the database are matched successfully, at least one control information corresponding to the successfully matched mapping data is determined as the control information of the target application; or, a target storage area of ​​the target application is determined based on the application data; at least one control information of the target application is obtained from the target storage area; or, the target application is identified based on a pre-trained model to obtain at least one control information of the target application, and the pre-trained model is trained based on an application interface training set.

[0151] Among them, when the control type of the control information belongs to the first type, the processor 502 determines the corresponding control component based on the control type of the control information; and draws the corresponding control in the target area of ​​the virtual screen based on the control component to obtain the first control interface, the target area is determined based on the control position of the control information, and the virtual screen is created based on the screen size of the second device.

[0152] Among them, the processor 502 determines a corresponding control component based on the control type of the control information when the control type of the control information belongs to the second type, wherein the second type of control information also includes optional item data; loads the corresponding optional item data into the control component; and draws the corresponding control in a target area in the virtual screen based on the control component after the optional item data is loaded, to obtain the first control interface, wherein the target area is determined based on the control position of the control information, and the virtual screen is created based on the screen size of the second device.

[0153] Among them, the processor 502 associates the first control interface with the second control interface in the application interface, and in response to receiving first operation information sent by the second device for operating the first control in the first control interface, maps the first operation information to second operation information of the second control in the second control interface, so as to manipulate the display content in the target application interface through the second control.

[0154] An embodiment of the present application also provides a second device, such as Fig.10 As shown, the second device 600 includes: a receiver 601, a processor 602, a display screen 603 and a transmitter 604;

[0155] The receiver 601 receives a first control interface of a target application interface sent by a first device, where the first control interface is different from the target application interface;

[0156] The processor 602 controls the display screen 603 to display the first control interface.

[0157] The transmitter 604 sends first operation information for operating the first control interface to the second device, so that the second device manipulates the display content in the target application interface based on the first operation information.

[0158] Among them, the transmitter 604 sends a request control instruction for the target application to the first device; or, determines that the display screen size of the second device is smaller than a preset threshold, sends a request control instruction for the target application to the first device; or, receives a control interface screen projection control instruction for the target application sent by the first device; or, receives a control interface screen projection control instruction for the target application sent by the first device when it is determined that the display screen size of the second device is smaller than a preset threshold.

[0159] In order to realize the above control method, Fig.11 As shown, an example of the present application provides a control device, including:

[0160] The first display module 701 is used to display the target application interface on the display screen;

[0161] Processing module 702, used for determining that a target condition is met, and sending a first control interface of the target application interface to the second device, where the first control interface is different from the target application interface;

[0162] The first communication module 703 is used to receive first operation information for operating the first control interface sent by the second device, and manipulate the display content in the target application interface based on the first operation information.

[0163] The first communication module 703 is also used to send the first control interface of the target application interface to the second device, the control information included in the first control interface is the same as the control information included in the second control interface in the target application interface, or the control information included in the first control interface belongs to a subset of the control information included in the second control interface in the target application interface, wherein the appearance of the first control included in the first control interface is the same as or similar to the second control included in the second control interface.

[0164] Among them, the first communication module 703 is also used to receive a request control instruction for the target application sent by the second device; or, receive a request control instruction for the target application sent by the second device, and the display screen size of the second device is smaller than a preset threshold; or, receive a control interface screen projection control instruction initiated by the first device for the target application; or, receive an application interface screen projection control instruction initiated by the first device for the target application, and the display screen size of the second device is smaller than a preset threshold.

[0165] The processing module 702 is further configured to respond to the control instruction, obtain at least one control information of the target application, and draw the first control interface based on the at least one control information of the target application, wherein obtaining the at least one control information of the target application includes one of the following methods:

[0166] The processing module 702 is also used to match the application information of the target application with multiple mapping data in the database; when the application information and the database are matched successfully, at least one control information corresponding to the successfully matched mapping data is determined as the control information of the target application; or, based on the application data, a target storage area of ​​the target application is determined; at least one control information of the target application is obtained from the target storage area; or, based on a pre-trained model, the target application is identified to obtain at least one control information of the target application, and the pre-trained model is trained based on an application interface training set.

[0167] The processing module 702 is also used to determine a corresponding control component based on the control type of the control information when the control type of the control information belongs to the first type; and to draw a corresponding control in a target area in the virtual screen based on the control component to obtain the first control interface, the target area being determined based on the control position of the control information, and the virtual screen being created based on the screen size of the second device.

[0168] Among them, the processing module 702 is also used to determine the corresponding control component based on the control type of the control information when the control type of the control information belongs to the second type, wherein the second type of control information also includes optional item data; load the corresponding optional item data into the control component; and draw the corresponding control in the target area of ​​the virtual screen based on the control component after the optional item data is loaded to obtain the first control interface, the target area is determined based on the control position of the control information, and the virtual screen is created based on the screen size of the second device.

[0169] Among them, the processing module 702 is also used to associate the first control interface with the second control interface in the application interface, and in response to receiving first operation information sent by the second device for operating the first control in the first control interface, map the first operation information to second operation information of the second control in the second control interface, so as to manipulate the display content in the target application interface through the second control.

[0170] In order to realize the above control method, Fig.12 As shown, an example of the present application also provides a control device, including:

[0171] The second communication module 801 is used to receive a first control interface of a target application interface sent by a first device, where the first control interface is different from the target application interface;

[0172] A second display module 802, used to display the first control interface;

[0173] The second communication module 801 is further used to send first operation information for operating the first control interface to the first device, so that the second device manipulates the display content in the target application interface based on the first operation information.

[0174] Among them, the second communication module 801 is also used to send a request control instruction for the target application to the first device; or, determine that the display screen size of the second device is smaller than a preset threshold, and send a request control instruction for the target application to the first device; or, receive a control instruction for the control interface projection screen for the target application sent by the first device; or, receive a control instruction for the control interface projection screen for the target application sent by the first device when it is determined that the display screen size of the second device is smaller than a preset threshold.

[0175] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), integrated systems on a chip (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0176] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0177] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0178] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0179] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0180] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0181] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0182] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of the features. In the description of the present disclosure, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0183] The above is only a specific embodiment of the present disclosure, but the protection scope of the present disclosure is not limited thereto. Any person skilled in the art who is familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present disclosure, which should be included in the protection scope of the present disclosure. Therefore, the protection scope of the present disclosure should be based on the protection scope of the claims.

Claims

1. A control method, applied to a first device, comprising: Display the target application interface on the display screen; Determining that the target condition is met, sending a first control interface of the target application interface to the second device, where the first control interface is different from the target application interface; Receive first operation information for operating the first control interface sent by the second device, and manipulate display content in the target application interface based on the first operation information.

2. According to the method according to claim 1, the control information included in the first control interface is the same as the control information included in the second control interface in the target application interface, or the control information included in the first control interface is a subset of the control information included in the second control interface in the target application interface, wherein the appearance of the first control included in the first control interface is the same as or similar to the second control included in the second control interface.

3. The method according to claim 1, wherein the target condition is met, including one of the following: receiving a request control instruction for the target application sent by the second device; or, receiving a request control instruction for the target application sent by the second device, and the display screen size of the second device is smaller than a preset threshold; or, receiving a screen projection control instruction for a control interface of the target application initiated by the first device; Or, an application interface projection control instruction for the target application initiated by the first device is received, and the display screen size of the second device is smaller than a preset threshold.

4. The method according to claim 3, further comprising: In response to the control instruction, obtaining at least one control information of the target application, and drawing the first control interface based on the at least one control information of the target application, wherein obtaining the at least one control information of the target application includes one of the following methods: Matching the application information of the target application with a plurality of mapping data in a database; if the application information and the database are successfully matched, determining at least one control information corresponding to the successfully matched mapping data as the control information of the target application; Or, determining a target storage area of ​​the target application based on the application data; acquiring at least one control information of the target application from the target storage area; Or, the target application is identified based on a pre-trained model to obtain at least one control information of the target application, and the pre-trained model is trained based on an application interface training set.

5. The method according to claim 4, wherein drawing display content based on the control information comprises: If the control type of the control information belongs to the first type; Determine a corresponding control component based on the control type of the control information; The first control interface is obtained by drawing a corresponding control in a target area of ​​the virtual screen based on the control component, the target area is determined based on the control position of the control information, and the virtual screen is created based on the screen size of the second device.

6. The method according to claim 4, wherein drawing display content based on the control information comprises: If the control type of the control information belongs to the second type, wherein the control information of the second type further includes optional item data; Determine a corresponding control component based on the control type of the control information; Loading corresponding optional data into the control component; Based on the control component after loading the optional item data, the corresponding control is drawn in the target area of ​​the virtual screen to obtain the first control interface, the target area is determined based on the control position of the control information, and the virtual screen is created based on the screen size of the second device.

7. The method according to claim 1, further comprising: Associate the first control interface with the second control interface in the application interface, and in response to receiving first operation information sent by the second device for operating the first control in the first control interface, map the first operation information to second operation information of the second control in the second control interface, so as to manipulate the display content in the target application interface through the second control.

8. A control method, applied to a second device, the method comprising: Receiving a first control interface of a target application interface sent by a first device, where the first control interface is different from the target application interface; Displaying the first control interface; First operation information for operating the first control interface is sent to the first device, so that the second device manipulates the display content in the target application interface based on the first operation information.

9. A first device, comprising: Display screen, processor and communication module; The display screen displays the target application interface; The processor determines that the target condition is met, controls the communication module to send a first control interface of the target application interface to the second device, where the first control interface is different from the target application interface; And controlling the communication module to receive first operation information for operating the first control interface sent by the second device, and manipulating the display content in the target application interface based on the first operation information.

10. A second device, comprising: Receivers, processors, displays, and transmitters; The receiver receives a first control interface of a target application interface sent by a first device, where the first control interface is different from the target application interface; The processor controls the display screen to display the first control interface, The transmitter sends first operation information for operating the first control interface to the second device, so that the second device manipulates the display content in the target application interface based on the first operation information.

Images