Cross-device cooperative operation method and device, medium and device

Abstract

The embodiment of the invention provides a cross-device collaborative operation method and device, a medium and equipment. The method comprises the following steps: acquiring a current display interface image of a second terminal connected with a first terminal; performing scene recognition processing on the display interface image to obtain a target use scene corresponding to the display interface image; displaying a corresponding target operation interface on the first terminal according to the target use scene, wherein the target operation interface is used for displaying associated information of the second terminal and / or controlling the second terminal; and controlling the second terminal device to execute a preset operation through the target operation interface. According to the cross-device cooperative operation method provided by the embodiment of the invention, the scene recognition is performed on the current display interface image of the second terminal, the corresponding target operation interface is automatically displayed on the first terminal according to the recognized target use scene, and the cross-device control on the second terminal is quickly realized based on the target operation interface.

Description

Technical Field

[0001] The present application relates to the field of communication technology, and in particular to the field of cross-device collaborative operation technology, and in particular to a cross-device collaborative operation method, apparatus, medium and equipment. Background Art

[0002] In common cross-device collaborative operation scenarios, such as using a remote control to search for information on a TV, the traditional remote control input method is inefficient, and users need to press multiple buttons to enter a complete search term on the TV's information input interface. Although some TVs have introduced far-field voice input technology, which can alleviate the problem of low efficiency of traditional remote control input to a certain extent, the voice recognition accuracy is low in noisy environments and cannot meet the needs of all users. For example, people with pronunciation disorders cannot use this function. Summary of the invention

[0003] The embodiments of the present application provide a cross-device collaborative operation method, apparatus, medium and device. The cross-device collaborative operation method provided by the embodiments of the present application is used to solve the problem of low efficiency of the current cross-device collaborative operation mode.

[0004] On the one hand, an embodiment of the present application provides a cross-device collaborative operation method, which is applied to a first terminal, and the method includes:

[0005] Acquire a current display interface image of a second terminal connected to the first terminal;

[0006] Performing scene recognition processing on the display interface image to obtain a target usage scene corresponding to the display interface image;

[0007] Displaying a corresponding target operation interface on the first terminal according to the target usage scenario, wherein the target operation interface is used to display associated information of the second terminal and / or control the second terminal;

[0008] The target operation interface controls the second terminal device to perform a preset operation.

[0009] Furthermore, in the cross-device collaborative operation method described in the embodiment of the present application, displaying a corresponding target operation interface on the first terminal according to the target usage scenario includes:

[0010] If the target usage scenario is an information input scenario, displaying a target operation interface corresponding to the display interface of the second terminal on the first terminal;

[0011] The controlling the second terminal device to perform a preset operation through the target operation interface includes:

[0012] The information input operation of the target operation interface is controlled to be performed synchronously with the information input operation of the display interface, so that the second terminal can be synchronously controlled to perform the information input operation through the first terminal.

[0013] Furthermore, in the cross-device collaborative operation method described in the embodiment of the present application, displaying a corresponding target operation interface on the first terminal according to the target usage scenario includes:

[0014] If the target usage scenario is a fault repair scenario, identifying the fault type according to the display interface image;

[0015] Obtaining a repair solution corresponding to the fault type;

[0016] A target operation interface including the repair solution is displayed on the first terminal.

[0017] Furthermore, in the cross-device collaborative operation method described in the embodiment of the present application, after displaying the target operation interface including the repair solution on the first terminal, the method further includes:

[0018] In response to a selection operation on the target repair solution on the target operation interface, generating an operation instruction including the target repair solution according to the selection operation;

[0019] The operation instruction is sent to the second terminal to control the second terminal to execute the operation instruction and perform fault repair processing.

[0020] Furthermore, in the cross-device collaborative operation method described in the embodiment of the present application, displaying a corresponding target operation interface on the first terminal according to the target usage scenario includes:

[0021] If the target usage scenario is a program information identification scenario, then identifying the playing program information according to the display interface image, wherein the playing program information includes one or more of a program name and a program source address;

[0022] A target operation interface including the broadcast program information is displayed on the first terminal.

[0023] Furthermore, in the cross-device collaborative operation method described in the embodiment of the present application, after displaying the target operation interface including the program information on the first terminal, the method further includes:

[0024] In response to a click operation on a sharing link creation function on the target operation interface, a sharing link including the program name and program source address is generated according to the click operation.

[0025] Furthermore, in the cross-device collaborative operation method described in the embodiment of the present application, before acquiring the current display interface image of the second terminal connected to the first terminal, the method further includes:

[0026] Acquire a QR code image provided by the second terminal;

[0027] Identify device information contained in the QR code image;

[0028] A connection is established with the second device according to the device information.

[0029] Correspondingly, another aspect of the embodiment of the present application further provides a cross-device collaborative operation apparatus, which is applied to a first terminal, and the cross-device collaborative operation apparatus includes:

[0030] An image acquisition module, used to acquire a current display interface image of a second terminal connected to the first terminal;

[0031] A scene recognition module, used to perform scene recognition processing on the display interface image to obtain a target usage scene corresponding to the display interface image;

[0032] An interface display module, used to display a corresponding target operation interface on the first terminal according to the target usage scenario, wherein the target operation interface is used to display the associated information of the second terminal and / or control the second terminal;

[0033] The device control module is used to control the second terminal device to perform a preset operation through the target operation interface.

[0034] Correspondingly, another aspect of the embodiment of the present application further provides a computer storage medium, wherein the computer storage medium stores a plurality of instructions, wherein the instructions are suitable for loading by a processor to execute the cross-device collaborative operation method as described above.

[0035] Correspondingly, another aspect of an embodiment of the present application further provides an electronic device, including a processor and a memory, wherein the memory stores a plurality of instructions, and the processor loads the instructions to execute the cross-device collaborative operation method as described above.

[0036] The embodiments of the present application provide a cross-device collaborative operation method, apparatus, medium and device, which obtains the current display interface image of the second terminal connected to the first terminal; performs scene recognition processing on the display interface image to obtain the target usage scenario corresponding to the display interface image; displays the corresponding target operation interface on the first terminal according to the target usage scenario, and the target operation interface is used to display the associated information of the second terminal and / or control the second terminal; and controls the second terminal device to perform preset operations through the target operation interface. The cross-device collaborative operation method provided by the embodiments of the present application is used to perform scene recognition on the current display interface image of the second terminal, and automatically display the corresponding target operation interface on the first terminal according to the identified target usage scenario, so as to realize quick cross-device control of the second terminal based on the target operation interface. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application, and those skilled in the art can obtain other drawings based on these drawings without creative work.

[0038] Figure 1 A flowchart of a cross-device collaborative operation method provided in an embodiment of the present application.

[0039] Figure 2 Another flowchart of the cross-device collaborative operation method provided in an embodiment of the present application.

[0040] Figure 3 A schematic diagram of the structure of a cross-device collaborative operation apparatus provided in an embodiment of the present application.

[0041] Figure 4 Another structural schematic diagram of the cross-device collaborative operation apparatus provided in an embodiment of the present application.

[0042] Figure 5 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0043] The technical solutions in the embodiments of the present application will be described clearly and completely below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present application.

[0044] An embodiment of the present application provides a cross-device collaborative operation method, which uses the method to perform scene recognition on the current display interface image of the second terminal, and automatically displays the corresponding target operation interface on the first terminal according to the recognized target usage scenario, so as to realize quick cross-device control of the second terminal based on the target operation interface.

[0045] The term "and / or" in this application can be a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this application generally indicates that the associated objects before and after are in an "or" relationship.

[0046] The terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable in appropriate circumstances, so that the embodiments described here can be implemented in a sequence other than the content illustrated or described here. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, the process, method, system, product or television set comprising a series of steps or modules need not be limited to the steps or modules clearly listed, but may include other steps or modules that are not clearly listed or inherent to these processes, methods, products or television sets. The naming or numbering of the steps in the present application does not mean that the steps in the method flow must be performed in the time / logic sequence indicated by the naming or numbering, and the process steps that have been named or numbered can change the execution order according to the technical purpose to be achieved, as long as the same or similar technical effects can be achieved. The division of modules in this application is a logical division. There may be other division methods when it is implemented in actual applications. For example, multiple modules can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection between modules can be electrical or other similar forms, which are not limited in this application. In addition, the modules or submodules described as separate components may or may not be physically separated, may or may not be physical modules, or may be distributed in multiple circuit modules, and some or all of the modules may be selected according to actual needs to achieve the purpose of the present application.

[0047] The cross-device collaborative operation method involved in the embodiment of the present application is mainly applied to electronic devices, which may be televisions, smart phones, tablet computers, laptops, desktop computers, and servers, etc., which are not limited here. Optionally, the server may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or an IoT cloud (Internet of Things cloud) that provides the ability to store, process and manage data generated by an IoT television, or a cloud server that provides cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), and basic cloud computing services such as big data and artificial intelligence platforms, which are not limited here.

[0048] For ease of understanding, the specific process in the embodiments of the present application is described below. Figure 1 , Figure 1 A flowchart of an embodiment of a cross-device collaborative operation method provided in an embodiment of the present application.

[0049] exist Figure 1 In the illustrated embodiment, the method may be applied to a first terminal, and specifically comprises the following steps:

[0050] S101: Acquire a current display interface image of a second terminal connected to the first terminal.

[0051] In this embodiment, the display interface image can be obtained based on the shooting function of the first terminal, and the display interface image can be obtained by directly shooting the current display interface of the second terminal by turning on the shooting function. For example, when the first terminal is a mobile phone and the second terminal is a TV, the user can turn on the shooting function of the mobile phone to shoot the current display interface image of the TV while maintaining a communication connection between the mobile phone and the TV, thereby obtaining the current display interface image of the TV.

[0052] In other embodiments, after shooting the display interface image of the second terminal using other devices other than the first terminal, the display interface image may be sent to the first terminal through other devices, that is, the first terminal does not participate in the generation process of the display interface image, but only serves as a receiving end of the display interface image. For example, when the first terminal is mobile phone A and the second terminal is a TV, the user may use other devices other than the mobile phone and the TV, such as mobile phone B, to shoot the current display interface of the TV through mobile phone B to obtain the display interface image, and send the display interface image shot by mobile phone B to mobile phone A while maintaining a communication connection between mobile phones A and B.

[0053] It should be noted that the content displayed by the display interface image of the second terminal can be determined according to the actual situation and is not limited here. For example, when the current display screen of the second terminal is an information input interface, the display interface image is an image containing the information input interface of the second terminal. For another example, when the current display screen of the second terminal is a fault error interface, the display interface image is an image containing the fault error interface of the second terminal. For another example, when the current display screen of the second terminal is a normal program playback interface, the display interface image is an image containing the program playback interface of the second terminal.

[0054] S102: Perform scene recognition processing on the display interface image to obtain a target usage scene corresponding to the display interface image.

[0055] In this embodiment, by performing scene recognition processing on the display interface image, the current target usage scenario of the second terminal can be determined, and then different collaborative operation strategies are selected in a targeted manner according to the identified target usage scenario to better respond to the actual needs of the user. For example, when the identified target usage scenario is an information input scenario, the collaborative operation strategy set in this solution can be used to give the first terminal the ability to synchronously control the second terminal to perform information input operations. When the first terminal uses a more operable mobile phone or tablet, the mobile phone or tablet can simulate the current display interface of the second terminal, such as the information input interface, and more conveniently control the second terminal to perform information input operations based on the simulated display interface on the mobile phone or tablet, replacing the traditional remote control method of controlling the TV. The specific implementation process will be further described below.

[0056] In some embodiments, in order to realize scene recognition for display interface images, a large model capable of recognizing scenes based on display interface images can be obtained based on conventional neural network training, such as R-CNN (Regions with CNN features) or YOLO (You Only Look Once). Since this solution does not make substantial improvements to existing neural networks, the specific training process will not be described here.

[0057] S103: Displaying a corresponding target operation interface on the first terminal according to the target usage scenario, where the target operation interface is used to display associated information of the second terminal and / or control the second terminal.

[0058] In this embodiment, different target operation interfaces can be displayed on the first terminal according to the identified target usage scenario, and based on different target operation interfaces, the user can view the associated information of the second terminal on the first terminal, and can also control the second terminal to perform a preset operation through the first terminal. It is understandable that the user can also view the associated information of the second terminal on the first terminal and control the second terminal to perform a preset operation through the first terminal at the same time through the target operation interface.

[0059] In this embodiment, by performing scene recognition on the current display interface image of the second terminal and automatically displaying the corresponding target operation interface on the first terminal according to the recognized target usage scenario, the second terminal can be quickly controlled across devices based on the target operation interface.

[0060] It should be explained that the associated information of the second terminal refers to the information provided by the second terminal and required by the user, and the specific content of the associated information is related to the target usage scenario and changes with different target usage scenarios. For example, when the target usage scenario is a program information identification scenario, the associated information of the second terminal includes one or more of the program name and program source address of the program currently playing on the second terminal.

[0061] S104: Control the second terminal device to execute a preset operation through the target operation interface.

[0062] In this embodiment, different target operation interfaces can be displayed on the first terminal according to the identified target usage scenarios, and based on different target operation interfaces, the user can view the associated information of the second terminal on the first terminal, and can also control the second terminal to perform preset operations through the first terminal.

[0063] For example, when the identified target usage scenario is an information input scenario, according to the collaborative operation strategy set in this solution, the first terminal can be given the ability to synchronously control the second terminal to perform information input operations. When the first terminal uses a more operable mobile phone or tablet, the mobile phone or tablet can simulate the current display interface of the second terminal, such as the information input interface, and more conveniently control the second terminal to perform information input operations based on the simulated display interface on the mobile phone or tablet, replacing the traditional remote control method of controlling the TV.

[0064] In some embodiments, the S103 specifically includes:

[0065] If the target usage scenario is an information input scenario, a target operation interface corresponding to the display interface of the second terminal is displayed on the first terminal.

[0066] The controlling the second terminal device to perform a preset operation through the target operation interface includes:

[0067] The information input operation of the target operation interface is controlled to be performed synchronously with the information input operation of the display interface, so that the second terminal can be synchronously controlled to perform the information input operation through the first terminal.

[0068] In this embodiment, when the target usage scenario is an information input scenario, the target operation interface of the display interface of the second terminal is displayed and simulated on the first terminal, and the information input operation of the target operation interface is controlled to be performed synchronously with the information input operation of the display interface, so that the user can synchronously control the second terminal to perform information input operations through the first terminal. For example, when the first terminal uses a more operable mobile phone or tablet, the mobile phone or tablet can simulate the current display interface of the second terminal, such as the information input interface, and more conveniently control the second terminal to perform information input operations based on the simulated target operation interface on the mobile phone or tablet. Compared with the traditional remote control input method, the user does not need to press multiple buttons to enter a complete search term on the information input interface of the TV.

[0069] In some embodiments, the S103 specifically includes:

[0070] If the target usage scenario is a fault repair scenario, identifying the fault type according to the display interface image;

[0071] Obtaining a repair solution corresponding to the fault type;

[0072] A target operation interface including the repair solution is displayed on the first terminal.

[0073] In this embodiment, when the target usage scenario is a fault repair scenario, the target operation interface containing the repair solution is displayed on the first terminal to assist the user in repairing the fault problem existing in the second terminal. Among them, the method of obtaining the repair solution can be to identify the fault type according to the display interface image, and then search for a repair solution matching the fault type from the cloud server or local database according to the fault type. It is understandable that the repair solution can be a solution such as prompting the user to restart the second terminal or clearing the background running memory of the second terminal, which is not limited here.

[0074] In some embodiments, controlling the second terminal device to perform a preset operation through the target operation interface includes:

[0075] In response to a selection operation on the target repair solution on the target operation interface, generating an operation instruction including the target repair solution according to the selection operation;

[0076] The operation instruction is sent to the second terminal to control the second terminal to execute the operation instruction and perform fault repair processing.

[0077] In this embodiment, the repair plan can be displayed on the target operation interface of the first terminal and exist in the form of an operation option. When the user selects an operation option corresponding to any target repair plan, the first terminal can generate an operation instruction containing the target repair plan according to the user's selection operation, and send the operation instruction to the second terminal to control the second terminal to execute the operation instruction and perform fault repair processing, that is, the second terminal attempts to self-repair and solve the fault problem.

[0078] In some embodiments, the S103 specifically includes:

[0079] If the target usage scenario is a program information identification scenario, then identifying the playing program information according to the display interface image, wherein the playing program information includes one or more of a program name and a program source address;

[0080] A target operation interface including the broadcast program information is displayed on the first terminal.

[0081] In this embodiment, when the target usage scenario is a program information identification scenario, by displaying a target operation interface containing playback program information on the first terminal, the user can know the playback program information of the program currently being played on the second terminal, such as one or more of the program name and program source address.

[0082] In some embodiments, in a program information recognition scenario, when a target operation interface including program information for playing can be displayed on a first terminal, a control capable of controlling a second terminal, such as a volume adjustment control, can be added based on the target operation interface.

[0083] It should be further explained that the broadcast program information contained in the display interface image can be obtained by using a pre-trained large model for intelligent recognition, or the broadcast program information of the currently broadcast program can be directly requested from the second terminal, which is not limited here.

[0084] In some embodiments, after displaying the target operation interface including the program information on the first terminal, the method further includes:

[0085] In response to a click operation on a sharing link creation function on the target operation interface, a sharing link including the program name and program source address is generated according to the click operation.

[0086] In this embodiment, a sharing link including the program name and program source address can be automatically generated through the sharing link creation function, and the user can decide to forward the sharing link to other people, so that users who receive the sharing link can click the sharing link to watch the program synchronously.

[0087] In some embodiments, Figure 2 As shown, before S101, the method further includes:

[0088] S201, obtaining a two-dimensional code image provided by the second terminal;

[0089] S202, identifying device information contained in the two-dimensional code image;

[0090] S203: Establish a connection with the second device according to the device information.

[0091] In this embodiment, the user can scan the QR code image provided by the second terminal through the shooting application or system camera function on the first terminal, and then identify the device information in the QR code image. The device information usually includes the device ID, IP address, port number, etc. of the second terminal. The first terminal establishes a connection with the second terminal through a preset communication protocol (such as Wi-Fi, Bluetooth, etc.) based on the parsed device information. After the connection is established, the first terminal can exchange data and cooperate with the second terminal.

[0092] In some embodiments, when a user wants the system to recommend a program, the user can obtain the current program list of the second terminal through the first terminal, and then upload it to the cloud server. The cloud server analyzes the user's preferences and the current program list of the second terminal, generates recommended programs, and displays the recommended programs on the first terminal or the second terminal.

[0093] It should be noted that a pre-trained program intelligent recommendation model can be installed on the cloud server, so that the model can automatically analyze user preferences based on user behavior data (such as the number of times and duration of viewing a certain type of program within a set time) through learning, and then search for corresponding programs from the program list as recommended programs based on user preferences.

[0094] The program intelligent recommendation model can be obtained based on conventional neural network training, such as R-CNN (Regions with CNN features) or YOLO (You Only Look Once). Since this solution does not make substantial improvements to the existing neural network, the specific training process will not be described here.

[0095] All the above optional technical solutions can be arbitrarily combined to form optional embodiments of the present application, which will not be described one by one here.

[0096] In specific implementation, the present application is not limited by the execution order of the various steps described. If no conflict occurs, some steps can be performed in other orders or simultaneously.

[0097] As can be seen from the above, the cross-device collaborative operation method provided by the embodiment of the present application obtains the current display interface image of the second terminal that establishes a connection with the first terminal; performs scene recognition processing on the display interface image to obtain the target usage scenario corresponding to the display interface image; and displays the corresponding target operation interface on the first terminal according to the target usage scenario, and the target operation interface is used to display the associated information of the second terminal and / or control the second terminal. Utilizing the cross-device collaborative operation method provided by the embodiment of the present application, by performing scene recognition on the current display interface image of the second terminal, and automatically displaying the corresponding target operation interface on the first terminal according to the identified target usage scenario, the second terminal can be quickly controlled across devices based on the target operation interface.

[0098] An embodiment of the present application also provides a cross-device collaborative operation apparatus, which can be integrated into an electronic device.

[0099] See also Figure 3 , Figure 3 A schematic diagram of the structure of a cross-device collaborative operation apparatus provided in an embodiment of the present application. The cross-device collaborative operation apparatus 30 may include:

[0100] An image acquisition module 31 is used to acquire a current display interface image of a second terminal connected to the first terminal;

[0101] A scene recognition module 32, configured to perform scene recognition processing on the display interface image to obtain a target usage scene corresponding to the display interface image;

[0102] An interface display module 33, configured to display a corresponding target operation interface on the first terminal according to the target usage scenario, wherein the target operation interface is used to display associated information of the second terminal and / or control the second terminal;

[0103] The device control module 34 is used to control the second terminal device to perform a preset operation through the target operation interface.

[0104] In some embodiments, the interface display module 33 is used to display a target operation interface corresponding to the display interface of the second terminal on the first terminal if the target usage scenario is an information input scenario; the device control module 34 is used to control the information input operation of the target operation interface to be performed synchronously with the information input operation of the display interface, so that the second terminal can be synchronously controlled to perform information input operations through the first terminal.

[0105] In some embodiments, the interface display module 33 is used to identify the fault type according to the display interface image if the target usage scenario is a fault repair scenario; obtain a repair solution corresponding to the fault type; and display a target operation interface including the repair solution on the first terminal.

[0106] In some embodiments, the device control module 34 is used to respond to a selection operation of a target repair solution on the target operation interface, generate an operation instruction containing the target repair solution according to the selection operation; and send the operation instruction to the second terminal to control the second terminal to execute the operation instruction and perform fault repair processing.

[0107] In some embodiments, the interface display module 33 is used to identify the playback program information according to the display interface image if the target usage scenario is a program information identification scenario, wherein the playback program information includes one or more of a program name and a program source address; and display a target operation interface containing the playback program information on the first terminal.

[0108] In some embodiments, the device further includes a creation module for responding to a click operation on a sharing link creation function on the target operation interface, and generating a sharing link including the program name and program source address according to the click operation.

[0109] In some embodiments, the apparatus further includes a connection module, configured to obtain a two-dimensional code image provided by the second terminal; identify device information contained in the two-dimensional code image; and establish a connection with the second device based on the device information.

[0110] In specific implementation, the above modules can be implemented as independent entities, or can be arbitrarily combined to be implemented as the same entity or several entities.

[0111] As can be seen from the above, the cross-device collaborative operation device 30 provided in the embodiment of the present application, wherein the image acquisition module 31 is used to acquire the current display interface image of the second terminal that establishes a connection with the first terminal; the scene recognition module 32 is used to perform scene recognition processing on the display interface image to obtain the target usage scene corresponding to the display interface image; the interface display module 33 is used to display the corresponding target operation interface on the first terminal according to the target usage scene, and the target operation interface is used to display the associated information of the second terminal and / or control the second terminal; the device control module 34 is used to control the second terminal device to perform a preset operation through the target operation interface. The cross-device collaborative operation device 30 provided in the embodiment of the present application, by performing scene recognition on the current display interface image of the second terminal, and automatically displaying the corresponding target operation interface on the first terminal according to the identified target usage scene, realizes quick cross-device control of the second terminal based on the target operation interface.

[0112] An embodiment of the present application also provides a cross-device collaborative operation apparatus, which can be integrated into an electronic device.

[0113] In specific implementation, the above modules can be implemented as independent entities, or can be arbitrarily combined to be implemented as the same entity or several entities.

[0114] See also Figure 4 , Figure 4 Another structural diagram of the cross-device collaborative operation apparatus provided in an embodiment of the present application, the cross-device collaborative operation apparatus 30 includes a memory 120, one or more processors 180, and one or more applications, wherein the one or more applications are stored in the memory 120 and configured to be executed by the processor 180; the processor 180 may include an image acquisition module 31, a scene recognition module 32, an interface display module 33, and a device control module 34. For example, the structure and connection relationship of the above components may be as follows:

[0115] The memory 120 can be used to store applications and data. The applications stored in the memory 120 include executable codes. The applications can be composed of various functional modules. The processor 180 executes various functional applications and data processing by running the applications stored in the memory 120. In addition, the memory 120 may include a high-speed random access memory and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices. Accordingly, the memory 120 may also include a memory controller to provide the processor 180 with access to the memory 120.

[0116] The processor 180 is the control center of the device, and uses various interfaces and lines to connect various parts of the entire terminal. By running or executing applications stored in the memory 120 and calling data stored in the memory 120, the processor 180 performs various functions of the device and processes data, thereby monitoring the device as a whole. Optionally, the processor 180 may include one or more processing cores; preferably, the processor 180 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and application programs.

[0117] Specifically in this embodiment, the processor 180 will load the executable code corresponding to the process of one or more applications into the memory 120 according to the following instructions, and the processor 180 will run the application stored in the memory 120 to implement various functions:

[0118] An image acquisition instruction, used to acquire a current display interface image of a second terminal connected to the first terminal;

[0119] A scene recognition instruction, used to perform scene recognition processing on the display interface image to obtain a target usage scene corresponding to the display interface image;

[0120] An interface display instruction, used to display a corresponding target operation interface on the first terminal according to the target usage scenario, wherein the target operation interface is used to display the associated information of the second terminal and / or control the second terminal;

[0121] The device control instruction is used to control the second terminal device to perform a preset operation through the target operation interface.

[0122] In some embodiments, the interface display instruction is used to display a target operation interface corresponding to the display interface of the second terminal on the first terminal if the target usage scenario is an information input scenario; the device control instruction is used to control the information input operation of the target operation interface to be performed synchronously with the information input operation of the display interface, so that the second terminal can be synchronously controlled to perform information input operations through the first terminal.

[0123] In some embodiments, the interface display instruction is used to identify the fault type according to the display interface image if the target usage scenario is a fault repair scenario; obtain a repair plan corresponding to the fault type; and display a target operation interface including the repair plan on the first terminal.

[0124] In some embodiments, the device control instruction is used to respond to a selection operation of a target repair solution on the target operation interface, generate an operation instruction containing the target repair solution according to the selection operation; and send the operation instruction to the second terminal to control the second terminal to execute the operation instruction and perform fault repair processing.

[0125] In some embodiments, the interface display instruction is used to identify the playback program information based on the display interface image if the target usage scenario is a program information identification scenario, wherein the playback program information includes one or more of a program name and a program source address; and display a target operation interface containing the playback program information on the first terminal.

[0126] In some embodiments, the program also includes a creation instruction for responding to a click operation on a sharing link creation function on the target operation interface, and generating a sharing link including the program name and program source address according to the click operation.

[0127] In some embodiments, the program further includes connection instructions for acquiring a two-dimensional code image provided by the second terminal; identifying device information contained in the two-dimensional code image; and establishing a connection with the second device according to the device information.

[0128] The present application also provides an electronic device. Figure 5 , Figure 5 The electronic device 1200 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application, and the electronic device can be used to implement the cross-device collaborative operation method provided in the above embodiment. The electronic device 1200 can be a television, a smart phone, or a tablet computer.

[0129] like Figure 5 As shown, the electronic device 1200 may include an RF (Radio Frequency) circuit 110, a memory 120 including one or more (only one is shown in the figure) computer-readable storage media, an input unit 130, a display unit 140, a sensor 150, an audio circuit 160, a transmission module 170, a processor 180 including one or more (only one is shown in the figure) processing cores, and a power supply 190. Those skilled in the art will understand that Figure 5 The structure of the electronic device 1200 shown in the figure does not constitute a limitation on the electronic device 1200, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0130] The RF circuit 110 is used to receive and send electromagnetic waves, realize the mutual conversion between electromagnetic waves and electrical signals, and thus communicate with a communication network or other televisions. The RF circuit 110 may include various existing circuit elements for performing these functions, such as an antenna, a radio frequency transceiver, a digital signal processor, an encryption / decryption chip, a user identity module (SIM) card, a memory, etc. The RF circuit 110 can communicate with various networks such as the Internet, an intranet, a wireless network, or communicate with other televisions via a wireless network.

[0131] The memory 120 can be used to store software programs and modules, such as the program instructions / modules corresponding to the cross-device collaborative operation method in the above-mentioned embodiment. The processor 180 executes various functional applications and data processing by running the software programs and modules stored in the memory 120. The vibration reminder mode can be automatically selected according to the current scene of the electronic device to perform cross-device collaborative operation, which can not only ensure that scenes such as meetings are not disturbed, but also ensure that users can perceive incoming calls, thereby improving the intelligence of the electronic device. The memory 120 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 120 may further include a memory remotely arranged relative to the processor 180, and these remote memories may be connected to the electronic device 1200 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0132] The input unit 130 can be used to receive input digital or character information, and generate keyboard, mouse, joystick, optical or trackball signal input related to user settings and function control. Specifically, the input unit 130 may include a touch-sensitive surface 131 and other input devices 132. The touch-sensitive surface 131, also known as a touch display screen or touch pad, can collect user touch operations on or near it (such as operations performed by users using fingers, styluses, or any other suitable objects or accessories on or near the touch-sensitive surface 131), and drive corresponding connection devices according to a pre-set program. Optionally, the touch-sensitive surface 131 may include a touch detection device and a touch controller. Among them, the touch detection device detects the user's touch position, detects the signal brought by the touch operation, and transmits the signal to the touch controller; the touch controller receives the touch information from the touch detection device, converts it into touch point coordinates, and then sends it to the processor 180, and can receive and execute commands sent by the processor 180. In addition, the touch-sensitive surface 131 may be implemented in various types such as resistive, capacitive, infrared, and surface acoustic wave. In addition to the touch-sensitive surface 131, the input unit 130 may further include other input devices 132. Specifically, the other input devices 132 may include, but are not limited to, one or more of a physical keyboard, a function key (such as a volume control key, a switch key, etc.), a trackball, a mouse, a joystick, and the like.

[0133] The display unit 140 can be used to display information input by the user or information provided to the user and various graphical user interfaces of the electronic device 1200, which can be composed of graphics, text, icons, videos and any combination thereof. The display unit 140 may include a display panel 141. Optionally, the display panel 141 may be configured in the form of LCD (Liquid Crystal Display), OLED (Organic Light-Emitting Diode), etc. Further, the touch-sensitive surface 131 may cover the display panel 141. When the touch-sensitive surface 131 detects a touch operation on or near it, it is transmitted to the processor 180 to determine the type of touch event. Subsequently, the processor 180 provides corresponding visual output on the display panel 141 according to the type of touch event. Although in Figure 5 In the embodiment, the touch-sensitive surface 131 and the display panel 141 are implemented as two independent components to realize the input and output functions. However, in some embodiments, the touch-sensitive surface 131 and the display panel 141 can be integrated to realize the input and output functions.

[0134] The electronic device 1200 may also include at least one sensor 150, such as a light sensor, a motion sensor, and other sensors. Specifically, the light sensor may include an ambient light sensor and a proximity sensor, wherein the ambient light sensor may adjust the brightness of the display panel 141 according to the brightness of the ambient light, and the proximity sensor may turn off the display panel 141 and / or the backlight when the electronic device 1200 is moved to the ear. As a type of motion sensor, the gravity acceleration sensor can detect the magnitude of acceleration in each direction (generally three axes), and can detect the magnitude and direction of gravity when stationary. It can be used for applications that identify the posture of the mobile phone (such as horizontal and vertical screen switching, related games, magnetometer posture calibration), vibration recognition related functions (such as pedometer, tapping), etc.; as for other sensors such as gyroscopes, barometers, hygrometers, thermometers, infrared sensors, etc. that can also be configured in the electronic device 1200, they will not be repeated here.

[0135] The audio circuit 160, the speaker 161, and the microphone 162 can provide an audio interface between the user and the electronic device 1200. The audio circuit 160 can transmit the received audio data to the speaker 161 after converting the received audio data into an electrical signal, which is converted into a sound signal for output by the speaker 161; on the other hand, the microphone 162 converts the collected sound signal into an electrical signal, which is received by the audio circuit 160 and converted into audio data, and then the audio data is processed by the output processor 180, and then sent to another terminal through the RF circuit 110, or the audio data is output to the memory 120 for further processing. The audio circuit 160 may also include an earplug jack to provide communication between an external headset and the electronic device 1200.

[0136] The electronic device 1200 can help users send and receive emails, browse web pages, and access streaming media through the transmission module 170 (such as a Wi-Fi module), which provides users with wireless broadband Internet access. Figure 5 The transmission module 170 is shown, but it is understandable that it is not an essential component of the electronic device 1200 and can be omitted as needed without changing the essence of the invention.

[0137] The processor 180 is the control center of the electronic device 1200. It uses various interfaces and lines to connect various parts of the entire mobile phone. By running or executing software programs and / or modules stored in the memory 120, and calling data stored in the memory 120, it executes various functions of the electronic device 1200 and processes data, thereby monitoring the mobile phone as a whole. Optionally, the processor 180 may include one or more processing cores; in some embodiments, the processor 180 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and application programs, etc., and the modem processor mainly processes wireless communications. It is understandable that the above-mentioned modem processor may not be integrated into the processor 180.

[0138] The electronic device 1200 also includes a power supply 190 for supplying power to various components. In some embodiments, the power supply can be logically connected to the processor 180 through a power management system, so that the power management system can realize functions such as managing discharge and power consumption. The power supply 190 can also include any components such as one or more DC or AC power supplies, recharging systems, power failure detection circuits, power converters or inverters, and power status indicators.

[0139] Although not shown, the electronic device 1200 may also include a camera (such as a front camera, a rear camera), a Bluetooth module, etc., which will not be described in detail here. Specifically in this embodiment, the display unit 140 of the electronic device 1200 is a touch screen display, and the electronic device 1200 also includes a memory 120, and one or more programs, wherein one or more programs are stored in the memory 120, and are configured to be executed by one or more processors 180. One or more programs include instructions for performing the following operations:

[0140] An image acquisition instruction, used to acquire a current display interface image of a second terminal connected to the first terminal;

[0141] A scene recognition instruction, used to perform scene recognition processing on the display interface image to obtain a target usage scene corresponding to the display interface image;

[0142] An interface display instruction, used to display a corresponding target operation interface on the first terminal according to the target usage scenario, wherein the target operation interface is used to display the associated information of the second terminal and / or control the second terminal;

[0143] The device control instruction is used to control the second terminal device to perform a preset operation through the target operation interface.

[0144] In some embodiments, the interface display instruction is used to display a target operation interface corresponding to the display interface of the second terminal on the first terminal if the target usage scenario is an information input scenario; the device control instruction is used to control the information input operation of the target operation interface to be performed synchronously with the information input operation of the display interface, so that the second terminal can be synchronously controlled to perform information input operations through the first terminal.

[0145] In some embodiments, the interface display instruction is used to identify the fault type according to the display interface image if the target usage scenario is a fault repair scenario; obtain a repair plan corresponding to the fault type; and display a target operation interface including the repair plan on the first terminal.

[0146] In some embodiments, the device control instruction is used to respond to a selection operation of a target repair solution on the target operation interface, generate an operation instruction containing the target repair solution according to the selection operation; and send the operation instruction to the second terminal to control the second terminal to execute the operation instruction and perform fault repair processing.

[0147] In some embodiments, the interface display instruction is used to identify the playback program information based on the display interface image if the target usage scenario is a program information identification scenario, wherein the playback program information includes one or more of a program name and a program source address; and display a target operation interface containing the playback program information on the first terminal.

[0148] In some embodiments, the program also includes a creation instruction for responding to a click operation on a sharing link creation function on the target operation interface, and generating a sharing link including the program name and program source address according to the click operation.

[0149] In some embodiments, the program further includes connection instructions for acquiring a two-dimensional code image provided by the second terminal; identifying device information contained in the two-dimensional code image; and establishing a connection with the second device according to the device information.

[0150] The embodiment of the present application also provides an electronic device, which may be a smart phone, a television, a computer, or other device.

[0151] As can be seen from the above, the embodiment of the present application provides an electronic device 1200, and the electronic device 1200 performs the following steps:

[0152] Acquire a current display interface image of a second terminal connected to the first terminal;

[0153] Performing scene recognition processing on the display interface image to obtain a target usage scene corresponding to the display interface image;

[0154] Displaying a corresponding target operation interface on the first terminal according to the target usage scenario, wherein the target operation interface is used to display associated information of the second terminal and / or control the second terminal;

[0155] The target operation interface controls the second terminal device to perform a preset operation.

[0156] An embodiment of the present application further provides a storage medium, in which a computer program is stored. When the computer program runs on a computer, the computer executes the cross-device collaborative operation method described in any of the above embodiments.

[0157] It should be noted that, for the cross-device collaborative operation method described in this application, ordinary testers in this field can understand that all or part of the process of implementing the cross-device collaborative operation method described in the embodiment of this application can be completed by controlling the relevant hardware through a computer program, and the computer program can be stored in a computer-readable storage medium, such as stored in the memory of an electronic device, and executed by at least one processor in the electronic device, and the execution process may include the process of the embodiment of the cross-device collaborative operation method. Among them, the storage medium can be a disk, an optical disk, a read-only memory (ROM, Read Only Memory), a random access memory (RAM, Random Access Memory), etc.

[0158] For the cross-device collaborative operation device of the embodiment of the present application, its various functional modules can be integrated into a processing chip, or each module can exist physically separately, or two or more modules can be integrated into one module. The above-mentioned integrated modules can be implemented in the form of hardware or in the form of software functional modules. If the integrated modules are implemented in the form of software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium, such as a read-only memory, a disk or an optical disk.

[0159] The above is a detailed introduction to the cross-device collaborative operation method, apparatus, medium and equipment provided by the embodiments of the present application. Specific examples are used herein to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method and core idea of ​​the present application; at the same time, for those skilled in the art, according to the idea of ​​the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A cross-device collaborative operation method, applied to a first terminal, characterized in that: include: Acquire a current display interface image of a second terminal connected to the first terminal; Performing scene recognition processing on the display interface image to obtain a target usage scene corresponding to the display interface image; Displaying a corresponding target operation interface on the first terminal according to the target usage scenario, wherein the target operation interface is used to display associated information of the second terminal and / or control the second terminal; The target operation interface controls the second terminal device to perform a preset operation.

2. The cross-device collaborative operation method according to claim 1, characterized in that: The displaying a corresponding target operation interface on the first terminal according to the target usage scenario includes: If the target usage scenario is an information input scenario, displaying a target operation interface corresponding to the display interface of the second terminal on the first terminal; The controlling the second terminal device to perform a preset operation through the target operation interface includes: The information input operation of the target operation interface is controlled to be performed synchronously with the information input operation of the display interface, so that the second terminal can be synchronously controlled to perform the information input operation through the first terminal.

3. The cross-device collaborative operation method according to claim 1, characterized in that: The displaying a corresponding target operation interface on the first terminal according to the target usage scenario includes: If the target usage scenario is a fault repair scenario, identifying the fault type according to the display interface image; Obtaining a repair solution corresponding to the fault type; A target operation interface including the repair solution is displayed on the first terminal.

4. The cross-device collaborative operation method according to claim 3, characterized in that: The controlling the second terminal device to perform a preset operation through the target operation interface includes: In response to a selection operation on the target repair solution on the target operation interface, generating an operation instruction including the target repair solution according to the selection operation; The operation instruction is sent to the second terminal to control the second terminal to execute the operation instruction and perform fault repair processing.

5. The cross-device collaborative operation method according to claim 1, characterized in that: The displaying a corresponding target operation interface on the first terminal according to the target usage scenario includes: If the target usage scenario is a program information identification scenario, then identifying the playing program information according to the display interface image, wherein the playing program information includes one or more of a program name and a program source address; A target operation interface including the broadcast program information is displayed on the first terminal.

6. The cross-device collaborative operation method according to claim 5, characterized in that: After displaying the target operation interface including the program information on the first terminal, the method further includes: In response to a click operation on a sharing link creation function on the target operation interface, a sharing link including the program name and program source address is generated according to the click operation.

7. The cross-device collaborative operation method according to claim 1, characterized in that: Before acquiring the current display interface image of the second terminal connected to the first terminal, the method further includes: Acquire a QR code image provided by the second terminal; Identify device information contained in the QR code image; A connection is established with the second device according to the device information.

8. A cross-device collaborative operation apparatus, applied to a first terminal, characterized in that: The cross-device collaborative operation apparatus comprises: An image acquisition module, used to acquire a current display interface image of a second terminal connected to the first terminal; A scene recognition module, used to perform scene recognition processing on the display interface image to obtain a target usage scene corresponding to the display interface image; An interface display module, used to display a corresponding target operation interface on the first terminal according to the target usage scenario, wherein the target operation interface is used to display the associated information of the second terminal and / or control the second terminal; The device control module is used to control the second terminal device to perform a preset operation through the target operation interface.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a plurality of instructions, and the instructions are suitable for being loaded by a processor to execute the cross-device collaborative operation method according to any one of claims 1 to 7.

10. An electronic device, characterized in that: It comprises a processor and a memory, wherein the memory stores a plurality of instructions, and the processor loads the instructions to execute the cross-device collaborative operation method according to any one of claims 1 to 7.

Images