Linkage method and device between devices, electronic device and storage medium

Abstract

The invention relates to a linkage method between devices and a device thereof, and belongs to the technical field of Internet of Things. The method comprises the following steps: receiving a first linkage request sent by a second device through a local area network channel; receiving a second linkage request of a second device forwarded by the cloud server, wherein the first linkage request and the second linkage request are used for requesting linkage with the first device; and carrying out linkage data interaction with the second equipment through the local area network channel or the cloud server. Therefore, according to the scheme, the first linkage request and the second linkage request can be received, two linkage modes of the local area network channel and the cloud server are supported at the same time, and the linkage stability and success rate are greatly improved.

Description

Technical Field

[0001] The present application relates to the technical field of Internet of Things, and in particular to a method, apparatus, electronic device and storage medium for linkage between devices. Background Art

[0002] At present, the stability of screen projection linkage between home devices is poor, the response time of screen projection linkage is long, and the user experience is poor. Moreover, when network abnormalities occur, the existing technology cannot realize screen projection linkage between devices. Summary of the invention

[0003] The present application provides a method, apparatus, electronic device and storage medium for inter-device linkage, which can be applied to the field of Internet of Things technology to solve the problem of poor stability when inter-device linkage is performed. The technical solution of the present application is as follows:

[0004] According to a first aspect of an embodiment of the present application, a method for linkage between devices is provided, including: receiving a first linkage request sent by a second device through a local LAN channel; receiving a second linkage request of the second device forwarded by a cloud server, wherein the first linkage request and the second linkage request are used to request linkage with the first device; and performing linkage data interaction with the second device through the local LAN channel or the cloud server.

[0005] In one embodiment of the present application, the linkage data interaction with the second device through the local LAN channel or the cloud server includes: preferentially receiving the second linkage data sent by the second device through the local LAN channel; if the local LAN channel is abnormal, receiving the second linkage data sent by the cloud server.

[0006] In one embodiment of the present application, before receiving the first linkage request sent by the second device through the local LAN channel, it also includes: receiving first configuration information sent by the cloud server, wherein the first configuration information includes attribute information of the second device that has a linkage relationship with the first device; or, requesting the first configuration information from the cloud server, and caching the requested first configuration information locally.

[0007] In one embodiment of the present application, before receiving the first linkage request sent by the second device through the local LAN channel, it also includes: based on the first configuration information, sending a first User Datagram Protocol UDP unicast packet in the local area network, attempting to establish the local LAN channel with the second device; after the connection is established through the local LAN channel, transmitting a heartbeat signal with the second device.

[0008] In one embodiment of the present application, before receiving the first linkage request sent by the second device through the local LAN channel, it also includes: sending and exchanging respective public keys with the second device; determining the first encryption key of the first device based on the first private key of the first device and the second public key of the second device.

[0009] In one embodiment of the present application, after determining the first encryption key of the first device based on the first private key of the first device and the second public key of the second device, it also includes: generating a first target encryption key of the first device based on the identification code of the second device and the first encryption key.

[0010] In one embodiment of the present application, the linkage data interaction with the second device through the local LAN channel or the cloud server includes: encrypting the linkage data with the second device based on the first encryption key or the first target encryption key to obtain first linkage data; and sending the first linkage data to the second device through the local LAN channel or the cloud server.

[0011] In one embodiment of the present application, the method also includes: in response to an update of the first IP address of the first device, obtaining an updated first IP address of the first device; based on the device identifier of the first device, the device identifier of the second device and the updated first IP address, generating and broadcasting a first UDP broadcast message of the first device.

[0012] In one embodiment of the present application, after the linkage data interaction with the second device through the local LAN channel or the cloud server, it also includes: determining the video data and the second audio data based on the second linkage data; based on the video data, displaying the video picture in a small window on the screen of the first device, and synchronously playing the second audio data.

[0013] In one embodiment of the present application, the method also includes: detecting a full-screen display instruction, and if the full-screen display instruction is detected, switching the video screen from the small window form to the full-screen form; during the display of the video screen, detecting an end instruction for ending the video screen, and if the end instruction is detected, closing the small window or full-screen window of the video screen.

[0014] In one embodiment of the present application, the method further includes: obtaining the screen aspect ratio of the first device; and determining the aspect ratio of the video picture in the small window and the aspect ratio of the video picture in the full-screen window according to the screen aspect ratio.

[0015] In one embodiment of the present application, the linked data interaction with the second device through the local LAN channel or the cloud server includes: collecting indoor sound to obtain first audio data; and sending the first audio data to the second device through the local LAN channel or the cloud server.

[0016] In one embodiment of the present application, collecting indoor sounds to obtain first audio data includes: collecting indoor sounds to obtain initial audio data, and performing voice change processing on the initial audio data to obtain the first audio data.

[0017] In one embodiment of the present application, the linked data interaction with the second device through the local LAN channel or the cloud server includes: displaying quick reply options, wherein the quick reply options include multiple candidate reply information; receiving information selection instructions, and determining selected reply information from the multiple candidate reply information based on the information selection instructions, and sending the selected reply information to the second device through the local LAN channel or the cloud server.

[0018] According to a second aspect of an embodiment of the present application, another method for linkage between devices is provided, including: sending a first linkage request to a first device through a local LAN channel, and sending a second linkage request to a cloud server, wherein the first linkage request and the second linkage request are used to request linkage with the first device; and performing linkage data interaction with the first device through the local LAN channel or the cloud server.

[0019] In one embodiment of the present application, before sending the first linkage request to the first device through the local LAN channel and sending the second linkage request to the cloud server, it also includes: receiving second configuration information sent by the cloud server, the second configuration information including attribute information of the first device that has a linkage relationship with the second device; or, requesting the second configuration information from the cloud server, and caching the requested second configuration information locally.

[0020] In one embodiment of the present application, before sending the first linkage request to the first device through the local LAN channel and sending the second linkage request to the cloud server, it also includes: based on the second configuration information, sending a second UDP unicast packet in the LAN, attempting to establish the local LAN channel with the first device; after the connection is established through the local LAN channel, performing heartbeat communication with the first device.

[0021] In one embodiment of the present application, before sending the first linkage request to the first device through the local LAN channel and sending the second linkage request to the cloud server, it also includes: sending and exchanging respective public keys with the first device; determining the second encryption key of the second device based on the second private key of the second device and the first public key of the first device.

[0022] In one embodiment of the present application, after determining the second encryption key of the second device based on the second private key of the second device and the first public key of the first device, it also includes: generating a second target encryption key of the second device based on the identification code of the first device and the second encryption key.

[0023] In one embodiment of the present application, sending a first linkage request to a first device through a local LAN channel, and sending a second linkage request to a cloud server, includes: detecting one or more first devices that have the linkage relationship with the second device; when there are multiple first devices, sending the first linkage request to the first device through the local LAN channel of each of the first devices, and sending the second linkage request to each of the first devices through the cloud server.

[0024] In one embodiment of the present application, the linkage data interaction with the first device through the local LAN channel or the cloud server includes: encrypting the linkage data of the second device based on the second encryption key or the second target encryption key to obtain second linkage data; and sending the second linkage data to the first device through the local LAN channel or the cloud server.

[0025] In one embodiment of the present application, the method also includes: in response to an update of the second IP address of the second device, obtaining an updated second IP address of the second device; based on the device identifier of the first device, the device identifier of the second device and the updated second IP address, generating and broadcasting a second UDP broadcast message of the second device.

[0026] In one embodiment of the present application, the linkage data interaction with the first device through the local LAN channel or the cloud server includes: collecting outdoor images and outdoor sounds as second linkage data of the second device, and sending them to the first device through the local LAN channel or the cloud server; receiving the first audio data and / or reply information sent by the first device through the local LAN channel or the cloud server.

[0027] According to a third aspect of an embodiment of the present application, a linkage device between devices is provided, including: a first receiving module, used to receive a first linkage request sent by a second device through a local LAN channel; a second receiving module, used to receive a second linkage request of the second device forwarded by a cloud server, wherein the first linkage request and the second linkage request are used to request linkage with the first device; an interaction module, used to perform linkage data interaction with the second device through the local LAN channel or the cloud server.

[0028] In one embodiment of the present application, the interaction module is further used to: preferentially receive the second linkage data sent by the second device through the local LAN channel; if the local LAN channel is abnormal, receive the second linkage data sent by the cloud server.

[0029] In one embodiment of the present application, the first receiving module is further used to: receive first configuration information sent by the cloud server, wherein the first configuration information includes attribute information of the second device that has a linkage relationship with the first device; or, request the first configuration information from the cloud server and cache the requested first configuration information locally.

[0030] In one embodiment of the present application, the first receiving module is also used to: based on the first configuration information, send a first User Datagram Protocol UDP unicast packet in the local area network, and attempt to establish the local LAN channel with the second device; after the connection is established in the local LAN channel, transmit a heartbeat signal with the second device.

[0031] In one embodiment of the present application, the first receiving module is further used to: send and exchange respective public keys with the second device; and determine a first encryption key of the first device based on a first private key of the first device and a second public key of the second device.

[0032] In one embodiment of the present application, the first receiving module is further used to: generate a first target encryption key of the first device based on the identification code of the second device and the first encryption key.

[0033] In one embodiment of the present application, the interaction module is also used to: encrypt the linkage data with the second device based on the first encryption key or the first target encryption key to obtain first linkage data; and send the first linkage data to the second device through the local LAN channel or the cloud server.

[0034] In one embodiment of the present application, the device also includes: in response to an update of the first IP address of the first device, obtaining an updated first IP address of the first device; based on the device identifier of the first device, the device identifier of the second device and the updated first IP address, generating and broadcasting a first UDP broadcast message of the first device.

[0035] In one embodiment of the present application, the interaction module is also used to: determine the video data and the second audio data based on the second linkage data; based on the video data, display the video screen in a small window on the screen of the first device, and synchronously play the second audio data.

[0036] In one embodiment of the present application, the device also includes: detecting a full-screen display instruction, and if the full-screen display instruction is detected, switching the video screen from the small window form to the full-screen form; during the display of the video screen, detecting an end instruction for ending the video screen, and if the end instruction is detected, closing the small window or full-screen window of the video screen.

[0037] In one embodiment of the present application, the apparatus further includes: obtaining the screen aspect ratio of the first device; and determining the aspect ratio of the video picture in the small window and the aspect ratio of the video picture in the full-screen window according to the screen aspect ratio.

[0038] In one embodiment of the present application, the interaction module is further used to: collect indoor sound to obtain first audio data; and send the first audio data to the second device through the local area network channel or the cloud server.

[0039] In one embodiment of the present application, the interaction module is further used to: collect indoor sound to obtain initial audio data, and perform voice change processing on the initial audio data to obtain the first audio data.

[0040] In one embodiment of the present application, the interaction module is also used to: display quick reply options, wherein the quick reply options include multiple candidate reply information; receive information selection instructions, and determine the selected reply information from the multiple candidate reply information based on the information selection instructions, and send the selected reply information to the second device through the local LAN channel or the cloud server.

[0041] According to the fourth aspect of an embodiment of the present application, another inter-device linkage device is provided, including: a sending module, used to send a first linkage request to a first device through a local LAN channel, and to send a second linkage request to a cloud server, wherein the first linkage request and the second linkage request are used to request linkage with the first device; an interaction module, used to perform linkage data interaction with the first device through the local LAN channel or the cloud server.

[0042] In one embodiment of the present application, the sending module is also used to: receive second configuration information sent by the cloud server, the second configuration information including attribute information of the first device that has a linkage relationship with the second device; or, request the second configuration information from the cloud server and cache the requested second configuration information locally.

[0043] In one embodiment of the present application, the sending module is also used to: send a second UDP unicast packet in the local area network based on the second configuration information, and attempt to establish the local LAN channel with the first device; after the connection is established in the local LAN channel, perform heartbeat communication with the first device.

[0044] In one embodiment of the present application, the sending module is further used to: send and exchange respective public keys with the first device; and determine the second encryption key of the second device based on the second private key of the second device and the first public key of the first device.

[0045] In one embodiment of the present application, the sending module is further used to: generate a second target encryption key of the second device based on the identification code of the first device and the second encryption key.

[0046] In one embodiment of the present application, the sending module is also used to: detect one or more first devices that have the linkage relationship with the second device; when there are multiple first devices, send the first linkage request to the first device through the local LAN channel of each of the first devices, and send the second linkage request to each of the first devices through the cloud server.

[0047] In one embodiment of the present application, the interaction module is also used to: encrypt the linkage data of the second device based on the second encryption key or the second target encryption key to obtain second linkage data; and send the second linkage data to the first device through the local LAN channel or the cloud server.

[0048] In one embodiment of the present application, the device also includes: in response to an update of the second IP address of the second device, obtaining an updated second IP address of the second device; based on the device identifier of the first device, the device identifier of the second device and the updated second IP address, generating and broadcasting a second UDP broadcast message of the second device.

[0049] In one embodiment of the present application, the interaction module is also used to: collect outdoor images and outdoor sounds as second linkage data of the second device, and send them to the first device through the local LAN channel or the cloud server; receive the first audio data and / or reply information sent by the first device through the local LAN channel or the cloud server.

[0050] According to the fifth aspect of the embodiments of the present application, there is provided an electronic device, comprising a processor; and a memory for storing instructions executable by the processor; wherein the processor is configured to implement the steps of the method of the first aspect of the embodiments of the present application.

[0051] According to the sixth aspect of an embodiment of the present application, there is provided an electronic device, comprising a processor; and a memory for storing instructions executable by the processor; wherein the processor is configured to implement the steps of the method of the second aspect of the embodiment of the present application.

[0052] According to the seventh aspect of the embodiments of the present application, a computer-readable storage medium is provided, on which computer program instructions are stored. When the program instructions are executed by a processor, the steps of the method of the first aspect of the embodiments of the present application are implemented.

[0053] According to an eighth aspect of an embodiment of the present application, a computer-readable storage medium is provided, on which computer program instructions are stored. When the program instructions are executed by a processor, the steps of the method of the second aspect of the embodiment of the present application are implemented.

[0054] According to the ninth aspect of the embodiments of the present application, a computer program product is provided, including a computer program, characterized in that when the computer program is executed by a processor of an electronic device, the steps of the method of the first aspect of the embodiments of the present application are implemented.

[0055] According to the tenth aspect of the embodiments of the present application, a computer program product is provided, including a computer program, characterized in that when the computer program is executed by a processor of an electronic device, the steps of the method of the second aspect of the embodiments of the present application are implemented.

[0056] The technical solution provided by the embodiment of the present application brings at least the following beneficial effects: in order to link the first device with the second device, a first linkage request sent by the second device through the local LAN channel and a second linkage request of the second device forwarded by the cloud server can be received to request linkage with the first device, and then linkage data interaction is performed with the second device through the local LAN channel or the cloud server. It supports both the local LAN channel and the cloud server linkage, which greatly improves the stability of the linkage. The linkage using the local LAN channel can achieve low latency, real-time and fast linkage. When the local LAN channel is not smooth, linkage is performed through the cloud server, which improves the success rate of linkage.

[0057] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the application.

[0059] Figure 1 It is a flow chart of a method for linkage between devices provided in an embodiment of the present application.

[0060] Figure 2 It is a flowchart of another method for linkage between devices provided in an embodiment of the present application.

[0061] Figure 3 It is a flowchart of another method for linkage between devices provided in an embodiment of the present application.

[0062] Figure 4 It is a flowchart of another method for linkage between devices provided in an embodiment of the present application.

[0063] Figure 5 It is a flowchart of another method for linkage between devices provided in an embodiment of the present application.

[0064] Figure 6 It is a flowchart of another method for linkage between devices provided in an embodiment of the present application.

[0065] Figure 7 This is a schematic diagram of the linkage with multiple first devices provided in an embodiment of the present application.

[0066] Figure 8 It is a flowchart of another method for linkage between devices provided in an embodiment of the present application.

[0067] Fig. 9It is a flowchart of another method for linkage between devices provided in an embodiment of the present application.

[0068] Fig.10 It is a flowchart of another method for linkage between devices provided in an embodiment of the present application.

[0069] Fig.11 It is a schematic diagram of the linkage between the smart doorbell and the smart TV provided in the embodiment of the present application.

[0070] Fig.12 It is a flow chart of the linkage between devices provided in the embodiment of the present application.

[0071] Fig.13 It is an interactive schematic diagram of a method for linkage between devices provided in an embodiment of the present application.

[0072] Fig.14 It is an interactive schematic diagram of another method for linking devices provided in an embodiment of the present application.

[0073] Fig.15 It is a structural schematic diagram of a linkage device between devices provided in an embodiment of the present application.

[0074] Fig.16 It is a structural schematic diagram of another inter-device linkage device provided in an embodiment of the present application.

[0075] Fig.17 It is a structural schematic diagram of an electronic device provided in an embodiment of the present application.

[0076] Fig.18 It is a schematic diagram of the structure of another electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0077] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0078] The terms used in the embodiments of the present application are only for the purpose of describing specific embodiments, and are not intended to limit the embodiments of the present application. The singular forms of "a" and "the" used in the embodiments of the present application and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0079] It should be understood that, although the terms first, second, third, etc. may be used to describe various information in the embodiments of the present application, these information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of the embodiments of the present application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining" for the purpose of brevity and ease of understanding, the terms used herein when characterizing the size relationship are "greater than" or "less than", "higher than" or "lower than". However, for those skilled in the art, it can be understood that the term "greater than" also covers the meaning of "greater than or equal to", and "less than" also covers the meaning of "less than or equal to"; the term "higher than" covers the meaning of "higher than or equal to", and "lower than" also covers the meaning of "lower than or equal to".

[0080] Figure 1 is a flow chart of a method for inter-device linkage provided by an embodiment of the present application. The method for inter-device linkage can be executed by a first device. Figure 1 As shown, the method may include but is not limited to the following steps:

[0081] S101, receiving a first linkage request sent by a second device through a local area network channel.

[0082] It should be noted that the execution subject of the inter-device linkage method provided in the embodiment of the present application is an electronic device, that is, a first device, which may be a terminal device or other intelligent device. Optionally, the terminal device includes but is not limited to a mobile phone, a tablet computer, an intelligent voice interaction device, etc. In the embodiment of the present application, the first device is located indoors and the second device is located outdoors. The embodiment of the present application does not specifically limit the terminal device.

[0083] It is understandable that a linkage request refers to a request for collaboration or information sharing between two or more devices. It can be transmitted through a network channel or other communication methods to achieve interaction and data exchange between devices. In this application, a linkage request refers to a request initiated when a first device and a second device are linked. For example, when the content on the second device is projected onto the first device, the second device can send a linkage request to the first device, and based on the linkage request, the displayed content can be transmitted to the first device, thereby achieving content sharing between different devices.

[0084] In some implementations, when the first device and the second device are in the same local area network, the first device may receive a first linkage request sent by the second device through the local area network channel to request the second device to be linked with the first device.

[0085] In some implementations, the local area network communication between the first device and the second device can be implemented based on a communication protocol and a communication interface. Optionally, the local area network communication between the first device and the second device can be established based on a communication protocol such as Transmission Control Protocol / Internet Protocol (TCP / IP), User Datagram Protocol (UDP) unicast packet, and a communication interface such as an Application Programming Interface (API) and an application.

[0086] S102, receiving a second linkage request from a second device forwarded by the cloud server, wherein the first linkage request and the second linkage request are used to request linkage with the first device.

[0087] In some implementations, in order to enable communication between devices in different local area networks, or to enable communication between devices when a local area network channel is abnormal, communication between a first device and a second device may be implemented based on a cloud server.

[0088] In some implementations, the first device may receive a second linkage request forwarded by the second device via a cloud server, requesting the second device to be linked to the first device. That is, the second device sends the second linkage request to the cloud server, and the cloud server forwards the second linkage request to the first device.

[0089] Optionally, communication between the first device and the cloud server, and between the second device and the cloud server can be achieved based on Hypertext Transfer Protocol Secure (HTTPS).

[0090] S103, performing linkage data interaction with the second device through a local area network channel or a cloud server.

[0091] It can be understood that the data linkage between the first device and the second device includes the first device sending first linkage data to the second device, and the second device sending second linkage data to the first device.

[0092] It is understandable that when a first device performs linkage data interaction with a second device, linkage data interaction may be performed by one or more first devices and the second device, wherein a linkage relationship exists between the second device and one or more first devices.

[0093] For example, the second device is a smart doorbell, and the first device is a smart TV, a mobile phone, a smart speaker, etc. The smart TV can interact with the smart doorbell in a linked manner, or the mobile phone can interact with the smart doorbell in a linked manner, or the smart speaker can interact with the smart doorbell in a linked manner, or the smart TV, mobile phone, and smart speaker can interact with the smart doorbell in a linked manner at the same time.

[0094] In some implementations, the linkage data interaction between the first device and the second device through the local LAN channel can be regarded as a local direct connection channel. The linkage data interaction between the first device and the second device through the cloud server can be regarded as a cloud direct connection channel.

[0095] Optionally, when the local direct connection channel and the cloud direct connection channel can communicate normally, the first device preferentially selects the local direct connection channel to perform linkage data interaction with the second device. When the local direct connection channel is abnormal, the first device selects the cloud direct connection channel to perform linkage data interaction with the second device.

[0096] In some implementations, the linkage data includes, but is not limited to, audio linkage data, screen display linkage data, information linkage data, etc. Optionally, based on the audio linkage data, the audio of the second device can be played on the first device. Based on the screen display linkage data, the display content of the second device can be displayed on the first device. Based on the information linkage data, the information content of the second device can be displayed on the first device.

[0097] In the linkage method between devices provided in the embodiment of the present application, in order to perform linkage between the first device and the second device, a first linkage request sent by the second device through the local LAN channel and a second linkage request of the second device forwarded by the cloud server can be received to request linkage with the first device, and then linkage data interaction is performed with the second device through the local LAN channel or the cloud server. Both local LAN channel and cloud server linkage modes are supported at the same time, which greatly improves the stability of linkage. Using the local LAN channel for linkage can achieve low-latency, real-time and fast linkage. When the local LAN channel is not smooth, linkage is performed through the cloud server, which improves the success rate of linkage.

[0098] Figure 2 is a flow chart of a method for inter-device linkage provided by an embodiment of the present application. The method for inter-device linkage can be executed by a first device. Figure 2 As shown, the method may include but is not limited to the following steps:

[0099] S201: Obtain first configuration information of a second device.

[0100] In some implementations, the first device can configure the linkage relationship between the first device and the second device by obtaining the first configuration information of the second device. Optionally, the first device can receive the first configuration information sent by the cloud server, wherein the first configuration information includes attribute information of the second device that has a linkage relationship with the first device. The first device can also request the first configuration information from the cloud server and cache the requested first configuration information locally. The first configuration information includes, but is not limited to: the unique code (Identity Document, ID) of the second device, the Internet Communication Protocol (Internet Protocol, IP) address, the device identification code (pincode), etc.

[0101] Furthermore, based on the first configuration information, a first User Datagram Protocol (UDP) unicast packet can be sent in the local area network to try to establish a local LAN channel with the second device. After the local LAN channel is connected, a heartbeat signal is transmitted with the second device to monitor the online status of the second device.

[0102] Optionally, the first device may periodically transmit a heartbeat signal to the second device, and if the heartbeat signal sent by the second device is received, it indicates that the second device is online and working normally. If the heartbeat signal sent by the second device is not received within a period of time, it indicates that the second device is offline or has a fault.

[0103] In some implementations, if the IP address of the first device changes, the second device may be notified via UDP broadcast. Optionally, in response to the first IP address of the first device being updated, the updated first IP address of the first device is obtained. Then, based on the device identifier of the first device, the device identifier of the second device, and the updated first IP address, a first UDP broadcast message of the first device is generated and broadcasted to inform the second device that the IP address of the first device has changed.

[0104] S202, receiving a first linkage request sent by a second device through a local area network channel.

[0105] In the embodiment of the present application, the implementation method of step S202 can be implemented by any method in the embodiments of the present application, which is not limited here and will not be repeated.

[0106] S203, receiving a second linkage request from a second device forwarded by the cloud server, wherein the first linkage request and the second linkage request are used to request linkage with the first device.

[0107] In the embodiment of the present application, the implementation method of step S203 can be implemented by any method in the embodiments of the present application, which is not limited here and will not be repeated.

[0108] S204, performing linkage data interaction with the second device through a local area network channel or a cloud server.

[0109] In some implementations, when the first device receives a linkage request from the second device, it can first determine whether local LAN communication is established between the first device and the second device. If local LAN communication exists, the second linkage data sent by the second device is received preferentially through the local LAN channel. If the local LAN channel is abnormal, the second linkage data sent by the cloud server is received.

[0110] In some implementations, when the first device receives linkage data from the second device, it can reply to the information contained in the linkage data. Optionally, by displaying a quick reply option on the first device, where the quick reply option includes multiple candidate reply information. Then, by receiving an information selection instruction, and according to the information selection instruction, determining the selected reply information from the multiple candidate reply information, and sending the selected reply information to the second device through a local LAN channel or a cloud server, the user can reply to the information of the second device on the first device.

[0111] In the inter-device linkage method provided in the embodiment of the present application, in order to perform linkage between the first device and the second device, communication between the first device and the second device can be established first, and then a first linkage request sent by the second device through the local LAN channel is received, and a second linkage request of the second device forwarded by the cloud server is received, so as to request linkage with the first device, and then perform linkage data interaction with the second device through the local LAN channel or the cloud server. Both local LAN channel and cloud server linkage modes are supported at the same time, which greatly improves the stability of linkage. Using the local LAN channel for linkage can achieve low-latency, real-time and fast linkage. When the local LAN channel is not smooth, linkage is performed through the cloud server, which improves the success rate of linkage.

[0112] Figure 3 is a flow chart of a method for inter-device linkage provided by an embodiment of the present application. The method for inter-device linkage can be executed by a first device. Figure 3 As shown, the method may include but is not limited to the following steps:

[0113] S301, sending and exchanging respective public keys with the second device.

[0114] S302: Determine a first encryption key of the first device based on a first private key of the first device and a second public key of the second device.

[0115] In some implementations, in order to ensure the real-time linkage between the second device and the first device, the first device may first exchange respective public keys with the second device, and then determine the first encryption key of the first device required for subsequent linkage. Optionally, the first encryption key may be determined based on the first private key of the first device and the second public key of the second device.

[0116] For example, assuming that the first private key of the first device is privateKeyA, the first public key of the first device is publicKeyA, the second public key of the second device is publicKeyB, and the first encryption key is shareKeyA, then the first encryption key of the first device shareKeyA=(publicKeyB, privateKeyA).

[0117] S303: Generate a first target encryption key of the first device based on the identification code of the second device and the first encryption key.

[0118] In some implementations, in order to ensure the security and confidentiality of communication with the second device, a more secure first target encryption key may be generated based on the identification code of the second device and the first encryption key, wherein the identification code of the second device may be obtained from the first configuration information.

[0119] Optionally, the identification code and the first encryption key of the second device may be encrypted based on a hash operation message authentication code (Hash Message Authentication Code-Secure Hash Algorithm 256bit, HMAC-SHA256) using a hash function to obtain the first target encryption key of the first device.

[0120] For example, assuming that the identification code of the second device is pincodeB, the first encryption key is shareKey=(publicKeyB, privateKeyA), and the first target encryption key is secureKeyA, then the first target encryption key secureKey A=SHA256HMAC(shareKeyA, pincodeB).

[0121] S304: Receive a first linkage request sent by the second device through the local area network channel.

[0122] In the embodiment of the present application, the implementation method of step S304 can be implemented by any method in the embodiments of the present application, which is not limited here and will not be repeated.

[0123] S305: Receive a second linkage request from a second device forwarded by the cloud server, wherein the first linkage request and the second linkage request are used to request linkage with the first device.

[0124] In the embodiment of the present application, the implementation method of step S305 can be implemented by any method in the embodiments of the present application, which is not limited here and will not be repeated.

[0125] S306: Encrypt the linkage data with the second device based on the first encryption key or the first target encryption key to obtain first linkage data.

[0126] It can be understood that encrypting the linkage data can ensure the confidentiality of the linkage data, prevent the leakage of the linkage data during the linkage process, and ensure the information security of users during the linkage process between devices.

[0127] In some implementations, an encryption algorithm may be used to encrypt the linkage data with the second device based on the first encryption key or the first target encryption key to obtain the first linkage data. Optionally, the linkage data may be encrypted using a symmetric encryption algorithm, an asymmetric encryption algorithm, a hybrid encryption algorithm, or the like to obtain the first linkage data. For example, the linkage data may be encrypted using encryption algorithms such as Advanced Encryption Standard (AES) and Data Encryption Standard (DES).

[0128] S307: Send the first linkage data to the second device through the local area network channel or the cloud server.

[0129] Optionally, if there is local area network communication, the first linkage data is preferentially sent to the second device through the local area network channel. If the local area network channel is abnormal, the first linkage data is sent to the second device through the cloud server.

[0130] In the linkage method between devices provided in the embodiment of the present application, in order to perform linkage between the first device and the second device, the linkage data can be encrypted first to ensure the security of the linkage data, and then the first linkage request sent by the second device through the local LAN channel is received, and the second linkage request of the second device forwarded by the cloud server is received, so as to request linkage with the first device, and then the linkage data is interacted with the second device through the local LAN channel or the cloud server. Both the local LAN channel and the cloud server are supported, which greatly improves the stability of the linkage. The linkage using the local LAN channel can achieve low latency, real-time and fast linkage. When the local LAN channel is not smooth, linkage is performed through the cloud server, which improves the success rate of linkage.

[0131] Figure 4 is a flow chart of a method for inter-device linkage provided by an embodiment of the present application. The method for inter-device linkage can be executed by a first device. Figure 4 As shown, the method may include but is not limited to the following steps:

[0132] S401, receiving a first linkage request sent by a second device through a local area network channel.

[0133] In the embodiment of the present application, the implementation method of step S401 can be implemented by any method in the embodiments of the present application, which is not limited here and will not be repeated.

[0134] S402, receiving a second linkage request from a second device forwarded by the cloud server, wherein the first linkage request and the second linkage request are used to request linkage with the first device.

[0135] In the embodiment of the present application, the implementation method of step S402 can be implemented by any method in the embodiments of the present application, which is not limited here and will not be repeated.

[0136] S403: preferentially receive second linkage data sent by the second device through the local area network channel.

[0137] In the embodiment of the present application, the implementation method of step S403 can be implemented by any method in the embodiments of the present application, which is not limited here and will not be repeated.

[0138] S404: If the local area network channel is abnormal, receive the second linkage data sent by the cloud server.

[0139] In the embodiment of the present application, the implementation method of step S404 can be implemented by any method in the embodiments of the present application, which is not limited here and will not be repeated.

[0140] S405: Determine the video data and the second audio data based on the second linkage data.

[0141] It is understandable that the second linkage data is linkage data sent by the second device and received by the first device through the local LAN channel or the cloud server. The second linkage data includes data that the second device needs to be linked with the first device, such as video data, audio data, etc.

[0142] In the embodiment of the present application, taking the screen projection linkage between the first device and the second device as an example, the video data and the second audio data that need to be displayed on the first device can be determined from the second linkage data.

[0143] For example, the second device is a video doorbell, the first device is a TV, and the first device (TV) receives a linkage request from the second device (video doorbell), and then links with the video doorbell. By receiving the second linkage data sent by the video doorbell, the video data and audio data that need to be displayed on the TV can be determined.

[0144] S406: Based on the video data, display the video image in a small window on the screen of the first device, and synchronously play the second audio data.

[0145] In some implementations, when the first device obtains video data and second audio data from the second device, the video image can be displayed in a small window on the screen of the first device, and the second audio data can be played at the same time, so that the linked data on the second device are all displayed on the first device.

[0146] In some implementations, the second linkage data may also be displayed in full screen on the first device. Optionally, a full screen display instruction may be detected, and if a full screen display instruction is detected, the video screen is switched from a small window to a full screen display.

[0147] Furthermore, during the display of the video screen, an end instruction for ending the video screen is detected. If the end instruction is detected, the small window or full-screen window of the video screen is closed, indicating that this linkage is ended.

[0148] In some implementations, the display ratio of the video picture can also be determined according to the screen ratio of the first device. Optionally, the screen aspect ratio of the first device can be obtained based on the attribute information of the first device. Then, according to the screen aspect ratio, the aspect ratio of the video picture in the small window and the aspect ratio in the full-screen window are determined. For example, if the screen aspect ratio of the first device is 16:9, the aspect ratio of the video picture in the small window and the aspect ratio in the full-screen window are both 16:9.

[0149] In the linkage method between devices provided in the embodiment of the present application, in order to perform linkage between the first device and the second device, a first linkage request sent by the second device through the local LAN channel and a second linkage request of the second device forwarded by the cloud server can be received to request linkage with the first device, and then linkage data is interacted with the second device through the local LAN channel or the cloud server to obtain the second linkage data of the second device, and the content of the second device can be displayed on the first device based on the second linkage data. Both the local LAN channel and the cloud server are supported, which greatly improves the stability of the linkage. The linkage using the local LAN channel can achieve low latency, real-time and fast linkage. When the local LAN channel is not smooth, linkage is performed through the cloud server, which improves the success rate of linkage.

[0150] Figure 5 is a flow chart of a method for inter-device linkage provided by an embodiment of the present application. The method for inter-device linkage can be executed by a first device. Figure 5 As shown, the method may include but is not limited to the following steps:

[0151] S501, receiving a first linkage request sent by a second device through a local area network channel.

[0152] In the embodiment of the present application, the implementation method of step S501 can be implemented by any method in the embodiments of the present application, which is not limited here and will not be repeated.

[0153] S502, receiving a second linkage request from a second device forwarded by the cloud server, wherein the first linkage request and the second linkage request are used to request linkage with the first device.

[0154] In the embodiment of the present application, the implementation method of step S502 can be implemented by any method in the embodiments of the present application, which is not limited here and will not be repeated.

[0155] S503: Collect indoor sound to obtain first audio data.

[0156] S504, sending the first audio data to the second device through the local area network channel or the cloud server.

[0157] In some implementations, when a user uses a first device to communicate with a user in front of a second device, the first device may send the first audio data to the second device. Alternatively, the initial audio data may be obtained by collecting sounds in the room, and the initial audio data may be voice-changed to obtain the first audio data, and then the first audio data may be sent to the second device via a local area network channel or a cloud server.

[0158] Optionally, if there is local area network communication, the first audio data is preferentially sent to the second device through the local area network channel. If the local area network channel is abnormal, the first audio data is sent to the second device through the cloud server.

[0159] For example, the first device is a mobile phone, and the second device is a video doorbell. When the delivery man rings the doorbell, the user can receive video images and audio data from the video doorbell on the mobile phone. The mobile phone can collect the audio of the user's reply to the delivery man and send it to the video doorbell as the first audio.

[0160] In some implementations, the user can perform voice change processing on the initial audio data based on the voice change instruction. Optionally, the voice change function component is displayed on the first device, and the selection operation of the voice change function component is detected. If the selection operation of the voice change function component is detected, it is determined that the voice change instruction is detected.

[0161] Furthermore, candidate voice change types are displayed, and the user can determine a target voice change type from the candidate voice change types. Then, according to the audio encoding parameters of the target voice change type, the initial audio data is voice-changed and encoded to obtain the first audio data.

[0162] In the linkage method between devices provided in the embodiment of the present application, in order to perform linkage between the first device and the second device, a first linkage request sent by the second device through the local LAN channel and a second linkage request of the second device forwarded by the cloud server can be received to request linkage with the first device, and then linkage data interaction is performed with the second device through the local LAN channel or the cloud server. Both local LAN channel and cloud server linkage modes are supported at the same time, which greatly improves the stability of linkage. Using the local LAN channel for linkage can achieve low-latency, real-time and fast linkage. When the local LAN channel is not smooth, linkage is performed through the cloud server, which improves the success rate of linkage.

[0163] Figure 6 1 is a flow chart of a method for inter-device linkage provided by an embodiment of the present application. The method for inter-device linkage can be executed by a second device. Figure 6 As shown, the method may include but is not limited to the following steps:

[0164] S601, sending a first linkage request to a first device through a local LAN channel, and sending a second linkage request to a cloud server, wherein the first linkage request and the second linkage request are used to request linkage with the first device.

[0165] In some implementations, the user can trigger a linkage event of the second device based on the application software. For example, the user can turn on the screen projection function on the second device to trigger the screen projection event. The second device then monitors the linkage trigger event in real time and determines the timing to project the screen to the first device.

[0166] In some implementations, in order to ensure the success rate of screen projection, when the second device monitors the linkage trigger event, it can directly send a first linkage request to the first device based on the local LAN channel. At the same time, it sends a second linkage request to the cloud server, where the first linkage request and the second linkage request are used to request linkage with the first device.

[0167] In some implementations, the second device can detect one or more first devices that have a linkage relationship with the second device. When there are multiple first devices, a first linkage request is sent to the first device through the local LAN channel of each of the first devices, and a second linkage request is sent to each first device through the cloud server.

[0168] In some implementations, in order to establish local LAN communication between the first device and the second device, second configuration information sent by the cloud server may be received, and the second configuration information includes attribute information of the first device that has a linkage relationship with the second device. The second configuration information is also requested from the cloud server, and the requested second configuration information is cached locally. The second configuration information includes, but is not limited to: a unique coding ID, an IP address, a device identification code pincode, etc. of the first device.

[0169] Further, based on the second configuration information, a second UDP unicast packet can be sent in the local area network to try to establish a local area network channel with the first device. After the local area network channel is connected, a heartbeat signal is transmitted with the first device to monitor the online status of the first device.

[0170] Optionally, the second device may periodically transmit a heartbeat signal to the first device. If the second device receives the heartbeat signal sent by the first device, it indicates that the first device is online and working normally. If the second device does not receive the heartbeat signal sent by the first device within a period of time, it indicates that the first device is offline or has a fault.

[0171] In some implementations, if the IP address of the second device changes, the second device may be notified via UDP broadcast. Optionally, in response to the second IP address of the second device being updated, the updated second IP address of the second device is obtained. Then, based on the device identifier of the second device, the device identifier of the second device, and the updated second IP address, a second UDP broadcast message of the second device is generated and broadcasted to inform the second device that the IP address of the second device has changed.

[0172] S602, performing linkage data interaction with the first device through a local area network channel or a cloud server.

[0173] It can be understood that the data linkage between the first device and the second device includes the first device sending first linkage data to the second device, and the second device sending second linkage data to the first device.

[0174] Optionally, if there is local area network communication, the linkage data interaction with the first device is performed preferentially through the local area network channel. If the local area network channel is abnormal, the linkage data interaction with the first device is performed through the cloud server.

[0175] In some implementations, the linkage data includes, but is not limited to, audio linkage data, screen display linkage data, information linkage data, etc. Optionally, based on the audio linkage data, the audio of the second device can be played on the first device. Based on the screen display linkage data, the display content of the second device can be displayed on the first device. Based on the information linkage data, the information content of the second device can be displayed on the first device.

[0176] For example, Figure 7 The schematic diagram of linkage with multiple first devices is shown in FIG. Among them, the doorbell is the second device, and the TV, the speaker with screen, and the mobile phone are all first devices. When someone presses the doorbell outside the door, the doorbell detects the device that has established a linkage relationship with it. If the TV, the speaker with screen, and the mobile phone are all in a linkage relationship with the doorbell, the doorbell will send a linkage request to the TV, the speaker with screen, and the mobile phone at the same time, requesting linkage, and the TV, the speaker with screen, and the mobile phone will display the video screen collected by the doorbell.

[0177] In the linkage method between devices provided in the embodiment of the present application, when the second device monitors the linkage trigger event, it can send a first linkage request through the local LAN channel, and forward the second linkage request through the cloud server, so as to request linkage with the first device, and then perform linkage data interaction with the first device through the local LAN channel or the cloud server. It supports both the local LAN channel and the cloud server linkage, which greatly improves the stability of the linkage. The linkage using the local LAN channel can achieve low latency, real-time and fast linkage. When the local LAN channel is not smooth, linkage is performed through the cloud server, which improves the success rate of linkage.

[0178] Figure 8 1 is a flow chart of a method for inter-device linkage provided by an embodiment of the present application. The method for inter-device linkage can be executed by a second device. Figure 8 As shown, the method may include but is not limited to the following steps:

[0179] S801: Detect one or more first devices that have a linkage relationship with a second device.

[0180] S802, when there are multiple first devices, sending a first linkage request to the first device through the local area network channel of each first device, and sending a second linkage request to each first device through the cloud server.

[0181] S803, when there is a first device, send a first linkage request to the first device through a local area network channel, and send a second linkage request to the cloud server.

[0182] It should be noted that steps S802 and S803 are two different implementation methods of sending linkage requests. When the second device monitors the linkage trigger event, it detects one or more first devices with which there is a linkage relationship. When the second device detects multiple first devices with which there is a linkage relationship, it sends linkage requests to the multiple first devices and interacts with the multiple first devices at the same time. When the second device detects only one first device with which there is a linkage relationship, it sends a linkage request to the first device and interacts with the linkage data.

[0183] S804, performing linkage data interaction with the first device through a local area network channel or a cloud server.

[0184] In the embodiment of the present application, the implementation method of steps S801-S804 can be implemented by any method in the embodiments of the present application, which is not limited here and will not be repeated.

[0185] In the linkage method between devices provided in the embodiment of the present application, when the second device monitors the linkage trigger event, it can send a first linkage request through the local LAN channel, and forward the second linkage request through the cloud server, so as to request linkage with the first device, and then perform linkage data interaction with the first device through the local LAN channel or the cloud server. It supports both the local LAN channel and the cloud server linkage, which greatly improves the stability of the linkage. The linkage using the local LAN channel can achieve low latency, real-time and fast linkage. When the local LAN channel is not smooth, linkage is performed through the cloud server, which improves the success rate of linkage.

[0186] Fig. 9 1 is a flow chart of a method for inter-device linkage provided by an embodiment of the present application. The method for inter-device linkage can be executed by a second device. Fig. 9 As shown, the method may include but is not limited to the following steps:

[0187] S901, sending and exchanging respective public keys with the first device.

[0188] S902: Determine a second encryption key of the second device based on a second private key of the second device and a first public key of the first device.

[0189] In some implementations, in order to ensure the real-time linkage between the second device and the first device, the second device may first exchange their respective public keys with the first device, and then determine the second encryption key of the second device required for subsequent linkage. Optionally, the second encryption key may be determined based on the second private key of the second device and the first public key of the first device.

[0190] For example, assuming that the second private key of the second device is privateKeyB, the second public key of the second device is publicKeyB, the first public key of the first device is publicKeyBA, and the second encryption key is shareKeyB, then the second encryption key of the second device shareKeyB=(publicKeyA, privateKeyB).

[0191] S903: Generate a second target encryption key for the second device based on the identification code and the second encryption key of the first device.

[0192] In some implementations, in order to ensure the security and confidentiality of communication with the first device, a more secure second target encryption key may be generated based on the identification code and the second encryption key of the first device, wherein the identification code of the first device may be obtained from the second configuration information.

[0193] Optionally, the identification code and the second encryption key of the first device may be encrypted based on the HMAC-SHA256 encryption algorithm to obtain the second target encryption key of the second device.

[0194] For example, assuming that the identification code of the first device is pincodeA, the second encryption key is shareKeyB=(publicKeyA, privateKeyB), and the second target encryption key is secureKeyB, then the second target encryption key secureKeyB=SHA256HMAC(shareKeyB, pincodeA).

[0195] S904, sending a first linkage request to the first device through the local area network channel, and sending a second linkage request to the cloud server.

[0196] In the embodiment of the present application, the implementation method of step S904 can be implemented by any method in the embodiments of the present application, which is not limited here and will not be repeated.

[0197] S905: Encrypt the linkage data of the second device based on the second encryption key or the second target encryption key to obtain second linkage data.

[0198] It can be understood that encrypting the linkage data can ensure the confidentiality of the linkage data, prevent the leakage of the linkage data during the linkage process, and ensure the information security of users during the linkage process between devices.

[0199] In some implementations, an encryption algorithm may be used to encrypt the linkage data with the first device based on the second encryption key or the second target encryption key to obtain the second linkage data. Alternatively, the linkage data may be encrypted using a symmetric encryption algorithm, an asymmetric encryption algorithm, a hybrid encryption algorithm, or the like to obtain the first linkage data. For example, the linkage data may be encrypted using an AES encryption algorithm, a DES encryption algorithm, or the like.

[0200] S906, sending the second linkage data to the first device through the local area network channel or the cloud server.

[0201] Optionally, if there is local area network communication, the second linkage data is preferentially sent to the first device through the local area network channel. If the local area network channel is abnormal, the second linkage data is sent to the first device through the cloud server.

[0202] In the linkage method between devices provided in the embodiment of the present application, in order to ensure the security of the linkage data, the linkage data can be encrypted first, and then the linkage trigger event can be monitored. When the second device monitors the linkage trigger event, it can send a first linkage request through the local LAN channel, and forward the second linkage request through the cloud server to request linkage with the first device, and then interact with the first device through the local LAN channel or the cloud server. Both local LAN channel and cloud server linkage methods are supported at the same time, which greatly improves the stability of the linkage. Using the local LAN channel for linkage can achieve low latency, real-time and fast linkage. When the local LAN channel is not smooth, linkage is performed through the cloud server, which improves the success rate of linkage.

[0203] Fig.10 1 is a flow chart of a method for inter-device linkage provided by an embodiment of the present application. The method for inter-device linkage can be executed by a second device. Fig.10 As shown, the method may include but is not limited to the following steps:

[0204] S1001, sending a first linkage request to a first device through a local area network channel, and sending a second linkage request to a cloud server.

[0205] In the embodiment of the present application, the implementation method of step S1001 can be implemented by any method in the embodiments of the present application, which is not limited here and will not be repeated.

[0206] S1002, collecting outdoor images and outdoor sounds as second linkage data of the second device, and sending them to the first device through a local LAN channel or a cloud server.

[0207] In some implementations, it is determined based on the linkage request that the second device is linked with the first device, and the second linkage data for screen projection includes video data and second audio data.

[0208] Optionally, the second device can collect the image outside the house as the video data based on its own camera. Meanwhile, the second device can also collect the sound outside the house based on its own microphone and use it as the second audio data.

[0209] Further, the second linkage data is sent to the first device. Optionally, if there is local area network communication, the second linkage data is preferentially sent to the first device through the local area network channel. If the local area network channel is abnormal, the second linkage data is sent to the first device through the cloud server.

[0210] S1003, receiving the first audio data and / or reply information sent by the first device through the local area network channel or the cloud server.

[0211] In some implementations, after the second device sends the second linkage data to the first device, the first device responds to the second linkage data. If the first device collects the sound in the house, the first audio data sent by the first device is received through the local LAN channel or the cloud server. If the second linkage data contains information data, the reply information sent by the first device is received through the local LAN channel or the cloud server.

[0212] Optionally, if there is local area network communication, the first audio data sent by the first device is preferentially received through the local area network channel. If the local area network channel is abnormal, the reply information sent by the first device is received through the cloud server.

[0213] In the linkage method between devices provided in the embodiment of the present application, when the second device monitors the linkage trigger event, it can send a first linkage request through the local LAN channel, and forward the second linkage request through the cloud server, so as to request linkage with the first device, and then perform linkage data interaction with the first device through the local LAN channel or the cloud server. It supports both the local LAN channel and the cloud server linkage, which greatly improves the stability of the linkage. The linkage using the local LAN channel can achieve low latency, real-time and fast linkage. When the local LAN channel is not smooth, linkage is performed through the cloud server, which improves the success rate of linkage.

[0214] like Fig.11The figure shows a schematic diagram of the linkage between the smart doorbell and the smart TV. The user pre-sets the linkage between the smart doorbell and the smart TV in the application software (Application, APP) on the mobile phone. The smart TV and the smart doorbell can actively obtain each other's ID, IP and other information from the cloud server, and the cloud server can also send each other's ID, IP and other information to establish a linkage relationship between the smart TV and the smart doorbell. When the user outdoors presses the doorbell, the doorbell screen projection event (message) is triggered, and a linkage request can be sent to the smart TV through the local LAN channel (local message 1) or the cloud server (cloud message 2). When the smart TV and the smart doorbell are linked, the smart doorbell sends the video screen to the smart TV through the local LAN channel (local screen projection 1) or the cloud server (cloud screen projection 2).

[0215] like Fig.12 What is shown is a flow chart of linkage between the first device and the second device, wherein the first device is the control end, the second device is the device end, and the cloud server is the cloud service end, and the linkage method can be screen projection linkage. The flow chart includes function configuration stage, local device discovery stage, local intercommunication channel, device trigger event, local direct connection channel, and cloud direct connection channel. The user can send a message to the cloud service end through the mobile phone APP to configure the device end and the control end for linkage, and then the device end and the control end can actively obtain each other's ID, IP and other information from the cloud server, and the cloud server can also send each other's ID, IP and other information to establish a linkage relationship between the device end and the control end.

[0216] Furthermore, a local LAN channel is established between the device and the control end through periodic UDP unicast packets, and whether the connection between the device and the control end is successful is determined through the transmission of heartbeat signals. When the IP address of the device changes, the control end is informed through the UDP broadcast message of the device. When the IP address of the control end changes, the device end is informed through the UDP broadcast message of the control end.

[0217] The device side can generate the device side target encryption key based on its own public key and the private key of the control side to encrypt the linkage data. The control side can generate the control side target encryption key based on its own public key and the private key of the device side to encrypt the linkage data.

[0218] When the user triggers the screen projection event on the device, the device can directly send a first linkage request to the control end through the local LAN channel, and send a second linkage request to the control end through the cloud server. When the control end receives the linkage request, it will firstly link the screen projection with the device through the local LAN channel; if the local LAN channel is abnormal, it will link the screen projection with the device through the cloud server.

[0219] Fig.13 Schematic diagram of an interaction method between devices provided in an embodiment of the present application. Fig.13 As shown, the method may include but is not limited to the following steps:

[0220] S1301: The second device sends a first linkage request to the first device through a local area network channel.

[0221] S1302, the second device sends a second linkage request to the cloud server, and the cloud server sends a second linkage request to the first device.

[0222] S1303: The first device obtains second linkage data of the second device through a local area network channel.

[0223] In the linkage method between devices provided in the embodiment of the present application, in order to perform linkage between the first device and the second device, a first linkage request sent by the second device through the local LAN channel and a second linkage request of the second device forwarded by the cloud server can be received to request linkage with the first device, and then linkage data interaction is performed with the second device through the local LAN channel or the cloud server. Both local LAN channel and cloud server linkage modes are supported at the same time, which greatly improves the stability of linkage. Using the local LAN channel for linkage can achieve low-latency, real-time and fast linkage. When the local LAN channel is not smooth, linkage is performed through the cloud server, which improves the success rate of linkage.

[0224] Fig.14 Schematic diagram of an interaction method between devices provided in an embodiment of the present application. Fig.14 As shown, the method may include but is not limited to the following steps:

[0225] S1401: The second device sends a first linkage request to the first device through a local area network channel.

[0226] S1402, the second device sends a second linkage request to the cloud server, and the cloud server sends a second linkage request to the first device.

[0227] S1403, the second device sends second linkage data to the cloud server, and the cloud server sends second linkage data to the first device.

[0228] In the linkage method between devices provided in the embodiment of the present application, in order to perform linkage between the first device and the second device, a first linkage request sent by the second device through the local LAN channel and a second linkage request of the second device forwarded by the cloud server can be received to request linkage with the first device, and then linkage data interaction is performed with the second device through the local LAN channel or the cloud server. Both local LAN channel and cloud server linkage modes are supported at the same time, which greatly improves the stability of linkage. Using the local LAN channel for linkage can achieve low-latency, real-time and fast linkage. When the local LAN channel is not smooth, linkage is performed through the cloud server, which improves the success rate of linkage.

[0229] Fig.15 is a structural block diagram of a linkage device between devices according to an exemplary embodiment. Fig.15 The device linkage apparatus 1500 according to the embodiment of the present application includes: a first receiving module 1501 , a second receiving module 1502 and an interaction module 1503 .

[0230] The first receiving module 1501 is used to receive a first linkage request sent by the second device through a local area network channel;

[0231] The second receiving module 1502 is used to receive a second linkage request of the second device forwarded by the cloud server, wherein the first linkage request and the second linkage request are used to request linkage with the first device;

[0232] The interaction module 1503 is used to perform linkage data interaction with the second device through the local area network channel or the cloud server.

[0233] In one embodiment of the present application, the interaction module 1503 is also used to: preferentially receive the second linkage data sent by the second device through the local LAN channel; if the local LAN channel is abnormal, receive the second linkage data sent by the cloud server.

[0234] In one embodiment of the present application, the first receiving module 1501 is further used to: receive first configuration information sent by the cloud server, wherein the first configuration information includes attribute information of the second device that has a linkage relationship with the first device; or, request the first configuration information from the cloud server and cache the requested first configuration information locally.

[0235] In one embodiment of the present application, the first receiving module 1501 is also used to: based on the first configuration information, send a first User Datagram Protocol UDP unicast packet in the local area network, and try to establish the local LAN channel with the second device; after the connection is established in the local LAN channel, transmit a heartbeat signal with the second device.

[0236] In one embodiment of the present application, the first receiving module 1501 is further used to: send and exchange respective public keys with the second device; and determine a first encryption key of the first device based on a first private key of the first device and a second public key of the second device.

[0237] In one embodiment of the present application, the first receiving module 1501 is further used to: generate a first target encryption key of the first device based on the identification code of the second device and the first encryption key.

[0238] In one embodiment of the present application, the interaction module 1503 is also used to: encrypt the linkage data with the second device based on the first encryption key or the first target encryption key to obtain first linkage data; and send the first linkage data to the second device through the local LAN channel or the cloud server.

[0239] In one embodiment of the present application, the device also includes: in response to an update of the first IP address of the first device, obtaining an updated first IP address of the first device; based on the device identifier of the first device, the device identifier of the second device and the updated first IP address, generating and broadcasting a first UDP broadcast message of the first device.

[0240] In one embodiment of the present application, the interaction module 1503 is also used to: determine the video data and the second audio data based on the second linkage data; based on the video data, display the video screen in a small window on the screen of the first device, and synchronously play the second audio data.

[0241] In one embodiment of the present application, the device also includes: detecting a full-screen display instruction, and if the full-screen display instruction is detected, switching the video screen from the small window form to the full-screen form; during the display of the video screen, detecting an end instruction for ending the video screen, and if the end instruction is detected, closing the small window or full-screen window of the video screen.

[0242] In one embodiment of the present application, the apparatus further includes: obtaining the screen aspect ratio of the first device; and determining the aspect ratio of the video picture in the small window and the aspect ratio of the video picture in the full-screen window according to the screen aspect ratio.

[0243] In one embodiment of the present application, the interaction module 1503 is further used to: collect indoor sound to obtain first audio data; and send the first audio data to the second device through the local area network channel or the cloud server.

[0244] In one embodiment of the present application, the interaction module 1503 is further used to: collect indoor sound to obtain initial audio data, and perform voice change processing on the initial audio data to obtain the first audio data.

[0245] In one embodiment of the present application, the interaction module 1503 is also used to: display quick reply options, wherein the quick reply options include multiple candidate reply information; receive information selection instructions, and determine the selected reply information from the multiple candidate reply information based on the information selection instructions, and send the selected reply information to the second device through the local LAN channel or the cloud server.

[0246] In the linkage method between devices provided in the embodiment of the present application, in order to perform linkage between the first device and the second device, a first linkage request sent by the second device through the local LAN channel and a second linkage request of the second device forwarded by the cloud server can be received to request linkage with the first device, and then linkage data interaction is performed with the second device through the local LAN channel or the cloud server. Both local LAN channel and cloud server linkage modes are supported at the same time, which greatly improves the stability of linkage. Using the local LAN channel for linkage can achieve low-latency, real-time and fast linkage. When the local LAN channel is not smooth, linkage is performed through the cloud server, which improves the success rate of linkage.

[0247] Fig.16 is a structural block diagram of a linkage device between devices according to an exemplary embodiment. Fig.16 The device linkage apparatus 1600 according to the embodiment of the present application includes: a sending module 1601 and an interaction module 1602 .

[0248] A sending module 1601, configured to send a first linkage request to a first device through a local area network channel, and to send a second linkage request to a cloud server, wherein the first linkage request and the second linkage request are used to request linkage with the first device;

[0249] The interaction module 1602 is used to perform linkage data interaction with the first device through the local area network channel or the cloud server.

[0250] In one embodiment of the present application, the sending module 1601 is also used to: receive second configuration information sent by the cloud server, the second configuration information including attribute information of the first device that has a linkage relationship with the second device; or, request the second configuration information from the cloud server and cache the requested second configuration information locally.

[0251] In one embodiment of the present application, the sending module 1601 is also used to: send a second UDP unicast packet in the local area network based on the second configuration information, and attempt to establish the local LAN channel with the first device; after the connection is established in the local LAN channel, perform heartbeat communication with the first device.

[0252] In one embodiment of the present application, the sending module 1601 is further used to: send and exchange respective public keys with the first device; and determine the second encryption key of the second device based on the second private key of the second device and the first public key of the first device.

[0253] In one embodiment of the present application, the sending module 1601 is further used to: generate a second target encryption key of the second device based on the identification code of the first device and the second encryption key.

[0254] In one embodiment of the present application, the sending module 1601 is also used to: detect one or more first devices that have the linkage relationship with the second device; when there are multiple first devices, send the first linkage request to the first device through the local LAN channel of each of the first devices, and send the second linkage request to each of the first devices through the cloud server.

[0255] In one embodiment of the present application, the interaction module 1602 is also used to: encrypt the linkage data of the second device based on the second encryption key or the second target encryption key to obtain second linkage data; and send the second linkage data to the first device through the local LAN channel or the cloud server.

[0256] In one embodiment of the present application, the device also includes: in response to an update of the second IP address of the second device, obtaining an updated second IP address of the second device; based on the device identifier of the first device, the device identifier of the second device and the updated second IP address, generating and broadcasting a second UDP broadcast message of the second device.

[0257] In one embodiment of the present application, the interaction module 1602 is also used to: collect outdoor images and outdoor sounds as second linkage data of the second device, and send them to the first device through the local LAN channel or the cloud server; receive the first audio data and / or reply information sent by the first device through the local LAN channel or the cloud server.

[0258] In the linkage method between devices provided in the embodiment of the present application, when the second device monitors the linkage trigger event, it can send a first linkage request through the local LAN channel, and forward the second linkage request through the cloud server, so as to request linkage with the first device, and then perform linkage data interaction with the first device through the local LAN channel or the cloud server. It supports both the local LAN channel and the cloud server linkage, which greatly improves the stability of the linkage. The linkage using the local LAN channel can achieve low latency, real-time and fast linkage. When the local LAN channel is not smooth, linkage is performed through the cloud server, which improves the success rate of linkage.

[0259] Fig.17 It is a block diagram of an electronic device according to an exemplary embodiment. Fig.17 The electronic device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0260] like Fig.17 As shown, the electronic device 1700 includes a processor 1701, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) 1702 or the program loaded from the memory 1706 to the random access memory (RAM) 1703. In the RAM 1703, various programs and data required for the operation of the electronic device 1700 are also stored. The processor 1701, the ROM 1702 and the RAM 1703 are connected to each other through a bus 1704. An input / output (I / O) interface 1705 is also connected to the bus 1704.

[0261] The following components are connected to the I / O interface 1705: a memory 1706 including a hard disk, etc.; and a communication part 1707 including a network interface card such as a local area network (LAN) card, a modem, etc., and the communication part 1707 performs communication processing via a network such as the Internet; a drive 1708 is also connected to the I / O interface 1705 as needed.

[0262] In particular, according to an embodiment of the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 1707. When the computer program is executed by the processor 1701, the above-mentioned functions defined in the method of the present application are executed.

[0263] In an exemplary embodiment, a storage medium including instructions is also provided, such as a memory including instructions, and the instructions can be executed by the processor 1701 of the electronic device 1700 to complete the above method. Optionally, the storage medium can be a non-transitory computer-readable storage medium, for example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0264] In the present application, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, apparatus, or device. In the present application, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in combination with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium may be transmitted using any suitable medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0265] Fig.18 It is a structural block diagram of an electronic device according to an exemplary embodiment. Fig.18 The electronic device shown is only an example and should not limit the functions and scope of use of the embodiments of the present application. Fig.18 As shown, the electronic device 1800 includes a processor 1801 and a memory 1802. The memory 1802 is used to store program codes, and the processor 1801 is connected to the memory 1802 to read program codes from the memory 1802 to implement the information acquisition method of the application program in the above embodiment.

[0266] Optionally, the number of processors 1801 may be one or more.

[0267] Optionally, the electronic device may further include an interface 1803, and the number of the interface 1803 may be multiple. The interface 1803 may be connected to an application program and may receive data from an external device such as a sensor.

[0268] A person skilled in the art may understand that the various numerical numbers such as first and second involved in the present application are only used for the convenience of description and are not used to limit the scope of the embodiments of the present application, and also indicate the order of precedence.

[0269] At least one in the present application can also be described as one or more, and a plurality can be two, three, four or more, which is not limited in the present application. In the embodiments of the present application, for a technical feature, the technical features in the technical feature are distinguished by "first", "second", "third", "A", "B", "C" and "D", etc., and there is no order of precedence or size between the technical features described by the "first", "second", "third", "A", "B", "C" and "D".

[0270] The corresponding relationships shown in each table in the present application can be configured or predefined. The values ​​of the information in each table are only examples and can be configured as other values, which are not limited by the present application. When configuring the corresponding relationship between the information and each parameter, it is not necessarily required to configure all the corresponding relationships illustrated in each table. For example, in the table in the present application, the corresponding relationships shown in some rows may not be configured. For another example, appropriate deformation adjustments can be made based on the above table, such as splitting, merging, etc. The names of the parameters shown in the titles of the above tables can also use other names that can be understood by the communication device, and the values ​​or representations of the parameters can also be other values ​​or representations that can be understood by the communication device. When implementing the above tables, other data structures can also be used, such as arrays, queues, containers, stacks, linear lists, pointers, linked lists, trees, graphs, structures, classes, heaps, hash tables or hash tables.

[0271] The predefined in the present application may be understood as defined, predefined, stored, pre-stored, pre-negotiated, pre-configured, solidified, or pre-burned.

[0272] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments applied for herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0273] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0274] The above are only specific implementations of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A linkage method between devices, characterized in that: Executed by a first device, the method includes: Receiving a first linkage request sent by a second device through a local area network channel; receiving a second linkage request of the second device forwarded by the cloud server, wherein the first linkage request and the second linkage request are used to request linkage with the first device; Linked data interaction is performed with the second device through the local area network channel or the cloud server.

2. The method according to claim 1, characterized in that The performing linkage data interaction with the second device through the local area network channel or the cloud server includes: preferentially receiving the second linkage data sent by the second device through the local area network channel; If the local area network channel is abnormal, the second linkage data sent by the cloud server is received.

3. The method according to claim 1, characterized in that Before receiving the first linkage request sent by the second device through the local area network channel, the method further includes: receiving first configuration information sent by the cloud server, wherein the first configuration information includes attribute information of the second device that has a linkage relationship with the first device; or, The first configuration information is requested from the cloud server, and the requested first configuration information is cached locally.

4. The method according to claim 3, characterized in that Before receiving the first linkage request sent by the second device through the local area network channel, the method further includes: Based on the first configuration information, sending a first User Datagram Protocol UDP unicast packet in the local area network, trying to establish the local local area network channel with the second device; After the local area network channel is connected, a heartbeat signal is transmitted to the second device.

5. The method according to claim 1, characterized in that Before receiving the first linkage request sent by the second device through the local area network channel, the method further includes: Sending and exchanging respective public keys with the second device; A first encryption key of the first device is determined based on a first private key of the first device and a second public key of the second device.

6. The method according to claim 5, characterized in that After determining the first encryption key of the first device based on the first private key of the first device and the second public key of the second device, the method further includes: A first target encryption key of the first device is generated based on the identification code of the second device and the first encryption key.

7. The method according to claim 6, characterized in that The performing linkage data interaction with the second device through the local area network channel or the cloud server includes: Encrypting the linkage data with the second device based on the first encryption key or the first target encryption key to obtain first linkage data; The first linkage data is sent to the second device through the local area network channel or the cloud server.

8. The method according to any one of claims 2 to 7, characterized in that: The method further comprises: In response to the first IP address of the first device being updated, obtaining an updated first IP address of the first device; Based on the device identification of the first device, the device identification of the second device and the updated first IP address, a first UDP broadcast message of the first device is generated and broadcast.

9. The method according to any one of claims 2 to 7, characterized in that: After the linkage data interaction with the second device through the local LAN channel or the cloud server, the method further includes: determining video data and second audio data based on the second linkage data; Based on the video data, the video picture is displayed in a small window on the screen of the first device, and the second audio data is played synchronously.

10. The method according to claim 9, characterized in that The method further comprises: Detecting a full-screen display instruction, and if the full-screen display instruction is detected, switching the video image from the small window form to a full-screen form; During the display of the video picture, an end instruction for ending the video picture is detected. If the end instruction is detected, the small window or full-screen window of the video picture is closed.

11. The method according to claim 10, characterized in that The method further comprises: Obtaining the screen aspect ratio of the first device; According to the screen aspect ratio, the aspect ratio of the video picture in the small window and the aspect ratio of the video picture in the full-screen window are determined.

12. The method according to claim 1, characterized in that The performing linkage data interaction with the second device through the local area network channel or the cloud server includes: Collecting indoor sound to obtain first audio data; The first audio data is sent to the second device through the local area network channel or the cloud server.

13. The method according to claim 12, characterized in that The collecting of indoor sound to obtain first audio data includes: The sound in the room is collected to obtain initial audio data, and the initial audio data is subjected to voice changing processing to obtain the first audio data.

14. The method according to claim 1, characterized in that The performing linkage data interaction with the second device through the local area network channel or the cloud server includes: Displaying quick reply options, wherein the quick reply options include a plurality of candidate reply information; Receive an information selection instruction, and determine the selected reply information from the multiple candidate reply information according to the information selection instruction, and send the selected reply information to the second device through the local area network channel or the cloud server.

15. A method for linkage between devices, characterized in that: Executed by a second device, the method includes: Sending a first linkage request to the first device through a local area network channel, and sending a second linkage request to the cloud server, wherein the first linkage request and the second linkage request are used to request linkage with the first device; Linked data interaction is performed with the first device through the local area network channel or the cloud server.

16. The method according to claim 15, characterized in that Before sending the first linkage request to the first device through the local area network channel and sending the second linkage request to the cloud server, the method further includes: receiving second configuration information sent by the cloud server, where the second configuration information includes attribute information of the first device that has a linkage relationship with the second device; or, The second configuration information is requested from the cloud server, and the requested second configuration information is cached locally.

17. The method according to claim 15, characterized in that Before sending the first linkage request to the first device through the local area network channel and sending the second linkage request to the cloud server, the method further includes: Based on the second configuration information, sending a second UDP unicast packet in the local area network, attempting to establish the local area network channel with the first device; After the local area network channel is connected, heartbeat communication is performed with the first device.

18. The method according to claim 15, characterized in that Before sending the first linkage request to the first device through the local area network channel and sending the second linkage request to the cloud server, the method further includes: Send and exchange respective public keys with the first device; A second encryption key of the second device is determined based on a second private key of the second device and a first public key of the first device.

19. The method according to claim 18, characterized in that After determining the second encryption key of the second device based on the second private key of the second device and the first public key of the first device, the method further includes: A second target encryption key for the second device is generated based on the identification code of the first device and the second encryption key.

20. The method according to claim 15, characterized in that The sending of the first linkage request to the first device through the local area network channel, and the sending of the second linkage request to the cloud server, comprises: Detecting one or more of the first devices that have the linkage relationship with the second device; When there are multiple first devices, the first linkage request is sent to the first device through the local LAN channel of each of the first devices, and the second linkage request is sent to each of the first devices through the cloud server.

21. The method according to claim 19, characterized in that The step of performing linkage data interaction with the first device through the local area network channel or the cloud server includes: encrypting the linkage data of the second device based on the second encryption key or the second target encryption key to obtain second linkage data; The second linkage data is sent to the first device through the local area network channel or the cloud server.

22. The method according to any one of claims 16 to 19, characterized in that The method further comprises: In response to the second IP address of the second device being updated, acquiring an updated second IP address of the second device; Based on the device identification of the first device, the device identification of the second device and the updated second IP address, a second UDP broadcast message of the second device is generated and broadcast.

23. The method according to claim 15, characterized in that The step of performing linkage data interaction with the first device through the local area network channel or the cloud server includes: Collecting outdoor images and outdoor sounds as second linkage data of the second device, and sending the data to the first device through the local area network channel or the cloud server; The first audio data and / or reply information sent by the first device is received through the local area network channel or the cloud server.

24. A linkage device between devices, characterized in that: The device comprises: A first receiving module, configured to receive a first linkage request sent by a second device through a local area network channel; A second receiving module, configured to receive a second linkage request of the second device forwarded by the cloud server, wherein the first linkage request and the second linkage request are used to request linkage with the first device; An interaction module is used to perform linkage data interaction with the second device through the local area network channel or the cloud server.

25. A linkage device between devices, characterized in that: The device comprises: A sending module, configured to send a first linkage request to a first device through a local area network channel, and to send a second linkage request to a cloud server, wherein the first linkage request and the second linkage request are used to request linkage with the first device; An interaction module is used to perform linkage data interaction with the first device through the local area network channel or the cloud server.

26. An electronic device, characterized in that: include: processor; a memory for storing processor-executable instructions; The processor is configured to implement the steps of the method of any one of claims 1-14, or to implement the steps of the method of any one of claims 15-23.

27. A computer-readable storage medium storing instructions, which, when executed, implement the method according to any one of claims 1 to 14, or implement the method according to any one of claims 15 to 23.

Images