Multi-screen interaction method, electronic equipment and computer program product

Abstract

The invention discloses a multi-screen interaction method, electronic equipment and a computer program product, and the multi-screen interaction method applied to a cloud terminal comprises the steps: responding to an interaction request initiated by target interaction equipment, and obtaining a to-be-interacted equipment identifier associated with the target interaction equipment; and sending the to-be-interacted content and the to-be-interacted device identifier associated with the target interaction device to a proxy server. The multi-screen interaction method applied to the proxy server comprises the following steps: receiving to-be-interacted content sent by a cloud terminal and a to-be-interacted device identifier associated with a target interaction device; and sending the to-be-interacted content to a to-be-interacted device corresponding to the to-be-interacted device identifier.

Description

Technical Field

[0001] The present application belongs to the field of communication technology, and specifically relates to a multi-screen interaction method, electronic equipment and computer program product. Background Art

[0002] As the current technological development trend of the Internet of Everything, multi-screen interaction has gradually become one of the essential needs of mass entertainment life. Multi-screen interaction refers to the sharing service and wireless screen sharing / screen projection functions for multimedia content. Through the connection of wireless network, a series of operations such as transmission, analysis, and display of multimedia content can be realized. Content can be shared and displayed on different terminal devices to realize media sharing entertainment interaction.

[0003] In the prior art, screen projection mainly forwards multimedia content from the main device to the auxiliary device, establishes a data communication link between the two devices in the same local area network, and the main device sends a screen projection request to the auxiliary device. After the screen projection request is responded to, the corresponding data transmission and synchronous display are executed to the auxiliary device. However, the main device and the auxiliary device need to be in the same network environment. The two devices of the screen projection are limited by the network environment and the dependence on the performance of both devices. If the main device and the auxiliary device are not in the same network environment, it is difficult to achieve multi-screen interaction. Summary of the invention

[0004] The embodiments of the present application provide a multi-screen interaction method, electronic device and computer program product, which can solve the problem that both devices of the screen projection are limited by the network environment and rely on the performance of both devices.

[0005] In a first aspect, an embodiment of the present application provides a multi-screen interaction method, which is applied to a cloud terminal. The method includes: in response to an interaction request initiated by a target interactive device, obtaining an identifier of a device to be interacted with that is associated with the target interactive device; and sending content to be interacted with and the identifier of the device to be interacted with that is associated with the target interactive device to a proxy server.

[0006] In the second aspect, an embodiment of the present application provides a multi-screen interaction method, which is applied to a proxy server, and the method includes: receiving content to be interacted with sent by a cloud terminal and a device identifier to be interacted with associated with a target interactive device; and sending the content to be interacted with to the device to be interacted with corresponding to the device identifier to be interacted with.

[0007] In a third aspect, an embodiment of the present application provides an electronic device, which includes a processor, a memory, and a program or instruction stored in the memory and executable on the processor, wherein the program or instruction, when executed by the processor, implements the steps of the method described in the first aspect.

[0008] In a fourth aspect, an embodiment of the present application provides a readable storage medium, on which a program or instruction is stored, and when the program or instruction is executed by a processor, the steps of the method described in the first aspect are implemented.

[0009] In a fifth aspect, an embodiment of the present application provides a chip, comprising a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run a program or instruction to implement the steps of the method described in the first aspect.

[0010] In a sixth aspect, an embodiment of the present application provides a computer program product, comprising a computer program stored on a non-transitory computer-readable storage medium, wherein the computer program comprises a program or instructions, and when the program or instructions are executed, the steps of the method described in the first aspect are implemented.

[0011] In an embodiment of the present application, the cloud terminal responds to an interaction request initiated by a target interactive device, obtains an identifier of the device to be interacted associated with the target interactive device, and sends the content to be interacted and the identifier of the device to be interacted associated with the target interactive device to a proxy server, which then sends the content to be interacted to the device to be interacted corresponding to the identifier of the device to be interacted, thereby realizing interactive control of the interactive device through the cloud terminal, clouding the interactive control process, releasing the computing resources of the device, reducing dependence on device performance, and transmitting the content to be interacted to the device to be interacted through the proxy server, realizing cross-network use, and breaking the limitation that multi-screen interaction must be in the same local area network environment. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 It is a flowchart of a multi-screen interaction method provided in an embodiment of the present application; Figure 2 It is a flowchart of a multi-screen interaction method provided in an embodiment of the present application; Figure 3 It is a flowchart of a method for training a speech recognition model provided in an embodiment of the present application; Figure 4 It is a structural diagram of a multi-screen interactive system provided in an embodiment of the present application; Figure 5 It is a flowchart of another multi-screen interaction method provided in an embodiment of the present application; Figure 6 It is a flowchart of another multi-screen interaction method provided in an embodiment of the present application; Figure 7 It is a flowchart of another multi-screen interaction method provided in an embodiment of the present application; Figure 8 It is a structural schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0013] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0014] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0015] In the following, in conjunction with the accompanying drawings, a multi-screen interaction method, an electronic device, and a computer program product provided by an embodiment of the present application are described in detail through specific embodiments and their application scenarios.

[0016] Figure 1 The flowchart of a multi-screen interactive method provided by an embodiment of the present application is shown, and the method can be applied to a cloud terminal. Figure 1 , the method may include the following steps.

[0017] Step 102: In response to an interaction request initiated by a target interactive device, an identifier of a device to be interacted with associated with the target interactive device is obtained.

[0018] The target interactive device is a device that initiates an interactive request, and may include a to-be-interacted device or a main interactive device. The main interactive device is a main device having content to be interacted with, and the to-be-interacted device is a receiving device that receives the content to be interacted with, and the to-be-interacted device may include multiple devices. Based on the established data communication link between the main interactive device, the cloud terminal, the proxy server, and the to-be-interacted device, initiating an interactive request may include the main interactive device actively initiating it to the to-be-interacted device, and may also include the to-be-interacted device actively initiating it to the main interactive device.

[0019] The cloud terminal is a virtual device that is logged in by the main interactive device in the cloud and has the same or similar functions as the main interactive device. The cloud terminal is controlled by the cloud terminal application on the main interactive device.

[0020] Step 104: Send the content to be interacted with and the identifier of the device to be interacted with associated with the target interactive device to a proxy server.

[0021] In an embodiment of the present application, the cloud terminal responds to an interaction request initiated by a target interactive device, obtains an identifier of the device to be interacted associated with the target interactive device, and sends the content to be interacted and the identifier of the device to be interacted associated with the target interactive device to a proxy server, which then sends the content to be interacted to the device to be interacted corresponding to the identifier of the device to be interacted, thereby realizing interactive control of the interactive device through the cloud terminal, clouding the interactive control process, releasing the computing resources of the device, reducing dependence on device performance, and transmitting the content to be interacted to the device to be interacted through the proxy server, realizing cross-network use, and breaking the limitation that multi-screen interaction must be in the same local area network environment.

[0022] In one implementation, the target interactive device includes the device to be interacted with or the main interactive device, and the above step 104 sends the content to be interacted with and the device to be interacted with identifier associated with the target interactive device to the proxy server, including the following steps.

[0023] Step 1041: When the interaction request is initiated by the device to be interacted with, receive a first response message from the main interaction device to the interaction request.

[0024] When the interaction request is initiated by the device to be interacted, the main interactive device confirms or denies the interaction request initiated by the device to be interacted, so as to control the multi-screen interaction with the device to be interacted.

[0025] Step 1042: When the first response message indicates confirmation of interaction, the content to be interacted in the primary interactive device and the identifier of the device to be interacted are sent to the proxy server.

[0026] Among them, the first response message indicates confirmation of interaction, indicating that the main interactive device allows multi-screen interaction with the device to be interacted, thereby sending the content to be interacted in the main interactive device and the device identification to be interacted to the proxy server, so that the proxy server forwards the content to be interacted to the device to be interacted corresponding to the device identification to be interacted.

[0027] Optionally, when the interaction request is initiated by the main interactive device, the main interactive device sends the content to be interacted and the device identifier to be interacted associated with the main interactive device to the proxy server, and the proxy server forwards the content to be interacted to the device to be interacted corresponding to the device identifier to be interacted.

[0028] In an embodiment of the present application, when the interaction request is initiated by the device to be interacted, the cloud terminal displays an inquiry message in the main interactive device in response to the interaction request to inquire whether to agree to send the content to be interacted to the device to be interacted, and to review and restrict the device to be interacted to ensure the safety and effectiveness of the interaction. After receiving the first response message of the main interactive device to the interaction request, it is determined whether to agree to send the content to be interacted to the device to be interacted for interaction according to the first response message. When the first response message indicates confirmation of interaction, the content to be interacted in the main interactive device and the identifier of the device to be interacted are sent to the proxy server, and then the proxy server forwards the content to be interacted to the device to be interacted corresponding to the identifier of the device to be interacted, so as to realize the control of multi-screen interaction through the cloud terminal.

[0029] In one implementation, after receiving a response message from the primary interactive device to the interaction request in the above-mentioned step 1041, the method may further include: step 1043, when the first response message indicates denial of interaction, sending a denial instruction generated by the primary interactive device to the proxy server.

[0030] In the embodiment of the present application, when the interaction request is initiated by the device to be interacted, the cloud terminal responds to the interaction request, receives the first response message of the main interactive device to the interaction request, and determines whether to agree to send the content to be interacted to the device to be interacted for interaction according to the first response message. When the first response message indicates that the interaction is denied, the denial instruction generated by the main interactive device is sent to the proxy server, and then the denial instruction is sent to the device to be interacted through the proxy server, so as to indicate to the device to be interacted that the main interactive device does not agree to interact with the device to be interacted, thereby realizing the control of multi-screen interaction through the cloud terminal.

[0031] In one implementation, when the interaction request is initiated by the device to be interacted, before obtaining the device to be interacted identifier associated with the target interactive device in response to the interaction request initiated by the target interactive device in the above step 102, the method further includes the following steps.

[0032] Step 1011, receiving target voice data sent by the proxy server.

[0033] The target voice data is received by the device to be interacted with; the device to be interacted with may actively initiate an interaction request by triggering a built-in instruction in the device to be interacted with, or by receiving the target voice data input by a user.

[0034] Step 1012: convert the target speech data into a second target text through a second speech recognition model, and send the second target text to the proxy server.

[0035] In the embodiment of the present application, when the interactive request is initiated by the device to be interacted, the device to be interacted receives the target voice data and uploads it to the proxy server and the cloud terminal, so that the proxy server and the cloud terminal recognize the target voice data, thereby confirming the interactive request of the device to be interacted. The target voice data is converted into a second target text through a second voice recognition model and the second target text is sent to the proxy server, which is then sent to the device to be interacted by the proxy server. The user of the device to be interacted confirms whether the second target text is correctly recognized. If the recognition is correct, the proxy server sends the interactive request initiated by the device to be interacted to the cloud terminal, and the cloud terminal performs multi-screen interaction based on the interactive request.

[0036] In one implementation, the second speech recognition model includes a multilingual speech recognition model. The method further includes the following steps.

[0037] Step 1061, obtaining a second voice data set, a second annotated voice data set, a second unannotated voice data set and a voice text data set of the region where the device to be interacted is located; wherein the second voice data set, the second annotated voice data set and the second unannotated voice data set all include voice data in multiple languages.

[0038] For example, in a home scenario, the voice data includes different types of languages, different dialects, and mixed languages ​​of the above languages.

[0039] Step 1062: Perform self-supervisory training on the multilingual speech recognition model using the second speech data set.

[0040] Step 1063: Perform supervised training on the multilingual speech recognition model after the self-supervised training using the second labeled speech dataset.

[0041] Step 1064: Perform semi-supervised training on the supervised-trained multilingual speech recognition model using the second unlabeled speech dataset to obtain an initial multilingual speech recognition model.

[0042] Step 1065 , supervised training is performed on the initial multilingual speech recognition model using the speech text dataset of the region where the device to be interacted is located, to obtain a trained multilingual speech recognition model.

[0043] In the embodiment of the present application, the multi-level performance optimization training of the multilingual speech recognition model is performed by obtaining the second speech data set, the second annotated speech data set, the second unannotated speech data set, and the speech text data set of the region where the device to be interacted is located, see Figure 3Compared with the existing single training mode, the training method provided by the embodiment of the present application can obtain a multilingual speech recognition model with higher recognition performance, and the initial multilingual speech recognition model is supervisedly trained through the speech text data set of the region where the device to be interacted is located, so that a multilingual speech recognition model that is more in line with the language characteristics of the region where the device to be interacted is located can be obtained, thereby enabling more accurate recognition of the target speech data input through the device to be interacted.

[0044] Figure 2 The flowchart of a multi-screen interaction method provided by an embodiment of the present application is shown, and the method can be applied to a proxy server. Figure 2 , the method may include the following steps.

[0045] Step 202: receiving the content to be interacted with sent by the cloud terminal and the device identifier to be interacted with associated with the target interactive device.

[0046] The device to be interacted is determined according to the device identifier to be interacted associated with the target interactive device.

[0047] Step 204: Send the content to be interacted with to the device to be interacted with corresponding to the device to be interacted with identifier.

[0048] In an embodiment of the present application, the cloud terminal responds to the interaction request initiated by the target interactive device, obtains the device identifier to be interacted associated with the target interactive device, and sends the content to be interacted and the device identifier to be interacted associated with the target interactive device to the proxy server. The proxy server receives the content to be interacted and the device identifier to be interacted, and then sends the content to be interacted to the device to be interacted corresponding to the device identifier to be interacted, thereby realizing interactive control of the device to be interacted through the cloud terminal, reducing dependence on device performance, and transmitting the content to be interacted to the device to be interacted through the proxy server, realizing cross-network use, and breaking the limitation that multi-screen interaction must be in the same local area network environment.

[0049] In one implementation, the above method may further include the following steps.

[0050] Step 206: receiving a denial instruction sent by the cloud terminal.

[0051] The deny instruction is used to indicate to the device to be interacted that the main interactive device does not agree to interact with the device to be interacted.

[0052] Step 208: Send the denial instruction to the device to be interacted with corresponding to the device to be interacted with identifier.

[0053] In an embodiment of the present application, when the interaction request is initiated by the device to be interacted, the cloud terminal responds to the interaction request, and when the received first response message indicates denial of interaction, the cloud terminal sends the denial instruction generated by the main interactive device to the proxy server, and the proxy server then sends the received denial instruction to the device to be interacted, so as to indicate to the device to be interacted that the main interactive device does not agree to interact with the device to be interacted, thereby realizing control of multi-screen interaction through the cloud terminal.

[0054] In one implementation, before receiving the content to be interacted with and the device identifier to be interacted with associated with the target interactive device sent by the cloud terminal in the above step 202, the method may further include the following steps.

[0055] Step 2011, receiving target voice data sent by the device to be interacted with.

[0056] The target voice data is received by the device to be interacted with, and the device to be interacted with triggers the initiation of an interaction request through the target voice data.

[0057] Step 2012: convert the target speech data into a first target text through a first speech recognition model.

[0058] Step 2013: determine the instruction to be determined that has the highest similarity to the first target text from the voice instruction database, and send the instruction to be determined to the device to be interacted with.

[0059] The voice command database stores commands that can be recognized by various devices. The first target text converted from the target voice data is compared with the commands in the voice command database to determine the command corresponding to the first target text, thereby triggering the corresponding command.

[0060] Step 2014: receiving a second response message from the device to be interacted with to the instruction to be determined, and sending the interaction request to the cloud terminal when the second response message indicates that the instruction to be determined is correct.

[0061] The interaction request includes the instruction to be determined.

[0062] In an embodiment of the present application, when the interactive request is initiated by the device to be interacted, the device to be interacted receives the target voice data and uploads it to the proxy server, so that the proxy server recognizes the target voice data, thereby confirming the interactive request of the device to be interacted. The first target text converted from the target voice data is compared with the instructions in the voice instruction database to determine the instruction to be determined with the highest similarity to the first target text, and the instruction to be determined is sent to the device to be interacted, so that the user of the device to be interacted can confirm whether the instruction to be determined is correctly identified. In the case where the second response message of the device to be interacted to the instruction to be determined indicates that the instruction to be determined is correct, the interactive request including the instruction to be determined is sent to the cloud terminal, and the cloud terminal then performs multi-screen interaction based on the interactive request.

[0063] In one implementation, after receiving a second response message from the device to be interacted with to the instruction to be determined in the above step 2014, the method further includes the following steps.

[0064] Step 2015: When the second response message indicates that the instruction to be determined is wrong, the target voice data is sent to the cloud terminal.

[0065] Among them, when the second response message indicates that the instruction to be determined is wrong, it means that the first speech recognition model of the proxy server recognizes the target voice data incorrectly, and then forwards the target voice data to the cloud terminal, and performs secondary recognition on the target voice data through the second speech recognition model of the cloud terminal, so as to obtain a more accurate text.

[0066] Step 2016, receiving a second target text sent by the cloud terminal, and sending the second target text to the device to be interacted with.

[0067] The second target text is obtained by converting the target voice data by the cloud terminal based on a second voice recognition model.

[0068] Step 2017: receiving a third response message from the device to be interacted with to the second target text, and sending the interaction request to the cloud terminal when the third response message indicates that the second target text is correct.

[0069] The interactive request includes the second target text.

[0070] In an embodiment of the present application, when the second response message received by the device to be interacted to the instruction to be determined indicates that the instruction to be determined is wrong, the target voice data is forwarded to the cloud terminal, and the target voice data is re-recognized by the second voice recognition model of the cloud terminal, and a more complex recognition process is performed at the cloud terminal in the cloud, further reducing the demand for device performance. The cloud terminal receives the second target text obtained by converting the target voice data through the second voice recognition model, and forwards it to the device to be interacted, so that the user using the device to be interacted can confirm whether the second target text is recognized correctly. When the third response message received by the device to be interacted to the second target text indicates that the second target text is correct, an interaction request including the second target text is sent to the cloud terminal, and the cloud terminal performs multi-screen interaction based on the interaction request.

[0071] In the embodiment of the present application, the recognition of the target voice data is performed through staged voice command recognition, and the difficulty of voice recognition is simplified through rough recognition and secondary recognition. Specifically, the rough recognition result is first obtained through rough recognition and similarity to meet the recognition requirements of small computing resources. Then, when the rough recognition is inaccurate, the secondary recognition is performed in the cloud, and the complex computing process is placed in the cloud. The cloud terminal in the cloud performs the secondary recognition, which can achieve high-precision recognition while reducing the demand for device performance. In addition, compared to directly placing the voice recognition process in the cloud, the embodiment of the present application can effectively shorten the voice data recognition process, reduce the call for complex operations, and simplify the voice data recognition process by first performing rough recognition.

[0072] In one implementation, when the third response message indicates that the second target text is correct, the method further includes: updating the second target text into the voice instruction database.

[0073] In an embodiment of the present application, when the second target text obtained by the cloud terminal through the second speech recognition model to recognize the target voice data is correct, the second target text is updated to the voice instruction database of the proxy server, thereby improving the recognition accuracy of the subsequent voice data recognized by the proxy server.

[0074] In one implementation, the first speech recognition model includes a multilingual speech recognition model. The method further includes the following steps.

[0075] Step 2101, obtain a first voice data set, a first annotated voice data set, a first unannotated voice data set and a voice text data set of the region where the device to be interacted is located; wherein the first voice data set, the first annotated voice data set and the first unannotated voice data set all include voice data in multiple languages.

[0076] Step 2102: Perform self-supervisory training on the multilingual speech recognition model using the first speech data set.

[0077] Step 2103: Perform supervised training on the multilingual speech recognition model after the self-supervised training using the first annotated speech dataset.

[0078] Step 2104: Perform semi-supervised training on the supervised-trained multilingual speech recognition model using the first unlabeled speech dataset to obtain an initial multilingual speech recognition model.

[0079] Step 2105: supervised training is performed on the initial multilingual speech recognition model using the speech text dataset of the region where the device to be interacted is located, to obtain a trained multilingual speech recognition model.

[0080] In the embodiment of the present application, the multi-level performance optimization training of the multilingual speech recognition model is performed by obtaining the first speech data set, the first annotated speech data set, the first unannotated speech data set, and the speech text data set of the region where the device to be interacted is located, see Figure 3 Compared with the existing single training mode, the training method provided by the embodiment of the present application can obtain a multilingual speech recognition model with higher recognition performance, and the initial multilingual speech recognition model is supervisedly trained through the speech text data set of the region where the device to be interacted is located, so that a multilingual speech recognition model that is more in line with the language characteristics of the region where the device to be interacted is located can be obtained, thereby enabling more accurate recognition of the target speech data input through the device to be interacted.

[0081] Figure 4 A schematic diagram of the structure of a multi-screen interactive system provided in an embodiment of the present application is shown. Figure 4 The multi-screen interactive system 400 includes a main device 41, a cloud phone 42, a cloud device server 43, a proxy server 44 and an auxiliary device 45.

[0082] The multi-screen interactive method provided in the embodiment of the present application includes active screen projection of the main device 41 to the auxiliary device 45, such as Figure 4 The process (1-2-3-4-5) shown by the solid line also includes the active request of the auxiliary device 45 to cast the screen to the main device 41, such as Figure 4 The process shown by the dotted line in the middle (6-7-8-9-10-11-12-13). Compared with the one-way data operation in the prior art, the embodiment of the present application can better meet the user's experience needs during actual use.

[0083] For example, in a home application scenario, in order to remotely control the video viewing time of children, the parent's device is used as the main device, and the video presentation terminal at home (such as TV, playback tablet and other display devices) is used as the auxiliary device. The main device controls the screen projection of the auxiliary device through the cloud phone, and the auxiliary device obtains the video content of the main device by actively sending requests.

[0084] Figure 5 In one implementation, a flowchart of another multi-screen interaction method provided by an embodiment of the present application is shown, which can be applied to Figure 4 The multi-screen interactive system 400 shown. Figure 5 , the method may include the following steps.

[0085] Step 501, building a data communication link between the main device, cloud phone, cloud device server, proxy server and auxiliary devices.

[0086] Step 502: The main device logs in to the cloud phone to initiate a screen projection request.

[0087] In step 503, the cloud phone forwards the screen projection request to the cloud device server. The cloud device server calls the cloud application according to the screen projection request to search for the auxiliary device associated with the main device and determine the device identifier of the auxiliary device to be interacted with.

[0088] Step 504: the cloud device server encapsulates the multimedia content and returns the encapsulated response data to the cloud phone.

[0089] Step 505 , the cloud phone sends the encapsulated response data and the device identification of the attached device to be interacted with to the proxy server.

[0090] Step 506: The proxy server sends the encapsulated response data to the accessory device to be interacted with corresponding to the device identifier.

[0091] Step 507: the accessory device to be interacted with receives the encapsulated response data and displays the multimedia content corresponding to the encapsulated response data.

[0092] In an embodiment of the present application, a screen projection request is actively initiated by the main device, and the multimedia content is sent to the accessory device to be interacted with through the cloud phone, cloud device server and proxy server and displayed in the accessory device, thereby realizing multi-screen interaction between the main device and the accessory device to be interacted with.

[0093] Figure 6 In one implementation, a flowchart of another multi-screen interaction method provided by an embodiment of the present application is shown, which can be applied to Figure 4 The multi-screen interactive system 400 shown. Figure 6 , the method may include the following steps.

[0094] Step 601, building a data communication link between the main device, cloud phone, cloud device server, proxy server and auxiliary devices.

[0095] Step 602: The auxiliary device initiates a screen projection request to the main device through the proxy server and the cloud phone.

[0096] Among them, the cloud phone parses the screen projection request and generates a corresponding application request notification according to the mapping relationship between the main device and the auxiliary device, so as to inform the main device through the application request notification that the auxiliary device initiates the screen projection request.

[0097] Step 603a, the main device sends a call request to the cloud phone in response to the approval operation of the screen projection request.

[0098] Step 604 , the cloud phone calls the cloud application according to the call request to search for the auxiliary device associated with the main device, and determines the device identifier of the auxiliary device to be interacted with.

[0099] Step 605 to step 608. Step 605 to step 608 are the same as the above steps 504 to 507, and the details can be found in the above description.

[0100] Step 603b: The main device responds to the disagreement operation on the screen projection request and generates a disagreement instruction, and executes step 609.

[0101] Step 609, the cloud phone sends the disagreement instruction of the main device to the auxiliary device through the proxy server, wherein the disagreement instruction is used to indicate that the main device disagrees with the multi-screen interaction with the auxiliary device.

[0102] In an embodiment of the present application, the screen projection request is actively initiated by the accessory device, and the main device needs to perform an agreement or disagreement operation on the screen projection request to control and restrict the accessory device. In the case where the main device performs an agreement operation, the multimedia content is sent to the accessory device to be interacted with through the cloud phone, cloud device server and proxy server and displayed in the accessory device, thereby realizing multi-screen interaction between the main device and the accessory device to be interacted with. In the case where the main device performs a disagreement operation, the cloud phone sends the disagreement instruction generated by the main device to the accessory device through the proxy server, indicating to the accessory device that the main device does not agree to multi-screen interaction with the accessory device.

[0103] Figure 7 In one implementation, a flowchart of another multi-screen interaction method provided by an embodiment of the present application is shown, which can be applied to Figure 4 The multi-screen interactive system 400 shown. Figure 7 , the method may include the following steps.

[0104] Step 701, building a data communication link between the main device, cloud phone, cloud device server, proxy server and auxiliary devices.

[0105] Step 702: The auxiliary device receives voice data and sends the voice data to the proxy server.

[0106] Among them, the auxiliary device is woken up by the voice assistant, and voice is input into the auxiliary device to call the command corresponding to the voice.

[0107] Step 703: The proxy server identifies the voice data, determines the instruction corresponding to the voice data, and feeds the instruction back to the attached device.

[0108] The speech data is converted into a first text through a first speech recognition model, and the similarity between the first text and the instructions in the speech instruction database is calculated, and the instruction with the highest similarity is determined to be the instruction corresponding to the speech data.

[0109] Step 704: the accessory device announces the instruction in the form of voice announcement.

[0110] Step 705a: the proxy server receives a response message of a command error sent by the auxiliary device, and executes steps 706 to 709.

[0111] Step 706: The proxy server sends the voice data to the cloud phone.

[0112] Step 707: The cloud phone calls the second speech recognition model from the cloud device server, and converts the speech data into a second text through the second speech recognition model.

[0113] Step 708: The cloud phone feeds back the second text to the attached device via the proxy server.

[0114] Step 709: the auxiliary device broadcasts the second text in the form of voice broadcast.

[0115] Step 705b: the proxy server receives a response message from the accessory device indicating that the instruction is correct, and executes steps 710 to 717.

[0116] Among them, step 710 to step 717 are the same as the above steps 602 to 609, and the details can be found in the above description.

[0117] Step 718: The proxy server updates the second text into the voice command database.

[0118] In an embodiment of the present application, the accessory device can actively initiate an interaction request by triggering a built-in instruction in the accessory device, or by receiving voice data input by the user into the accessory device and identifying the instruction corresponding to the voice data to trigger the initiation of the interaction request. For example, when the user of the accessory device is not capable of operating according to the built-in instructions, such as a young child, an elderly person, or a person who is performing another activity, an interaction request can also be triggered by voice input. Due to the diversity and complexity of the types of user input languages, the embodiment of the present application performs a rough recognition through the first voice recognition model on the proxy server, and then performs a secondary recognition through the second voice recognition model on the cloud phone. While being able to perform high-precision recognition of voice data, it can also reduce the demand for device performance. After the interaction request is triggered by voice input, multi-screen interaction between the main device and the accessory device is achieved based on the interaction request through the cloud phone and the proxy server.

[0119] The embodiment of the present application also provides an electronic device, which is used to execute the above-mentioned multi-screen interaction method. Figure 8 A schematic diagram of the structure of an electronic device for implementing various embodiments of the present application. Electronic devices may have relatively large differences due to different configurations or performances, and may include a processor (processor) 801, a communication interface (CommunicationsInterface) 802, a memory (memory) 803 and a communication bus 804, wherein the processor 801, the communication interface 802, and the memory 803 communicate with each other through the communication bus 804. The processor 801 can call a computer program stored in the memory 803 and can be run on the processor 801 to execute the various steps of the above-mentioned multi-screen interaction method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0120] The above electronic device structure does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently. For example, the input unit may include a graphics processing unit (GPU) and a microphone, and the display unit may be configured with a display panel in the form of a liquid crystal display, an organic light-emitting diode, etc. The user input unit includes a touch panel and at least one of other input devices. The touch panel is also called a touch screen. Other input devices may include, but are not limited to, a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, and a joystick, which will not be repeated here.

[0121] The memory can be used to store software programs and various data. The memory may mainly include a first storage area for storing programs or instructions and a second storage area for storing data, wherein the first storage area may store an operating system, an application program or instructions required for at least one function (such as a sound playback function, an image playback function, etc.), etc. In addition, the memory may include a volatile memory or a non-volatile memory, or the memory may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. Volatile memory can be random access memory (RAM), static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM) and direct memory bus random access memory (DRRAM).

[0122] The processor may include one or more processing units; optionally, the processor integrates an application processor and a modem processor, wherein the application processor mainly processes operations related to the operating system, user interface, and application programs, and the modem processor mainly processes wireless communication signals, such as a baseband processor. It is understandable that the modem processor may not be integrated into the processor.

[0123] An embodiment of the present application also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, each process of the above-mentioned multi-screen interaction method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0124] The processor is a processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0125] An embodiment of the present application further provides a chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned multi-screen interaction method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0126] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0127] An embodiment of the present application also provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium, and the computer program includes a program or instructions. When the program or instructions are executed, the various processes of the above-mentioned multi-screen interaction method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, they are not repeated here.

[0128] It should be noted that, in this article, the terms "comprise", "include" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise one..." do not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0129] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0130] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

Claims

1. A multi-screen interactive method, characterized in that: Applied to cloud terminals, including: In response to an interaction request initiated by a target interactive device, obtaining an identifier of a device to be interacted with associated with the target interactive device; The content to be interacted with and the identifier of the device to be interacted with associated with the target interactive device are sent to a proxy server.

2. The method according to claim 1, characterized in that The target interactive device includes the device to be interacted or the main interactive device; the sending of the content to be interacted and the identifier of the device to be interacted associated with the target interactive device to the proxy server includes: In a case where the interaction request is initiated by the device to be interacted, receiving a first response message of the main interaction device to the interaction request; In the case where the first response message indicates confirmation of interaction, the content to be interacted with in the primary interactive device and the identifier of the device to be interacted with are sent to the proxy server.

3. The method according to claim 2, characterized in that After receiving a first response message from the primary interactive device to the interactive request, the method may further include: In the case where the first response message indicates denial of interaction, the denial instruction generated by the primary interactive device is sent to the proxy server.

4. The method according to claim 2, characterized in that: In the case where the interaction request is initiated by the device to be interacted, before obtaining the device to be interacted identifier associated with the target interactive device in response to the interaction request initiated by the target interactive device, the method further includes: Receiving target voice data sent by the proxy server; wherein the target voice data is received by the device to be interacted with; The target speech data is converted into a second target text through a second speech recognition model, and the second target text is sent to the proxy server.

5. A multi-screen interaction method, characterized in that: Applicable to proxy servers, including: Receiving the content to be interacted with sent by the cloud terminal and the device identifier to be interacted with associated with the target interactive device; The content to be interacted with is sent to the device to be interacted with corresponding to the device to be interacted with identifier.

6. The method according to claim 5, characterized in that The method further comprises: receiving a denial instruction sent by the cloud terminal; The denial instruction is sent to the device to be interacted with corresponding to the device to be interacted with identifier.

7. The method according to claim 5, characterized in that Before receiving the content to be interacted with and the device identifier to be interacted with associated with the target interactive device sent by the cloud terminal, the method further includes: Receiving target voice data sent by the device to be interacted with; Converting the target speech data into a first target text through a first speech recognition model; Determine the instruction to be determined with the highest similarity to the first target text from the voice instruction database, and send the instruction to be determined to the device to be interacted with; receiving a second response message of the device to be interacted with to the instruction to be determined, and sending the interaction request initiated by the device to be interacted with to the cloud terminal when the second response message indicates that the instruction to be determined is correct; The interaction request includes the instruction to be determined.

8. The method according to claim 7, characterized in that After receiving a second response message from the to-be-interacted device to the to-be-determined instruction, the method further includes: When the second response message indicates that the to-be-determined instruction is wrong, sending the target voice data to the cloud terminal; Receiving a second target text sent by the cloud terminal, and sending the second target text to the device to be interacted with; wherein the second target text is obtained by the cloud terminal converting the target voice data based on a second voice recognition model; receiving a third response message from the device to be interacted to the second target text, and sending the interaction request initiated by the device to be interacted to the cloud terminal when the third response message indicates that the second target text is correct; The interactive request includes the second target text.

9. An electronic device, characterized in that: It includes a processor, a memory, and a program or instruction stored in the memory and executable on the processor, wherein the program or instruction, when executed by the processor, implements the steps of the multi-screen interaction method as described in any one of claims 1 to 4, or implements the steps of the multi-screen interaction method as described in any one of claims 5 to 8.

10. A computer program product, characterized in that The computer program product includes a computer program stored on a non-transitory computer-readable storage medium, and the computer program includes a program or instructions, which, when executed, implements the steps of the multi-screen interaction method as described in any one of claims 1 to 4, or implements the steps of the multi-screen interaction method as described in any one of claims 5 to 8.

Images