Terminal interaction method and system, electronic device and computer readable storage medium

Abstract

The application discloses a terminal interaction method and system, an electronic device and a computer readable storage medium. The method comprises the following steps: sending multimedia data to a control terminal; receiving an operation instruction fed back by the control terminal, wherein the operation instruction is generated by the control terminal according to a trigger operation for the multimedia data; determining a trigger position and operation information of the trigger operation in a display screen of the control terminal according to the operation instruction; determining a display position of the trigger operation in a display screen of a controlled terminal according to the trigger position; and outputting the operation information at the display position. The technical scheme provided by the application realizes the synchronization of the multimedia data of the controlled terminal to the control terminal, the synchronization of the control operation of the control terminal for the multimedia data to the controlled terminal, and further realizes the double-screen interaction. The operation mode is more convenient and fast, more diversified information interaction can be performed, and the user experience is effectively improved.

Description

Technical Field

[0001] This application relates to the field of multimedia technology, and in particular to a terminal interaction method, system, electronic device, and computer-readable storage medium. Background Technology

[0002] With the rapid development of science and technology, multimedia devices such as smart TVs have become increasingly popular, allowing users to watch online movies, TV series, and other programs via wired or wireless networks. However, in terms of interaction, smart TVs are not significantly different from traditional TVs. Both rely on remote controls or remote-controlled apps on mobile devices to control the displayed content. Unlike watching videos on a mobile phone, which allows for direct touchscreen interaction, smart TVs are not convenient to use and their interactive features are limited.

[0003] Therefore, how to achieve more convenient, faster, and more diverse information interaction with multimedia devices is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0004] In view of this, in order to solve the above-mentioned technical problems, this application provides a terminal interaction method, system, electronic device and computer-readable storage medium.

[0005] In a first aspect, this application provides a terminal interaction method applied to a controlled terminal, comprising:

[0006] Send multimedia data to the control terminal;

[0007] The system receives operation instructions fed back from the control terminal, which are generated by the control terminal based on a trigger operation targeting the multimedia data.

[0008] The triggering position and operation information of the triggering operation on the control terminal display screen are determined according to the operation instructions;

[0009] The display position of the trigger operation on the controlled terminal screen is determined based on the trigger position, and the operation information is output at the display position.

[0010] Optionally, a first communication channel and a second communication channel are established between the control terminal and the controlled terminal;

[0011] Sending multimedia data to the control terminal includes:

[0012] The multimedia data is sent to the control terminal through the first communication channel;

[0013] The receiving of operation instructions from the control terminal includes:

[0014] The operation instructions are received from the control terminal via the second communication channel.

[0015] Optionally, sending multimedia data to the control terminal includes:

[0016] The controlled terminal display screen is recorded to obtain the multimedia data;

[0017] The multimedia data is sent to the control terminal.

[0018] Optionally, determining the display position of the trigger operation on the controlled terminal display screen based on the trigger position includes:

[0019] The display position corresponding to the trigger position is determined based on the coordinate mapping relationship, which is the mapping relationship between the coordinate system of the control terminal display screen and the coordinate system of the controlled terminal display screen.

[0020] Optionally, it also includes:

[0021] Monitor the playback progress of the multimedia data on the controlled terminal display screen;

[0022] When the playback progress reaches a preset playback node, push information about the multimedia data is sent to the control terminal, so that the control terminal outputs the push information to the control terminal display screen, and the push information is superimposed on the multimedia data for display.

[0023] Optionally, it also includes:

[0024] Receive push requests sent by the control terminal;

[0025] Determine push information about the multimedia data based on the push request;

[0026] The push information is sent to the control terminal, so that the control terminal outputs the push information to the control terminal display screen, and the push information is superimposed on the multimedia data for display.

[0027] Secondly, this application provides another terminal interaction method for use in a control terminal, including:

[0028] Receive multimedia data sent by the controlled terminal;

[0029] Obtain the trigger operation for the multimedia data;

[0030] An operation instruction is generated based on the trigger operation, and the operation instruction includes the trigger position and operation information of the trigger operation on the control terminal display screen.

[0031] The operation instruction is sent to the controlled terminal so that the controlled terminal determines the display position of the trigger operation on the controlled terminal display screen according to the trigger position, and outputs the operation information at the display position.

[0032] Thirdly, this application provides a terminal interaction system, including a controlled terminal and a control terminal, wherein...

[0033] The control terminal is used to receive multimedia data sent by the controlled terminal, obtain a trigger operation for the multimedia data, generate an operation instruction according to the trigger operation, and send the operation instruction to the controlled terminal.

[0034] The controlled terminal is configured to send the multimedia data to the control terminal, and determine the trigger position and operation information of the trigger operation on the control terminal display screen according to the operation instruction, determine the display position of the trigger operation on the controlled terminal display screen according to the trigger position, and output the operation information at the display position.

[0035] Fourthly, this application provides an electronic device, comprising:

[0036] Memory, used to store computer programs;

[0037] A processor, when executing the computer program, implements the steps of any of the terminal interaction methods described above.

[0038] Fifthly, this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of any of the terminal interaction methods described above.

[0039] The terminal interaction scheme provided in this application sends multimedia data to a control terminal; receives operation instructions from the control terminal, which are generated by the control terminal based on the trigger operation for the multimedia data; determines the trigger position and operation information on the control terminal's display screen based on the operation instructions; determines the display position of the trigger operation on the controlled terminal's display screen based on the trigger position; and outputs the operation information at the display position. This achieves the synchronization of multimedia data from the controlled terminal to the control terminal, and the synchronization of control operations from the control terminal to the controlled terminal for the multimedia data, thereby realizing dual-screen interaction. This not only makes the operation more convenient and faster, but also allows for more diverse information interaction, effectively improving the user experience. Attached Figure Description

[0040] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0041] To more clearly illustrate the technical solutions in the prior art and the embodiments of this application, the accompanying drawings used in the description of the prior art and the embodiments of this application will be briefly introduced below. Of course, the accompanying drawings described below with respect to the embodiments of this application are only a part of the embodiments in this application. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort, and such other drawings also fall within the protection scope of this application.

[0042] Figure 1 This is a schematic diagram of the structure of a terminal interaction system provided in an embodiment of this application;

[0043] Figure 2 A schematic flowchart illustrating a terminal interaction method applied to a controlled terminal, provided as an embodiment of this application;

[0044] Figure 3 A flowchart illustrating another terminal interaction method applied to a controlled terminal provided in an embodiment of this application;

[0045] Figure 4 A flowchart illustrating another terminal interaction method applied to a controlled terminal provided in an embodiment of this application;

[0046] Figure 5 A flowchart illustrating another terminal interaction method applied to a controlled terminal provided in an embodiment of this application;

[0047] Figure 6 A flowchart illustrating a terminal interaction method applied to a control terminal, provided as an embodiment of this application;

[0048] Figure 7 This application provides an illustration of a terminal interaction method according to an embodiment of the present application.

[0049] Figure 8 An application scenario diagram for another terminal interaction method provided in the embodiments of this application;

[0050] Figure 9 This is a schematic diagram of the structure of a terminal interactive device provided in an embodiment of this application;

[0051] Figure 10 This is a schematic diagram of another terminal interactive device provided in an embodiment of this application;

[0052] Figure 11 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0053] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0054] The terminal interaction method provided in this application is applied to a terminal interaction system. Please refer to [reference needed]. Figure 1 , Figure 1 This is a schematic diagram of a terminal interaction system provided in an embodiment of this application. The terminal interaction system may include a control terminal 100 and a controlled terminal 200, which are used to realize information interaction between the control terminal 100 and the controlled terminal 200. The information interaction mainly includes synchronizing multimedia data of the controlled terminal 200 to the control terminal 100, and synchronizing the control operations of the control terminal 100 on the multimedia data to the controlled terminal 200, thereby realizing dual-screen interaction.

[0055] Among them, the controlled terminal 100 can be a smart TV or other terminal device that can play multimedia data, and the control terminal 200 can be a mobile phone, tablet or other terminal device that can be operated by touch screen.

[0056] It is understandable that the number of control terminals 100 is not unique. When there are multiple terminals, multi-terminal interaction can be realized. For example, a chat room can be formed for multimedia data in the controlled terminals 200 to facilitate topic discussion, interaction, voting, etc.

[0057] This application provides a terminal interaction method.

[0058] Please refer to Figure 2 , Figure 2 This application provides a flowchart illustrating a terminal interaction method applied to a controlled terminal, which may include the following steps S101 to S104:

[0059] S101: Send multimedia data to the control terminal.

[0060] This step aims to transmit multimedia data, specifically, the controlled terminal sends multimedia data to the controlling terminal. In particular, upon receiving a communication request from the controlling terminal, multimedia data is obtained. The communication request instructs the controlling terminal to interact with the controlled terminal. In this embodiment, the multimedia data includes, but is not limited to, video and images.

[0061] Specifically, after the controlled terminal obtains multimedia data, it can first compress the multimedia data to obtain compressed multimedia data, namely the compressed data mentioned above. Then, the compressed data is sent to the control terminal for further processing.

[0062] Correspondingly, after receiving compressed data sent by the controlled terminal, the control terminal can first decompress the compressed data to obtain multimedia data, and then perform subsequent operations on the multimedia data.

[0063] As can be seen, the terminal interaction method provided in this application transmits multimedia data in the form of compressed packets when the controlled terminal and the control terminal interact, which can effectively improve data transmission efficiency and further ensure the real-time performance of screen sharing.

[0064] S102: Receive operation instructions from the control terminal. The operation instructions are generated by the control terminal based on the trigger operation for multimedia data.

[0065] This step aims to receive operation instructions, which are instructions for operating multimedia data and are generated by the control terminal based on trigger operations related to the multimedia data. Specifically, after receiving multimedia data sent by the controlled terminal, the control terminal outputs it to its display screen, enabling synchronous display of multimedia data between the control terminal and the controlled terminal. Furthermore, during the display, users can perform various operations on the multimedia data via touchscreen, such as liking, posting comments, sending emojis, and voting. Simultaneously, the control terminal collects these user trigger operations on the multimedia data and generates corresponding operation instructions. Finally, the operation instructions are fed back to the controlled terminal.

[0066] S103: Determine the trigger position and operation information of the trigger operation on the control terminal display screen according to the operation instruction.

[0067] This step aims to parse the operation instructions. Specifically, after receiving the operation instructions from the control terminal, the trigger position and operation information of the triggered operation on the control terminal's display screen can be obtained through instruction parsing. The operation information refers to the specific action performed by the user on multimedia data, such as liking, text commenting, or emoji commenting; the trigger position is the specific location (which can be coordinate information) where the operation data occurs on the control terminal's display screen, such as the location of the like, the display position of the comment text, or the display position of the emoji. This trigger position is used to determine the specific location of the operation data on the controlled terminal's display screen.

[0068] S104: Determine the display position of the trigger operation on the controlled terminal screen according to the trigger position, and output the operation information at the display position.

[0069] This step aims to execute the operation instructions. After parsing the operation instructions, based on the determined trigger location and operation information, the operation information can be displayed on the controlled terminal's screen. This enables synchronous display of operation events related to multimedia data between the control terminal and the controlled terminal.

[0070] For example, when the control terminal triggers a "like" action on multimedia data, the controlled terminal's display screen will simultaneously show a "like" animation effect; when the control terminal triggers a "comment" action on multimedia data, the controlled terminal's display screen will simultaneously show the comment content; and when the control terminal triggers a "comment" action on multimedia data using emojis, the controlled terminal's display screen will simultaneously show the same emoji animation effect.

[0071] As can be seen, the terminal interaction method provided in this application sends multimedia data to the control terminal; receives operation instructions from the control terminal, the operation instructions are generated by the control terminal based on the trigger operation for the multimedia data, the trigger position and operation information of the trigger operation on the control terminal display screen are determined according to the operation instructions, the display position of the trigger operation on the controlled terminal display screen is determined according to the trigger position, and the operation information is output at the display position. This realizes the synchronization of multimedia data from the controlled terminal to the control terminal and the synchronization of control operations of the control terminal for the multimedia data to the controlled terminal, thereby realizing dual-screen interaction. This not only makes the operation more convenient and faster, but also allows for more diverse information interaction, effectively improving the user experience.

[0072] Please refer to Figure 3 , Figure 3 This is a flowchart illustrating another terminal interaction method applied to a controlled terminal, provided as an embodiment of this application. The method includes the following steps:

[0073] S201: Record the screen of the controlled terminal to obtain multimedia data;

[0074] In this embodiment, when terminal interaction is required (i.e., when a communication request is received from the control terminal), the control terminal can first establish a communication connection with the controlled terminal, then the controlled terminal records its screen to obtain multimedia data, and finally sends the multimedia data to the control terminal through the communication connection. In this embodiment, screencasting technology can be used to record the data displayed on the controlled terminal's screen.

[0075] S202: Send multimedia data to the control terminal through the first communication channel.

[0076] S203: Receive operation instructions from the control terminal via the second communication channel.

[0077] In this embodiment, for steps S202 and S203, to achieve data communication between the control terminal and the controlled terminal, a first communication channel and a second communication channel are established between the control terminal and the controlled terminal. Information interaction is achieved through the first and second communication channels. A communication channel can be established in advance between the control terminal and the controlled terminal. Therefore, multimedia data can be sent to the control terminal and operation commands can be received from the control terminal using this communication channel. Specifically, when establishing the communication channel, the control terminal can use the address information of the controlled terminal, which can be the IP address of the controlled terminal, thereby establishing the communication channel between the control terminal and the controlled terminal.

[0078] In one possible implementation, the address information may include a first port address and a second port address. The control terminal establishes a communication channel with the controlled terminal based on the address information of the controlled terminal, which may include the following steps:

[0079] The steps include: the control terminal establishing a first communication channel with the controlled terminal based on the first port address, and using the first communication channel to send multimedia data to the controlled terminal;

[0080] A second communication channel is established with the controlled terminal based on the second port address, so as to execute the steps of receiving operation instructions from the control terminal using the second communication channel.

[0081] Specifically, dual communication channels can be established, including the aforementioned first and second communication channels. This allows the controlled terminal to send multimedia data to the control terminal via the first communication channel, and for the controlled terminal to receive operation commands from the control terminal via the second communication channel. The address information of the controlled terminal can include a first port address and a second port address. For example, the first port address can be the controlled terminal's IP address plus the port number of the first port, and the second port address can be the controlled terminal's IP address plus the port number of the second port. Thus, the control terminal can establish a first communication channel with the controlled terminal using the first port address for multimedia data transmission, and a second communication channel using the second port address for operation command transmission.

[0082] Furthermore, the second communication channel can also be used by the controlled terminal to send data information other than multimedia data to the control terminal. For example, when the multimedia data is a movie, the controlled terminal can send movie data to the control terminal through the first communication channel, and at the same time, send information about movie characters to the control terminal through the second communication channel. Similarly, when there are multiple controlled terminals interacting, the second communication channel can be used to send trigger operations from other control terminals regarding multimedia data to this control terminal. Therefore, the second communication channel can be a bidirectional communication channel. Based on this, to ensure that the multimedia data can be clearly displayed on the control terminal's screen, the first communication channel can be used only for transmitting multimedia data; that is, the first communication channel can be a unidirectional communication channel.

[0083] As can be seen, the terminal interaction method provided in this application, based on the realization of synchronizing multimedia data of the controlled terminal to the control terminal and synchronizing the control operations of the control terminal on the multimedia data to the controlled terminal, thereby realizing terminal interaction, can effectively ensure the real-time performance of terminal interaction by establishing dual communication channels for data transmission, avoiding the data congestion and transmission delay problems that may be caused by using a single communication channel for data transmission; in addition, the separate establishment of a first communication channel for unidirectional transmission is only used to realize the transmission of multimedia data, which can further ensure the clarity of the multimedia data displayed on the control terminal display screen.

[0084] S204: Determine the trigger position and operation information of the trigger operation on the control terminal display screen according to the operation instruction.

[0085] In this embodiment, step S204 is the same as step S103, and will not be described in detail here. For details, please refer to the description above.

[0086] S205: Determine the display position corresponding to the trigger position according to the coordinate system mapping relationship, and output the operation information at the display position. The coordinate system mapping relationship is the mapping relationship between the coordinate system of the control terminal display screen and the coordinate system of the controlled terminal display screen.

[0087] Specifically, a mapping relationship can be pre-established between the coordinate system of the control terminal display screen and the coordinate system of the controlled terminal display screen, i.e., the aforementioned coordinate mapping relationship. This allows for determining the display position of the multimedia data trigger operation on the control terminal display screen at the controlled terminal display screen based on this coordinate mapping relationship. In implementation, after receiving the operation command, it can be parsed to obtain the trigger position and operation information it carries. Further, the display position on the controlled terminal display screen corresponding to the trigger position is determined by referring to the coordinate mapping relationship. This trigger position is understood to be the display position of the operation data on the control terminal display screen. Finally, the operation information is output at the display position on the controlled terminal display screen, so that the operation information is displayed at the corresponding position on the controlled terminal display screen.

[0088] As can be seen, the terminal interaction method provided in this application, based on the realization of synchronizing the multimedia data of the controlled terminal to the control terminal and synchronizing the control operations of the control terminal on the multimedia data to the controlled terminal, thereby realizing terminal interaction, further achieves this by pre-establishing a mapping relationship between the coordinate system of the control terminal display screen and the coordinate system of the controlled terminal display screen. This allows the control terminal display screen and the controlled terminal display screen to display the exact same image, achieving proportional scaling of the displayed image and helping to improve the user experience.

[0089] The terminal interaction method provided in this embodiment can also push similar peripherals related to multimedia data to the control terminal during the interaction between the controlled terminal and the control terminal. The interaction process between the controlled terminal and the control terminal can be understood as the process of synchronously displaying multimedia data between the control terminal and the controlled terminal. Specifically, pushing similar peripherals related to multimedia data to the control terminal can be achieved in the following manner.

[0090] In one possible implementation, refer to Figure 4 During the interaction between the controlled terminal and the controlling terminal, the terminal interaction method provided in this embodiment further includes:

[0091] S301: Monitor the playback progress of multimedia data on the controlled terminal display screen.

[0092] S302: When the playback progress reaches the preset playback node, push information about multimedia data is sent to the control terminal, so that the control terminal outputs the push information to the control terminal display screen, so that the push information is superimposed on the multimedia data for display.

[0093] In this embodiment, the push information refers to related merchandise items associated with the multimedia data. For example, the push information could be a link to purchase a magazine or shirt related to the multimedia data. The specific format of the push information can be set according to actual needs, and this embodiment does not impose any specific limitations. For multimedia data, a preset correspondence between preset playback nodes and push information can be stored in the controlled terminal. By monitoring the playback progress of the multimedia data on the controlled terminal's display screen, the playback node corresponding to the monitoring time can be obtained. When the playback node is a preset playback node, the push information can be obtained from the preset correspondence.

[0094] As can be seen, the terminal interaction method provided in this application, based on the realization of synchronizing multimedia data of the controlled terminal to the control terminal and synchronizing the control operations of the control terminal on the multimedia data to the controlled terminal, thereby realizing terminal interaction, allows the controlled terminal to monitor the playback progress of the multimedia data on the controlled terminal's display screen and send the obtained push information to the control terminal, so that the push information is displayed on the multimedia data displayed on the control terminal's display screen, which helps users to place orders quickly, shortens the purchase path, and does not interrupt the playback of the multimedia data on the control terminal when pushing push information, further improving the user experience.

[0095] In another possible implementation, refer to Figure 5 During the interaction between the controlled terminal and the controlling terminal, the terminal interaction method provided in this embodiment further includes:

[0096] S401: Receive push request sent by the control terminal.

[0097] S402: Determine push information about multimedia data based on the push request.

[0098] S403: Send push information to the control terminal so that the control terminal outputs the push information to the control terminal display screen, so that the push information is superimposed on the multimedia data for display.

[0099] The push information is the same as that in the previous embodiment, and will not be repeated here. In this embodiment, in order to prevent frequent push information to the control terminal and affect the user's interactive experience, the push information does not need to be actively pushed by the controlled terminal. It can be pushed after receiving a relevant request from the control terminal. For example, the push request can be generated by the control terminal when it detects a trigger operation on the push control. The push control is displayed on the control terminal screen for the user to select.

[0100] As can be seen, the terminal interaction method provided in this application, based on the realization of synchronizing multimedia data of the controlled terminal to the control terminal and synchronizing the control operations of the control terminal on the multimedia data to the controlled terminal, thereby realizing terminal interaction, receives push requests sent by the control terminal and sends the obtained push information corresponding to the push request to the control terminal, so that the control terminal displays the push information on its screen to help users place orders quickly, making the purchase path shorter. It does not require the controlled terminal to actively push push information, thus improving the user experience. Furthermore, when pushing push information, it does not interrupt the playback of multimedia data on the control terminal, further improving the user experience.

[0101] Please refer to Figure 6 , Figure 6 This application provides a flowchart illustrating a terminal interaction method applied to a control terminal, which may include the following steps S501 to S504:

[0102] S501: Receives multimedia data sent by the controlled terminal.

[0103] This step aims to receive multimedia data, which refers to the data information displayed on the screen of the controlled terminal, including but not limited to video and images.

[0104] In this embodiment, the method for acquiring multimedia data can be referred to in step S201, and will not be repeated here. The control terminal receives multimedia data sent by the controlled terminal through the first communication channel. The establishment of the first communication channel can be referred to the above description, and will not be repeated here.

[0105] S502: Obtain the trigger operation for multimedia data.

[0106] This step aims to trigger the acquisition of operations. After receiving multimedia data sent by the controlled terminal, it can be output to the display screen of the control terminal, realizing the synchronous display of multimedia data between the control terminal and the controlled terminal. Furthermore, during the display process, users can perform various processing on the multimedia data through touch screen operations, such as liking, posting comments, sending emoticons, voting, etc. At the same time, the control terminal will collect these user trigger operations on the multimedia data.

[0107] S503: Generate an operation instruction based on the trigger operation. The operation instruction includes the trigger position and operation information of the trigger operation on the control terminal display screen.

[0108] This step aims to generate operation instructions. After the control terminal collects the user's trigger operation on multimedia data, it will generate corresponding operation instructions based on the trigger operation. These operation instructions instruct the controlled terminal to simulate the corresponding trigger operation and display it on the controlled terminal's screen, thereby enabling synchronous display of operation events related to multimedia data between the control terminal and the controlled terminal.

[0109] S504: Send the operation command to the controlled terminal so that the controlled terminal can determine the display position of the triggered operation on the controlled terminal screen according to the trigger position, and output the operation information at the display position.

[0110] This step aims to send operation instructions to the controlled terminal so that the controlled terminal can execute the operation instructions. Specifically, after generating the corresponding operation instructions based on the trigger operation, they can be sent to the controlled terminal via a communication connection. Upon receiving the operation instructions, the controlled terminal can parse the instructions to obtain the display location and operation information, and simultaneously output the operation information to the display screen at the designated display location. This achieves synchronous display of operation events for multimedia data between the control terminal and the controlled terminal. In this embodiment, the operation instructions can be sent via a second communication channel. The establishment of the second communication channel is described above and will not be repeated in this embodiment.

[0111] As can be seen, the terminal interaction method provided in this application receives multimedia data sent by the controlled terminal, obtains trigger operations for the multimedia data, generates operation instructions based on the trigger operations, and sends the operation instructions to the controlled terminal. This allows the controlled terminal to determine the display position of the trigger operation on its screen based on the trigger position and output operation information at the display position. This achieves the synchronization of multimedia data from the controlled terminal to the control terminal and the synchronization of control operations from the control terminal to the controlled terminal for the multimedia data, thereby realizing dual-screen interaction. This not only makes the operation more convenient and faster but also allows for more diverse information interaction, effectively improving the user experience.

[0112] This application provides a terminal interaction method for a specific application scenario, using information interaction between a TV and a mobile phone as an example. The TV is the controlled terminal, and the mobile phone is the controlling terminal. Please refer to... Figure 7 , Figure 7 This application scenario diagram illustrates a terminal interaction method provided in an embodiment of this application. Its implementation process may include:

[0113] First, install a desktop application on the TV as the server, and install a corresponding desktop application on the mobile phone as the client.

[0114] Furthermore, once the TV is launched, the desktop application, acting as the server, will create two TCP (Transmission Control Protocol) connection channels (Socket connections) and bind them to two ports on the TV. Simultaneously, the desktop application on the mobile phone is launched. When the mobile phone and the TV are on the same network (such as connected to the same Wi-Fi), they can establish two communication channels through the two ports to conduct data communication.

[0115] Finally, after establishing the communication channels, one of the communication channels is a one-way channel, used to push the display screen (i.e., the aforementioned multimedia data) from the TV to the mobile phone for synchronous display; the other communication channel is a two-way channel, used to send the trigger operation of the mobile phone on the display screen to the TV for synchronous display, and can also receive other relevant data information about the display screen from the TV.

[0116] The implementation process of data transmission based on the first communication channel is as follows:

[0117] 1. Establish a non-blocking server on the TV using the NIO (New Input / Output) ServerSocketChannel class.

[0118] 2. The server creates a Socket and listens on port 6000, waiting for a connection.

[0119] 3. The mobile client establishes a connection with the TV via "TV IP + port 6000" and passes in the answer method, receiving byte stream data from the server through the onStringAvailable() callback function.

[0120] 4. The server acquires the byte stream data and prepares to output it to the client; the byte stream data is the content currently displayed on the server's screen, which can be obtained based on screen casting technology. The implementation process may include:

[0121] (1) Request screen recording permission and create an mMediaProjection object;

[0122] (2) Create a MediaCodec class to compress and encode multimedia data to implement screen recording function;

[0123] (3) In the loop, continuously obtain encodedData and send it out through Socket.

[0124] 5. The client decodes the received byte stream using MediaCodec and then displays the decoded data on the SurfaceView.

[0125] This completes the process of transmitting and displaying multimedia data from the server (TV) to the client (mobile).

[0126] II. The implementation process of data transmission based on the second communication channel is as follows:

[0127] 1. The mobile client establishes a connection with the TV via "TV IP + port 6059".

[0128] 2. After the server and client establish a connection, the socket instance is assigned to touchSocket.

[0129] 3. The client sends data to the server through the touchSocket instance to complete the interaction.

[0130] The data sent from the client to the server is a JSON-formatted string containing the coordinates of the triggered operation on the mobile screen.

[0131] 4. After receiving the corresponding data, the server first parses the data according to the JSON format, and then uses the InputManager class to simulate the corresponding trigger operation, such as clicking, swiping, dragging, etc.

[0132] This completes the process of transmitting and displaying multimedia data from the client (mobile device) to the server (TV device). Of course, the server can also use this communication channel to send information about the multimedia data to the client, enabling bidirectional communication via the second communication channel.

[0133] Furthermore, the TV, acting as a server, can connect to multiple mobile devices, such as... Figure 8 As shown, Figure 8 This is an application scenario diagram of another terminal interaction method provided in the embodiments of this application, which supports multiple mobile phones to operate the same TV terminal at the same time, thereby realizing multi-terminal interaction.

[0134] As can be seen, the terminal interaction method provided in this application embodiment realizes the synchronization of multimedia data from the TV to the mobile phone, and the synchronization of control operations on the multimedia data from the mobile phone to the TV, thereby realizing dual-screen interaction. This not only makes the operation more convenient and faster, but also enables more diverse information interaction, effectively improving the user experience.

[0135] This application provides a terminal interaction device.

[0136] Please refer to Figure 9 , Figure 9 This is a schematic diagram of a terminal interaction device provided in an embodiment of this application. The terminal interaction device is applied to a controlled terminal and may include:

[0137] The first transmitting module 1 is used to send multimedia data to the control terminal;

[0138] The first receiving module 2 is used to receive operation instructions fed back by the control terminal. The operation instructions are generated by the control terminal based on the trigger operation for multimedia data.

[0139] Module 3 is used to determine the trigger position and operation information of the trigger operation on the control terminal display screen according to the operation instruction;

[0140] The determination module 3 is also used to determine the display position of the trigger operation on the controlled terminal display screen based on the trigger position;

[0141] Output module 4 is used to output the operation information at the display location.

[0142] As can be seen, the terminal interaction device provided in this application embodiment sends multimedia data to the control terminal; receives operation instructions from the control terminal, the operation instructions are generated by the control terminal based on the trigger operation for the multimedia data, the trigger position and operation information of the trigger operation on the control terminal display screen are determined according to the operation instructions, the display position of the trigger operation on the controlled terminal display screen is determined according to the trigger position, and the operation information is output at the display position. This realizes the synchronization of multimedia data from the controlled terminal to the control terminal and the synchronization of control operations of the control terminal for the multimedia data to the controlled terminal, thereby realizing dual-screen interaction. Not only is the operation method more convenient and faster, but it also allows for more diverse information interaction, effectively improving the user experience.

[0143] In one embodiment of this application, a first communication channel and a second communication channel are established between the control terminal and the controlled terminal.

[0144] The first sending module 1 is also used to send multimedia data to the control terminal through the first communication channel.

[0145] The first receiving module 2 is also used to receive operation instructions fed back by the control terminal through the second communication channel.

[0146] In one embodiment of this application, the first sending module 1 is further configured to record the screen of the controlled terminal display to obtain multimedia data;

[0147] The first receiving module 2 is also used to send multimedia data to the control terminal.

[0148] In one embodiment of this application, the determining module 3 is further configured to determine the display position corresponding to the trigger position based on the coordinate mapping relationship, wherein the coordinate mapping relationship is the mapping relationship between the coordinate system of the control terminal display screen and the coordinate system of the controlled terminal display screen.

[0149] In one embodiment of this application, the terminal interaction device may further include:

[0150] The monitoring module is used to monitor the playback progress of multimedia data on the controlled terminal display screen.

[0151] The first sending module 1 is also used to send push information about multimedia data to the control terminal when the playback progress reaches a preset playback node, so that the control terminal outputs the push information to the control terminal display screen, so that the push information is superimposed on the multimedia data for display.

[0152] In one embodiment of this application, the first receiving module 2 is further configured to receive a push request sent by the control terminal.

[0153] Module 3 is also used to determine push information about multimedia data based on the push request.

[0154] The first sending module 1 is also used to send push information to the control terminal, so that the control terminal outputs the push information to the control terminal display screen, so that the push information is superimposed on the multimedia data for display.

[0155] For a description of the apparatus provided in the embodiments of this application, please refer to the above method embodiments; the embodiments of this application will not be repeated here.

[0156] This application provides another terminal interaction device.

[0157] Please refer to Figure 10 , Figure 10 This is a schematic diagram of another terminal interaction device provided in an embodiment of this application. The terminal interaction device is applied to a control terminal and may include:

[0158] The second receiving module 5 is used to receive multimedia data sent by the controlled terminal.

[0159] Module 6 is used to acquire trigger operations for multimedia data.

[0160] The generation module 7 is used to generate operation instructions based on the trigger operation. The operation instructions include the trigger position and operation information of the trigger operation on the control terminal display screen.

[0161] The second sending module 8 is used to send operation instructions to the controlled terminal so that the controlled terminal can determine the display position of the triggered operation on the controlled terminal display screen according to the trigger position, and output the operation information at the display position.

[0162] As can be seen, the terminal interaction device provided in this application embodiment sends multimedia data to the control terminal; receives operation instructions from the control terminal, the operation instructions are generated by the control terminal based on the trigger operation for the multimedia data, the trigger position and operation information of the trigger operation on the control terminal display screen are determined according to the operation instructions, the display position of the trigger operation on the controlled terminal display screen is determined according to the trigger position, and the operation information is output at the display position. This realizes the synchronization of multimedia data from the controlled terminal to the control terminal and the synchronization of control operations of the control terminal for the multimedia data to the controlled terminal, thereby realizing dual-screen interaction. Not only is the operation method more convenient and faster, but it also allows for more diverse information interaction, effectively improving the user experience.

[0163] For a description of the apparatus provided in the embodiments of this application, please refer to the above method embodiments; the embodiments of this application will not be repeated here.

[0164] This application provides an electronic device.

[0165] Please refer to Figure 11 , Figure 11 This application provides a schematic diagram of the structure of an electronic device, which may include:

[0166] Memory, used to store computer programs;

[0167] A processor, used to execute computer programs, can implement the steps of any of the terminal interaction methods described above.

[0168] like Figure 11 The diagram shows the structural composition of an electronic device, which may include a processor 10, a memory 11, a communication interface 12, and a communication bus 13. The processor 10, memory 11, and communication interface 12 all communicate with each other through the communication bus 13.

[0169] In this embodiment, the processor 10 may be a central processing unit (CPU), an application-specific integrated circuit, a digital signal processor, a field-programmable gate array, or other programmable logic devices. The processor 10 may call programs stored in the memory 11; specifically, the processor 10 may execute operations in the embodiments of the terminal interaction method.

[0170] The memory 11 is used to store one or more programs. The programs may include program code, which includes computer operation instructions. In this embodiment, the memory 11 stores at least a program for implementing the following functions:

[0171] Send multimedia data to the control terminal; receive operation instructions from the control terminal. The operation instructions are generated by the control terminal based on the trigger operation for the multimedia data. Based on the operation instructions, determine the trigger position and operation information of the trigger operation on the control terminal display screen. Based on the trigger position, determine the display position of the trigger operation on the controlled terminal display screen and output the operation information at the display position.

[0172] Alternatively, at least a program for implementing the following functions is stored:

[0173] The system receives multimedia data sent by the controlled terminal, obtains the trigger operation for the multimedia data, generates an operation instruction based on the trigger operation, and sends the operation instruction to the controlled terminal so that the controlled terminal can determine the display position of the trigger operation on the controlled terminal screen according to the trigger position, and outputs the operation information at the display position.

[0174] In one possible implementation, the memory 11 may include a program storage area and a data storage area, wherein the program storage area may store the operating system and applications required for at least one function; and the data storage area may store data created during use.

[0175] In addition, memory 11 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device or other volatile solid-state storage device.

[0176] Communication interface 12 can be an interface for the communication module, used to connect with other devices or systems.

[0177] Of course, it should be noted that, Figure 11 The structure shown does not constitute a limitation on the electronic device in the embodiments of this application. In practical applications, the electronic device may include more than Figure 11 More or fewer components as shown, or combinations of certain components.

[0178] This application provides a computer-readable storage medium storing a computer program, which, when executed by a processor, can implement the steps of any of the terminal interaction methods described above.

[0179] The computer-readable storage medium may include various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0180] For a description of the computer-readable storage medium provided in the embodiments of this application, please refer to the above method embodiments; the embodiments of this application will not be repeated here.

[0181] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to in the method section.

[0182] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0183] The technical solutions provided in this application have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are only for the purpose of helping to understand the methods and core ideas of this application. It should be noted that those skilled in the art can make several improvements and modifications to this application without departing from the principles of this application, and these improvements and modifications also fall within the protection scope of this application.

Claims

1. A terminal interaction method, characterized in that, Applied to controlled terminals, including: Send multimedia data to the control terminal; The system receives operation instructions fed back from the control terminal, which are generated by the control terminal based on a trigger operation targeting the multimedia data. The triggering position and operation information of the triggering operation on the control terminal display screen are determined according to the operation instructions; The display position of the trigger operation on the controlled terminal screen is determined according to the trigger position, and the operation information is output at the display position. The control terminal and the controlled terminal establish a first communication channel and a second communication channel. Sending multimedia data to the control terminal includes: The multimedia data is sent to the control terminal through the first communication channel; The receiving of operation instructions from the control terminal includes: The operation instructions are received from the control terminal through the second communication channel; The method further includes: The control terminal receives a push request, which is generated by the control terminal when it detects a trigger operation on the push control, and the push control is displayed on the control terminal's screen. Determine push information about the multimedia data based on the push request; The push information is sent to the control terminal, so that the control terminal outputs the push information to the control terminal display screen, and the push information is superimposed on the multimedia data for display.

2. The method according to claim 1, characterized in that, Sending multimedia data to the control terminal includes: The controlled terminal display screen is recorded to obtain the multimedia data; The multimedia data is sent to the control terminal.

3. The method according to claim 1, characterized in that, Determining the display position of the trigger operation on the controlled terminal screen based on the trigger position includes: The display position corresponding to the trigger position is determined based on the coordinate mapping relationship, which is the mapping relationship between the coordinate system of the control terminal display screen and the coordinate system of the controlled terminal display screen.

4. The method according to any one of claims 1 to 3, characterized in that, Also includes: Monitor the playback progress of the multimedia data on the controlled terminal display screen; When the playback progress reaches a preset playback node, push information about the multimedia data is sent to the control terminal, so that the control terminal outputs the push information to the control terminal display screen, and the push information is superimposed on the multimedia data for display.

5. A terminal interaction method, characterized in that, Applications to control terminals include: Receive multimedia data sent by the controlled terminal; Obtain the trigger operation for the multimedia data; An operation instruction is generated based on the trigger operation, and the operation instruction includes the trigger position and operation information of the trigger operation on the control terminal display screen. The operation instruction is sent to the controlled terminal, so that the controlled terminal determines the display position of the trigger operation on the controlled terminal display screen according to the trigger position, and outputs the operation information at the display position; The control terminal and the controlled terminal establish a first communication channel and a second communication channel. The receiving of multimedia data sent by the controlled terminal includes: Multimedia data sent by the controlled terminal is received through the first communication channel. Sending the operation command to the controlled terminal includes: The operation command is sent to the controlled terminal through the second communication channel; The method further includes: A push request is sent to the controlled terminal so that the controlled terminal can determine push information about the multimedia data based on the push request. The push request is generated by the control terminal when it detects a trigger operation on the push control. The push control is displayed on the control terminal's screen. Receive the push information sent by the controlled terminal; The push information is output to the control terminal display screen so that the push information is superimposed on the multimedia data for display.

6. A terminal interaction system, characterized in that, It includes a controlled terminal and a control terminal, wherein the control terminal and the controlled terminal establish a first communication channel and a second communication channel; wherein, The control terminal is configured to receive multimedia data sent by the controlled terminal through the first communication channel, obtain a trigger operation for the multimedia data, generate an operation instruction based on the trigger operation, and send the operation instruction to the controlled terminal through the second communication channel. The controlled terminal is configured to send the multimedia data to the control terminal through the first communication channel, and determine the trigger position and operation information of the trigger operation on the display screen of the control terminal according to the operation instruction, determine the display position of the trigger operation on the display screen of the controlled terminal according to the trigger position, and output the operation information at the display position; The controlled terminal is further configured to receive a push request sent by the control terminal, so as to determine push information about the multimedia data according to the push request. The push request is generated by the control terminal when it detects a trigger operation on the push control. The push control is displayed on the display screen of the control terminal. The control terminal is also configured to receive the push information sent by the controlled terminal and output the push information to the control terminal display screen, so that the push information is superimposed on the multimedia data for display.

7. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor, when executing the computer program, implements the steps of the terminal interaction method as described in any one of claims 1 to 4 or the steps of the terminal interaction method as described in claim 5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the terminal interaction method as described in any one of claims 1 to 4 or the steps of the terminal interaction method as described in claim 5.

Images