Content service relay interruption method and terminal device

Abstract

This embodiment provides a content service relay interruption method and terminal device, wherein the relay terminal device receives a service relay instruction from an initiating relay terminal device, the service relay instruction carries content service relay data, and the relay terminal device can execute the content service relay operation corresponding to the content service relay data. The relay terminal device then receives the relay interruption instruction, and according to the relay interruption instruction, the relay terminal device does not execute the content service relay operation corresponding to the content service relay data. In this way, even if the relay terminal device has received the service relay instruction, it can also not execute the relay operation according to the relay interruption instruction, that is, if the user mistakenly triggers the content service relay process, the content service relay is interrupted in time, thereby improving the user experience.

Description

[0001] This application claims priority to the Chinese patent application filed with the China Patent Office on August 5, 2021, with application number 202110896120.4, and invention name “Content service relay method and terminal device of terminal device”, and claims priority to the Chinese patent application filed with the China Patent Office on September 18, 2021, with application number 202111097923.X, and invention name “Display device”, all contents of which are incorporated by reference in this application. Technical Field

[0002] The present application relates to the technical field of terminal equipment, and in particular to a content service relay interruption method and a terminal equipment. Background Art

[0003] With the development of smart terminal products and the improvement of people's family living standards, more and more families have multiple home smart terminal devices. Home smart devices can relay content services to each other. For example, the video playback service on the smart TV can be relayed to the display screen of the refrigerator, and the video playback service on the refrigerator display screen can also be relayed to the smart TV.

[0004] The existing screen projection function is a cross-terminal video control function, which requires the projection end and the projection end to support specific protocols, such as DLNA, Miracast, etc. There are two main screen projection methods. One is represented by DLNA, which is mostly used for online video playback. After the user selects the screen projection function on the playback page, the projection end finds the projection end that supports this protocol in the local area network according to the predetermined protocol. By directly passing the video playback address to the projection end through the local area network, the projection end plays the video according to the video playback address. At this time, the projection end no longer performs video data analysis and can sleep or shut down without affecting the playback of the projection end. The other is represented by Miracast, which belongs to the operation of screen mirroring, that is, the projection end still displays the image after executing the screen projection, and the displayed image is mirrored to the projection end through the local area network. The sleep or shutdown of the projection end will affect the playback of the projection end.

[0005] Therefore, the current screen projection function can only realize the cross-terminal control function of the video, and cannot realize the content service relay operation other than the video service. Content service relay applications have emerged. The content service relay process is usually that the user selects the target device, i.e. the relay terminal device, on the initiating relay terminal device. After confirming the triggering of the relay, the initiating relay terminal device sends a service relay instruction to the relay terminal device. The relay terminal device then performs the relay operation according to the service relay instruction.

[0006] However, in actual usage scenarios, the relay process may be triggered by mistake, and at this time the relay-initiating terminal device has already sent a service relay instruction to the relay terminal device, so the content service relay process cannot be interrupted, resulting in a poor user experience. Summary of the invention

[0007] The present application provides a content service relay interruption method and terminal device, which are used to solve the problem that in current actual usage scenarios, the relay process may be triggered by mistake, and at this time the initiating relay terminal device has already sent a service relay instruction to the relay terminal device, so the content service relay process cannot be interrupted, resulting in a poor user experience.

[0008] In a first aspect, this embodiment provides a terminal device, wherein the terminal device is a relay terminal device, and the relay terminal device is a target device of a content service relay, including:

[0009] The controller is configured as:

[0010] Receiving a service relay instruction from a relay initiating terminal device, wherein the service relay instruction carries content service relay data, and the relay terminal device can execute a content service relay operation corresponding to the content service relay data;

[0011] A relay interruption instruction is received, and according to the relay interruption instruction, a content service relay operation corresponding to the content service relay data is not performed.

[0012] In a second aspect, this embodiment provides a terminal device, where the terminal device is a relay initiating terminal device, including:

[0013] The controller is configured as:

[0014] Sending a service relay instruction to a relay terminal device, wherein the service relay instruction carries content service relay data, and the relay terminal device can execute a content service relay operation corresponding to the content service relay data;

[0015] A relay interruption instruction is sent to the relay terminal device, so that the relay terminal device does not perform the content service relay operation corresponding to the content service relay data according to the relay interruption instruction.

[0016] In a third aspect, this embodiment provides a content service relay interruption method, which is applied to a relay terminal device, where the relay terminal device is a target device of the content service relay, including:

[0017] Receiving a service relay instruction from a relay initiating terminal device, wherein the service relay instruction carries content service relay data, and the relay terminal device can execute a content service relay operation corresponding to the content service relay data;

[0018] A relay interruption instruction is received, and according to the relay interruption instruction, a content service relay operation corresponding to the content service relay data is not performed.

[0019] In a fourth aspect, this embodiment provides a content service relay interruption method, which is applied to a relay initiating terminal device, including:

[0020] Sending a service relay instruction to a relay terminal device, wherein the service relay instruction carries content service relay data, and the relay terminal device can execute a content service relay operation corresponding to the content service relay data;

[0021] A relay interruption instruction is sent to the relay terminal device, so that the relay terminal device does not perform the content service relay operation corresponding to the content service relay data according to the relay interruption instruction.

[0022] This embodiment provides a content service relay interruption method and terminal device. The relay terminal device receives a service relay instruction from an initiating relay terminal device. The service relay instruction carries content service relay data. The relay terminal device can execute the content service relay operation corresponding to the content service relay data. The relay terminal device then receives the relay interruption instruction. According to the relay interruption instruction, the relay terminal device does not execute the content service relay operation corresponding to the content service relay data. In this way, even if the relay terminal device has received the service relay instruction, it can also not execute the relay operation according to the relay interruption instruction, that is, if the user mistakenly triggers the content service relay process, the content service relay is interrupted in time, thereby improving the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0024] Figure 1 shows a usage scenario of a display device according to some embodiments;

[0025] Figure 2 shows a hardware configuration block diagram of a control device 100 according to some embodiments;

[0026] Figure 3 shows a hardware configuration block diagram of a display device 200 according to some embodiments;

[0027] Figure 4shows a software configuration diagram in a display device 200 according to some embodiments;

[0028] Figure 5 shows an icon control interface display diagram of an application in a display device 200 according to some embodiments;

[0029] Figure 6 A schematic diagram of a terminal device content service relay system framework according to some embodiments is exemplarily shown;

[0030] Figure 7 A schematic diagram of a user interface of a relay initiating terminal device according to some embodiments is exemplarily shown;

[0031] Figure 8 A schematic diagram of a user interface of another relay initiating terminal device according to some embodiments is exemplarily shown;

[0032] Fig. 9 A schematic diagram of a user interface of another relay initiating terminal device according to some embodiments is exemplarily shown;

[0033] Fig.10 A schematic diagram of a user interface of another relay initiating terminal device according to some embodiments is exemplarily shown;

[0034] Fig.11 A schematic diagram of a user interface of a relay terminal device according to some embodiments is exemplarily shown;

[0035] Fig.12 A schematic diagram of a user interface of another relay initiating terminal device according to some embodiments is exemplarily shown;

[0036] Fig.13 A schematic diagram of a user interface of another relay terminal device according to some embodiments is exemplarily shown;

[0037] Fig.14 A schematic diagram of a service relay business process according to some embodiments is exemplarily shown;

[0038] Fig.15 A schematic diagram of a terminal device content service relay application scenario according to some embodiments is exemplarily shown;

[0039] Fig.16 illustratively showing a content service relay device matching dimension scene graph according to some embodiments;

[0040] Fig.17 The following is a schematic diagram showing a timing diagram of content service relay signaling processing according to some embodiments;

[0041] Fig.18A schematic diagram of another content service relay signaling processing timing according to some embodiments is exemplarily shown;

[0042] Fig.19 Another content service relay device matching dimension scene graph according to some embodiments is exemplarily shown;

[0043] Fig. 20 Another content service relay device matching dimension scene graph according to some embodiments is exemplarily shown;

[0044] Fig.21 Another content service relay device matching dimension scene graph according to some embodiments is exemplarily shown;

[0045] Fig. 22 A schematic diagram of a user interface of another relay initiating terminal device according to some embodiments is exemplarily shown;

[0046] Fig.23 A schematic diagram of another content service relay signaling processing timing according to some embodiments is exemplarily shown;

[0047] Fig.24 A schematic diagram of a user interface of another relay initiating terminal device according to some embodiments is exemplarily shown;

[0048] Fig.25 A schematic diagram of a user interface of another relay initiating terminal device according to some embodiments is exemplarily shown;

[0049] Fig.26 A schematic diagram of a user interface of another relay initiating terminal device according to some embodiments is exemplarily shown;

[0050] Fig. 27 A schematic diagram of a business process for displaying a relay button control on a user interface of a relay-initiating terminal device according to some embodiments is exemplified;

[0051] Fig.28 A schematic diagram of a business process for implementing service relay interruption according to some embodiments is exemplified;

[0052] Fig.29 The signaling timing diagram of relay return according to some embodiments is exemplarily shown;

[0053] Fig.30 A flowchart of a method for interrupting a content service relay according to some embodiments is exemplified;

[0054] Fig.31 A signaling diagram of a method for interrupting a content service relay according to some embodiments is exemplified. DETAILED DESCRIPTION

[0055] In order to make the purpose and implementation method of the present application clearer, the exemplary implementation method of the present application will be clearly and completely described below in conjunction with the drawings in the exemplary embodiments of the present application. Obviously, the described exemplary embodiments are only part of the embodiments of the present application, rather than all the embodiments.

[0056] It should be noted that the brief description of terms in this application is only for the convenience of understanding the embodiments described below, and is not intended to limit the embodiments of this application. Unless otherwise specified, these terms should be understood according to their common and usual meanings.

[0057] The terms "first", "second", "third", etc. in the specification and claims of this application and the above drawings are used to distinguish similar or similar objects or entities, and do not necessarily mean to limit a specific order or sequence, unless otherwise noted. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances.

[0058] The terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover but not exclude inclusion, for example, a product or device comprising a list of components is not necessarily limited to all the components expressly listed but may include other components not expressly listed or inherent to such product or device.

[0059] The display device provided in the embodiments of the present application may have various implementation forms, for example, it may be a television, a smart television, a laser projection device, a monitor, an electronic bulletin board, an electronic table, etc. Figure 1 and Figure 2 This is a specific implementation of the display device of the present application.

[0060] Figure 1 FIG. 1 is a schematic diagram of an operation scenario between a display device and a control device according to an embodiment. Figure 1 As shown, the user can operate the display device 200 through the smart device 300 or the control apparatus 100 .

[0061] In some embodiments, the control device 100 may be a remote controller, and the communication between the remote controller and the display device includes infrared protocol communication or Bluetooth protocol communication, and at least one of other short-range communication methods, to control the display device 200 wirelessly or wired. The user may control the display device 200 by inputting user commands through at least one of buttons on the remote controller, voice input, and control panel input.

[0062] In some embodiments, a smart device 300 (such as a mobile terminal, a tablet computer, a computer, a laptop computer, etc.) may also be used to control the display device 200. For example, the display device 200 is controlled using an application running on the smart device.

[0063] In some embodiments, the smart device 300 may also be other terminals that can communicate, such as air conditioners, refrigerators, smart central controls, smart access control and other devices.

[0064] In some embodiments, the display device may not use the above-mentioned smart device or control device to receive instructions, but may receive user control through touch or gestures.

[0065] In some embodiments, the display device 200 can also be controlled in a manner other than the control device 100 and the smart device 300. For example, the user's voice command control can be directly received through a module for obtaining voice commands configured inside the display device 200, or the user's voice command control can be received through a voice control device set outside the display device 200.

[0066] In some embodiments, the display device 200 also communicates data with the server 400. The display device 200 may be allowed to communicate via a local area network (LAN), a wireless local area network (WLAN), and other networks. The server 400 may provide various content and interactions to the display device 200. The server 400 may be one cluster or multiple clusters, and may include one or more types of servers.

[0067] Figure 2 Schematically shows a block diagram of a configuration of the control device 100 according to an exemplary embodiment. Figure 2 As shown, the control device 100 includes a controller 110, a communication interface 130, a user input / output interface 140, a memory, and a power supply. The control device 100 can receive input operation instructions from the user, and convert the operation instructions into instructions that the display device 200 can recognize and respond to, playing the role of an interactive intermediary between the user and the display device 200.

[0068] like Figure 3 The display device 200 includes at least one of a tuner and demodulator 210, a communicator 220, a detector 230, an external device interface 240, a controller 250, a display 260, an audio output interface 270, a memory, a power supply, and a user interface.

[0069] In some embodiments, the user may input a user command in a graphical user interface (GUI) displayed on the display 260, and the user input interface receives the user input command through the graphical user interface (GUI). Alternatively, the user may input a user command by inputting a specific sound or gesture, and the user input interface receives the user input command by recognizing the sound or gesture through a sensor.

[0070] "User interface" is the medium interface for interaction and information exchange between application programs or operating systems and users. It realizes the conversion between the internal form of information and the form acceptable to users. The most common form of user interface is the Graphical User Interface (GUI), which refers to the user interface related to computer operation displayed in a graphical way. It can be an interface element such as an icon, window, control, etc. displayed on the display screen of an electronic device, where the control can include icons, buttons, menus, tabs, text boxes, dialog boxes, status bars, navigation bars, widgets, etc.

[0071] See also Figure 4 In some embodiments, the system is divided into four layers, from top to bottom, namely, the application layer (Applications) layer (referred to as "application layer"), the application framework layer (Application Framework) layer (referred to as "framework layer"), the Android runtime (Android runtime) and system library layer (referred to as "system runtime library layer"), and the kernel layer.

[0072] In some embodiments, at least one application is running in the application layer, and these applications can be window programs, system settings programs, clock programs, etc. provided by the operating system, or applications developed by third-party developers. In specific implementations, the application packages in the application layer are not limited to the above examples.

[0073] like Figure 4As shown, in some embodiments of the present application, the application framework layer includes managers, content providers, etc., wherein the manager includes at least one of the following modules: an activity manager for interacting with all activities running in the system; a location manager for providing system services or applications with access to system location services; a package manager for retrieving various information related to application packages currently installed on the device; a notification manager for controlling the display and clearing of notification messages; and a window manager for managing icons, windows, toolbars, wallpapers, and desktop widgets on the user interface.

[0074] In some embodiments, the activity manager is used to manage the life cycle of each application and the common navigation back function, such as controlling the exit, opening, and back of the application. The window manager is used to manage all window programs, such as obtaining the display screen size, determining whether there is a status bar, locking the screen, capturing the screen, and controlling the display window changes (for example, reducing the display window, shaking the display, distorting the display, etc.).

[0075] In some embodiments, after the display device is started, the interface of the preset video-on-demand program can be directly entered. The video-on-demand program is the default system program of the system, and its interface can be as follows: Figure 5 As shown in , at least a navigation bar 510 and a content display area located below the navigation bar 510 are included, and the content displayed in the content display area changes with the change of the selected control in the navigation bar. The program in the application layer can be integrated into the video-on-demand program and displayed through a control in the navigation bar, or further displayed after the application control in the navigation bar is selected.

[0076] In some embodiments, after the display device is started, it can directly enter the display interface of the last selected signal source, or the signal source selection interface, where the signal source can be a preset video on demand program, or at least one of an HDMI interface, a live TV interface, etc. After the user selects a different signal source, the display can display the content obtained from different signal sources.

[0077] With the development of smart terminal products and the improvement of people's family living standards, more and more families have multiple home smart terminal devices. Home smart devices can relay content services to each other. For example, the video playback service on the smart TV can be relayed to the display screen of the refrigerator, and the video playback service on the display screen of the refrigerator can also be relayed to the smart TV. This application provides a terminal device content service relay system, such as Figure 6 The schematic diagram of the terminal device content service relay system framework is shown. The terminal devices 100 can transfer data through the server 200, or directly send and receive data between the terminal devices. In the content service relay system of the present application, all terminal devices 100 can be used as both relay initiating terminal devices and relay terminal devices. The relay initiating terminal device is a terminal device that initiates the content service relay process, and the relay terminal device is the target terminal device of the content service relay process. The server of the embodiment of the present application can be an AIOT (Artificial Intelligence & Internet of Things) server.

[0078] The content service relay business scenarios of the embodiments of the present application include two types: one is to trigger the relay service through a button control, and the other is a voice wake-up relay service.

[0079] In some embodiments, Figure 7 The user interface diagram of the relay initiating terminal device shown in the figure shows that, based on the business application currently running in the foreground of the relay initiating terminal device, if there is a relay terminal device, a relay button control is displayed on the user interface of the relay initiating terminal device. The following will specifically explain how to determine whether there is a relay terminal device. It should be noted that to determine whether there is a relay terminal device here, it is not necessary to query all relayable terminal devices.

[0080] Figure 7 The business scenario in the example shown is a business scenario in which a relay service is triggered by a button control. After the user clicks the relay button in the user interface, the relay-initiating terminal device receives the user operation and generates a relay instruction. The generated relay instruction carries the device information of the relay-initiating terminal device and the business information of the current business of the relay-initiating terminal device. According to the relay instruction, the relay-initiating terminal device will query the business of the current relay-initiating terminal device, and then query the relayable terminal device based on the business information of the current business. In the voice-wake-up relay service business scenario, the user can directly input voice, and the relay-initiating terminal device receives the voice input by the user and can also generate a relay instruction. The relay-initiating terminal device queries the relayable terminal device based on the generated relay instruction, such as Figure 8The user interface shown in the figure displays a prompt "Looking for executable devices". At this time, the service currently being run by the relay-initiating terminal device can be suspended or continued. After the relay-initiating terminal device is found, the information of the relay-initiating terminal device is displayed on the relay-initiating terminal device. In some embodiments, if multiple relay-initiating terminal devices are found, a list of relay-initiating devices is displayed on the user interface, such as Fig. 9 The user interface shown in Fig. 9 The user interface shown also displays an option bar of "delay 5 seconds to play" and an option bar of "continue playing on original device".

[0081] When the user selects a relayable terminal device in the relayable device list, for example, selects "bedroom TV", and turns on the switch of the "delay 5 seconds to play" option bar at the same time, Fig.10 As shown, a prompt "will be pushed to the bedroom TV" and a countdown dialog box are displayed on the user interface of the relay initiating terminal device. The countdown dialog box includes a return button, and the user can cancel the relay before the 5-second countdown by clicking the return button.

[0082] In addition, if Fig.11 The user interface of the relay terminal device shown, that is, the user interface of the device being pushed, "bedroom TV", also displays the prompt "about to receive push from XX device" and a countdown dialog box. Among them, XX device is the terminal device that initiates the relay. The countdown dialog box displayed on the user interface of the relay terminal device includes a return button and an OK button. If the user clicks the return button on the user interface of the relay terminal device, the relay is canceled. If the user clicks the OK button on the user interface of the relay terminal device, the relay is confirmed.

[0083] After the relay is determined, Fig.12 As shown, a prompt "Relayed to bedroom TV" is displayed on the user interface of the relay-initiating terminal device, and the user interface of the relay-initiating terminal device jumps back to the interface before the relay service content is performed. Fig.13 As shown, a prompt "Content of XX device has been relayed" is displayed on the user interface of the relay terminal device, and at the same time, the user interface of the relay terminal device starts to relay service content, such as playing a video.

[0084] In the service relay business scenario where a button control is used to trigger a relay service, such as Fig.14 As shown, the business process of the above embodiment specifically includes:

[0085] A1: The business application of the relay-initiating terminal device registers the relay capability of the business application with the unified relay service application, and the unified relay service application controls the scene and data transmission during the relay process. After the relay-initiating terminal device system enters the relay business scene (entering the relay business scene can be the user interface entering a specific page, such as the video playback application entering the video playback page from the main page. A relay button control is set in the video playback page.) After the user clicks the relay button in the user interface, the relay-initiating terminal device receives the user operation and generates a relay instruction. The unified relay service application can determine the current relay business scene based on the operation input by the user.

[0086] A2: After receiving the relay instruction, the unified relay service application carries the business information (the information of the business application currently running in the foreground of the relay-initiating terminal device) to the AIOT server to query the current account (which can be the account that the user has registered on the relay service platform in advance, and then the user binds the terminal device under the same account) for the relay terminal device that can relay the current business.

[0087] A3: The AIOT server returns a list of relay terminal devices under the current account that support the current business relay based on the status of the current business and all devices under the current account (only online devices are returned, not offline devices).

[0088] The process of forming the relay terminal device list is as follows: the AIOT server stores the device capabilities of all terminal devices in advance. When the AIOT server receives the business information of the current business that initiates the relay terminal device, it traverses the stored device capabilities and finds the device capabilities that match the business information of the current business. If they match, they are determined to be relay-capable terminal devices. List all relay-capable terminal devices in a list, which is the relay terminal device list. If there is only one relay-capable terminal device, there is only one terminal device listed in the relay terminal device list. If there are multiple relay-capable terminal devices, there are also multiple terminal devices listed in the relay terminal device list.

[0089] A4: After receiving the returned data (including device ID, device name and other data), the unified relay service application displays a list of relay terminal devices based on the returned data for the user to select.

[0090] A5: After the user selects a device, the device information of the selected device will be transmitted to the unified relay service application. (The main device information transmitted is the device ID, which is used to indicate the relay terminal device selected by the user)

[0091] A6: The unified relay service collects the business information of the current business and the device information of the selected device, assembles all the information, and passes it to the AIOT server.

[0092] A7: After receiving the assembled information, the AIOT server parses the information and then transmits the business information (content information that needs to be relayed) to the target device according to the device information, that is, to the relay terminal device selected by the user.

[0093] A8: After receiving the relay content information, the relay terminal device performs a corresponding relay action according to the content information, wherein the initiator of the relay action is the unified relay service application of the relay terminal device, and the executor of the relay action is an application with corresponding functions.

[0094] A9: After the relay terminal device executes the relay action, the unified relay service application of the relay terminal device will feed back the relay execution result to the initiating relay terminal device through the AIOT server.

[0095] A10: The relay initiating terminal device receives the relay execution result and displays different relay states to the user according to the relay execution result. For example, if the relay terminal device successfully executes the relay action, the relay success state is displayed. If the relay terminal device has not successfully executed the relay action but is waiting for the relay countdown, the relay countdown start state is displayed.

[0096] In the business scenario of voice wake-up relay service, such as Fig.14 As shown, the business process of the above embodiment specifically includes:

[0097] B1: This step is the speech recognition process. After the speech application collects the speech signal input by the user, it sends the speech signal to the speech server, which performs semantic analysis on the speech signal, that is, recognizes the user's intention from the speech signal.

[0098] B2: After the voice server recognizes from the voice signal that the user intends to provide relay service, it calls the AIOT server to query the relayable terminal devices (when calling the AIOT server, the voice server sends a call request to the AIOT server, and the call request carries the service information for querying the relayable terminal devices).

[0099] B3: After receiving the call request, the AIOT server queries the relay terminal device that can relay the current business under the current account according to the business information, and feeds back the device information of the queried relay terminal device to the voice server.

[0100] It should be noted that the voice server can only be used to identify the user's intention from the voice signal, and then feed the user's intention back to the unified service relay application. The unified service relay application then interacts with the AIOT server. Some embodiments of the present application forward data through the voice server, which can shorten the link and improve the system response speed.

[0101] B4: The voice server returns the device information queried from the AIOT server to the voice application.

[0102] B5: The voice application calls the interface of the unified relay service application and transmits the voice recognition information and the acquired device information to the unified relay service application. The voice recognition information may include the relay service instruction, and the device information includes the device ID.

[0103] B6: After receiving the device information of the relay terminal device, the unified relay service application transmits the above information (including business information, device information, relay service instructions, etc.) to the AIOT server in combination with the business information. The relay service instructions may carry the business information.

[0104] B7: The AIOT server sends the business information and relay instructions to the relay terminal device corresponding to the device information.

[0105] B8: After receiving the relay service instruction, the unified relay service application of the relay terminal device executes the corresponding relay action according to the service information carried in the relay service instruction, wherein the relay action is consistent with the content of the current service application of the initiating relay terminal device.

[0106] B9: After the relay terminal device executes the relay action, the unified relay service application of the relay terminal device will feed back the relay execution result to the initiating relay terminal device through the AIOT server.

[0107] B10: The relay initiating terminal device receives the relay execution result and displays different relay states to the user according to the relay execution result. For example, if the relay terminal device successfully executes the relay action, the relay success state is displayed. If the relay terminal device has not successfully executed the relay action but is waiting for the relay countdown, the relay countdown start state is displayed.

[0108] The content service relay process is usually that the user selects the target device, i.e. the relay terminal device, on the relay initiating terminal device, and after determining to trigger the relay, the relay initiating terminal device sends a service relay instruction to the relay terminal device. The relay terminal device then performs the relay operation according to the service relay instruction.

[0109] Before the user selects the relay terminal device on the initiating relay terminal device, the initiating relay terminal device and the relay terminal device need to be matched. The device matching is based on the service capabilities of each terminal device. Here, the service capability corresponds to the service parameters of the application installed on the terminal device. For example, if the display device is installed with a video playback application, the service capability it has is video playback capability. If the speaker is installed with an audio playback application, the service capability it has is audio playback capability. If the tablet is installed with a video call application, the service capability it has is video call capability.

[0110] Each terminal device may report the service capability of the local device to the server, or may directly send the report to other terminal devices. Some embodiments of the present application take the terminal device reporting the service capability of the local device to the server as an example to elaborate on the service capability reporting of the terminal device.

[0111] The terminal device can report business capabilities to the server. The server can be an AIOT (Artificial Intelligence & Internet of Things) system or a capability management system. When the terminal device system starts, it can automatically send a business capability registration request to the server. The capability registration request carries the business capabilities of the terminal device, thereby realizing the reporting of business capabilities. It can also be that the terminal devices report business capabilities to each other. The process of terminal devices reporting business capabilities to each other is not elaborated in detail in this application.

[0112] For example, applications (A1, A2...An), (B1, B2...Bn) and applications (C1, C2...Cn) are installed on terminal devices A, B and C respectively. After the service capability reporting process, the service capability list (a1, a2...an) of terminal device A, the service capability list (b1, b2...bn) of terminal device B and the service capability list (c1, c2...cn) of terminal device C are saved on the server. Among them, the service capabilities (a1, a2...an) in the service capability list correspond to applications (A1, A2...An), the service capabilities (b1, b2...bn) correspond to applications (B1, B2...Bn), and the service capabilities (c1, c2...cn) correspond to applications (C1, C2...Cn).

[0113] In some embodiments, when the A1 application on the terminal device A is running in the foreground, the business capability that matches the application capability a1 corresponding to A1 is searched according to the business parameters of the A1 application. If other business capabilities that match the business capability a1 are found, the relay terminal device is matched accordingly. If no other business capabilities that match the business capability a1 are found, the relay terminal device is not matched. For example, when the business capabilities a1 match b1, the terminal device B is determined to be the relay terminal device. When the business capabilities a1 match c2, the terminal device C is determined to be the relay terminal device.

[0114] For example, if the terminal device is running a video playback application A1, it can be determined that the terminal device A is performing the video playback function, and the video playback capability can be marked as a1 in the device capability list of the terminal device A. When determining the device that can be relayed, it is found that b1 matches a1, that is, b1 indicates that the terminal device B can play the video, and B is used as the relay device. For another example, if the terminal device is running a video playback application A1, it can be determined that the terminal device A is performing the video playback function, and the video playback application A1 corresponds to the first authorization information, and the video playback capability that can play the video corresponding to the first authorization information can be marked as a1 in the device capability list of the terminal device A. When determining the device that can be relayed, it is found that b1 matches a1, that is, b1 indicates that the terminal device B can play the video corresponding to the first authorization information, and B is used as the relay device. It is found that c1 indicates that the terminal device C can play the video, but the played video corresponds to the second authorization information, and C cannot be used as the relay device.

[0115] The business parameters of the application installed on the terminal device will change as the user uses it. For example, as the application is installed or uninstalled, the business capabilities supported by the terminal device will also increase or decrease. Therefore, during the content service relay process of some embodiments of the present application, it is necessary to monitor whether the business parameters of the application installed on the terminal device have changed, and report according to the changed results. This is to avoid the situation where the content service relay process is not carried out in accordance with the actual business capabilities of the terminal device. If the content service relay process is not carried out in accordance with the actual business capabilities of the terminal device, it is possible that the terminal device has a certain business capability, but the system determines that the terminal device is not a relay terminal device. Or, the terminal device does not have a certain business capability, and because the business capability list has not been updated, the system mistakenly determines that the terminal device is a relay terminal device.

[0116] In some embodiments, if it is determined that there is a change in the service parameters of the application installed on the terminal device, a service capability update request is generated according to the change. The update request carries the identification information of the newly added or reduced capabilities of the terminal in the table and the identification of the terminal device, and sends the service capability update request to the server, and the server updates the service capability list corresponding to the identification of the device according to the identification information in the service capability update request. The updated service capability list is used to record the service capabilities of the terminal device, and the recorded service capabilities correspond to the service parameters of the application installed on the terminal device.

[0117] If it is determined that the service parameters of the application installed on the terminal device have not changed, no service capability update request is generated. Therefore, there is no need to update the service capability list on the server. In this way, when the content service is relayed between terminal devices, it is carried out according to the actual service capabilities of the terminal devices, avoiding confusion in the content service relay process.

[0118] In some embodiments, when the service parameters of the application installed on the terminal device change, there are several possible situations: application installation, application version change, and application uninstallation. Application installation, application version change, and application download will cause the service parameters of the application to change.

[0119] It should be noted that after the terminal device reports its business capabilities to the server, the server saves a list of business capabilities of the terminal device. For example, in the above example, after the three terminal devices A, B, and C report their business capabilities, the server saves a list of business capabilities corresponding to the device information of the terminal devices: the business capability list of terminal device A (a1, a2...an), the business capability list of terminal device B (b1, b2...bn), and the business capability list of terminal device C (c1, c2...cn). If the application installed on the terminal device undergoes at least one of the following changes: application installation, application version change, and application uninstallation, the business parameters of the application will be changed accordingly. After the above-mentioned business capability update request, the records in the business capability list will also be changed accordingly.

[0120] If the service parameter change is application installation, the service capability update request is a service capability addition request. The corresponding server adds a service capability to the service capability list according to the service capability addition request. The added service capability corresponds to the service parameter of the newly installed application.

[0121] If the business parameter change is an application version change, the business capability update request is a business capability modification request. The corresponding server modifies the business capability in the business capability list according to the business capability modification request. The modified business capability corresponds to the business parameter of the application with the changed version.

[0122] If the service parameter change is application uninstallation, the service capability update request is a service capability deletion request. The corresponding server elaborates the service capability in the service capability list according to the service capability deletion request. The deleted service capability corresponds to the service parameter of the uninstalled application.

[0123] For example, when a new application A3 is installed on terminal device A, the service parameter of terminal device A changes to application installation, and terminal device A has a new service capability a3. At the same time, terminal device A sends a service capability addition request to the server, and after receiving the service capability addition request, the server adds service capability a3 to the service capability list of terminal device A.

[0124] For example, if the terminal device A does not have a broadcasting application originally, it does not have the service capability of broadcasting. The service capability list of the terminal device A saved in the server does not include the service capability of broadcasting. If the broadcasting application is newly installed in the terminal device A, the terminal device A has the service capability of broadcasting. At the same time, the terminal device A sends a service capability increase request to the server, and the server adds the service capability of broadcasting to the service capability list of the terminal device A according to the request. In this way, when the content service is relayed between the terminal devices, it can be carried out according to the actual service capability of the terminal device A.

[0125] When the version of application B1 on terminal device B is upgraded from version 1.0 to version 2.0, the service parameter of terminal device B changes to version change. Version 1.0 of application B1 only supports video services, while version 2.0 of application B1 supports both video services and audio services, and other applications installed on terminal device B do not support audio services. Therefore, the service capability of terminal device B newly adds audio playback service capability. Similarly, the server receives a request to modify the service capability and modifies the service capability list of terminal device B to include audio playback service capability.

[0126] When the application C1 is uninstalled on the terminal device C, the service parameter of the terminal device C changes to application uninstallation, and the terminal device C loses the service capability c1. Similarly, the server receives a request to delete the service capability and deletes the service capability c1 from the service capability list of the terminal device C.

[0127] In some embodiments, when the system of the current terminal device is started, the controller automatically queries whether there is a difference between the current business parameters of the application installed on the terminal device and the business parameters saved last time. If there is a difference between the current business parameters and the business parameters saved last time, it is determined that there is a change in the business parameters of the application on the terminal. It should be noted that when the terminal device starts the system, it not only collects the business parameter information of the application that starts when the computer is turned on, but also collects the business parameter information of the application that starts when the computer is not turned on.

[0128] For example, application A1 and application A2 are installed on terminal device A, application A1 is an application that starts automatically when the device is powered on, and application A2 is an application that does not start automatically when the device is powered on. When terminal device A is powered on, the controller automatically collects the current business parameters of application A1 and application A2, and compares them with the business parameters saved last time. If there is a difference between the current business parameters of application A1 and the business parameters saved last time, the difference or the current business parameters will be reported to the server so that the business capabilities related to terminal device A and application A1 are updated on the server. If there is no difference between the current business parameters of application A2 and the business capability parameters saved last time, the relevant information will not be reported to the server.

[0129] In some embodiments, after the current terminal device is started, if the application installed on the terminal device changes, the updated relevant business parameters will be automatically sent to the server. For example, application A1 installed on terminal device A, application A1 can be an application running in the foreground, an application running in the background, or an application that is not started. Regardless of the state of the application, if the business parameters of application A1 change, application A1 automatically reports the changed business parameters to the server. If the business parameters of application A1 do not change, no relevant information is reported to the server. The specific implementation process of the above business capability reporting method is:

[0130] After each application on the terminal device is installed on the terminal device, the business parameters of the current version will be written to the ANDROIDMANIFEST.XML file of its APK and placed in the META-DATA attribute. The reporting of business capabilities is completed by the unified relay service application. When the terminal device is turned on for the first time, the unified relay service application collects the business capabilities of all business applications in the system. After the collection is completed, the business capabilities of all applications collected will be reported to the server, and a business capability record of each application will be saved locally. The business capability record is consistent with the business capability list saved in the server.

[0131] The business capabilities of the terminal device change as the business parameters of the business application change. Changes in the business parameters of the business application refer to changes such as the installation, version update, and uninstallation of the application. When these changes occur in the application, an update broadcast will be sent to the unified relay service application. After receiving the update broadcast, the unified relay service application will re-collect the business parameters of the application that issued the update broadcast, and generate a business capability update request based on the re-collected business parameters. The unified relay service application reports the business capability update request to the server. The server updates the business capability list of the terminal device based on the business capability update request. Since not all applications are self-starting applications, the unified relay service application is used to collect the business parameters of each application, and the business capabilities are uniformly reported to the server based on the business parameters, so that the update changes of all applications on the terminal device can be monitored.

[0132] In some embodiments, the unified relay service application also collects business parameters of applications that have been uninstalled but whose data has not been cleared. For example, although application A1 on terminal device A has been uninstalled, the data has not been cleared. When terminal device B is the relay initiating terminal device, the query of application A1 can enable terminal device A to have business capability a1, and business capability a1 matches the business parameters of application B1 running in the foreground of terminal device B. A prompt can pop up on terminal device A, prompting the user to install application A1, so that terminal device A can serve as a relay terminal device to complete the relay. In this way, the success rate of content service relay can be improved, and the user experience can be further enhanced.

[0133] It should be noted that some embodiments of the present application utilize a unified relay service application to uniformly collect parameters of each application. Specifically, after the system is started, the unified relay service application queries the information of all applications in the system, and stores the information of all applications after the query. If the system is in a state of just starting up, the process of fully querying application information will be restarted. If the application information queried this time is different from the application information saved last time, it means that the application has changed. If the system remains started, when the application changes, the changed application automatically sends a corresponding broadcast to the unified relay service application.

[0134] In some embodiments, after the device capability list of the terminal changes, the server determines whether there has been any feedback on the relay-capable terminal device within the preset time period before the change to the terminal initiating the relay. If feedback has been given, feedback will be given again, otherwise feedback will not be given again. This situation mainly occurs when the capabilities supported by the relay-capable device are added or deleted within the preset time period after the last relay was triggered. This may be because the user intends to use or not use the relay-capable device to perform the currently ongoing relay operation. Therefore, by actively prompting the updated relay-capable device, the user's selection can be facilitated. After the preset time period after the last relay was triggered, the user's operation to change the capabilities supported by the relay device is weakly correlated with the last relay operation, and the changed relay-capable device may not be reminded.

[0135] In some embodiments, before feeding back the updated relayable device, the server also needs to determine whether the updated application is the application used by the user in the last relay. If so, feedback is given, otherwise, no feedback is given. In this case, the application used in the last relay is a specific application. If the unrelated application is changed, the association with the implementation of the last relay is weak, and no reminder of the change is required. If the application in use is upgraded or downgraded, it is strongly associated with the implementation of the last relay, and a reminder of the change is required. Exemplarily, if the V1 version of the A application in terminal device A does not support the first capability, and the V2 version of the A application in terminal device B supports the first capability, then in the last relay process, terminal device B is used as a relayable device and fed back to the terminal that initiated the relay. When the A application in terminal device A is upgraded from version V1 to version V2, terminal device A can also be used as a relayable device. In this way, the updated relayable device can be resent. When the C application is added to terminal device A, it is weakly associated with the implementation of the last relay, and no reminder of the change is required.

[0136] In some embodiments, before feeding back the updated relayable device, the server also needs to determine whether the capability to be updated is related to the capability used by the user in the last relay. If so, feedback is given; otherwise, no feedback is given.

[0137] In some embodiments, before feeding back the updated relayable device, the server also needs to determine whether the changed capability after the update is the capability used by the user in the last relay. If so, feedback is given, otherwise, no feedback is given. In this case, the application used in the last relay is a specific capability. If the changed irrelevant capability is weakly associated with the implementation of the last relay, no reminder of the change is required. If there is a new addition or reduction of related capabilities, it is strongly associated with the implementation of the last relay, and a reminder of the change is required. Exemplarily, if the A application in terminal device A does not support the first capability, and the B application in terminal device B supports the first capability, then in the last relay process, terminal device B is used as a relayable device and fed back to the terminal that initiated the relay. If the B application is installed in terminal device A and the first capability is added, terminal device A can also be used as a relayable device. In this way, the updated relayable device can be resent. If the C application is installed in terminal device A and the second capability is added, the terminal device still cannot be used as a relayable device. In this way, the updated relayable device can be resent.

[0138] In some embodiments, the terminal receiving the updated relay-capable device may be the first terminal initiating the first relay, so that the user can determine whether to switch the relay device through the first terminal.

[0139] In some embodiments, the terminal receiving the updated relayable device may be the second terminal receiving the first relay. This is because the first terminal may have been out of the user's use, such as sleeping, shutting down, etc., and the second terminal is being used by the user. Therefore, the second terminal is selected for display so that the user can determine whether to switch the relay device through the second terminal.

[0140] Unlike DLNA, since DLNA discovers the terminal after the user initiates it, the device discovery and user operation time are strongly related, and there is no need to pre-set a list of supported screen projection.

[0141] In some embodiments of the present application, since the reminder of the relayable device is an automatic reminder, the reminder is given before the user casts the screen, and the relayable device is not necessarily in the same local area network, it is more convenient to implement the function by managing the capabilities supported by the terminal through a capability list.

[0142] After the above service capability reporting process, the server stores the service capabilities of each terminal device. This service capability can be used to display a list of devices that can respond to the relay to the initiator after receiving the relay request.

[0143] It should be noted that in some embodiments of the present application, when an application with a relay service function is running in the foreground and the current page enters a specific page that requires a relay service operation, the relay-capable terminal device will be automatically queried. If the application running in the foreground does not have a relay service function, or although the application running in the foreground has a relay service function, but the current page has not entered a specific page that may require a relay service operation, the relay-capable terminal device will not be automatically queried.

[0144] In some embodiments, when a terminal device in the system (i.e., the relay-initiating terminal device) is running an application in the foreground, the relay terminal device is searched for according to the application data of the application, which is the device matching process. It should be noted that during the relay service operation, although the terminal devices can relay each other, some embodiments of the present application perform the relay service operation through the server.

[0145] In one scenario, Fig.15 In the application scenario diagram shown, the relay initiating terminal device is the display device 100-1, and the application running in the foreground is the video playback application. The current scenario is that the video playback application of the display device is playing a video, and the application data of the application is the data related to the video playback. The system also has terminal devices such as the display device 100-2, the speaker 100-3, and the smart alarm clock 100-4.

[0146] Among them, the device capability parameters of display device 100-2 are playing video and playing audio, the device capability parameters of speaker 100-3 are playing audio, and the device capability parameters of smart alarm clock 100-4 are alarm reminders. After the video playback application on display device 100-1 runs in the foreground, it searches the server for a terminal device that can perform the video playback operation. At this time, the application data is to play the video, and the stored device capability parameters are traversed according to the application data to play the video. Finally, the device capability parameters of display device 100-2 are found to play the video, which matches the current application data, and display device 100-2 is determined as a relay terminal device.

[0147] There are two business scenarios for relay service: the scenario where the relay service is triggered by a button and the scenario where the relay service is invoked by voice. In the scenario where the relay service is invoked by voice, if the voice signal input by the user includes a clear relay device, the content service can be relayed to the relay device. If the voice signal input by the user does not include a clear relay device, the system can only return a list of devices that can be relayed. The user needs to manually select the relay device in the list of devices that can be relayed to achieve the purpose of content service relay, resulting in a poor user experience.

[0148] It should be noted here that device capabilities refer to the functions supported by the device, such as audio and video playback, audio playback, temperature control, lighting switches, washing machine operation, etc. In this application, the server side screens and provides feedback on relayable devices based on device capabilities. This is different from the voice control in the prior art. In the voice control currently used, it can correspond to a single device, but when corresponding to multiple devices, the user needs to explicitly control the device, such as: TV volume up, air conditioner off, and so on. Take the smart speaker as an example. The smart speaker will parse the specified device name, and then control the device according to the device name. When corresponding to multiple devices, if you just say increase the volume, it will be difficult to determine the object to be adjusted, or only adjust the smart speaker (default setting) itself.

[0149] In some embodiments of the present application, in the scenario of voice-evoked relay service, the terminal device may include a sound collector, which is used to collect the voice signal input by the user. After collecting the voice signal input by the user, the voice signal is sent to the voice recognition server so that the voice text is recognized from the voice signal. Recognizing the voice text from the voice signal is a prior art, and the process of voice recognition is not elaborated in detail in this application.

[0150] In some embodiments, if the voice text includes device information, the relay device is determined based on the device information. The device information points to a unique relay device. Therefore, a unique relay device can be determined based on the voice text. The device information can be any one of the device's location information, device ID, device name, device type, device capability, etc. After determining the unique relay device, a relay relationship is established between the terminal device and the relay device, thereby completing the device matching process, and the content service relay process can be further performed.

[0151] In some embodiments, the above process can be implemented by a server or by the terminal device itself. In some embodiments, the above voice server can be a part of the server or can be independent.

[0152] In some embodiments, the server can parse out the capabilities required by the voice command based on the voice text, and then determine the relay device based on the determined capabilities. For example. The voice command of "air supply" can parse out the "air supply" capability, and the corresponding "air supply" capability can be air conditioners, fresh air blowers, fans, etc. "Start the rinsing function" can parse out the "rinsing" capability, and the corresponding "rinsing" capability can be washing machines, dishwashers, etc. For "volume plus", the "sound adjustment" capability can be parsed, and the corresponding "sound adjustment" capability can be TVs, smart speakers, refrigerator screens, etc. For "start sweeping", the "sweeping" capability can be parsed, and the corresponding "sweeping" capability can only be a sweeping robot.

[0153] If the voice text only includes device type information, it is not possible to determine a unique relay device based on the device type information, and only at least two relay terminal devices can be determined. Then, the final relay device is determined from these relay terminal devices based on the relationship between the relay dimension information of these relay terminal devices and the initiating dimension information of the relay terminal device that initiated the relay. For example, the voice text only includes "television", and multiple relay terminal devices may be determined based on the voice text. Then, based on the relay dimension information of multiple televisions and the initiating dimension information of the relay terminal device that initiated the relay, it is determined that the final relay television is the television with the same initiating dimension information as the initiating relay terminal device. In this way, even if the voice signal input by the user does not include a clear relay device, the system can automatically determine the final relay device to achieve the purpose of content service relay without the need for the user to manually select the relay device, thereby improving the user experience.

[0154] In some scenarios, there may be only one relay terminal device, and there is no need to compare the relay dimension information of the relay terminal device and the initiating dimension information of the relay terminal device, and directly determine the relay terminal device as the final relay device. In other scenarios, it is possible to determine multiple relay devices based on the relay dimension information of the relay terminal device and the initiating dimension information of the relay terminal device. At this time, the relay device with the highest frequency of use can be determined as the final relay device based on the frequency of use of these relay devices by users. Alternatively, a dialog box pops up for the user to select the required final relay device from these relay devices. Alternatively, the required final relay device is selected from these relay devices based on the user's location, where the location can refer to the current location, or it can be the location to which the user will move based on the user's moving path. In some embodiments, the user's location information can be determined by sensors on one or more devices.

[0155] It should be noted that the relay device is the terminal device that ultimately relays the content service from the current terminal device, and the relay terminal device is the device that matches the voice text. At the same time, the relay terminal device is assumed to be a terminal device that can perform operations corresponding to the application data of the application running in the foreground of the current terminal device.

[0156] In some embodiments, the dimension information of the device includes a first dimension, a second dimension, and a third dimension. The first dimension includes the second dimension, that is, the second dimension is within the range of the first dimension. The second dimension includes the third dimension, that is, the third dimension is within the range of the second dimension, and the third dimension is also within the range of the first dimension. The relay device is determined based on the relationship between the relay dimension information of the relay terminal device and the initiating dimension information of the initiating relay terminal device, which can be divided into the following situations:

[0157] If there is a relay terminal device whose first dimension, second dimension and third dimension respectively match the first dimension, second dimension and third dimension of the initiating relay terminal device, that is, the relay terminal dimension information of the relay terminal device completely matches the initiating dimension information of the initiating relay terminal device, then the relay terminal device is determined as the final relay device.

[0158] If there is no relay terminal device whose first dimension, second dimension and third dimension respectively match the first dimension, second dimension and third dimension of the initiating relay terminal device, but there is a relay terminal device whose first dimension and second dimension match the first dimension and second dimension of the initiating relay device, that is, the relay terminal device and the initiating relay terminal device are only within the same second dimension range, but in different third dimension ranges. However, there is no relay terminal device within the same third dimension range as the initiating relay terminal device, then the relay terminal device is determined as the final relay device.

[0159] If there is no relay terminal device whose first dimension, second dimension and third dimension respectively match the first dimension, second dimension and third dimension of the initiating relay terminal device, and there is no relay terminal device whose first dimension and second dimension respectively match the first dimension and second dimension of the initiating relay terminal device, but there is a relay terminal device whose first dimension matches the first dimension of the initiating relay terminal device, that is, there are only relay terminal devices within the same first dimension range, the relay terminal device is determined as the final relay device.

[0160] For example, Fig.16 , Fig.19 , Fig. 20 , Fig.21 In the dimensional scene diagram shown, the first dimension is the account number, the second dimension is the room attribute identifier, for example, it can be the user's house in different regions, and the third dimension is the room area identifier, for example, it can be the user's different rooms in different houses. The account number is the account ID registered by the user in the unified relay service application. The user can create multiple homes under each account dimension. For example, the user owns multiple houses, which are located in places A, B and C respectively. The houses in places A, B and C are the second dimension in the framework diagram. The third dimension is multiple physical spaces established based on the home dimension, such as master bedrooms, living rooms, second bedrooms, kitchens and other spaces.

[0161] exist Fig.16 In the scenario shown, TV A and entrance screen A are located in the master bedroom of home A, entrance screen B is located in the living room of home A, and refrigerator A with screen is located in the kitchen of home A. TV B is located in the living room of home B. Entrance screen C is located in the living room of home C, and refrigerator B with screen is located in the kitchen of home C. Homes A, B, and C all belong to account A.

[0162] If the voice text includes device information, for example, "relay to TV B in home B's living room", the voice text includes clear device information. No matter which terminal device is currently in use, the device information points to a unique relay device, and the final relay device is TV B in home B's living room.

[0163] The signaling processing sequence of the above embodiment is as follows: Fig.17 As shown in the signaling diagram, the relay-initiating terminal device receives the voice signal input by the user. The relay-initiating terminal device sends the voice signal and service information to the voice server. The voice server recognizes the voice text including the device information from the voice signal. The device information points to the only relay device, TV B in the living room of home B. The voice server sends the device information and service information to the AIOT server. The AIOT server searches for the device ID based on the device information, and sends the service information to TV B in the living room of home B based on the device ID, so that TV B in the living room of home B performs the corresponding relay operation according to the service information.

[0164] exist Fig.16 In the scenario shown, if the voice text only includes device type information, for example, the user inputs a voice signal to TV A, and the voice text recognized from the voice signal includes "Relay to the porch screen", then the voice text does not include clear device information, but only includes device type information "porch screen". The device type information "porch screen" refers to multiple porch screens. Fig.16 In the scenario shown, there is a porch screen A in the master bedroom of home A. The porch screen A and TV A are located in the same account, the same home, and the same physical space, that is, the first dimension, the second dimension, and the third dimension of the porch screen A match the first dimension, the second dimension, and the third dimension of TV A. Therefore, the porch screen A is determined as the final relay device, and the purpose of content service relay is finally achieved. The content service relay can be a relay video playback.

[0165] The signaling processing sequence of the above embodiment is as follows: Fig.18 As shown in the signaling diagram, the relay-initiating terminal receives the voice signal input by the user. The relay-initiating terminal device sends the voice signal to the voice server, and the voice server recognizes that the voice text includes device type information from the voice signal, and the device type information points to multiple relay devices. The voice server sends the device information and business information of multiple devices to the AIOT server. The AIOT server matches the dimensional information of multiple devices with the dimensional information of the relay-initiating terminal device, and after matching, the porch screen is obtained as the final relay device. After the AIOT server searches for the device ID of porch screen A, it sends business information to porch screen A according to the device ID, so that porch screen A performs the corresponding relay operation according to the business information.

[0166] exist Fig.19In the scene shown, TV A is located in the master bedroom of home A, entrance screen B is located in the living room of home A, and refrigerator A with screen is located in the kitchen of home A. TV B is located in the living room of home B. Entrance screen C is located in the living room of home C, and refrigerator B with screen is located in the kitchen of home C.

[0167] If the voice text only includes device type information, for example, the user inputs a voice signal to TV A, and the voice text recognized from the voice signal includes "relay to the porch screen", then the voice text does not include clear device information, but only includes device type information "porch screen". The device type information "porch screen" refers to multiple porch screens. TV A is located in the master bedroom of home A, and Fig.19 In the scene shown, there is no porch screen in the master bedroom of home A. Therefore, there is no relay terminal device in the current scene with the same account, the same family, and the same physical space as TV A. Then, in the current scene, search in the upper dimension whether there is a relay terminal device with the same account, the same family, but not in the same physical space as TV A. In the current scene, it is found that there is a porch screen B in the living room of home A. Porch screen B and TV A are in the same account, the same family, but not in the same physical space. Therefore, in the current scene, porch screen B is determined as the final relay device.

[0168] The signaling processing sequence of the above embodiment is Fig.18 The signaling processing sequence of the illustrated embodiment is similar, and the signaling processing sequence of this embodiment will not be described in detail.

[0169] exist Fig. 20 In the scene shown, TV A is located in the master bedroom of home A, and refrigerator A with a screen is located in the kitchen of home A. TV B is located in the living room of home B. Hallway screen C is located in the living room of home C, and refrigerator B with a screen is located in the kitchen of home C.

[0170] If the voice text only includes device type information, for example, the user inputs a voice signal to TV A, and the voice text recognized from the voice signal includes "relay to the porch screen", then the voice text does not include clear device information, but only includes device type information "porch screen". The device type information "porch screen" refers to multiple porch screens. TV A is located in the master bedroom of home A, and Fig. 20In the scene shown, there is no porch screen in the master bedroom of home A. Therefore, there is no relay terminal device in the current scene with the same account, the same family, and the same physical space as TV A. Then, in the current scene, search in the upper dimension whether there is a relay terminal device with the same account and the same family as TV A but not in the same physical space. In the current scene, there is no porch screen in the master bedroom, living room, and kitchen of home A. Therefore, there is no relay terminal device with the same account and the same family as TV A but not in the same physical space in the current scene. Then, search in the upper dimension in the current scene whether there is a relay terminal device with the same account as TV A but not in the same family. In the current scene, it is found that there is a porch screen C in home C under the same account. The porch screen C is under the same account as TV A, but not in the same family, and also not in the same physical space. Therefore, in the current scene, the porch screen C is determined as the final relay device.

[0171] The signaling processing sequence of the above embodiment is Fig.18 The signaling processing sequence of the illustrated embodiment is similar, and the signaling processing sequence of this embodiment will not be described in detail.

[0172] In some embodiments, if there is no relay terminal device whose first dimension matches the first dimension of the current terminal device, but there is a relay terminal device whose second dimension matches the second dimension of the current device, or there is a relay terminal device whose third dimension matches the third dimension of the current device, a prompt message is generated to enable the user to add the relay terminal device to the first dimension according to the prompt message, and then determine the relay terminal device as a relay device.

[0173] For example, in Fig.21 In the scene shown, TV A is located in the master bedroom of home A, and refrigerator A with screen is located in the kitchen of home A. TV B is located in the living room of home B. Refrigerator B with screen is located in the kitchen of home C. The porch screen D is located in the living room of home A, but TV A is under account A, while porch screen D is under account B or porch screen D is not added to any account. In the current scene, the first dimension of porch screen D does not match the first dimension of TV A, but porch screen D matches the voice text and belongs to the relay terminal device. At this time, a prompt message indicating that porch screen D is added to account A can be popped up on the display of the current terminal device. The user can operate according to the prompt information, and finally add porch screen D to account A, so that the first dimension of porch screen D matches the first dimension of TV A. At the same time, since porch screen D is located in the living room of home A, after adding porch screen D to account A, porch screen D and TV A are in the same account and the same home. That is to say, the first dimension and second dimension of porch screen D are matched with the first dimension and second dimension of TV A respectively.

[0174] The signaling processing sequence of the above embodiment is Fig.18The signaling processing sequence of the illustrated embodiment is similar, and the signaling processing sequence of this embodiment will not be described in detail.

[0175] After the above device matching process, if there is a relay terminal device, a relay prompt is generated to prompt the user that there is currently a matching relay terminal device. If there is no relay terminal device, no relay prompt may be generated. If the relay-initiating terminal device has a display, a relay button control may be displayed on the display, for example, Figure 7 The user interface diagram is shown in FIG. The user can trigger the content service relay process by clicking the relay button control. The relay prompt can also be in the form of voice broadcast.

[0176] It should be noted that in some embodiments, if it is determined that there is a relay terminal device, a relay button is displayed on the user interface. When the relay-initiating terminal device receives the user input to click the relay button, a query instruction is generated. The relay-initiating terminal device sends the query instruction to the AIOT server. After the AIOT server queries all relay-capable terminal devices in the system according to the query instruction, all relay-capable terminal devices are fed back to the relay-initiating terminal device. Then, the information of all relay-capable terminal devices is displayed on the user interface of the relay terminal device, that is, the relay terminal device list is displayed.

[0177] In some embodiments, the user can set to add a relay button to be displayed in a specific page of a specific application. For example, in the video playback page of the video playback application, it is set to display the relay button if there is a relay terminal device. In the teaching-related page of the learning application, it is set to display the relay button if there is a relay terminal device, but in the self-learning-related page of the learning application, it is not set to display the relay button if there is a relay terminal device.

[0178] In some embodiments, if there is no relay terminal device, a relay guide button control may be displayed on the display. The user may trigger the display of content service relay guide information on the display by clicking the relay guide button control. Here, the content service relay guide information may be introduction information of the content service relay, or information guiding the user to perform the next related operation of the content service relay. In this way, if there is no relay terminal device, even if the relay button control is not displayed, content related to the content service relay may be displayed on the user interface of the first application, thereby improving the user experience.

[0179] In such Figure 8In the application scenario shown, the user inputs a relay operation by selecting the target device display device 100-2, i.e., the relay terminal device. After receiving the relay operation input by the user, the display device 100-1 generates a relay instruction, which carries the application data and media data of the video playback application, and the relay instruction also carries the device ID of the relay terminal device. The relay-initiating terminal device sends the relay instruction to the display device 100-2 according to the device ID through the AIOT server. After receiving the relay instruction, the display device 100-2 continues to play the video according to the application data and media data of the video playback application.

[0180] In some embodiments, if the application currently running in the foreground of the relay-initiating terminal device (hereinafter referred to as the first application) is a video application, the corresponding relay button control is a video relay button control. If the first application is an audio application, the corresponding relay button control is an audio relay button control. If the first application is a picture display application, the corresponding relay button control is a picture display relay button control. The video relay button control, the audio relay button control, and the picture display relay button control can be displayed in different forms. This makes it convenient for users to view the type of application that currently needs to be relayed.

[0181] For example, Fig. 22 In the user interface of the music application shown, if there is a relay terminal device, an audio relay button control is displayed on the user interface of the audio application. The audio relay button control can be displayed as a music symbol icon. Similarly, the video relay button control can be displayed as a video icon, and the picture display relay button control can be displayed as a picture symbol icon.

[0182] The signaling processing sequence of the above embodiment is as follows: Fig.23 As shown in the signaling diagram, when the relay-initiating terminal device is on the audio application user interface, the relay terminal device query instruction is sent to the AIOT server through the unified relay service application. The query instruction carries the application data of the audio application. The AIOT server queries whether there is a terminal device that can execute the same or similar application data based on the application data of the audio application, and feeds back the query result to the relay-initiating terminal device. If the query result is that there is a terminal device that can execute the same or similar application data, the audio relay button control is displayed on the audio application user interface of the relay-initiating terminal device. Otherwise, the audio relay button control is not displayed. It should be noted that the audio application on the relay terminal device can be a different application than the application on the relay-initiating terminal device, but based on the application data, it can be determined that it is an application with the same function.

[0183] In some embodiments, the name of the second application on the relay terminal device may also be displayed on the user interface of the first application. The second application is an application that performs operations corresponding to the application data of the first application. Fig.24 In the user interface shown, the first application on the current display device is NetEase Cloud Music, and the device with the audio application installed is a relay terminal device. If the audio application installed on the relay terminal device is QQ Music, the words "QQ Music" are displayed near the relay button control. If the audio application installed on the relay terminal device is Migu Music, the words "Migu Music" are displayed near the relay button control. If QQ Music and Migu Music are installed on the relay terminal device at the same time, the words "QQ Music" and "Migu Music" can be displayed at the same time. If the user selects "QQ Music", the relay operation will be performed by QQ Music on the relay terminal device accordingly. If the user selects "Migu Music", the relay operation will be performed by Migu Music on the relay terminal device accordingly.

[0184] In some embodiments, if there are multiple relay terminal devices that can perform relay operations, a "plus sign" can be displayed on the relay button control to remind the user that there are multiple relay terminal devices. When the user clicks the relay button control, a list of relay terminal devices can be displayed on one side of the relay button control. The relay terminal device list includes device names, and the name of the application that can perform the relay operation is displayed on one side of each device name.

[0185] For example, Fig.25 In the user interface shown, when the first application on the current display device is NetEase Cloud Music, there are multiple devices with audio applications installed: device A, device B, and device C. The names of the applications that can perform relay operations are displayed on the side of the device name: Kugou Music, Migu Music, and QQ Music. Furthermore, if there are multiple applications that can perform relay operations on a relay terminal device, they are all displayed after the device name.

[0186] In some embodiments, a thumbnail of the user interface of the second application may also be displayed on the user interface of the first application. Fig.26 In the user interface shown, the first application on the current display device is NetEase Cloud Music, and the application installed on the relay terminal device is QQ Music, so a thumbnail of the QQ Music user interface is displayed on the side of the relay button control. In this way, the user can view the user interface of the second application after the relay, which is convenient for the user to decide whether to click the relay button control to carry out content service relay, further improving the user experience.

[0187] It should be noted that if the audio application on the relay-initiating terminal device is different from the audio application on the relay terminal device, the relay operation can be transmitted based on the name, singer, time and other information of the audio media data. For example, the audio application on the relay-initiating terminal device is NetEase Cloud Music, and the song played is "Invisible Wings". When the relay is passed to the relay terminal device, the corresponding song can be searched for on the audio application QQ Music on the relay terminal device based on the name of "Invisible Wings", the singer "Angela Chang", the time "2006" and other information. And the song continues to be played based on the current playing time of the song on the relay-initiating terminal device.

[0188] In terms of displaying a thumbnail of the user interface of the second application, the AIOT server may query the application on the relay terminal device and automatically send a screenshot request to the relay terminal device. After receiving the screenshot request, the relay terminal device sends a previously saved screenshot of the second application interface to the AIOT server. The AIOT server feeds back the screenshot of the second application interface to the initiating relay terminal device. After receiving the screenshot of the second application interface, the initiating relay terminal device displays a thumbnail of the screenshot of the second application interface on the first application interface.

[0189] In some embodiments, the display device and the determined relay terminal device need to be under the same account. Adding terminal devices to the same account is a prior art and is not elaborated in detail in this application. During the content service relay process, only devices under the same account can perform the content service relay process with each other. This can avoid interference from devices that are not under the same account.

[0190] In the above embodiment, the implementation process of displaying the relay button control on the user interface of the first application is as follows: Fig. 27The process shown specifically includes: C1, after the business application running in the foreground of the terminal device enters the relay business scene (entering the relay business scene can be the user interface entering a specific page, such as the video playback application entering the video playback page from the main page), the business application carries its relay scene identifier to request binding to the unified relay service application, so that the unified relay service application knows the business application currently in the relay business scene. C2, after receiving the binding request of the business application, the unified relay service application sends a query request (carrying business application data) to the AIOT server (which can be a device capability storage system). The AIOT server searches for terminal devices that support the same business under the same account according to the business application data carried in the query request, that is, queries the relay terminal device. C3, the server feeds back the device information of the relay terminal device found and the application information that can execute the relay to the unified relay service application. C4, the unified relay service application returns the device information of the relay terminal device found and the application information that can execute the relay to the business application. C5, the business application performs different UI displays on the user interface of the business application according to the device information of the relay terminal device and the application information that can execute the relay.

[0191] For example, if the service application in the relay service scenario on the relay-initiating terminal device is a video application, the application information of the service application of the relay terminal device that is searched is also a video application, and the video relay control is displayed. If the service application in the relay service scenario on the relay-initiating terminal device is an audio application, the application information of the service application of the relay terminal device that is searched is also an audio application, and the audio relay control is displayed.

[0192] If the user clicks the relay button control in the above embodiment, Figure 7 The user interface shown jumps to Fig. 9 The user interface shown. Fig. 9 The user interface shown lists the names of at least two relay terminal devices (if there is only one relay-capable terminal device, the relay terminal device list only displays the information of one relay-capable terminal device). The user can select one of the relay terminal devices, that is, determine the relay terminal device from multiple relay terminal devices. The initiating relay terminal generates a service relay instruction based on the user's operation. At the same time, the service relay instruction is sent to the determined relay terminal device. The service relay instruction carries content service relay data (corresponding to the application data of the first application running in the foreground of the initiating relay terminal device). After receiving the content service relay data, the relay terminal can execute the content service relay operation corresponding to the content service relay data.

[0193] If the current service relay instruction is sent by the user due to an error, for example, the user accidentally clicks the relay button or Fig. 9If a relay terminal device is accidentally clicked, the initiating relay terminal device sends a service relay instruction to the selected relay terminal device, and the current relay terminal device is the relay terminal device. The relay terminal device receives the receiving service relay instruction and can execute the content service relay operation corresponding to the content service relay data carried by the service relay instruction.

[0194] When the user finds that the current service relay instruction is sent by mistake, he can continue to input the relay interruption operation, and the device generates a relay interruption instruction based on the interruption operation. The relay interruption instruction carries business information and the device ID of the relay terminal device. The AIOT server sends the interruption instruction and business information to the corresponding relay terminal device according to the device ID. According to the relay interruption instruction, the relay terminal device does not perform the content service relay operation corresponding to the content service relay data. It should be noted that the relay terminal device receives the relay interruption instruction before the relay terminal device executes the content service relay operation corresponding to the content service relay data. It should also be noted that the application installed on the relay terminal device for the user to perform the relay operation can be in an open state or an unopened state. If no interruption instruction is received after receiving the service relay instruction, the relay terminal device opens the unopened relay application according to the relay service instruction to perform the corresponding relay operation.

[0195] In some embodiments, the relay interruption instruction may be an instruction generated according to an interruption operation input by a user on a relay terminal device. Specifically, after receiving the relay service instruction, the relay device may display a relay dialog box on the relay device, and the relay dialog box includes a confirm relay button control and a cancel relay button control. If the user clicks the confirm relay button control, the content service relay operation corresponding to the content service relay data is executed. If the user clicks the cancel relay button control, a relay interruption instruction is generated according to the user operation, and the relay device does not execute the content service relay operation corresponding to the content service relay data according to the relay interruption instruction.

[0196] The relay countdown dialog box can also be displayed on the relay terminal device, and the countdown time confirmation relay button control and the cancel relay button control are displayed on the relay countdown dialog box. The user clicks the confirm relay button control or the cancel relay button control to enter the operation of confirming the relay or canceling the relay, and the relay terminal device generates a confirmation relay instruction or a relay interruption instruction accordingly. If the time exceeds the countdown time and the user does not click any button control, a relay service instruction can be automatically generated.

[0197] In some embodiments, the relay interruption instruction may also be an instruction generated according to the interruption operation input by the user on the initiating relay terminal device. Specifically, after the relay device receives the relay service instruction, before executing the content service relay operation corresponding to the content service relay data, the relay dialog box is displayed on the initiating relay terminal device (the trigger condition for displaying the relay dialog box may be that the relay terminal device receives the relay service instruction and then feeds back the result of receiving the relay service instruction to the initiating relay terminal device. The trigger condition for displaying the relay dialog box may also be that the relay service instruction is issued by itself). The relay dialog box displays a cancel relay button control and a confirm relay button control. If the user clicks the cancel relay button control, a relay interruption instruction is generated according to the user operation. And the relay interruption instruction is sent to the relay terminal device. If the user clicks the confirm relay button control, a confirm relay instruction is generated according to the user operation. It should be noted that the relay dialog box can display a countdown. For example, if the user does not click any button control within 5 seconds, the display of the relay dialog box is canceled. The interruption operations in the embodiments of the present application are all assumed to be performed before the relay terminal device performs the relay operation.

[0198] A sound collector may be set on the relay-initiating terminal device, and the sound collector receives the voice signal input by the user. For example, the voice text "cancel relay" is recognized from the voice signal input by the user. The relay-initiating terminal device generates a relay interruption instruction carrying business information and the device ID of the relay terminal device according to the voice text, and the AIOT server sends the interruption instruction and business information to the corresponding relay terminal device according to the device ID. The relay terminal device does not perform the content service relay operation corresponding to the content service relay data according to the relay interruption instruction.

[0199] In some embodiments, after the relay terminal device does not perform the content service relay operation corresponding to the content service relay data, it also feeds back the relay result to the initiating relay terminal device. The initiating relay terminal device can continue to perform the content service relay operation corresponding to the content service relay data according to the fed-back relay result. That is, it continues to perform the operation before the relay. It should be noted that after the initiating relay terminal device sends the service relay instruction to the relay terminal device, the initiating relay terminal device may interrupt the previous operation or not.

[0200] For example, the display device is playing a video before the relay. When sending the service relay instruction to the relay terminal device, the operation of playing the video may be interrupted or not. If the operation of playing the video is interrupted, the operation of playing the video may be continued after receiving the relay result fed back by the relay terminal device (the content service relay operation corresponding to the content service relay data is not executed). A dialog box may also pop up on the display of the relay-initiating terminal device, prompting the user whether to continue the operation of playing the video on the relay-initiating terminal device. A "Yes" button control and a "No" button control may be set on the pop-up dialog box. If the user selects the "Yes" button control, the operation of playing the video will continue according to the operation input by the user. If the user selects the "No" button control, the operation of playing the video will not continue according to the operation input by the user.

[0201] The following is combined with Fig.28 The implementation process diagram of service relay interruption shown in the figure specifically explains the process of service relay interruption: first, after the system is started, the unified relay service application compiles the relay capabilities of each business application in the entire system, and reports the relay capabilities of each business application to the AIOT server.

[0202] D1 to D8 are similar to the processes B1 to B8 in the previous embodiment, and the processes D1 to D8 are not described in detail in this example.

[0203] In the embodiment of the present application, the relay activity can be interrupted before the relay terminal device performs the corresponding action. Fig.29 D9-D10 in FIG. 1 is the process of interrupting the service relay at the relay terminal device.

[0204] D9: Before the D8 process, if the user inputs an interruption command on the relay terminal device, the relay terminal device will not perform the corresponding action, that is, the D8 process will not be executed. The unified relay service application of the relay terminal device will send the assembly data after the interruption to the AIOT server. The assembly data includes the interruption result information, business information, and the device information of the initiating relay terminal device.

[0205] D10: AIOT sends the interruption result information and business information to the unified relay service application of the relay-initiating terminal device according to the device information. Then, according to the interruption result information and business information, the relay service interruption prompt information is displayed on the relay-initiating terminal device. If other applications have been switched on the current relay-initiating terminal device, the relay service interruption prompt information related to the interruption result information and business information can also be displayed on the relay-initiating terminal device.

[0206] like Fig.19 D11-D12 in the figure is the process of interrupting the service relay by the initiating relay terminal device.

[0207] D11: The user interrupts the relay by pressing the interrupt relay button or voice input on the relay initiating terminal device, triggering the relay interruption, and sends the relay interruption instruction to the AIOT server. The relay interruption instruction carries the business information (application service ID) and the device information of the relay terminal device.

[0208] D12: After the AIOT server finds the relay terminal device according to the device information, it sends the relay interruption instruction to the relay terminal device. The unified relay service application of the relay terminal device queries the corresponding application service according to the application service relay ID and cancels the startup of the application service, so that the relay terminal device does not perform the corresponding relay action.

[0209] After the relay terminal device starts to perform corresponding operations according to the content service relay data, the user can end the relay at any time. For example, the user can input an end relay instruction on the relay terminal device, and the relay terminal device stops performing corresponding operations according to the content service relay data after receiving the end relay instruction. Or the user inputs an end relay instruction on the initiating relay terminal device, and the initiating relay terminal device sends the end relay instruction to the relay terminal device. After the relay terminal device receives the end relay instruction from the initiating relay terminal device, it stops performing corresponding operations according to the content service relay data. It should be noted that the user inputs the end relay instruction by voice input, selection button input, gesture input, etc., and this application does not limit this.

[0210] After the relay terminal device finishes performing the corresponding operation according to the content service relay data, that is, after the content service relay operation is finished, the relay terminal device can generate a relay end feedback result and send the relay end feedback result to the initiating relay terminal device to implement the content service relay return operation. The relay end feedback result includes the execution progress data of the content service relay operation performed by the relay terminal device, and the initiating relay terminal device can continue to perform the content service relay operation according to the execution progress data. For example, the media content can be continued to be played on the initiating relay terminal device according to the playback progress of the media content on the relay terminal device.

[0211] In some embodiments, if the relay initiating terminal device is currently in the first state, the relay initiating terminal device can continue to perform the content service relay operation according to the execution progress data, then the relay initiating terminal device continues to perform the content service relay operation according to the execution progress data.

[0212] In some embodiments, if the relay initiating terminal device is currently in the second state and the relay initiating terminal device cannot continue to perform the content service relay operation according to the execution progress data, the relay initiating terminal device does not continue to perform the content service relay operation according to the execution progress data.

[0213] The signaling processing sequence of the above embodiment is as follows: Fig.29 The signaling diagram shown:

[0214] Step 1: The relay-initiating terminal device sends a relay service instruction to the AIOT server (carrying the business information of the current foreground running application of the relay-initiating terminal device and the device information of the relay terminal device).

[0215] Step 2: The AIOT server sends the business information and relay service instructions to the corresponding relay terminal device according to the device information.

[0216] Step 3: The relay terminal device receives the service information and performs the corresponding relay operation according to the service information. The service information may include basic information of the service and execution progress data of the service on the relay terminal device.

[0217] For example, if the application currently running in the foreground of the relay-initiating terminal device is a media asset playback-related application, the service information may include basic information such as the name of the media asset data, and also include the playback time point of the media asset on the relay-initiating terminal device during the relay. The relay terminal device continues to play the media asset based on the above basic information and playback time point.

[0218] Step 4: After receiving the instruction to interrupt the relay operation, the relay terminal device no longer continues to perform the corresponding relay operation. After interrupting the relay operation, the relay terminal device generates a relay end feedback result. The relay end feedback result may include basic information of the service and the execution progress data of the service when the relay terminal device interrupts the relay operation.

[0219] For example, the relay end feedback result includes service information and device information of the relay initiating terminal device. The service information at this time may also include the name of the media asset data and the playing time point of the media asset on the relay terminal device when the relay terminal device interrupts the relay operation.

[0220] Step 5: The AIOT server receives the relay end feedback result sent by the relay terminal device, and sends the relay end feedback result to the relay initiating terminal device.

[0221] Step 6: If the relay-initiating terminal device is in the first state when receiving the relay-ending feedback result, the relay-initiating terminal device continues to execute the relay operation according to the execution progress data. When the relay-initiating terminal device is in the first state, the relay-initiating terminal device can continue to execute the content service relay operation according to the execution progress data.

[0222] Step 7: If the relay-initiating terminal device is in the second state when receiving the relay-ending feedback result, the relay-initiating terminal device does not continue to perform the relay operation according to the execution progress data. When the relay-initiating terminal device is in the second state, the relay-initiating terminal device cannot continue to perform the content service relay operation according to the execution progress data.

[0223] In this way, in the content service relay return scenario, the current state of the initiating relay terminal device can be fully considered to avoid affecting the normal use of the initiating relay terminal device, thereby improving the user experience. It should be noted that the initiating relay terminal device and the relay terminal device can directly transmit instructions and data, or forward instructions and data through the server, which is not limited in this application.

[0224] When the relay initiating terminal device is in the first state, it can be in the following different situations:

[0225] The first situation is that when receiving the relay end feedback result, the relay initiating terminal device is currently in the first service content scene, wherein the first service content scene is the service content scene in which the relay initiating terminal device was before the relay.

[0226] For example, before the relay, the application running in the foreground on the relay-initiating terminal device is a video application. When receiving the relay end feedback result, the application running in the foreground on the relay-initiating terminal device is still a video application. Then the relay-initiating terminal device can continue to perform the content service relay operation according to the execution progress data, that is, continue to play the media content.

[0227] It should be noted that if the application currently running on the relay-initiating terminal device is a video application, the media content played by the video application of the relay-initiating terminal device is not the media content played before the relay. After the relay-initiating terminal device finishes playing the currently played media content, the previous media content can be played according to the execution progress data.

[0228] The second situation is that when receiving the relay end feedback result, the relay initiating terminal device is not currently in any business content scenario. For example, there is no application running in the foreground on the relay initiating terminal device. Then the relay initiating terminal device can continue to execute the content service relay operation according to the execution progress data.

[0229] When the relay initiating terminal device is in the second state, it can be in the following different situations:

[0230] The first scenario is that when receiving the relay end feedback result, the relay-initiating terminal device is currently in the second business content scenario, and the second business content scenario is different from the first business content scenario. The first business content scenario is the business content scenario that the relay-initiating terminal device was in before the relay. For example, before the relay, the application running in the foreground on the relay-initiating terminal device is a video application. When receiving the relay end feedback result, the application running in the foreground on the relay-initiating terminal device is an audio application. Then the relay-initiating terminal device cannot continue to execute the content service relay operation according to the execution progress data, that is, the relay-initiating terminal device cannot continue to play the media content.

[0231] The second scenario is that the relay-initiating terminal is currently in an offline state. At this time, the relay-initiating terminal cannot continue to perform the content service relay operation according to the execution progress data, that is, the relay-initiating terminal device cannot continue to play the media content. It should be noted that the relay end feedback result can be saved on the server, and when the relay-initiating terminal device enters the online state from the offline state, a state change request can be sent to the server. After the server receives the state change request of the relay-initiating terminal device and determines that the state of the relay-initiating terminal device is online, it sends the relay end feedback result to the relay-initiating terminal device. Furthermore, if the relay-initiating terminal device is not in any business content scenario or is in the first business content scenario after entering the online state, the content service relay operation can continue to be executed according to the execution progress data.

[0232] In some embodiments, when receiving the relay end feedback result, if the relay initiating terminal device is currently in the first state in the above embodiment, but according to the execution progress data, the relay terminal device has completed the corresponding content service relay operation according to the content service relay data, then the relay initiating terminal device also does not continue to execute the content service relay operation according to the execution progress data.

[0233] In some embodiments, if the relay-initiating terminal device is in the first state and receives the relay-ending feedback result, the relay-initiating terminal device can continue to perform the content service relay operation according to the execution progress data. Then the user can perceive the operation changes on the relay-initiating terminal device. For example, if the media content continues to be played according to the execution progress data, the user can watch the media content that continues to be played on the relay-initiating terminal device. However, if the relay-initiating terminal device is in the second state, after receiving the relay-ending feedback result, the relay-initiating terminal device cannot continue to perform the content service relay operation according to the execution progress data. Then the user cannot perceive the operation changes on the relay-initiating terminal device.

[0234] At this time, a return failure prompt may be generated on the relay-initiating terminal device to prompt the user that the current content service relay return has failed. For example, a dialog box may pop up on the relay terminal device and / or the relay-initiating terminal device, and the dialog box may include the prompt "content service relay return failed". The relay terminal device and / or the relay-initiating terminal device may also voice broadcast the return failure prompt, such as the voice broadcast prompt "content service relay return failed", thereby further improving the user's interactive experience.

[0235] Based on the above embodiments, the present application also provides a content service relay interruption method, such as Fig.30 The method flow chart and Fig.31 The method signaling diagram shown, the method comprises the following steps:

[0236] Step S101: The relay terminal device receives a relay service instruction from the initiating relay terminal device. The relay service instruction carries content service relay data, and the relay terminal device can execute the content service relay operation corresponding to the content service relay data.

[0237] Step S102: before the relay terminal device executes the content service relay operation corresponding to the content service relay data, if the relay terminal device receives a relay interruption instruction, the relay terminal device does not execute the content service relay operation corresponding to the content service relay data according to the relay interruption instruction.

[0238] In this way, even if the relay terminal device has received the service relay instruction, it can also interrupt the relay instruction according to the relay and not perform the relay operation, that is, if the user accidentally triggers the content service relay process, the content service relay is interrupted in time, thereby improving the user experience. It should be noted that in the method embodiment of the present application, between terminal devices, instructions and data can be transferred through the server, or they can be transferred directly between terminal devices without being transferred through the server.

[0239] The relay interruption instruction may be an instruction generated according to an interruption operation input by a user on a relay terminal device. For example, after receiving the relay service instruction, the relay device may display a relay dialog box on the relay device, and the relay dialog box includes a confirm relay button control and a cancel relay button control. If the user clicks the confirm relay button control, the content service relay operation corresponding to the content service relay data is executed. If the user clicks the cancel relay button control, a relay interruption instruction is generated according to the user operation.

[0240] The relay interruption instruction may also be an instruction generated according to an interruption operation input by the user on the relay initiating terminal device. For example, after the relay device receives the relay service instruction, before executing the content service relay operation corresponding to the content service relay data, a relay dialog box is displayed on the relay initiating terminal device. A cancel relay button control is displayed on the relay dialog box. If the user clicks the cancel relay button control, a relay interruption instruction is generated according to the user operation. And the relay interruption instruction is sent to the relay terminal device.

[0241] The same or similar contents between the various embodiments of the present application can be referenced to each other, and the relevant embodiments will not be described in detail.

[0242] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

[0243] For the convenience of explanation, the above description has been made in conjunction with specific embodiments. However, the above exemplary discussion is not intended to be exhaustive or limit the embodiments to the specific forms disclosed above. Based on the above teachings, various modifications and variations can be obtained. The selection and description of the above embodiments are to better explain the principles and practical applications, so that those skilled in the art can better use the embodiments and various different variations of the embodiments suitable for specific use considerations.

Claims

1. A terminal device, characterized in that: The terminal device is a relay terminal device, and the relay terminal device is a target device of the content service relay, including: monitor; The controller is configured as: Receiving a service relay instruction from a relay initiating terminal device, wherein the service relay instruction carries content service relay data, and the relay terminal device can execute a content service relay operation corresponding to the content service relay data; After receiving the service relay instruction, starting the first countdown function, and displaying a dialog box for confirming the relay on the display; If a first relay interruption instruction sent by the relay initiating terminal device is received, then according to the first relay interruption instruction, the content service relay operation corresponding to the content service relay data is not performed, the first relay interruption instruction is used to interrupt the relay terminal device from performing the operation corresponding to the service relay instruction, and the first relay interruption instruction is an instruction generated by the relay initiating terminal device according to the interruption operation input by the user after the second countdown function is started and before the second countdown ends; If a confirmation operation of canceling the relay button control in the relay confirmation dialog box is received before the first countdown ends, a second relay interruption instruction is generated, and according to the second relay interruption instruction, the content service relay operation corresponding to the content service relay data is not performed; If a confirmation operation on the relay confirmation button control in the relay confirmation dialog box is received before the first countdown ends, executing a content service relay operation corresponding to the content service relay data; After executing the content service relay operation, receiving a relay end instruction input by a user, stopping the execution of the content service relay operation, and generating a relay end feedback result, wherein the relay end feedback result includes execution progress data of the content service relay operation; sending the relay end feedback result to the relay initiating terminal device, so that the relay initiating terminal device continues to perform the content service relay operation according to the execution progress data when the relay initiating terminal device is in a first service content scenario or is not in any service content scenario, wherein the first service content scenario includes the audio or video service scenario in which the relay initiating terminal device is before the relay; and does not continue to perform the content service relay operation according to the execution progress data when the relay initiating terminal device is in a second service content scenario, wherein the first service content scenario is different from the second service content scenario; The terminal device and the relay initiating terminal device are fixed in different physical spaces.

2. The terminal device according to claim 1, characterized in that: The terminal device includes a television or a refrigerator with a screen, and the relay initiating terminal includes a television or a refrigerator with a screen.

3. The terminal device according to claim 1, characterized in that: The controller is further configured to: After not executing the content service relay operation corresponding to the content service relay data, feeding back a relay result to the relay initiating terminal device, so that the relay initiating terminal device continues to execute the content service relay operation corresponding to the content service relay data according to the relay result.

4. A terminal device, characterized in that: The terminal device is a relay initiating terminal device, including: The controller is configured as: Sending a service relay instruction to the relay terminal device, so that the relay terminal device starts the first countdown function and displays a dialog box for confirming the relay on the display, wherein the service relay instruction carries content service relay data, and the relay terminal device can execute the content service relay operation corresponding to the content service relay data; Start the second countdown function; If an interruption operation input by the user is received before the end of the second countdown, a first relay interruption instruction is generated, and the first relay interruption instruction is sent to the relay terminal device, so that the relay terminal device does not perform the content service relay operation corresponding to the content service relay data according to the first relay interruption instruction, and the first relay interruption instruction is used to interrupt the relay terminal device from performing the operation corresponding to the service relay instruction; If no interruption operation input by the user is received before the end of the second countdown, the first relay interruption instruction is not sent to the relay terminal device; if the relay terminal device receives a confirmation operation for canceling the relay button control in the confirmation relay dialog box before the end of the first countdown, the second relay interruption instruction is generated, and according to the second relay interruption instruction, the content service relay operation corresponding to the content service relay data is not executed; if a confirmation operation for confirming the relay button control in the confirmation relay dialog box is received before the end of the first countdown, the content service relay operation corresponding to the content service relay data is executed; receiving a relay end feedback result sent by the relay terminal device, wherein the relay end feedback result is generated after the relay terminal device receives a relay end instruction input by a user after executing the content service relay operation and stops executing the content service relay operation, and the relay end feedback result includes execution progress data of the content service relay operation; When in the first service content scenario or not in any service content scenario, continue to execute the content service relay operation according to the execution progress data, wherein the first service content scenario is the audio or video service scenario in which the relay-initiating terminal device is located before the relay; When in the second service content scenario, the content service relay operation is not continued according to the execution progress data, and the first service content scenario is different from the second service content scenario; Wherein, the relay terminal device and the terminal device are fixed in different physical spaces.

5. The terminal device according to claim 4, characterized in that: The controller is further configured to: After not executing the content service relay operation corresponding to the content service relay data, receiving a feedback relay result sent by the relay terminal device, and continuing to execute the content service relay operation corresponding to the content service relay data according to the relay result.

6. A content service relay interruption method, characterized in that: Applied to a relay terminal device, the relay terminal device is a target device for content service relay, including: Receiving a service relay instruction from a relay initiating terminal device, wherein the service relay instruction carries content service relay data, and the relay terminal device can execute a content service relay operation corresponding to the content service relay data; After receiving the service relay instruction, starting the first countdown function and displaying a relay confirmation dialog box on the display; If a first relay interruption instruction sent by the relay initiating terminal device is received, then according to the first relay interruption instruction, the content service relay operation corresponding to the content service relay data is not performed, the first relay interruption instruction is used to interrupt the relay terminal device from performing the operation corresponding to the service relay instruction, and the first relay interruption instruction is an instruction generated by the relay initiating terminal device according to the interruption operation input by the user after the second countdown function is started and before the second countdown ends; If a confirmation operation of canceling the relay button control in the relay confirmation dialog box is received before the first countdown ends, a second relay interruption instruction is generated, and according to the second relay interruption instruction, the content service relay operation corresponding to the content service relay data is not performed; If a confirmation operation on the relay confirmation button control in the relay confirmation dialog box is received before the first countdown ends, executing a content service relay operation corresponding to the content service relay data; After executing the content service relay operation, receiving a relay end instruction input by a user, stopping the execution of the content service relay operation, and generating a relay end feedback result, wherein the relay end feedback result includes execution progress data of the content service relay operation; sending the relay end feedback result to the relay initiating terminal device, so that the relay initiating terminal device continues to perform the content service relay operation according to the execution progress data when the relay initiating terminal device is in a first service content scenario or is not in any service content scenario, wherein the first service content scenario includes the audio or video service scenario in which the relay initiating terminal device is before the relay; and does not continue to perform the content service relay operation according to the execution progress data when the relay initiating terminal device is in a second service content scenario, wherein the first service content scenario is different from the second service content scenario; The relay terminal device and the initiating relay terminal device are fixed in different physical spaces.

7. A content service relay interruption method, characterized in that: Applicable to the relay initiating terminal equipment, including: Sending a service relay instruction to the relay terminal device, so that the relay terminal device starts the first countdown function and displays a dialog box for confirming the relay on the display, wherein the service relay instruction carries content service relay data, and the relay terminal device can execute the content service relay operation corresponding to the content service relay data; Start the second countdown function; If an interruption operation input by the user is received before the end of the second countdown, a first relay interruption instruction is generated, and the first relay interruption instruction is sent to the relay terminal device, so that the relay terminal device does not perform the content service relay operation corresponding to the content service relay data according to the first relay interruption instruction, and the first relay interruption instruction is used to interrupt the relay terminal device from performing the operation corresponding to the service relay instruction; If no interruption operation input by the user is received before the end of the second countdown, the first relay interruption instruction is not sent to the relay terminal device; if the relay terminal device receives a confirmation operation for canceling the relay button control in the confirmation relay dialog box before the end of the first countdown, the second relay interruption instruction is generated, and according to the second relay interruption instruction, the content service relay operation corresponding to the content service relay data is not executed; if a confirmation operation for confirming the relay button control in the confirmation relay dialog box is received before the end of the first countdown, the content service relay operation corresponding to the content service relay data is executed; receiving a relay end feedback result sent by the relay terminal device, wherein the relay end feedback result is generated after the relay terminal device receives a relay end instruction input by a user after executing the content service relay operation and stops executing the content service relay operation, and the relay end feedback result includes execution progress data of the content service relay operation; When in the first service content scenario or not in any service content scenario, continue to execute the content service relay operation according to the execution progress data, wherein the first service content scenario is the audio or video service scenario in which the relay-initiating terminal device is located before the relay; When in the second service content scenario, the content service relay operation is not continued according to the execution progress data, and the first service content scenario is different from the second service content scenario; Wherein, the relay terminal device and the terminal device are fixed in different physical spaces.

Images