Matching method of terminal device and terminal device

Abstract

The embodiment provides a matching method of a terminal device and the terminal device. The terminal device receives a voice signal input by a user collected by a sound collector, and identifies a voice text from the voice signal. If the voice text includes device information, a relay device is determined according to the device information, and a relay relationship between the current terminal device and the determined relay device is established. If the voice text includes device type information, a final relay device is determined from a relayable terminal device according to a relationship between relay dimension information of the relayable terminal device and initiation dimension information of the current terminal device. Finally, a relay relationship between the current terminal device and the determined relay device is established. In this way, even if the voice signal input by the user does not include explicit relay device information, the relay device can be determined to achieve the content service relay purpose without manual selection of the relay device by the user, thereby improving the user experience.

Description

[0001] This application claims priority to Chinese Patent Application No. 202110896120.4, filed on August 5, 2021, and titled "Content Service Relay Method of Terminal Device and Terminal Device", and to Chinese Patent Application No. 202111097923.X, filed on September 18, 2021, and titled "Display Device", the entire contents of which are incorporated herein by reference. TECHNICAL FIELD

[0002] The present application relates to the technical field of terminal devices, and in particular to a matching method of a terminal device and a terminal device. BACKGROUND

[0003] With the development of intelligent terminal products and the improvement of people's living standards, more and more families have multiple home intelligent terminal devices. Home intelligent devices can relay content services to each other. For example, video playback services on a smart TV can be relayed to the display screen of a refrigerator, and video playback services on the display screen of the refrigerator can also be relayed to the smart TV.

[0004] The existing screen projection function is a cross-terminal control function of video, which requires the screen projection end and the projected end to support a specific protocol, such as DLNA, Miracast, etc. There are two main screen projection methods. One is represented by DLNA, which is mainly used for online video playback. After the user selects the screen projection function on the playback page, the screen projection end discovers the supported projected end in the local area network according to the predetermined protocol. The video playback address is directly transmitted to the projected end through the local area network, and the projected end plays the video according to the video playback address. At this time, the screen projection end no longer needs to analyze the video data, and can be hibernated or closed without affecting the playback of the projected end. The other is represented by Miracast, which belongs to screen mirroring operation. After the screen projection end performs screen projection, it still displays the image. The displayed image is mirrored to the projected end through the local area network. The hibernation or closure of the screen projection end will affect the playback of the projected end.

[0005] Therefore, the current screen projection function can only realize the cross-terminal control function of video, and cannot realize the content service relay operation except for video services. The content service relay application arises at the historic moment. The content service relay between home intelligent devices can be realized by inputting a voice signal. In the above voice relay scenario, if the user input voice signal includes an explicit relay device, the content service can be relayed to the relay device for operation.

[0006] However, if the voice signal input by the user does not include an explicit relay device, the system can only return a list of devices that can be relayed, and the user needs to manually select a relay device in the list of devices that can be relayed to achieve the purpose of content service relay, resulting in a poor user experience. SUMMARY

[0007] The present application provides a matching method of a terminal device and a terminal device, which are used to solve the problem that the current terminal device, if the voice signal input by the user does not include an explicit relay device, the system can only return a list of devices that can be relayed, and the user needs to manually select a relay device in the list of devices that can be relayed to achieve the purpose of content service relay, resulting in a poor user experience.

[0008] In a first aspect, the present embodiment provides a terminal device, comprising:

[0009] a sound collector configured to collect a voice signal input by a user;

[0010] a controller configured to:

[0011] receive the voice signal from the sound collector and identify voice text from the voice signal;

[0012] when the voice text includes device information, determine a relay device according to the device information, and establish a relay relationship between the terminal device and the relay device, wherein the device information points to only one relay device;

[0013] when the voice text includes device type information, determine a terminal device that can be relayed according to the device type information, and determine a relay device from the terminal device that can be relayed according to the relationship between the relay dimension information of the terminal device that can be relayed and the initiation dimension information of the terminal device, and establish a relay relationship between the terminal device and the relay device, wherein the device type information points to at least two terminal devices that can be relayed.

[0014] In a second aspect, the present embodiment provides a matching method of a terminal device, comprising:

[0015] receiving a voice signal from a sound collector and identifying voice text from the voice signal;

[0016] when the voice text includes device information, determining a relay device according to the device information, and establishing a relay relationship between the terminal device and the relay device, wherein the device information points to only one relay device;

[0017] When the device type information is included in the voice text, a relayable terminal device is determined according to the device type information, a relay device is determined from the relayable terminal device according to a relationship between relay dimension information of the relayable terminal device and initiation dimension information of the terminal device, and a relay relationship between the terminal device and the relay device is established, wherein the device type information points to at least two relayable terminal devices.

[0018] The embodiment provides a matching method of a terminal device and the terminal device. A voice signal input by a user is collected by a sound collector, and voice text is recognized from the voice signal. If device information is included in the voice text, the device information points to a unique relay device, a relay device is determined according to the device information, and a relay relationship between the current terminal device and the determined relay device is established. If device type information is included in the voice text, the device type information points to at least two relayable terminal devices. Then, a final relay device is determined from the relayable terminal devices according to a relationship between relay dimension information of the relayable terminal devices and initiation dimension information of the current terminal device. Finally, a relay relationship between the current terminal device and the determined relay device is established. In this way, even if the voice signal input by the user does not include explicit relay device information, the relay device can be determined to achieve the content service relay purpose without manual selection of the relay device by the user, thereby improving the user experience. BRIEF DESCRIPTION OF DRAWINGS

[0019] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed in the embodiments. Obviously, the drawings in the following description are only some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained from these drawings without creative labor.

[0020] Figure 1 A use scenario of a display device according to some embodiments is shown;

[0021] Figure 2 A hardware configuration block diagram of a control device 100 according to some embodiments is shown;

[0022] Figure 3 A hardware configuration block diagram of a display device 200 according to some embodiments is shown;

[0023] Figure 4 A software configuration diagram in the display device 200 according to some embodiments is shown;

[0024] Figure 5 An icon control interface display diagram of an application program in the display device 200 according to some embodiments is shown;

[0025] Figure 6 FIG. 1 illustrates a schematic diagram of a terminal device content service relay system framework, according to some embodiments;

[0026] Figure 7 FIG. 2 illustrates a schematic diagram of a user interface of a terminal device initiating a relay, according to some embodiments;

[0027] Figure 8 FIG. 3 illustrates a schematic diagram of another user interface of a terminal device initiating a relay, according to some embodiments;

[0028] Figure 9 FIG. 4 illustrates a schematic diagram of another user interface of a terminal device initiating a relay, according to some embodiments;

[0029] Figure 10 FIG. 5 illustrates a schematic diagram of another user interface of a terminal device initiating a relay, according to some embodiments;

[0030] Figure 11 FIG. 6 illustrates a schematic diagram of a user interface of a relay terminal device, according to some embodiments;

[0031] Figure 12 FIG. 7 illustrates a schematic diagram of another user interface of a relay terminal device, according to some embodiments;

[0032] Figure 13 FIG. 8 illustrates a schematic diagram of another user interface of a relay terminal device, according to some embodiments;

[0033] Figure 14 FIG. 9 illustrates a schematic diagram of a service relay service flow, according to some embodiments;

[0034] Figure 15 FIG. 10 illustrates a schematic diagram of a terminal device content service relay application scenario, according to some embodiments;

[0035] Figure 16 FIG. 11 illustrates a schematic diagram of a content service relay device matching dimension scenario, according to some embodiments;

[0036] Figure 17 FIG. 12 illustrates a schematic diagram of a content service relay signaling processing timing, according to some embodiments;

[0037] Figure 18 FIG. 13 illustrates a schematic diagram of another content service relay signaling processing timing, according to some embodiments;

[0038] Figure 19 FIG. 14 illustrates a schematic diagram of another content service relay device matching dimension scenario, according to some embodiments;

[0039] Figure 20An exemplary diagram illustrates yet another content service relay device matching dimension scenario, according to some embodiments;

[0040] Figure 21 An exemplary diagram illustrates yet another content service relay device matching dimension scenario, according to some embodiments;

[0041] Figure 22 An exemplary diagram illustrates yet another user interface diagram of initiating a relay terminal device, according to some embodiments;

[0042] Figure 23 An exemplary diagram illustrates yet another content service relay signaling processing timing diagram, according to some embodiments;

[0043] Figure 24 An exemplary diagram illustrates yet another user interface diagram of initiating a relay terminal device, according to some embodiments;

[0044] Figure 25 An exemplary diagram illustrates yet another user interface diagram of initiating a relay terminal device, according to some embodiments;

[0045] Figure 26 An exemplary diagram illustrates yet another user interface diagram of initiating a relay terminal device, according to some embodiments;

[0046] Figure 27 An exemplary diagram illustrates yet another service flow diagram of displaying a relay button control on a user interface of a terminal device, according to some embodiments;

[0047] Figure 28 An exemplary diagram illustrates yet another service flow diagram of implementing service relay interruption, according to some embodiments;

[0048] Figure 29 An exemplary diagram illustrates yet another signaling timing diagram of relay return, according to some embodiments;

[0049] Figure 30 An exemplary diagram illustrates yet another matching method flowchart of a terminal device, according to some embodiments. DETAILED DESCRIPTION

[0050] For the purpose of making the objects and implementation of the present application more clear, the present application will be described in detail below with reference to the accompanying drawings of the exemplary embodiments of the present application. Obviously, the described exemplary embodiments are only a part of the embodiments of the present application, but not all the embodiments of the present application.

[0051] It should be noted that the brief description of the terms in this application is only for the convenience of understanding the embodiments described below, and is not intended to limit the embodiments of the application. Unless otherwise specified, these terms should be understood according to their ordinary and general meanings.

[0052] The terms "first", "second", "third" and the like in the specification and claims of the present application and the above-described drawings are used to distinguish similar or like objects or entities, and do not necessarily mean a specific order or sequence, unless otherwise noted. It should be understood that the terms used in this way can be interchanged under appropriate circumstances.

[0053] The terms "include" and "have" and any variations thereof are intended to cover but not exclusive inclusion, for example, a product or device including a series of components does not necessarily limit to all components clearly listed, but can include other components not clearly listed or inherent to these products or devices.

[0054] The display device provided by the embodiments of the present application can have various implementation forms, for example, can be a television, a smart television, a laser projection device, a monitor, an electronic bulletin board, an electronic table, etc. Figure 1 And Figure 2 As a specific embodiment of the display device of the present application.

[0055] Figure 1 As a schematic diagram of the operation scenario between the display device and the control device according to the embodiments. As shown in Figure 1 The user can operate the display device 200 through the smart device 300 or the control device 100.

[0056] In some embodiments, the control device 100 can be a remote controller, and the communication between the remote controller and the display device includes at least one of infrared protocol communication or Bluetooth protocol communication, and other short distance communication methods, to control the display device 200 by wireless or wired way. The user can control the display device 200 by at least one input user instruction such as button on the remote controller, voice input, control panel input, etc.

[0057] In some embodiments, the smart device 300 (such as a mobile terminal, a tablet computer, a computer, a notebook computer, etc.) can also be used to control the display device 200. For example, using an application running on the smart device to control the display device 200.

[0058] In some embodiments, the smart device 300 can also be other terminals that can communicate, for example, air conditioners, refrigerators, intelligent central control, intelligent access control, etc.

[0059] In some embodiments, the display device can receive the instruction without using the smart device or the control device described above, but by touch or gesture, etc.

[0060] In some embodiments, the display device 200 can also be controlled in ways other than the control device 100 and the smart device 300, for example, the display device 200 can receive the user's voice instruction control directly through the voice instruction module configured inside the display device 200, or through the voice control device set outside the display device 200.

[0061] In some embodiments, the display device 200 also communicates with the server 400. The display device 200 can be allowed to communicate through a local area network (LAN), a wireless local area network (WLAN), and other networks. The server 400 can provide various content and interaction to the display device 200. The server 400 can be a cluster or multiple clusters, and can include one or more types of servers.

[0062] Figure 2 An exemplary configuration block diagram of the control device 100 according to an exemplary embodiment is shown. As shown, the control device 100 includes a controller 110, a communication interface 130, a user input / output interface 140, a memory, a power supply. The control device 100 can receive the user's input operation instruction, and convert the operation instruction into an instruction that the display device 200 can recognize and respond to, acting as an intermediary between the user and the display device 200. Figure 2

[0063] As shown, the display device 200 includes a tuner and demodulator 210, a communicator 220, a detector 230, an external device interface 240, a controller 250, a display 260, an audio output interface 270, a memory, a power supply, at least one of a user interface. Figure 3

[0064] In some embodiments, the user can input the user command on the graphical user interface (GUI) displayed on the display 260, and then the user input interface receives the user input command through the graphical user interface (GUI). Alternatively, the user can input the user command by inputting a specific sound or gesture, and then the user input interface receives the user input command by recognizing the sound or gesture through the sensor.

[0065] ​​A "user interface" is a medium interface for interaction and information exchange between an application or an operating system and a user, which realizes the conversion between the internal form of information and the form acceptable by the user. A commonly used form of a user interface is a graphic user interface (GUI), which refers to a user interface related to computer operation displayed in a graphic manner. It can be an icon, a window, a control element, etc. displayed in the display screen of an electronic device, wherein the control element can include an icon, a button, a menu, a tab, a text box, a dialog box, a status bar, a navigation bar, a widget, etc.

[0066] Referring to Figure 4 In some embodiments, the system is divided into four layers, from top to bottom, an Applications layer (referred to as "the application layer"), an Application Framework layer (referred to as "the framework layer"), an Android runtime and system library layer (referred to as "the system runtime library layer"), and a kernel layer.

[0067] In some embodiments, at least one application program is running in the application layer, which can be a window program, a system setting program, or a clock program, etc. provided by the operating system; or an application program developed by a third-party developer. In a specific implementation, the application program package in the application layer is not limited to the above examples.

[0068] As Figure 4 shown, the application framework layer in some embodiments of the present application includes managers (Managers) and content providers (Content Provider), wherein the managers include at least one of the following modules: an activity manager (Activity Manager) for interacting with all activities running in the system; a location manager (LocationManager) for providing access to system location services for system services or applications; a file package manager (PackageManager) for retrieving various information related to application program packages currently installed on the device; a notification manager (Notification Manager) for controlling the display and clearing of notification messages; and a window manager (Window Manager) for managing icons, windows, toolbars, wallpapers, and desktop components on the user interface.

[0069] In some embodiments, the activity manager is used to manage the life cycle of each application and the general navigation back function, such as controlling the exit, opening, back, etc. of the application. The window manager is used to manage all window programs, such as obtaining the size of the display screen, determining whether there is a status bar, locking the screen, intercepting the screen, controlling the display window change (such as reducing the display window, shaking the display, twisting the display, etc.), etc.

[0070] In some embodiments, the display device can directly enter the interface of the preset video on demand program after starting. The video on demand program is the default system program of the system, and the interface thereof can include at least a navigation bar 510 and a content display area below the navigation bar 510, as shown in FIG. 5. The content displayed in the content display area changes with the change of the selected control in the navigation bar. The programs in the application layer can be integrated in the video on demand program and displayed through a control in the navigation bar, or further displayed after the application control in the navigation bar is selected. Figure 5

[0071] In some embodiments, the display device can directly enter the display interface of the last selected signal source or the signal source selection interface after starting. The signal source can be at least one of the preset video on demand program, the HDMI interface, the live TV interface, etc. After the user selects different signal sources, the display can display the content obtained from different signal sources.

[0072] With the development of intelligent terminal products and the improvement of people's living standards, more and more families have multiple home intelligent terminal devices. The home intelligent devices can provide content service relay to each other. For example, the video playing service on the smart TV can be relayed to the display screen of the refrigerator, and the video playing service on the display screen of the refrigerator can also be relayed to the smart TV. The present application provides a terminal device content service relay system, as shown in FIG. 6. The terminal devices 100 can transmit and receive data between the terminal devices through the server 200. In the content service relay system of the present application, all terminal devices 100 can act as an initiating relay terminal device and a relay terminal device. The initiating relay terminal device is the terminal device that initiates the content service relay process, and the relay terminal device is the target terminal device of the content service relay process. The server of the present application can be an AIOT (Artificial Intelligence & Internet of Things) server. Figure 6

[0073] ​​The content service relay scenarios in this application include two types: one is triggering the relay service through a button control, and the other is voice wake-up relay service.

[0074] In some embodiments, such as Figure 7 The diagram shown illustrates the user interface of the initiating relay terminal device. Based on the business application currently running in the foreground of the initiating relay terminal device, a relay button control is displayed on the user interface of the initiating relay terminal device if a relay terminal device exists. The following section will explain in detail how to determine whether a relay terminal device exists. It should be noted that determining the existence of a relay terminal device here does not require querying all available relay terminal devices.

[0075] Figure 7 The example shown illustrates a business scenario where a relay service is triggered via a button control. After a user clicks the relay button in the user interface, the initiating relay terminal device receives the user's action and generates a relay command. This command carries the device information of the initiating relay terminal device and the service information of its current business. Based on the relay command, the initiating relay terminal device queries its current business and then searches for available relay terminals based on the service information. In a voice-activated relay service scenario, the user can directly input voice commands. The initiating relay terminal device receives the user's voice input and also generates a relay command. Based on the generated command, the initiating relay terminal device searches for available relay terminals, such as... Figure 8 The user interface shown displays the message "Searching for executable devices". At this time, the service being run by the initiating relay terminal device can be either stopped or continued. After a relay terminal device is found, its information is displayed on the initiating relay terminal device. In some embodiments, if multiple relay terminal devices are found, a list of relay devices is displayed on the user interface, such as... Figure 9 The user interface shown. Figure 9 The user interface also displays a "Delay playback for 5 seconds" option bar and a "Continue playback on original device" option bar.

[0076] When a user selects a handover terminal device from the list of handover devices, for example, "Bedroom TV," and simultaneously turns on the "Delay 5-second playback" option, then as follows: Figure 10 As shown, the user interface of the device initiating the relay displays the message "Sending to the bedroom TV soon" and a countdown dialog box. The countdown dialog box includes a back button, which the user can click to cancel the relay before the 5-second countdown ends.

[0077] In addition, such as Figure 11The prompt "About to receive push from XX device" and the countdown dialog box are also displayed on the user interface of the relay terminal device, i.e., the user interface of the device "Bedroom TV" that is pushed. The XX device is the initiating terminal device. The countdown dialog box displayed on the user interface of the relay terminal device includes a back button and a confirm button. If the user clicks the back button on the user interface of the relay terminal device, the relay is cancelled. If the user clicks the confirm button on the user interface of the relay terminal device, the relay is determined.

[0078] After the relay is determined, as shown in FIG. 6, a prompt "Relayed to Bedroom TV" is displayed on the user interface of the initiating terminal device, and the user interface of the initiating terminal device jumps back to the interface before the relay service content is played. Figure 12 Figure 13 As shown in FIG. 7, a prompt "Relayed the content of XX device" is displayed on the user interface of the relay terminal device, and the user interface of the relay terminal device starts to play the relay service content, for example, a video.

[0079] In a service relay service scenario triggered by a button control, as shown in FIG. 8, the service flow of the above embodiment specifically includes: Figure 14

[0080] A1: The service application of the initiating terminal device registers the relay capability of the service application with the unified relay service application, and the unified relay service application controls the transmission of scenarios and data in the relay process. After the system of the initiating terminal device enters the relay service scenario (entering the relay service scenario can be that the user interface enters a specific page, for example, the video playing application enters the video playing page from the main page. The relay button control is set in the video playing page.), the user clicks the relay button in the user interface, and the initiating terminal device receives the user operation to generate a relay instruction. The unified relay service application can determine the current relay service scenario according to the user input operation.

[0081] A2: After receiving the relay instruction, the unified relay service application carries the service information (the information of the service application currently running in the foreground of the initiating terminal device) to query the AIOT server for the relay terminal device that can relay the current service under the current account (which can be an account registered by the user in the relay service platform in advance, and then the user binds the terminal device under the same account).

[0082] A3: The AIOT server returns a list of relay terminal devices that support the relay of the current service under the current account according to the current service and the state of all devices under the current account (only online devices are returned, and offline devices are not returned).

[0083] ​​The process of forming the relay terminal device list is as follows: The AIoT server pre-stores the device capabilities of all terminal devices. When the AIoT server receives the service information of the current service initiating the relay, it iterates through the stored device capabilities and finds the device capability that matches the service information of the current service. If a match is found, it is determined to be a relayable terminal device. All relayable terminal devices are listed in a list, which is the relay terminal device list. If there is only one relayable terminal device, then only one terminal device will be listed in the relay terminal device list. If there are multiple relayable terminal devices, then multiple terminal devices will be listed in the relay terminal device list.

[0084] A4: After receiving the returned data (including device ID, device name, etc.), the unified relay service application displays a list of relay terminal devices for users to select from.

[0085] A5: After a user selects a device, the device information of the selected device will be transmitted to the unified relay service application. (The main device information transmitted is the device ID, which identifies the relay terminal device selected by the user.)

[0086] A6: The unified relay service collects the business information of the current business and the equipment information of the selected device, assembles all the information, and transmits it to the AIoT server.

[0087] A7: After receiving the assembled information, the AIoT server parses the information. Then, based on the device information, it transmits the service information (the content information to be relayed) to the target device, that is, to the relay terminal device selected by the user.

[0088] A8: After receiving the relay content information, the relay terminal device executes the corresponding relay action based on the content information. The initiator of the relay action is the unified relay service application of the relay terminal device, and the executor of the relay action is an application with the corresponding function.

[0089] A9: After the relay terminal device executes the relay action, the unified relay service application of the relay terminal device will then feed back the relay execution result to the initiating relay terminal device through the AIoT server.

[0090] A10: The relay terminal device receives the relay execution result and displays different relay statuses to the user based on the result. For example, if the relay terminal device successfully executes the relay action, it displays a "Relay Successful" status. If the relay terminal device has not yet successfully executed the relay action and is waiting for the relay countdown, it displays a "Relay Countdown Begins" status.

[0091] In business scenarios where services are activated via voice, such as Figure 14 As shown, the business process of the above embodiment specifically includes:

[0092] B1: This step is a voice recognition process. After the voice application collects the voice signal input by the user, the voice signal is sent to the voice server, and the voice server performs semantic analysis on the voice signal, that is, identifies the user's intention from the voice signal.

[0093] B2: After the voice server identifies the user's intention from the voice signal as a relay service, the AIOT server is called to query the relay terminal device (when the AIOT server is called, the voice server sends a call request to the AIOT server, and the call request carries service information used to query the relay terminal device).

[0094] B3: After the AIOT server receives the call request, the relay terminal device that can relay the current service under the current account is queried according to the service information. The device information of the queried relay terminal device is fed back to the voice server.

[0095] It should be noted that the voice server can be used only to identify the user's intention from the voice signal, and then feed back the user's intention to the unified service relay application. Then the unified service relay application interacts with the AIOT server. Some embodiments of the present application can shorten the link and improve the system response speed by forwarding data through the voice server.

[0096] B4: The voice server returns the device information queried from the AIOT server to the voice application.

[0097] B5: The voice application calls the interface of the unified relay service application, and delivers the information recognized by the voice and the obtained device information to the unified relay service application. The information recognized by the voice can include a relay service instruction, and the device information includes a device ID.

[0098] B6: After the unified relay service application receives the device information of the relay terminal device, the above information (including service information, device information, and relay service instructions) is delivered to the AIOT server in combination with the service information. The relay service instruction can carry the service information.

[0099] B7: The AIOT server sends the service information and the relay instruction to the relay terminal device corresponding to the device information.

[0100] B8: After the unified relay service application of the relay terminal device receives the relay service instruction, the relay action is performed according to the service information carried by the relay service instruction, wherein the relay action is consistent with the content of the current service application of the initiating relay terminal device.

[0101] B9: After the relay terminal device performs the relay action, the unified relay service application of the relay terminal device feeds back the relay execution result to the initiating relay terminal device through the AIOT server.

[0102] B10: The initiating relay terminal device receives the relay execution result and displays different relay states to the user according to the relay execution result. For example, if the relay terminal device successfully performs the relay action, a relay success state is displayed. If the relay terminal device has not successfully performed the relay action, but is waiting for the relay countdown, a start relay countdown state is displayed.

[0103] The content service relay process is generally that the user selects a target device, i.e., a relay terminal device, on the initiating relay terminal device, determines to trigger the relay, and then sends a service relay instruction from the initiating relay terminal device to the relay terminal device. The relay terminal device then performs a relay operation according to the service relay instruction.

[0104] Before the user selects the relay terminal device on the initiating relay terminal device, the initiating relay terminal device and the relay terminal device need to be matched. The device matching is based on the service capability of each terminal device. Here, the service capability corresponds to the service parameter of the application installed on the terminal device. For example, a display device has a video playback application installed, and has a video playback capability. A sound box has an audio playback application installed, and has an audio playback capability. A tablet computer has a video call application installed, and has a video call capability.

[0105] Each terminal device can report the local service capability to the server, or directly send it to other terminal devices. Some embodiments of the present application take the terminal device reporting the local service capability to the server as an example to elaborate the service capability reporting of the terminal device.

[0106] The terminal device can report the service capability to the server, which can be an AIOT (Artificial Intelligence & Internet of Things) system or a capability management system. The terminal device can automatically send a service capability registration request to the server when the terminal device system starts, and the capability registration request carries the service capability of the terminal device, thereby realizing the reporting of the service capability. The terminal devices can also report the service capability to each other. The process of reporting the service capability between terminal devices is not elaborated.

[0107] For example, the terminal device A, B and C are respectively installed with the application (A1, A2...An), (B1, B2...Bn) and the application (C1, C2...Cn), and after the process of reporting the service capability, the service capability list (a1, a2...an) of the terminal device A, the service capability list (b1, b2...bn) of the terminal device B and the service capability list (c1, c2...cn) of the terminal device C are saved on the server. The service capability (a1, a2...an) in the service capability list respectively corresponds to the application (A1, A2...An), the service capability (b1, b2...bn) respectively corresponds to the application (B1, B2...Bn), and the service capability (c1, c2...cn) respectively corresponds to the application (C1, C2...Cn).

[0108] In some embodiments, when the A1 application on the terminal device A is running in the foreground, the service capability matching the application capability a1 corresponding to A1 is found according to the service parameter of the A1 application. If other service capabilities matching the service capability a1 are found, the corresponding matching relay terminal device is determined. If no other service capabilities matching the service capability a1 are found, the relay terminal device is not matched. For example, when the service capability a1 matches b1, the terminal device B is determined as the relay terminal device. When the service capability a1 matches c2, the terminal device C is determined as the relay terminal device.

[0109] For example, the video playing application A1 is running on the terminal device, and it is determined that the terminal device A is performing the function of video playing. The video playing capability can be marked as a1 in the terminal device A device capability list. When determining the relayable device, it is found that b1 and a1 match, i.e., b1 represents that the terminal device B can play the video, and B is determined as the relayable device. For another example, the video playing application A1 is running on the terminal device, and it is determined that the terminal device A is performing the function of video playing. The video playing application A1 corresponds to the first authorization information, and the video playing capability corresponding to the first authorization information can be marked as a1 in the terminal device A device capability list. When determining the relayable device, it is found that b1 and a1 match, i.e., b1 represents that the terminal device B can play the video corresponding to the first authorization information, and B is determined as the relayable device. It is found that c1 represents that the terminal device C can play the video, but the played video corresponds to the second authorization information, and C cannot be determined as the relayable device.

[0110] The service parameters of the applications installed on the terminal device change with the use of the user. For example, with the installation or uninstallation of the applications, the service capabilities supported by the terminal device increase or decrease. Therefore, in the content service relay process of some embodiments of the present application, it is necessary to monitor whether the service parameters of the applications installed on the terminal device change, and report the changed results. In order to avoid the content service relay process not being performed according to the actual service capabilities of the terminal device. If the content service relay process is not performed according to the actual service capabilities of the terminal device, it is possible that the terminal device has a certain service capability, but the system judges that the terminal device is not a relay terminal device. Or, the terminal device does not have a certain service capability, but the system mistakenly judges that the terminal device is a relay terminal device due to the fact that the service capability list is not updated.

[0111] In some embodiments, if it is determined that the service parameters of the applications installed on the terminal device change, a service capability update request is generated according to the change. The update request carries the identification information of the newly added or reduced capabilities of the terminal in the table and the identification of the terminal device, and sends the service capability update request to the server. The server updates the service capability list corresponding to the identification of the device according to the identification information in the service capability update request. The updated service capability list is used to record the service capabilities of the terminal device, and the recorded service capabilities correspond to the service parameters of the applications installed on the terminal device.

[0112] If it is determined that the service parameters of the applications installed on the terminal device do not change, no service capability update request is generated. Therefore, there is no need to update the service capability list on the server. In this way, when the content service is relayed between terminal devices, it is performed according to the actual service capabilities of the terminal device, avoiding the situation that the content service relay process is chaotic.

[0113] In some embodiments, when the service parameters of the applications installed on the terminal device change, there are several possible situations: application installation, application version change and application uninstallation. Application installation, application version change and application download will all cause the service parameters of the applications to change.

[0114] It should be noted that after the terminal device reports the service capability to the server, the service capability list of the terminal device is saved in the server. For example, in the above example, the terminal devices A, B and C report the service capability, and the service capability list corresponding to the terminal device information is saved in the server: the service capability list of the terminal device A (a1, a2...an), the service capability list of the terminal device B (b1, b2...bn), and the service capability list of the terminal device C (c1, c2...cn). If at least one of the application installation, application version change and application uninstallation of the application installed in the terminal device changes, the service parameter of the application will be changed accordingly. After the above service capability update request, the record in the service capability list will also be changed accordingly.

[0115] If the service parameter changes to application installation, the service capability update request is a service capability addition request. The server adds the service capability to the service capability list according to the service capability addition request. The added service capability corresponds to the service parameter of the newly installed application.

[0116] If the service parameter changes to application version change, the service capability update request is a service capability modification request. The server modifies the service capability in the service capability list according to the service capability modification request. The modified service capability corresponds to the service parameter of the application with version change.

[0117] If the service parameter changes to application uninstallation, the service capability update request is a service capability deletion request. The server deletes the service capability in the service capability list according to the service capability deletion request. The deleted service capability corresponds to the service parameter of the uninstalled application.

[0118] For example, when a new application A3 is installed in the terminal device A, the service parameter of the terminal device A changes to application installation, and the terminal device A has a new service capability a3. At the same time, the terminal device A sends a service capability addition request to the server, and the server adds the service capability a3 to the service capability list of the terminal device A after receiving the service capability addition request.

[0119] For example, the terminal device A does not have the service capability of playing broadcast because the terminal device A does not have the broadcast playing application originally. The service capability list of the terminal device A saved in the server does not include the service capability of playing broadcast. If the broadcast playing application is newly installed in the terminal device A, the terminal device A has the service capability of playing broadcast. At the same time, the terminal device A sends a service capability addition request to the server, and the server adds the service capability of playing broadcast to the service capability list of the terminal device A according to the request. In this way, when the content service is relayed between the terminal devices, the actual service capability of the terminal device A can be used.

[0120] When the version of application B1 on terminal device B is upgraded from 1.0 to 2.0, the service parameter of terminal device B changes due to the version change. The application B1 of version 1.0 only supports video service, while the application B1 of version 2.0 supports both video service and audio service, and other applications installed on terminal device B also do not support audio service. Therefore, the service capability of terminal device B is added with audio playing service capability. Similarly, the server receives the request of modifying service capability, and modifies the service capability list of terminal device B to include audio playing service capability.

[0121] When the application C1 is uninstalled on terminal device C, the service parameter of terminal device C changes due to the uninstallation of application, and terminal device C loses service capability c1. Similarly, the server receives the request of deleting service capability, and deletes service capability c1 from the service capability list of terminal device C.

[0122] In some embodiments, when the system of the current terminal device is started, the controller automatically queries whether the current service parameter of the application installed on the terminal device is different from the last saved service parameter. If the current service parameter is different from the last saved service parameter, it is determined that the service parameter of the application on the terminal device has changed. It should be noted that when the system of the terminal device is started, the service parameter information of the application that is started automatically at startup is collected, and the service parameter information of the application that is not started automatically at startup is also collected.

[0123] For example, terminal device A is installed with application A1 and application A2, application A1 is an application that is started automatically at startup, and application A2 is an application that is not started automatically at startup. When terminal device A is started, the controller automatically collects the current service parameter of application A1 and application A2, and compares the current service parameter with the last saved service parameter. If the current service parameter of application A1 is different from the last saved service parameter, the difference or the current service parameter is reported to the server, so as to update the service capability of terminal device A related to application A1 in the server. If the current service parameter of application A2 is not different from the last saved service parameter, no related information is reported to the server.

[0124] In some embodiments, after the current terminal device is started, if the application installed on the terminal device changes, the server is automatically sent the updated related service parameter. For example, application A1 is installed on terminal device A, and application A1 can be an application running in the foreground, an application running in the background, or an application that is not started. Regardless of the state of the application, if the service parameter of application A1 changes, application A1 automatically reports the changed service parameter to the server. If the service parameter of application A1 does not change, no related information is reported to the server. The specific implementation process of the above service capability reporting method is as follows:

[0125] After each application on the terminal device is installed on the terminal device, the current version of the service parameter of the application is written to the ANDROIDMANIFEST.XML file of the APK of the application and is placed in the META-DATA attribute. Reporting of the service capability is completed by the unified relay service application. When the terminal device is started for the first time, the unified relay service application collects the service capability of the service application of the whole system, after the collection is completed, reports the collected service capability of all the applications to the server, and saves a record of the service capability of each application locally, which is consistent with the service capability list saved in the server.

[0126] The service capability of the terminal device changes with the change of the service parameter of the service application. The change of the service parameter of the service application refers to the change of installation, version update, uninstallation and the like of the application. When the application changes, the application sends an update broadcast to the unified relay service application. After the unified relay service application receives the update broadcast, the service parameter of the application sending the update broadcast is re-collected, and a service capability update request is generated according to the re-collected service parameter. The unified relay service application reports the service capability update request to the server. The server updates the service capability list of the terminal device according to the service capability update request. Since not all applications are self-starting applications, the service parameter of each application is collected by the unified relay service application, and the service capability is uniformly reported to the server according to the service parameter, so that the update change of all the applications on the terminal device can be monitored.

[0127] In some embodiments, the unified relay service application also collects the service parameter of the application that is uninstalled but not data-cleared. For example, application A1 on terminal device A is uninstalled but not data-cleared. When terminal device B is the initiating relay terminal device, it is found that application A1 can make terminal device A have service capability a1, and service capability a1 matches the service parameter of foreground running application B1 of terminal device B. A prompt can be popped up on terminal device A to prompt the user to install application A1, so that terminal device A can serve as a relay terminal device and complete the relay. In this way, the success rate of content service relay can be improved, and the user experience can be further improved.

[0128] It should be noted that some embodiments of the present application use the unified relay service application to uniformly collect the parameters of each application. Specifically, after the system is started, the unified relay service application queries the information of all the applications in the system, and after the query, stores the information of all the applications. If the system is in a state of just starting, the process of querying the application information will be started again, and if the application information queried this time is different from the application information saved last time, it indicates that the application has changed. If the system is in a state of keeping starting, when the application changes, the changed application automatically sends a corresponding broadcast to the unified relay service application.

[0129] In some embodiments, after the device capability list of the terminal changes, the server determines whether the terminal initiating the baton has fed back the batonable terminal device within a preset time period before the change. If yes, the server re-feeds back; otherwise, the server does not re-feed back. This situation mainly occurs when the supported capability of the batonable device is added or deleted within the preset time period after the last triggering of the baton. The user may intend to use or not use the batonable device to perform the current baton operation. Therefore, by actively prompting the updated batonable device, the user can be facilitated to make a selection. After the preset time period after the last triggering of the baton, the operation of the user changing the supported capability of the batonable device is weakly associated with the last baton operation, and the changed batonable device can not be prompted.

[0130] In some embodiments, before feeding back the updated batonable device, the server also needs to determine whether the updated application is the application used by the user in the last baton. If yes, the server feeds back; otherwise, the server does not feed back. In this case, the application used in the last baton is a specific application. If the changed application is irrelevant, the application is weakly associated with the implementation of the last baton, and the changed application can not be prompted. If the used application is upgraded or downgraded, the application is strongly associated with the implementation of the last baton, and the changed application needs to be prompted. For example, if the V1 version of the A application in the terminal device A does not support the first capability, the V2 version of the A application in the terminal device B supports the first capability. Therefore, in the last baton process, the terminal device B is fed back as the batonable device to the terminal initiating the baton. When the A application in the terminal device A is upgraded from the V1 version to the V2 version, the terminal device A can also be used as the batonable device. In this way, the updated batonable device is re-sent. When the C application is added in the terminal device A, the application is weakly associated with the implementation of the last baton, and the changed application can not be prompted.

[0131] In some embodiments, before feeding back the updated batonable device, the server also needs to determine whether the updated capability is the capability used by the user in the last baton. If yes, the server feeds back; otherwise, the server does not feed back.

[0132] In some embodiments, the server needs to determine whether the changed capability after the update is the capability used by the user in the last relay, and if so, feedback is given, otherwise, no feedback is given. In this case, the application used in the last relay is the specific capability, and if the changed capability is irrelevant, the association with the implementation of the last relay is weak, and no change reminder can be given. If there is an increase or decrease in relevant capabilities, the association with the implementation of the last relay is strong, and a change reminder needs to be given. For example, if the A application in the terminal device A does not support the first capability, and the B application in the terminal device B supports the first capability, so in the last relay process, the terminal device B is used as the relayable device and feedback is given to the terminal that initiates the relay. After the B application is installed in the terminal device A and the first capability is added, the terminal device A can also be used as a relayable device. In this way, the updated relayable device can be re-sent. If the C application is installed in the terminal device A and the second capability is added, the terminal device cannot be used as a relayable device. In this way, the updated relayable device is not re-sent.

[0133] In some embodiments, the terminal that receives the updated relayable device can be the first terminal that initiates the first relay. In order to determine whether the user wants to switch the relayable device through the first terminal.

[0134] In some embodiments, the terminal that receives the updated relayable device can be the second terminal that receives the first relay. This is because the first terminal may have been removed from the user's use, such as hibernation, shutdown, etc., and the second terminal is being used by the user. Therefore, the second terminal is displayed to facilitate the user to determine whether to switch the relayable device through the second terminal.

[0135] Unlike DLNA, since DLNA discovers the terminal after the user initiates, the device discovery and user operation are strongly associated in time, and there is no need to pre-set a list of devices that can support screen projection.

[0136] In some embodiments of the present application, the reminder of the relayable device is an automatic reminder, which is given before the user performs screen projection, and the relayable device is not necessarily in the same local area network, so it is more convenient to manage the capabilities supported by the terminal through the capability list.

[0137] After the above service capability reporting process, the server stores the service capabilities of each terminal device. This service capability can be used to display a list of devices that can respond to the relay to the initiator after receiving the relay request.

[0138] It should be noted that in some embodiments of this application, when an application with relay service functionality runs in the foreground and the current page enters a specific page requiring relay service operation, it will automatically query available relay terminal devices. If the application running in the foreground does not have relay service functionality, or if the application running in the foreground does have relay service functionality but the current page has not entered a specific page that may require relay service operation, then available relay terminal devices will not be automatically queried.

[0139] In some embodiments, while an application is running in the foreground of a terminal device in the system (i.e., the initiating terminal device), the system searches for the next terminal device based on the application data of that application; this is the device matching process. It should be noted that although terminal devices can relay each other during the relay service operation, some embodiments of this application perform the relay service operation through a server.

[0140] In a scenario, such as Figure 15 The application scenario diagram shown depicts display device 100-1 as the initiating terminal device, and a video playback application running in the foreground. Therefore, the current scenario involves the video playback application on the display device playing a video, and the application data for this application is the data related to the video playback. Other terminal devices in the system include display device 100-2, speaker 100-3, and smart alarm clock 100-4.

[0141] Among them, the device capability parameters of display device 100-2 are video playback and audio playback, the device capability parameter of speaker 100-3 is audio playback, and the device capability parameter of smart alarm clock 100-4 is alarm reminder. After the video playback application on display device 100-1 runs in the foreground, it searches the server for terminal devices that can perform video playback operations. At this time, the application data is video playback, so the stored device capability parameters are traversed according to the application data. Finally, the device capability parameters of display device 100-2 are found to play video, and if they match the current application data, then display device 100-2 is determined as the relay terminal device.

[0142] The handover service can be implemented in two scenarios: button-triggered and voice-activated. In the voice-activated scenario, if the user's voice input includes a specific handover device, the content service can be transferred to that device. If the user's voice input does not include a specific handover device, the system only returns a list of available devices. The user must then manually select a device from this list to complete the content service handover, resulting in a poor user experience.

[0143] It should be noted that the device capability refers to the function supported by the device, such as audio and video playing, audio playing, temperature control, lighting switch, washing machine operation, and the like. In the present application, the server side performs screening and feedback of the relayable device according to the device capability. It is different from the voice control in the prior art. In the voice control used at present, a single device can be corresponded, but when multiple devices are corresponded, the user needs to explicitly control the device, for example, TV volume up, air conditioner off, and the like. Taking the smart sound box as an example. The smart sound box parses the specified device name, and then controls the device according to the device name. When multiple devices are corresponded, if only the volume up is said, it is difficult to determine the adjustment object, or only the smart sound box (default setting) itself is adjusted.

[0144] In some embodiments of the present application, in the scenario of voice calling the relay service, the terminal device can include a sound collector for collecting the voice signal input by the user. After the voice signal input by the user is collected, the voice signal is sent to the voice recognition server, so that the voice text is recognized from the voice signal. The process of voice recognition is prior art, and will not be described in detail in the present application.

[0145] In some embodiments, if the device information is included in the voice text, the relay device is determined according to the device information. The device information points to a unique relay device. Therefore, the unique relay device can be determined according to the voice text. The device information can be any one of the device location information, device ID, device name, device type, device capability, and the like. After the unique relay device is determined, the terminal device and the relay device establish a relay relationship, thereby completing the device matching process, and the content service relay process can be further performed.

[0146] In some embodiments, the above process can be implemented by the server, or can be implemented by the terminal device itself. In some embodiments, the voice server described above can be a part of the server, or can be independent.

[0147] In some embodiments, the server can parse the capability required by the voice instruction according to the voice text, and then determine the relay device according to the determined capability. For example, the voice instruction of “air supply” can parse the “air supply” capability, and the air conditioner, fresh air machine, fan, and the like can correspond to the “air supply” capability. The voice instruction of “start the rinsing function” can parse the “rinsing” capability, and the washing machine and the dish washer can correspond to the “rinsing” capability. For the voice instruction of “volume up”, the “sound adjustment” capability can be parsed, and the TV set, smart sound box, refrigerator screen, and the like can correspond to the “sound adjustment” capability. For the voice instruction of “start sweeping”, the “sweeping” capability can be parsed, and only the sweeping robot can correspond to the “sweeping” capability.

[0148] If the voice text only includes the device type information, the device type information cannot determine a unique relay device, but can only determine at least two relay terminal devices. Then, the final relay device is determined from the relay terminal devices according to the relationship between the relay dimension information of the relay terminal devices and the initiation dimension information of the initiation relay terminal device. For example, the voice text only includes "television", and according to the voice text, multiple relay terminal devices can be determined. Then, the final relay television is determined according to the relay dimension information of the multiple televisions and the initiation dimension information of the initiation relay terminal device. That is, even if the voice signal input by the user does not include an explicit relay device, the system can automatically determine the final relay device to achieve the purpose of content service relay, without the need for the user to manually select the relay device, thereby improving the user experience.

[0149] In some scenarios, there can be only one relay terminal device, and then there is no need to compare the relay dimension information of the relay terminal device and the initiation dimension information of the initiation relay terminal device, and the relay terminal device is directly determined as the final relay device. In other scenarios, there can be multiple relay devices determined according to the relay dimension information of the relay terminal device and the initiation dimension information of the initiation relay terminal device. At this time, the most frequently used relay device can be determined as the final relay device according to the user's usage frequency. Alternatively, a dialog box is popped up for the user to select the required final relay device from the relay devices. Alternatively, the required final relay device is selected from the relay devices according to the user's location, wherein the location can refer to the current location, or the location to which the user will move according to the user's moving path. In some embodiments, the user's location information can be determined by one or more sensors on the device.

[0150] It should be noted that the relay device is the terminal device that finally relays the content service from the current terminal device, and the relay terminal device is the device matched with the voice text. Meanwhile, it is assumed here that the relay terminal device is the terminal device that can perform the operation corresponding to the application data of the application running in the foreground of the current terminal device.

[0151] In some embodiments, the dimension information of the device includes a first dimension, a second dimension, and a third dimension. The first dimension includes the second dimension, that is, the second dimension is within the range of the first dimension. The second dimension includes the third dimension, that is, the third dimension is within the range of the second dimension, and the third dimension is also within the range of the first dimension. According to the relationship between the relay dimension information of the relay terminal device and the initiation dimension information of the initiation relay terminal device, the relay device can be determined in the following situations:

[0152] If there is a baton terminal device whose first dimension, second dimension and third dimension respectively match the first dimension, second dimension and third dimension of the initiating baton terminal device, i.e., the baton terminal dimension information of the baton terminal device completely matches the initiating dimension information of the initiating baton terminal device, the baton terminal device is determined as the final baton device.

[0153] If there is no baton terminal device whose first dimension, second dimension and third dimension respectively match the first dimension, second dimension and third dimension of the initiating baton terminal device, but there is a baton terminal device whose first dimension and second dimension match the first dimension and second dimension of the initiating baton device, i.e., the baton terminal device and the initiating baton terminal device are only in the same second dimension range, but in different third dimension ranges. But there is no baton terminal device in the same third dimension range as the initiating baton terminal device, the baton terminal device is determined as the final baton device.

[0154] If there is no baton terminal device whose first dimension, second dimension and third dimension respectively match the first dimension, second dimension and third dimension of the initiating baton terminal device, and there is no baton terminal device whose first dimension and second dimension respectively match the first dimension and second dimension of the initiating baton terminal device, but there is a baton terminal device whose first dimension matches the first dimension of the initiating baton terminal device, i.e., there is only a baton terminal device in the same first dimension range, the baton terminal device is determined as the final baton device.

[0155] For example, as shown in the dimension scenario diagram of Figure 16 , Figure 19 , Figure 20 , Figure 21 The first dimension is an account, the second dimension is a room attribute identifier, which can be a user's house in different regions, and the third dimension is a room area identifier, which can be different rooms in the user's different houses. The account is the account ID registered by the user in the unified baton service application. The user can create multiple homes under each account dimension, for example, the user has multiple houses, which are located in A, B and C respectively. The houses in A, B and C are the second dimensions in the framework diagram. The third dimension is multiple physical spaces established based on the home dimension, such as master bedroom, living room, secondary bedroom, kitchen, etc.

[0156] In the scenario shown in Figure 16 , TV A and entrance screen A are located in the master bedroom of home A, entrance screen B is located in the living room of home A, and screen-equipped refrigerator A is located in the kitchen of home A. TV B is located in the living room of home B. Entrance screen C is located in the living room of home C, and screen-equipped refrigerator B is located in the kitchen of home C. Home A, home B and home C all belong to account A.

[0157] If the voice text includes device information, for example, includes "relay to the TV B in the living room of home B", the voice text includes explicit device information, then no matter which current terminal device is, the device information points to a unique relay device, then the final relay device is the TV B in the living room of home B.

[0158] The signaling processing timing of the above embodiment is as shown in the signaling diagram, Figure 17 The initiating relay terminal device receives the voice signal input by the user. The initiating relay terminal device sends the voice signal and the service information to the voice server. The voice server identifies from the voice signal that the voice text includes device information, which points to a unique relay device, the TV B in the living room of home B. The voice server sends the device information and the service information to the AIOT server. The AIOT server finds the device ID according to the device information, and sends the service information to the TV B in the living room of home B according to the device ID, so that the TV B in the living room of home B performs the corresponding relay operation according to the service information.

[0159] In Figure 16 the scenario shown in the figure, if the voice text only includes device type information, for example, the user inputs the voice signal to the TV A, and the voice text identified from the voice signal includes "relay to the porch screen", at this time, the voice text does not include explicit device information, but only includes device type information "porch screen". The device type information "porch screen" points to multiple porch screens. Figure 16 In the scenario shown in the figure, there is a porch screen A in the master bedroom of home A, and the porch screen A and the TV A are located in the same account, the same home and the same physical space, that is, the first dimension, the second dimension and the third dimension of the porch screen A respectively match the first dimension, the second dimension and the third dimension of the TV A. Therefore, the porch screen A is determined as the final relay device, and finally the purpose of content service relay is achieved, and the content service relay can be relay playing video.

[0160] The signaling processing timing of the above embodiment is as shown in the signaling diagram, Figure 18 The initiating relay terminal receives the voice signal input by the user. The initiating relay terminal device sends the voice signal to the voice server. The voice server identifies from the voice signal that the voice text includes device type information, which points to multiple relay devices. The voice server sends the device information and the service information of the multiple devices to the AIOT server. The AIOT server matches the dimension information of the multiple devices with the dimension information of the initiating relay terminal device, and obtains the porch screen as the final relay device after matching. After finding the device ID of the porch screen A, the AIOT server sends the service information to the porch screen A according to the device ID, so that the porch screen A performs the corresponding relay operation according to the service information.

[0161] In Figure 19In the scene shown, TV A is located in the master bedroom of home A, entryway screen B is located in the living room of home A, and refrigerator A with a screen is located in the kitchen of home A. TV B is located in the living room of home B. Entryway screen C is located in the living room of home C, and refrigerator B with a screen is located in the kitchen of home C.

[0162] If the voice text only includes device type information, for example, if a user inputs a voice signal to TV A, and the voice text recognized from that voice signal includes "relay to the entryway screen," then this voice text does not include explicit device information, only the device type information "entryway screen." This device type information "entryway screen" refers to multiple entryway screens. TV A is located in the master bedroom of home A, while... Figure 19 In the scenario shown, there is no entryway screen in the master bedroom of house A. Therefore, there is no relay terminal device in the current scenario that shares the same account, company, and physical space as TV A. We then search the upper dimension of the current scenario for a relay terminal device that shares the same account and company as TV A but is not in the same physical space. We find that there is an entryway screen B in the living room of house A. Entryway screen B shares the same account and company as TV A but is not in the same physical space. Therefore, entryway screen B is determined as the final relay device in the current scenario.

[0163] The signaling processing timing of the above embodiments and Figure 18 The signaling processing timing is similar in the embodiments shown, and will not be described again in this embodiment.

[0164] exist Figure 20 In the scene shown, TV A is located in the master bedroom of home A, and refrigerator A with a screen is located in the kitchen of home A. TV B is located in the living room of home B. The entryway screen C is located in the living room of home C, and refrigerator B with a screen is located in the kitchen of home C.

[0165] If the voice text only includes device type information, for example, if a user inputs a voice signal to TV A, and the voice text recognized from that voice signal includes "relay to the entryway screen," then this voice text does not include explicit device information, only the device type information "entryway screen." This device type information "entryway screen" refers to multiple entryway screens. TV A is located in the master bedroom of home A, while... Figure 20In the shown scenario, there is no hall screen in the master bedroom of the house A. Therefore, there is no relay terminal device in the same account, the same house and the same physical space as the TV A in the current scenario. Then, in the current scenario, a search is made in the upper dimension to find whether there is a relay terminal device in the same account and the same house but not in the same physical space as the TV A. In the current scenario, there is no hall screen in the master bedroom, the living room and the kitchen of the house A. Therefore, there is also no relay terminal device in the same account, the same house but not in the same physical space as the TV A in the current scenario. Then, in the current scenario, a search is made in the upper dimension to find whether there is a relay terminal device in the same account but not in the same house as the TV A. In the current scenario, a hall screen C is found in the house C in the same account, the hall screen C is in the same account but not in the same house as the TV A, and also not in the same physical space. Therefore, in the current scenario, the hall screen C is determined as the final relay device.

[0166] The signaling processing timing of the above embodiment is similar to Figure 18 The signaling processing timing of the above embodiment is similar to

[0167] In some embodiments, if there is no relay terminal device in the first dimension and the first dimension of the current terminal device matches, but at the same time there is a relay terminal device in the second dimension and the second dimension of the current device matches, or there is a relay terminal device in the third dimension and the third dimension of the current device matches, a prompt information is generated to make the user add the relay terminal device to the first dimension, and then determine the relay terminal device as the relay device according to the prompt information.

[0168] For example, in Figure 21 In the shown scenario, the TV A is located in the master bedroom of the house A, and the refrigerator with screen A is located in the kitchen of the house A. The TV B is located in the living room of the house B. The refrigerator with screen B is located in the kitchen of the house C. The hall screen D is located in the living room of the house A, but the TV A is in the account A, and the hall screen D is in the account B or the hall screen D is not added to any account. In the current scenario, the first dimension of the hall screen D does not match the first dimension of the TV A, but the hall screen D matches the voice text and belongs to the relay terminal device. At this time, a prompt information indicating that the hall screen D is added to the account A can be popped up on the display of the current terminal device. The user can operate according to the prompt information, and finally add the hall screen D to the account A, so that the first dimension of the hall screen D matches the first dimension of the TV A. At the same time, since the hall screen D is located in the living room of the house A, after the hall screen D is added to the account A, the hall screen D and the TV A are in the same account and the same house. That is, finally, the first dimension and the second dimension of the hall screen D respectively match the first dimension and the second dimension of the TV A.

[0169] The signaling processing timing of the above embodiment is similar to Figure 18The signaling processing timing of the illustrated embodiment is similar, and the signaling processing timing of the present embodiment is not described in detail.

[0170] After the above device matching process, if there is a relay terminal device, a relay prompt is generated to prompt the user that there is a matched relay terminal device. If there is no relay terminal device, no relay prompt can be generated. If the initiating relay terminal device has a display, a relay button control can be displayed on the display, for example, as shown in the user interface diagram. Figure 7 The user can trigger the content service relay process by clicking the relay button control. The relay prompt can also be in the form of a voice broadcast.

[0171] It should be noted that in some embodiments, if it is determined that there is a relay terminal device, a relay button is displayed on the user interface. When the initiating relay terminal device receives a user input operation of clicking the relay button, a query instruction is generated. The initiating relay terminal device sends the query instruction to the AIOT server, and the AIOT server queries all the relayable terminal devices in the system according to the query instruction, and then feeds back all the relayable terminal devices to the initiating relay terminal device. Then, the information of all the relayable terminal devices is displayed on the user interface of the relay terminal device, that is, the relay terminal device list is displayed.

[0172] In some embodiments, the user can set to add a display relay button in a specific page of a specific application. For example, in the video playback page of the video playback application, if there is a relay terminal device, a relay button is displayed. In the teaching related page of the learning application, if there is a relay terminal device, a relay button is displayed, while in the self-learning related page of the learning application, no relay button is set to be displayed if there is a relay terminal device.

[0173] In some embodiments, if there is no relay terminal device, a relay guide button control can be displayed on the display. The user can trigger the display of content service relay guide information on the display by clicking the relay guide button control. Here, the content service relay guide information can be introduction information of the content service relay, or information guiding the user to perform the next related operation of the content service relay. In this way, if there is no relay terminal device, even if the relay button control is not displayed, content related to the content service relay can be displayed on the user interface of the first application, thereby improving the user experience.

[0174] In the above embodiments, the content service relay process is triggered by clicking the relay button control. In some embodiments, the content service relay process can be triggered by clicking the relay guide button control. Figure 8In the application scenario shown, the user inputs the relay operation by selecting the target device display device 100-2, i.e., the relay terminal device. After the display device 100-1 receives the relay operation input by the user, a relay instruction is generated, which carries application data and media data of the video playing application, and also carries the device ID of the relay terminal device. The initiating relay terminal device sends the relay instruction to the display device 100-2 through the AIOT server according to the device ID. After the display device 100-2 receives the relay instruction, it continues to play the video according to the application data and media data of the video playing application.

[0175] In some embodiments, if the application currently running in the foreground of the initiating relay terminal device (hereinafter referred to as the first application) is a video application, the corresponding relay button control is a video relay button control. If the first application is an audio application, the corresponding relay button control is an audio relay button control. If the first application is a picture display application, the corresponding relay button control is a picture display relay button control. The video relay button control, the audio relay button control, and the picture display relay button control can be displayed in different forms. In this way, the user can conveniently view the type of application that currently needs to be relayed.

[0176] For example, as shown in the user interface of the music application, Figure 22 If there is a relay terminal device, an audio relay button control is displayed on the user interface of the audio application. The audio relay button control can be displayed in the form of a music symbol icon. Similarly, the video relay button control can be displayed in the form of a video icon, and the picture display relay button control can be displayed in the form of a picture symbol icon.

[0177] The signaling processing timing of the above-mentioned embodiments is shown in the signaling diagram as shown in Figure 23 When the initiating relay terminal device is in the user interface of the audio application, the initiating relay terminal device sends a relay terminal device query instruction to the AIOT server through the unified relay service application. The query instruction carries the application data of the audio application. The AIOT server queries whether there is a terminal device that can execute the same or similar application data according to the application data of the audio application, and feeds back the query result to the initiating relay terminal device. If the query result is that there is a terminal device that can execute the same or similar application data, the audio relay button control is displayed on the user interface of the audio application of the initiating relay terminal device. Otherwise, the audio relay button control is not displayed. It should be noted that the audio application on the relay terminal device can be a different application from the application on the initiating relay terminal device, but it can be determined that the two applications have the same function according to the application data.

[0178] In some embodiments, the name of the second application on the relay terminal device can also be displayed on the user interface of the first application. The second application is an application that performs the operation corresponding to the application data of the first application. For example, as shown inFigure 24 The user interface shown depicts NetEase Cloud Music as the primary application on the currently displayed device. Devices with installed audio applications are considered relay terminal devices. If the audio application installed on the relay terminal device is QQ Music, the word "QQ Music" will be displayed near the relay button control. If the audio application installed on the relay terminal device is Migu Music, the word "Migu Music" will be displayed near the relay button control. If both QQ Music and Migu Music are installed on the relay terminal device, both "QQ Music" and "Migu Music" will be displayed simultaneously. If the user selects "QQ Music," the relay operation will be performed by QQ Music on the relay terminal device. If the user selects "Migu Music," the relay operation will be performed by Migu Music on the relay terminal device.

[0179] In some embodiments, if multiple relay terminal devices are available to perform the relay operation, a "plus" sign can be superimposed on the relay button control to indicate to the user that multiple relay terminal devices are currently available. When the user clicks the relay button control, a list of relay terminal devices can be displayed to one side of the button control. This list includes device names, and the name of the application capable of performing the relay operation is displayed next to each device name.

[0180] For example, such as Figure 25 The user interface shown, when NetEase Cloud Music is the first application on the currently displayed device, includes multiple devices with audio applications installed: Device A, Device B, and Device C. Next to the device name, the names of the applications that can perform the relay operation are displayed: Kugou Music, Migu Music, and QQ Music. Furthermore, if multiple applications that can perform the relay operation exist on a single relay terminal device, they are all displayed after the device name.

[0181] In some embodiments, a thumbnail of the user interface of the second application may also be displayed on the user interface of the first application. For example... Figure 26 The user interface shown depicts NetEase Cloud Music as the primary application on the currently displayed device and QQ Music as the application installed on the relay terminal device. A thumbnail of the QQ Music user interface is then displayed next to the relay button control. This allows users to view the user interface of the second application after the relay, making it easier for them to decide whether to click the relay button control to relay content services, further enhancing the user experience.

[0182] It should be noted that if the audio application on the initiating terminal device is different from the audio application on the relay terminal device, the relay operation can be performed according to the name, singer, time, and other information of the audio media data. For example, the audio application on the initiating terminal device is NetEase Cloud Music, and the song being played is "Invisible Wings". When relayed to the relay terminal device, the information of "Invisible Wings", such as the name, singer "Zhang Aoshan", and time "2006" can be used to search for the corresponding song on QQ Music, the audio application on the relay terminal device. According to the current playing time of the song on the initiating terminal device, the song can be continuously played.

[0183] In terms of displaying the thumbnail of the user interface of the second application, the AIOT server can query the application on the relay terminal device and automatically send a screenshot request to the relay terminal device. After receiving the screenshot request, the relay terminal device sends the previously saved screenshot of the user interface of the second application to the AIOT server. The AIOT server feeds back the screenshot of the user interface of the second application to the initiating terminal device. After receiving the screenshot of the user interface of the second application, the initiating terminal device displays a thumbnail of the screenshot of the user interface of the second application on the user interface of the first application.

[0184] In some embodiments, the display device and the determined relay terminal device need to be under the same account. Adding terminal devices to the same account is a prior art, and will not be described in detail in this application. During the content service relay process, only devices under the same account can perform the content service relay process with each other. In this way, interference from devices not under the same account can be avoided.

[0185] The implementation process of displaying the relay button control on the user interface of the first application in the above embodiments is as follows Figure 27The flow shown specifically includes: C1, after the service application running in the foreground of the terminal device enters the relay service scene (entering the relay service scene can be that the user interface enters a specific page, for example, the video playing application enters the video playing page from the main page), the service application carries the relay scene identifier thereof to request binding the unified relay service application, so that the unified relay service application knows the service application currently in the relay service scene. C2, after the unified relay service application receives the binding request of the service application, the unified relay service application sends a query request (carrying the service application data) to the AIOT server (which can be a device capability storage system). The AIOT server queries the terminal device supporting the same service under the same account according to the service application data carried in the query request, that is, queries the relay terminal device. C3, the server feeds back the device information and the application information executable for relay of the relay terminal device found to the unified relay service application. C4, the unified relay service application returns the device information and the application information executable for relay of the relay terminal device found to the service application. C5, the service application performs different UI display on the user interface of the service application according to the device information and the application information executable for relay of the relay terminal device.

[0186] For example, if the service application of the initiating relay terminal device in the relay service scene is a video application, and the application information of the service application of the found relay terminal device is also a video application, the video relay control is displayed. If the service application of the initiating relay terminal device in the relay service scene is an audio application, and the application information of the service application of the found relay terminal device is also an audio application, the audio relay control is displayed.

[0187] If the user clicks the relay button control in the above embodiment, the user interface shown in Figure 7 The user interface shown in Figure 9 The user interface shown in Figure 9 The names of at least two relay terminal devices are listed in the user interface shown, and if there is only one relayable terminal device, the relay terminal device list only displays the information of the one relayable terminal device. The user can select one of the relay terminal devices, that is, determine the relay terminal device from the multiple relay terminal devices. The initiating relay terminal generates a service relay instruction according to the operation of the user. At the same time, the service relay instruction is sent to the determined relay terminal device. The service relay instruction carries content service relay data (corresponding to the application data of the first application running in the foreground of the initiating relay terminal device). After the relay terminal receives the content service relay data, the content service relay operation corresponding to the content service relay data can be executed.

[0188] If the current service relay instruction is an instruction sent by user misoperation, for example, the user mis-clicks the relay button or in Figure 9When a user clicks a relay terminal device by mistake, the relay terminal device initiates a service relay instruction to send the selected relay terminal device, and the current relay terminal device is the relay terminal device. After the relay terminal device receives the service relay instruction, the relay terminal device can execute the content service relay operation corresponding to the content service relay data carried by the service relay instruction.

[0189] When a user finds that the current service relay instruction is sent by mistake, the user can continue to input a relay interruption operation, and the device generates a relay interruption instruction according to the interruption operation. The relay interruption instruction carries service information and a device ID of the relay terminal device, and the AIOT server sends the interruption instruction and the service information to the corresponding relay terminal device according to the device ID. According to the relay interruption instruction, the relay terminal device does not execute the content service relay operation corresponding to the content service relay data. It should be noted that the relay terminal device receives the relay interruption instruction before the relay terminal device executes the content service relay operation corresponding to the content service relay data. In addition, it should be noted that the application for executing the relay operation installed on the relay terminal device can be in an open state or an unopened state. If the relay terminal device does not receive the interruption instruction after receiving the service relay instruction, the relay terminal device opens the unopened relay application according to the relay service instruction, so as to execute the corresponding relay operation.

[0190] In some embodiments, the relay interruption instruction can be an instruction generated according to the interruption operation input by the user on the relay terminal device. Specifically, after the relay device receives the relay service instruction, the relay device can display a relay dialog box on the relay device, and the relay dialog box includes a confirm relay button control and a cancel relay button control. If the user clicks the confirm relay button control, the content service relay operation corresponding to the content service relay data is executed. If the user clicks the cancel relay button control, the relay interruption instruction is generated according to the user operation, and the relay device does not execute the content service relay operation corresponding to the content service relay data according to the relay interruption instruction.

[0191] The relay terminal device can also display a relay countdown dialog box, and the relay countdown dialog box displays a countdown time confirm relay button control and a cancel relay button control. The user clicks the confirm relay button control or the cancel relay button control to input a confirm relay operation or a cancel relay operation, and the relay terminal device correspondingly generates a confirm relay instruction or a relay interruption instruction. If the time exceeds the countdown time, the user does not click any button control, and the relay service instruction can be automatically generated.

[0192] In some embodiments, the relay interruption instruction can also be generated according to a user inputted interruption operation on the initiating relay terminal device. Specifically, after receiving the relay service instruction, the relay device displays a relay dialog box on the initiating relay terminal device before executing the content service relay operation corresponding to the content service relay data (the trigger condition for displaying the relay dialog box can be that the initiating relay terminal device receives the relay service instruction, the trigger condition for displaying the relay dialog box can also be that the initiating relay terminal device sends the relay service instruction). The relay dialog box displays a cancel relay button control and a confirm relay button control. If the user clicks the cancel relay button control, the relay interruption instruction is generated according to the user operation, and the relay interruption instruction is sent to the relay terminal device. If the user clicks the confirm relay button control, a confirm relay instruction is generated according to the user operation. It should be noted that the relay dialog box can display a countdown, for example, if the user does not click any button control within 5 seconds, the relay dialog box is cancelled. The interruption operation in the embodiments of the present application is assumed to be performed before the relay terminal device executes the relay operation.

[0193] The initiating relay terminal device can be provided with a sound collector, which receives a user inputted voice signal. For example, the voice text "cancel relay" is recognized from the user inputted voice signal. The initiating relay terminal device generates a relay interruption instruction carrying service information and a device ID of the relay terminal device according to the voice text, and the AIOT server sends the interruption instruction and the service information to the corresponding relay terminal device according to the device ID. The relay terminal device does not execute the content service relay operation corresponding to the content service relay data according to the relay interruption instruction.

[0194] In some embodiments, the relay terminal device also feeds back the relay result to the initiating relay terminal device after not executing the content service relay operation corresponding to the content service relay data. The initiating relay terminal device can continue to execute the content service relay operation corresponding to the content service relay data according to the feedback relay result, that is, continue to execute the operation before the relay. It should be noted that after the initiating relay terminal device sends the service relay instruction to the relay terminal device, the initiating relay terminal device can interrupt the previous operation or can not interrupt the previous operation.

[0195] For example, the display device is playing a video before the relay. When the service relay instruction is sent to the relay terminal device, the display device can continue playing the video or can interrupt playing the video. If the display device interrupts playing the video, after receiving the relay result (the content service relay operation corresponding to the content service relay data is not performed) fed back by the relay terminal device, the display device can continue playing the video. A dialog box can also be popped up on the display of the initiating relay terminal device to prompt a user whether to continue playing the video on the initiating relay terminal device. The dialog box can be provided with a "Yes" button control and a "No" button control. If the user selects the "Yes" button control, the display device continues playing the video according to the operation input by the user. If the user selects the "No" button control, the display device does not continue playing the video according to the operation input by the user.

[0196] The process of the service relay interruption will be described below in combination with a service relay interruption implementation process diagram as shown in Figure 28 First, after the system is started, the unified relay service application compiles the relay capabilities of each service application in the entire system, and reports the relay capabilities of each service application to the AIOT server.

[0197] D1 to D8 are similar to B1 to B8 in the foregoing embodiment, and the process of D1 to D8 will not be described again in this example.

[0198] In the embodiment of the present application, the relay activity can be interrupted before the relay terminal device performs the corresponding action. Specifically, D9 to D10 in Figure 29 are processes of interrupting the service relay on the relay terminal device.

[0199] D9: Before the process of D8 is performed, the user inputs an interruption instruction on the relay terminal device, and the relay terminal device does not perform the corresponding action, that is, does not perform the process of D8. The unified relay service application of the relay terminal device sends the assembled data after the interruption to the AIOT server. The assembled data includes interruption result information, service information, and device information of the initiating relay terminal device.

[0200] D10: The AIOT sends the interruption result information and the service information to the unified relay service application of the initiating relay terminal device according to the device information. Then, according to the interruption result information and the service information, the AIOT displays prompt information of the service relay interruption on the initiating relay terminal device. If other applications have been switched on the initiating relay terminal device, the AIOT can also display prompt information of the service relay interruption related to the interruption result information and the service information on the initiating relay terminal device.

[0201] D11 to D12 in Figure 19 are processes of interrupting the service relay on the initiating relay terminal device.

[0202] D11: The user triggers the relay break on the initiating relay terminal device by breaking the relay button or voice input, and sends the relay break instruction to the AIOT server. The relay break instruction carries the service information (application service ID) and the device information of the relay terminal device.

[0203] D12: The AIOT server finds the relay terminal device according to the device information, and sends the relay break instruction to the relay terminal device. The unified relay service application of the relay terminal device queries the corresponding application service according to the application service relay ID, and cancels the start of the application service, so that the relay terminal device does not perform the corresponding relay action.

[0204] After the relay terminal device starts to perform the corresponding operation according to the content service relay data, the user can end the relay at any time. For example, the user can input an end relay instruction on the relay terminal device, and the relay terminal device stops performing the corresponding operation according to the content service relay data after receiving the end relay instruction. Or the user inputs an end relay instruction on the initiating relay terminal device, and the initiating relay terminal device sends the end relay instruction to the relay terminal device. The relay terminal device stops performing the corresponding operation according to the content service relay data after receiving the end relay instruction from the initiating relay terminal device. It should be noted that the user can input the end relay instruction in the form of voice input, button selection input, gesture input, etc., and the application does not limit this.

[0205] After the relay terminal device ends the operation according to the content service relay data, that is, after the content service relay operation ends, the relay terminal device can generate a relay end feedback result and send it to the initiating relay terminal device to realize the content service relay return operation. The relay end feedback result includes execution progress data of the content service relay operation performed by the relay terminal device, and the initiating relay terminal device can continue to perform the content service relay operation according to the execution progress data. For example, the media content can be played on the initiating relay terminal device according to the playing progress of the media content on the relay terminal device.

[0206] In some embodiments, if the initiating relay terminal device is currently in the first state, the initiating relay terminal device can continue to perform the content service relay operation according to the execution progress data, and the initiating relay terminal device continues to perform the content service relay operation according to the execution progress data.

[0207] In some embodiments, if the initiating relay terminal device is currently in the second state, the initiating relay terminal device cannot continue to perform the content service relay operation according to the execution progress data, and the initiating relay terminal device does not continue to perform the content service relay operation according to the execution progress data.

[0208] The signaling processing timing of the above embodiment is a signaling diagram as shown in Figure 29

[0209] Step one, the initiating baton terminal device sends a baton service instruction (carrying the service information of the application currently running in the foreground of the initiating baton terminal device and the device information of the baton terminal device) to the AIOT server.

[0210] Step two, the AIOT server sends the service information and the baton service instruction to the corresponding baton terminal device according to the device information.

[0211] Step three, the baton terminal device receives the service information and executes the corresponding baton operation according to the service information. The service information can include basic information of the service and execution progress data of the service on the baton terminal device.

[0212] For example, if the application currently running in the foreground of the initiating baton terminal device is a media play related application, the service information can include basic information such as the name of the media data, and also includes the play time point of the media on the initiating baton terminal device when the baton is performed. The baton terminal device continues to play the media according to the above basic information and the play time point.

[0213] Step four, after the baton terminal device receives the instruction to interrupt the baton operation, it no longer continues to execute the corresponding baton operation. After interrupting the execution of the baton operation, the baton terminal device generates a baton end feedback result. The baton end feedback result can include basic information of the service and execution progress data of the service when the baton terminal device interrupts the execution of the baton operation.

[0214] For example, the baton end feedback result includes service information and device information of the initiating baton terminal device. At this time, the service information can also include the name of the media data, and also includes the play time point of the media on the baton terminal device when the baton terminal device interrupts the execution of the baton operation.

[0215] Step five, the AIOT server receives the baton end feedback result sent by the baton terminal device, and sends the baton end feedback result to the initiating baton terminal device.

[0216] Step six, if the initiating baton terminal device is in the first state when receiving the baton end feedback result, it continues to execute the baton operation according to the execution progress data, wherein when the initiating baton terminal device is in the first state, the initiating baton terminal device can continue to execute the content service baton operation according to the execution progress data.

[0217] ​Step seven, if the relay end feedback result is received, the initiating relay terminal device is in the second state, and the content service relay operation is not continued according to the execution progress data, wherein, when the initiating relay terminal device is in the second state, the initiating relay terminal device cannot continue to execute the content service relay operation according to the execution progress data.

[0218] In this way, in the content service relay return scenario, the current state of the initiating relay terminal device can be fully considered, the normal use of the initiating relay terminal device is avoided from being affected, and therefore the user experience is improved. It should be noted that the initiating relay terminal device and the relay terminal device can directly transmit instructions and data, or the instructions and data can be forwarded through a server, and the application does not make any limitation.

[0219] When the initiating relay terminal device is in the first state, the following different situations can be provided:

[0220] The first situation is that when the relay end feedback result is received, the initiating relay terminal device is currently in the first business content scenario, wherein the first business content scenario is the business content scenario in which the initiating relay terminal device is before the relay.

[0221] For example, before the relay, the application running in the foreground of the initiating relay terminal device is a video application. When the relay end feedback result is received, the application running in the foreground of the initiating relay terminal device is still the video application. The initiating relay terminal device can continue to execute the content service relay operation according to the execution progress data, that is, the playing of the media asset content is continued.

[0222] It should be noted that if the application running in the foreground of the initiating relay terminal device is a video application, but the media asset content played by the video application of the initiating relay terminal device is not the media asset content played before the relay. The media asset content played before the relay can be played after the media asset content currently played by the initiating relay terminal device is played.

[0223] The second situation is that when the relay end feedback result is received, the initiating relay terminal device is not currently in any business content scenario. For example, the initiating relay terminal device can have no application running in the foreground. The initiating relay terminal device can continue to execute the content service relay operation according to the execution progress data.

[0224] When the initiating relay terminal device is in the second state, the following different situations can be provided:

[0225] The first case is that the initiating relay terminal device is currently in a second service content scenario when the relay end feedback result is received, the second service content scenario is different from the first service content scenario, and the first service content scenario is the service content scenario in which the initiating relay terminal device is before the relay. For example, before the relay, the application running in the foreground on the initiating relay terminal device is a video application. When the relay end feedback result is received, the application running in the foreground on the initiating relay terminal device is an audio application. Then the initiating relay terminal device cannot continue to perform the content service relay operation according to the execution progress data, that is, the initiating relay terminal device cannot continue to play the media asset content.

[0226] The second case is that the initiating relay terminal device is currently in an offline state. At this time, the initiating relay terminal device also cannot continue to perform the content service relay operation according to the execution progress data, that is, the initiating relay terminal device cannot continue to play the media asset content. It should be noted that the relay end feedback result can be saved on the server, and when the initiating relay terminal device enters an online state from an offline state, a state change request can be sent to the server. After the server receives the state change request of the initiating relay terminal device and determines that the state of the initiating relay terminal device is an online state, the relay end feedback result is sent to the initiating relay terminal device. Further, if the initiating relay terminal device enters an online state, is not in any service content scenario or is in the first service content scenario, the content service relay operation can be continued according to the execution progress data.

[0227] In some embodiments, when the relay end feedback result is received, if the initiating relay terminal device is currently in the first state in the above embodiments, but it is determined according to the execution progress data that the relay terminal device has already performed the corresponding content service relay operation according to the content service relay data. Then the initiating relay terminal device also does not continue to perform the content service relay operation according to the execution progress data.

[0228] In some embodiments, if the initiating relay terminal device is in the first state and the relay end feedback result is received, the initiating relay terminal device can continue to perform the content service relay operation according to the execution progress data. Then the user can perceive the change in the operation on the initiating relay terminal device. For example, the media asset content is continued to be played according to the execution progress data, and then the user can watch the continued playing of the media asset content on the initiating relay terminal device. However, if the initiating relay terminal device is in the second state and the relay end feedback result is received, the initiating relay terminal device cannot continue to perform the content service relay operation according to the execution progress data. Then the user cannot perceive the change in the operation on the initiating relay terminal device.

[0229] At this time, a return failure prompt can be generated on the initiating terminal device to prompt the user that the current content service relay return fails. For example, a dialog box including the prompt "content service relay return fails" can be popped up on the relay terminal device and / or the initiating terminal device. The relay terminal device and / or the initiating terminal device can also voice broadcast the return failure prompt, for example, voice broadcast the prompt "content service relay return fails", thereby further improving the user's interactive experience.

[0230] Based on the above embodiments, the application further provides a matching method of a terminal device, applied to a content service relay scenario, as shown in the method flowchart, the method comprises the following steps: Figure 30

[0231] Step S101, the current terminal device (initiating terminal device) receives a voice signal from a sound collector, and identifies a voice text from the voice signal (identifies the voice text through a voice server).

[0232] Step S102, if the voice text includes device information, determine a relay device according to the device information, wherein the device information points to a unique relay device.

[0233] Step S103, if the voice text includes device type information, determine a relayable terminal device according to the device type information, and determine a relay device from the relayable terminal device according to the relationship between the relay dimension information of the relayable terminal device and the initiating dimension information of the current terminal device, wherein the device type information points to at least two relayable terminal devices.

[0234] Step S104, establish a relay relationship between the determined relay device and the current terminal device.

[0235] In some embodiments, the dimension information of the device includes a first dimension, a second dimension, and a third dimension, the first dimension includes the second dimension, and the second dimension includes the third dimension. The first dimension is an account, the second dimension is a room attribute identifier, for example, it can be a user's house in different regions, and the third dimension is a room area identifier, for example, it can be different rooms in the user's different houses. The account is an account ID registered by the user in a unified relay service application. The user can create multiple homes under each account dimension, for example, the user has multiple houses, respectively located in A, B and C. The houses in A, B and C are the second dimension in the framework diagram. The third dimension is multiple physical spaces established based on the home dimension, for example, master bedroom, living room, secondary bedroom, kitchen, etc. Then, the specific steps of determining the relay device according to the relationship between the relay dimension information of the relayable terminal device and the second space of the terminal device can be:

[0236] ​If there is a relayable terminal device whose first dimension, second dimension and third dimension match the first dimension, second dimension and third dimension of the terminal device respectively, the relayable terminal device is determined as the relay device;

[0237] Or, if there is no relayable terminal device whose first dimension, second dimension and third dimension match the first dimension, second dimension and third dimension of the terminal device respectively, and there is a relayable terminal device whose first dimension and second dimension match the first dimension and second dimension of the terminal device respectively, the relayable terminal device is determined as the relay device.

[0238] Or, if there is no relayable terminal device whose first dimension, second dimension and third dimension match the first dimension, second dimension and third dimension of the terminal device respectively, and there is no relayable terminal device whose first dimension and second dimension match the first dimension and second dimension of the terminal device respectively, and there is a relayable terminal device whose first dimension matches the first dimension of the terminal device, the relayable terminal device is determined as the relay device.

[0239] That is, in determining the relay device, first, the relayable terminal device whose first dimension, second dimension and third dimension match the first dimension, second dimension and third dimension of the current terminal device respectively is considered. If there is no relayable terminal device whose three dimensions match, then the relayable terminal device whose first dimension and second dimension match the first dimension and second dimension of the current terminal device respectively is further considered. If there is no relayable terminal device whose first dimension and second dimension match the first dimension and second dimension of the current terminal device respectively, then the relayable terminal device whose first dimension matches the first dimension of the current terminal device is finally considered. If there is no relayable terminal device whose first dimension matches the first dimension of the current terminal device, it means that there is no terminal device that can be relayed in the system. In this way, the relay device that matches the same room area identifier or the same room attribute identifier as the current device as much as possible can be matched.

[0240] The same or similar contents among various embodiments of the present application can be referred to each other, and the related embodiments will not be described again.

[0241] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for part or all of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present application.

[0242] For the sake of explanation, the foregoing descriptions have been presented in terms of specific embodiments. However, it is to be appreciated that specific embodiments described herein are not intended to limit the scope of the present application, which is defined with reference to the following claims. Various modifications and changes can be made thereto by those skilled in the art which fall within the scope of the present application as defined by the following claims. The embodiments were chosen and described in order to explain the principles of the application and the practical application and to enable others skilled in the art to understand for implementing various embodiments and with various modifications as are suited to the particular use contemplated.

Claims

1. A terminal device, characterized by comprising: The method comprises the following steps: a sound collector configured to collect a voice signal input by a user; a controller configured to: receive the voice signal from the sound collector, and identify voice text from the voice signal; when the voice text comprises device information, determine a relay device according to the device information, and establish a relay relationship between the terminal device and the relay device, wherein the device information points to a unique relay device; when the voice text comprises device type information, determine a relayable terminal device according to the device type information, and determine a relay device from the relayable terminal device according to the relationship between the relay dimension information of the relayable terminal device and the initiation dimension information of the terminal device, and establish a relay relationship between the terminal device and the relay device, wherein the device type information points to at least two relayable terminal devices, and the dimension information of a device comprises a first dimension, a second dimension, and a third dimension, the first dimension is a user account, the second dimension is a room attribute identifier, and the third dimension is a room area identifier; if there is no relayable terminal device matching the first dimension of the terminal device, but there is a relayable terminal device matching the second dimension of the terminal device, or there is a relayable terminal device matching the third dimension of the terminal device, generate prompt information to enable the user to add the relayable terminal device to the first dimension according to the prompt information, and determine the relayable terminal device as the relay device.

2. The terminal device according to claim 1, characterized by The specific steps for determining a relay device according to the relationship between the relay dimension information of the relayable terminal device and the initiation dimension information of the terminal device are as follows: when there are relayable terminal devices respectively matching the first dimension, the second dimension, and the third dimension of the terminal device, the relayable terminal devices are determined as the relay device.

3. The terminal device according to claim 1, characterized by The specific steps for determining a relay device according to the relationship between the relay dimension information of the relayable terminal device and the initiation dimension information of the terminal device are as follows: when there are no relayable terminal devices respectively matching the first dimension, the second dimension, and the third dimension of the terminal device, but there are relayable terminal devices respectively matching the first dimension and the second dimension of the terminal device, the relayable terminal devices are determined as the relay device.

4. The terminal device according to claim 1, characterized by The specific steps for determining a relay device according to the relationship between the relay dimension information of the relayable terminal device and the initiation dimension information of the terminal device are as follows: when there are no relayable terminal devices respectively matching the first dimension, the second dimension, and the third dimension of the terminal device, and there are no relayable terminal devices respectively matching the first dimension and the second dimension of the terminal device, but there is a relayable terminal device matching the first dimension of the terminal device, the relayable terminal device is determined as the relay device.

5. A matching method of a terminal device, characterized by, The method comprises the following steps: receive a voice signal from a sound collector, and identify voice text from the voice signal; When the device information is included in the voice text, a relay device is determined according to the device information, a relay relationship between the terminal device and the relay device is established, wherein the device information points to the unique relay device; When the device type information is included in the voice text, a relayable terminal device is determined according to the device type information, and a relay device is determined from the relayable terminal device according to the relationship between the relay dimension information of the relayable terminal device and the initiation dimension information of the terminal device, and a relay relationship between the terminal device and the relay device is established, wherein the device type information points to at least two relayable terminal devices, the dimension information of the device includes a first dimension, a second dimension and a third dimension, the first dimension is a user account, the second dimension is a room attribute identifier, and the third dimension is a room area identifier; If there is no relayable terminal device matching the first dimension of the terminal device, but there is a relayable terminal device matching the second dimension of the terminal device, or there is a relayable terminal device matching the third dimension of the terminal device, a prompt information is generated to enable a user to add the relayable terminal device to the first dimension according to the prompt information, and the relayable terminal device is determined as the relay device.

6. The matching method of terminal equipment according to claim 5, characterized in that, The specific steps for determining the relay device according to the relationship between the relay dimension information of the relayable terminal device and the initiation dimension information of the terminal device are as follows: When there are relayable terminal devices respectively matching the first dimension, the second dimension and the third dimension of the terminal device, the relayable terminal devices are determined as the relay device.

7. The matching method of terminal equipment according to claim 5, characterized in that, The specific steps for determining the relay device according to the relationship between the relay dimension information of the relayable terminal device and the initiation dimension information of the terminal device are as follows: When there are no relayable terminal devices respectively matching the first dimension, the second dimension and the third dimension of the terminal device, but there are relayable terminal devices respectively matching the first dimension and the second dimension of the terminal device, the relayable terminal devices are determined as the relay device.

8. The matching method of terminal equipment according to claim 5, characterized in that, The specific steps for determining the relay device according to the relationship between the relay dimension information of the relayable terminal device and the initiation dimension information of the terminal device are as follows: When there are no relayable terminal devices respectively matching the first dimension, the second dimension and the third dimension of the terminal device, and there are no relayable terminal devices respectively matching the first dimension and the second dimension of the terminal device, but there is a relayable terminal device matching the first dimension of the terminal device, the relayable terminal device is determined as the relay device.

Images