Data relay playback method and intelligent device

Through communication between smart devices and controller configuration, automatic relay playback is achieved, solving the problem of cumbersome operation when users switch devices in different areas and improving the efficiency of media data sharing and relay playback.

CN114189729BActive Publication Date: 2024-11-26HISENSE VISUAL TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202111531719.4
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-12-14
Publication Date
2024-11-26
Estimated Expiration
2041-12-14

AI Technical Summary

Technical Problem

In home settings, when users switch between smart devices in different areas, they need to manually search for and adjust media content, which is cumbersome and affects the efficiency of relay playback of media data.

Method used

Through communication between smart devices and configuration of the controller, automatic relay playback of media data is achieved. This includes a playback device, a communicator, and a controller. The system automatically acquires media data and playback progress from the target device and jumps directly to the specified playback progress on the new device to continue playback.

Benefits of technology

Users no longer need to repeat operations on different devices, enabling rapid sharing and relay playback of media data between devices, thus improving user experience and efficiency.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114189729B_ABST
    Figure CN114189729B_ABST
Patent Text Reader

Abstract

The present invention discloses a data relay playback method and a smart device, which receives an operation instruction to continue playing the content currently being played by the first target device on the smart device, and sends a first data relay request; the first data relay request is used to obtain the media data and playback progress of the content currently being played from the first target device; after receiving the media data and playback progress, the device automatically jumps to the playback progress to continue playing the media data. This application realizes the sharing of media data among multiple smart devices, and users can view the audio-visual content played on other devices without having to operate separately on different devices, breaking the information islands between devices. When the user switches areas and devices, the relay playback can be quickly completed by simply indicating the target device object of the desired relay to the smart device in the new area, without the need for the user to perform tedious operations, and the playback method is flexible and fast, which improves the efficiency of media relay playback and user experience.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of smart homes, and in particular to a data relay playback method and a smart device. Background Art

[0002] A variety of smart devices can be installed in the home scene, and these smart devices are distributed in different areas of the home. For example, TV 1 is placed in the living room, TV 2 is placed in the bedroom, a smart refrigerator is placed in the kitchen, and a smart door mirror is set on the door. With the development of intelligent home appliances and home equipment, these devices may have at least one of the functions of video playback, audio playback, image playback, image acquisition, sound acquisition, etc. As the user moves, the user's location area will shift. For example, if the user previously watched video A played on TV 1 in the living room, if the user moves to the bedroom and wants to continue watching video A, he needs to search for video A again on TV 2, click to play video A after searching, and adjust video A on TV 2 to the same playback progress as TV 1, so as to achieve relay playback of media data when the user's activity area and available devices are switched, but the user operation is more complicated under this implementation method. Summary of the invention

[0003] In order to solve the problems existing in the above-mentioned background technology, the present invention provides a data relay playback method and an intelligent device to realize the sharing and interaction of media data among multiple devices, and improve the speed of media data relay playback when switching intelligent devices.

[0004] The smart device provided in the first aspect includes:

[0005] A playback device, used for playing media data;

[0006] A communicator, used for communicating with a server or other intelligent devices;

[0007] The controller is configured to execute:

[0008] receiving an operation instruction on the smart device to continue playing the content currently being played by the first target device, and sending a first data relay request; the first data relay request is used to obtain media data and playback progress of the content currently being played by the first target device from the first target device;

[0009] The media data and the playing progress are received, and the playing device is controlled to automatically jump to the playing progress to continue playing the media data.

[0010] In an exemplary implementation of the first aspect, the playback device includes a display, and the controller is further configured to execute:

[0011] When receiving an access request to the relay application page, querying the media information of the content currently being played by the relay-capable device associated with the smart device, the media information including the media type and the media name;

[0012] Controlling the display to display the generated playlist in the relay application page; wherein the playlist includes a number of relay options, and the relay options include media information and device information of the relayable device;

[0013] A selection operation of a target relay option in the playlist is received, and the first data relay request is generated according to a first target device corresponding to the target relay option.

[0014] In an exemplary implementation of the first aspect, the smart device further includes a sound collection device, and the controller is further configured to execute:

[0015] Acquire voice information input by a user collected by a sound collection device, and analyze the voice information;

[0016] If it is analyzed that the user intends to continue playing the media data of the target device, query the target device ID corresponding to the name of the first target device according to the device list; wherein the device list records the correspondence between the name of the currently relayable device and the device ID;

[0017] The first data relay request is generated according to the target device ID.

[0018] In an exemplary implementation of the first aspect, the controller is configured to: send the first data relay request to the first target device; or, send the first data relay request to the server, wherein the first data relay request is also used to instruct the server to forward the first data relay request to the first target device, and to instruct the server to send the media data and playback progress received from the first target device to the smart device.

[0019] In an exemplary implementation of the first aspect, the controller is further configured to execute:

[0020] Upon receiving an operation instruction to play the real-time media data collected by the second target device, a second data relay request is sent; the second data relay request is used to instruct the second target device to collect the real-time media data, and encode the real-time media data and send it to the smart device;

[0021] The real-time media data is received and decoded, and the playback device is controlled to play the decoded real-time media data.

[0022] In an exemplary implementation of the first aspect, the controller is further configured to: upon receiving an operation instruction to exit the relay, send a termination instruction, wherein the termination instruction is used to instruct the second target device to stop collecting and encoding the real-time media data.

[0023] In an exemplary implementation of the first aspect, the controller is configured to: send the second data relay request and the termination instruction to the second target device; or, send the second data relay request and the termination instruction to the server, wherein the second data relay request is used to instruct the server to forward the second data relay request to the second target device, and to instruct the server to send the real-time media data received from the second target device to the smart device, and the termination instruction is also used to instruct the server to forward the termination instruction to the second target device.

[0024] Smart devices in home scenarios can communicate and interact with each other, for example, by direct communication between devices in a local area network, or indirect communication through a server. In the first aspect, the smart device is the end that initiates the relay request, and the target device is the end that is relayed, that is, the smart device obtains media data from the target device and then relays and continues to play it on this end. For example, the user selects the target device through the device's special configuration relay application page, and then the smart device sends a data relay request to the target device. The target device responds to the data relay request and sends the currently playing media data and its playback progress to the smart device. The smart device can use the playback progress as a relay node benchmark and directly jump to the playback progress to continue playing the media data after that, thereby realizing relay and continued playback of the same media asset in the device switching scenario. For example, the TV in the living room is playing media A, and the user moves from the living room to the bedroom, and hopes to relay media A on the bedroom TV. In this case, the TV in the living room is selected as the target device, and the TV in the living room sends the data of media A to the bedroom TV. For example, if the video duration of media A is 40 minutes, and the TV in the living room is currently playing to the 20th minute (half of the progress), the playback progress is sent to the bedroom TV, and the bedroom TV is directly positioned at the 20th minute playback node, and continues to play the media data in the interval from the 20th minute to the 40th minute of media A. The first aspect is to achieve the sharing of media data among multiple smart devices. Users can view the audio-visual content played on other devices without having to operate on different devices separately, breaking the information islands between devices. When users switch areas and devices, they only need to indicate the target device object they want to relay to the smart devices in the new area, and the relay playback can be completed quickly without the user having to perform tedious operations. In addition, the playback method is flexible and fast, which improves the efficiency of media relay playback and user experience.

[0025] The smart device provided in the second aspect includes:

[0026] A playback device, used for playing media data;

[0027] A communicator, used for communicating with a server or other intelligent devices;

[0028] The controller is configured to execute:

[0029] receiving a data relay request sent by the first device, and sending the currently playing media data and its playing progress to the first device, wherein the playing progress is used to enable the first device to jump to the playing progress and continue playing the media data;

[0030] Alternatively, upon receiving a data relay request sent by the server, the currently playing media data and its playing progress are conveyed to the first device via the server.

[0031] In an exemplary implementation of the second aspect, the controller is configured to send the media data in the following manner:

[0032] Upon receiving the data relay request, querying the source of the currently playing media data;

[0033] If the media data is from local storage, a network access path corresponding to the local storage path is generated, and the network access path is sent to the first device, or the network access path is forwarded to the first device via a server;

[0034] receiving an access request from the first device to the network access path, parsing the local storage path from the network access path, converting media data in the local storage path into a network media stream, and sending the network media stream to the first device, or forwarding the network media stream to the first device via a server;

[0035] Alternatively, if the media data originates from a network resource, the URL address of the media data is sent to the first device, or the URL address is forwarded to the first device via a server.

[0036] The smart device in the second aspect corresponds to the first target device being relayed in the first aspect, and the first device corresponds to the smart device that initiates the relay request in the first aspect. When the smart device in the second aspect receives the data relay request, it queries the currently broadcasting media, and then directly sends its media data and the current playback progress to the first device or conveys it to the first device via the server, so that the first device can continue to broadcast the same media. The smart device in the second aspect can also realize the transmission and sharing of media data to the first device according to the information such as the source of the currently broadcasting media (such as local or network), the type of media (such as video, audio, image, etc.). The second aspect realizes the sharing of media data among multiple smart devices. Users can view the audio-visual content played on other devices without operating separately on different devices, breaking the information islands between devices. After the user switches the area and device, he only needs to indicate to the first device which smart device he wants to relay, and the relay playback can be completed quickly, without the user having to perform other redundant and cumbersome operations, and the playback method is flexible and fast, which improves the efficiency of media relay playback and user experience.

[0037] The smart device provided by the third aspect includes:

[0038] A collection device, used for collecting real-time media data, wherein the collection device includes an image collection device and / or a sound collection device;

[0039] A communicator, used for communicating with a server or other intelligent devices;

[0040] The controller is configured to execute:

[0041] receiving a data relay request sent by the first device, and controlling the collection device to collect real-time media data;

[0042] encoding the real-time media data and sending the encoded real-time media data to the first device so that the first device decodes and plays the real-time media data; or, if the data relay request is sent by the server, forwarding the encoded real-time media data to the first device via the server;

[0043] When receiving a termination instruction generated by the first device when exiting the relay, stop collecting and encoding the real-time media data.

[0044] The smart device in the third aspect corresponds to the second target device that is relayed in the first aspect, and the first device corresponds to the smart device that initiates the relay request in the first aspect. In the scenario of the third aspect, some smart devices are equipped with acquisition devices such as cameras and microphones, which can collect real-time media data. For example, a camera is set on the smart door mirror, which can monitor the scene outside the door to ensure home safety. If the user wants to view the monitored scene on smart devices in other areas, taking the smart refrigerator with a display in the kitchen as an example, the smart refrigerator initiates a data relay request to the smart door mirror (i.e., the first device), the smart door mirror starts the camera and collects video data, and encodes the video data to facilitate video transmission, and then sends the encoded video data to the smart refrigerator, which decodes and plays the video data. In this way, even if the user moves from the door body to the kitchen, he can view the video picture taken by the door mirror camera through the kitchen refrigerator, without being limited to viewing near the door body. The third aspect is for smart devices with real-time data acquisition capabilities, so that the real-time collected media data can also be relayed and played as the user moves and regional devices switch.

[0045] The data relay playback method provided in the fourth aspect includes:

[0046] receiving an operation instruction to continue playing the content currently being played by the first target device on the smart device, and sending a first data relay request; the first data relay request is used to obtain media data and playback progress of the content currently being played by the first target device from the first target device;

[0047] The media data and the playback progress are received, and the playback progress is automatically jumped to continue playing the media data.

[0048] The data relay playback method provided in the fifth aspect includes:

[0049] receiving a data relay request sent by the first device, and sending the currently playing media data and its playing progress to the first device, wherein the playing progress is used to enable the first device to jump to the playing progress and continue playing the media data;

[0050] Alternatively, upon receiving a data relay request sent by the server, the currently playing media data and its playing progress are conveyed to the first device via the server.

[0051] A sixth aspect provides a data relay playback method, comprising:

[0052] When receiving the data relay request sent by the first device, controlling the collection device to collect real-time media data;

[0053] encoding the real-time media data and sending the encoded real-time media data to the first device so that the first device decodes and plays the real-time media data; or, if the data relay request is sent by the server, forwarding the encoded real-time media data to the first device via the server;

[0054] When receiving a termination instruction generated by the first device when exiting the relay, stop collecting and encoding the real-time media data.

[0055] Other optional implementation methods and beneficial effects of the method parts of the fourth to sixth aspects can be referred to the equipment parts of the first to third aspects mentioned above, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 shows a usage scenario of a display device according to some embodiments;

[0057] Figure 2 shows a hardware configuration block diagram of a control device 100 according to some embodiments;

[0058] Figure 3 shows a hardware configuration block diagram of a display device 200 according to some embodiments;

[0059] Figure 4 shows a software configuration diagram in a display device 200 according to some embodiments;

[0060] FIG5( a ) exemplarily shows a communication architecture between smart devices in a home scene;

[0061] FIG5( b ) exemplarily shows a schematic diagram of relay distribution of smart devices in a home scene;

[0062] Figure 6 A schematic diagram exemplarily shows a user accessing a relay application from an application center;

[0063] Figure 7 The following is an exemplary UI diagram showing a playlist in a relay application page;

[0064] Figure 8 The schematic diagram of the relay application page is shown as an example when there is no relay device;

[0065] Fig. 9 A schematic diagram showing an exemplary voice relay control logic;

[0066] Fig.10 The following is an exemplary diagram showing the UI display of the smart device that initiates the relay request when the target device finishes playing the current media asset;

[0067] Fig.11 The following is an exemplary logical diagram showing that when the media data of the relay playback is from the local storage of the target device, the smart device initiating the relay obtains the media stream;

[0068] FIG. 12 ( a1 ) exemplarily shows the logical flow of voice relay interaction between devices A and B when the media data played by device B is from local storage;

[0069] FIG. 12 ( a2 ) exemplarily shows the logical flow of voice relay interaction between devices A and B when the media data played by device B is from a network resource;

[0070] FIG. 12 ( b1 ) exemplarily shows the logic flow of UI relay interaction between devices A and B when the media data played by device B is from local storage;

[0071] FIG. 12 ( b2 ) exemplarily shows the logic flow of UI relay interaction between devices A and B when the media data played by device B is from network resources;

[0072] Fig.13 The example shows the relay interaction logic of device B relaying the playback of real-time media data collected by device C;

[0073] Fig.14 A flowchart of a data relay playback method executed by a relay initiating terminal device is exemplarily shown;

[0074] Fig.15 A flowchart of a data relay playback method executed by a relayed target device is exemplarily shown;

[0075] Fig.16 The flowchart exemplarily shows another data relay playback method executed by the target device being relayed. DETAILED DESCRIPTION

[0076] Figure 1 Schematic diagram of a usage scenario of a display device according to an embodiment. Figure 1 As shown, the display device 200 also performs data communication with the server 400 , and the user can operate the display device 200 through the smart device 300 or the control apparatus 100 .

[0077] In an exemplary implementation, the control device 100 may be a remote controller, and the communication between the remote controller and the display device includes infrared protocol communication or Bluetooth protocol communication, and at least one of other short-range communication methods, to control the display device 200 wirelessly or wired. The user may control the display device 200 by inputting user commands through at least one of buttons on the remote controller, voice input, and control panel input.

[0078] In an exemplary implementation, the smart device 300 may include any one of a mobile terminal, a tablet computer, a computer, a laptop computer, an AR / VR device, etc.

[0079] In an exemplary implementation, the smart device 300 may also be used to control the display device 200. For example, the display device 200 is controlled using an application running on the smart device.

[0080] In an exemplary implementation, the smart device 300 and the display device may also be used to communicate data.

[0081] In an exemplary implementation, the display device 200 may also be controlled in a manner other than the control device 100 and the smart device 300. For example, the user's voice command control may be directly received through a module for acquiring voice commands configured inside the display device 200, or the user's voice command control may be received through a voice control device provided outside the display device 200.

[0082] In an exemplary implementation, the display device 200 also communicates data with the server 400. The display device 200 may be allowed to communicate via a local area network (LAN), a wireless local area network (WLAN), and other networks. The server 400 may provide various content and interactions to the display device 200. The server 400 may be one cluster or multiple clusters, and may include one or more types of servers.

[0083] Figure 2 Schematically shows a block diagram of a configuration of the control device 100 according to an exemplary embodiment. Figure 2 As shown, the control device 100 includes a controller 110, a communication interface 130, a user input / output interface 140, a memory, and a power supply. The control device 100 can receive input operation instructions from the user, and convert the operation instructions into instructions that the display device 200 can recognize and respond to, playing the role of an interactive intermediary between the user and the display device 200.

[0084] Figure 3 The hardware configuration block diagram of the display device 200 according to an exemplary embodiment is shown. The display device 200 includes at least one of a tuner 210, a communicator 220, a detector 230, an external device interface 240, a controller 250, a display 260, an audio output interface 270, a memory, a power supply, and a user interface.

[0085] In an exemplary implementation, the controller includes a central processing unit, a video processor, an audio processor, a graphics processor, a RAM, a ROM, and a first interface to an nth interface for input / output.

[0086] In an exemplary implementation, the display 260 includes a display screen component for presenting images, and a driving component for driving image display, which is used to receive image signals output from a controller, display video content, image content, and a menu control interface component and a user control UI interface, etc.

[0087] In an exemplary implementation, the display 260 may be at least one of a liquid crystal display, an OLED display, and a projection display, and may also be a projection device and a projection screen.

[0088] In an exemplary implementation, the communicator 220 is a component for communicating with an external device or server according to various communication protocol types. For example, the communicator may include at least one of a Wifi module, a Bluetooth module, a wired Ethernet module, or other network communication protocol chips or near field communication protocol chips, and an infrared receiver. The display device 200 may establish transmission and reception of control signals and data signals with the control device 100 or the server 400 through the communicator 220.

[0089] In an exemplary implementation, the detector 230 is used to collect signals from the external environment or the external interaction. For example, the detector 230 includes a light receiver, a sensor for collecting the intensity of ambient light; or, the detector 230 includes an image collector, such as a camera, which can be used to collect external environment scenes, user attributes or user interaction gestures; or, the detector 230 includes a sound collector, such as a microphone, etc., for receiving external sounds.

[0090] In an exemplary implementation, the external device interface 240 may include, but is not limited to, any one or more of the following: a high-definition multimedia interface (HDMI), an analog or digital high-definition component input interface (component), a composite video input interface (CVBS), a USB input interface (USB), an RGB port, etc. It may also be a composite input / output interface formed by the above multiple interfaces.

[0091] In an exemplary implementation, the controller 250 controls the operation of the display device and responds to user operations through various software control programs stored in the memory. The controller 250 controls the overall operation of the display device 200. For example, in response to receiving a user command for selecting a UI object to be displayed on the display 260, the controller 250 can perform operations related to the object selected by the user command.

[0092] In an exemplary implementation, the object may be any one of the selectable objects, such as a hyperlink, an icon or other operable controls. Operations related to the selected object include: displaying a page, document, image, etc. connected to the hyperlink, or executing a program corresponding to the icon.

[0093] In an exemplary implementation, the controller includes at least one of a central processing unit (CPU), a video processor, an audio processor, a graphics processing unit (GPU), RAM (Random Access Memory, RAM), ROM (Read-Only Memory, ROM), a first interface to an nth interface for input / output, a communication bus (Bus), etc.

[0094] In an exemplary implementation, the user may input a user command in a graphical user interface (GUI) displayed on the display 260, and the user input interface receives the user input command through the graphical user interface (GUI). Alternatively, the user may input a user command by inputting a specific sound or gesture, and the user input interface receives the user input command by recognizing the sound or gesture through a sensor.

[0095] In an exemplary implementation, "user interface" is a medium interface for interaction and information exchange between an application or operating system and a user, which realizes the conversion between the internal form of information and the form acceptable to the user. The commonly used form of user interface is the graphical user interface (GUI), which refers to a user interface related to computer operation displayed in a graphical manner. It can be an interface element such as an icon, window, control, etc. displayed on the display screen of an electronic device, where the control can include at least one of the visual interface elements such as icons, buttons, menus, tabs, text boxes, dialog boxes, status bars, navigation bars, widgets, etc.

[0096] In an exemplary implementation, the user interface 280 is an interface that can be used to receive control input (such as a physical button on the display device body, or other interfaces).

[0097] See also Figure 4In an exemplary implementation, the system is divided into four layers, from top to bottom, namely, the application layer (Applications) layer (referred to as "application layer"), the application framework layer (Application Framework) layer (referred to as "framework layer"), the Android runtime (Android runtime) and system library layer (referred to as "system runtime library layer"), and the kernel layer.

[0098] In an exemplary implementation, at least one application is running in the application layer, and these applications can be window programs, system settings programs, clock programs, etc. provided by the operating system, or applications developed by third-party developers. In specific implementations, the application packages in the application layer are not limited to the above examples.

[0099] The framework layer provides application programming interfaces (APIs) and programming frameworks for applications in the application layer. The application framework layer includes some predefined functions. The application framework layer is equivalent to a processing center that determines the actions that applications in the application layer take. Through the API interface, applications can access system resources and obtain system services during execution.

[0100] like Figure 4 As shown, in the embodiment of the present application, the application framework layer includes managers, content providers, etc., wherein the manager includes at least one of the following modules: an activity manager (ActivityManager) is used to interact with all activities running in the system; a location manager (Location Manager) is used to provide system services or applications with access to system location services; a package manager (Package Manager) is used to retrieve various information related to the application package currently installed on the device; a notification manager (NotificationManager) is used to control the display and clearing of notification messages; a window manager (Window Manager) is used to manage icons, windows, toolbars, wallpapers, and desktop components on the user interface.

[0101] In an exemplary implementation, the kernel layer is a layer between hardware and software. Figure 4 As shown, the kernel layer includes at least one of the following drivers: audio driver, display driver, Bluetooth driver, camera driver, WIFI driver, USB driver, HDMI driver, sensor driver (such as fingerprint sensor, temperature sensor, pressure sensor, etc.), and power driver, etc.

[0102] The above embodiments introduce the hardware / software architecture and functional implementation of the display device. The display device has the capabilities of video / image / text playback, audio playback, image acquisition, sound acquisition, etc. For other smart devices, the relevant hardware can be configured according to their own device type and functional requirements. When the smart device has at least one of the above capabilities, it can support the relay playback function of multimedia data. The "relay" mentioned in this application refers to the ability of a smart device to continue playing media data from other devices. For example, device A transmits the currently playing video data and its playback progress to device B, and device B jumps to the playback progress to continue playing the subsequent video content, thereby presenting a relay effect.

[0103] In some exemplary implementations, for example, if the smart device is required to support voice control or recording and other similar functions, a sound acquisition device is configured for the smart device. For example, a microphone; if the smart device is required to support video surveillance and other similar functions, an image acquisition device is configured for the smart device, such as a camera; if the smart device is required to have the function of displaying multimedia files such as videos, images, and texts, a display is configured for the smart device; if the smart device is required to have a sound playback function, a sound player is configured for the smart device, such as a speaker or other external audio output device (such as a speaker, headphones, etc.). Smart devices can have one or more multimedia playback / collection capabilities.

[0104] In some exemplary implementations, for home scenes, in order to achieve fast relay playback of multimedia data as the user moves and the available devices in the area are switched, associations can be established for multiple smart devices in the home.

[0105] In some exemplary implementations, the association can be reflected in that any two smart devices can directly communicate end-to-end, and the way to establish a communication connection between devices is not limited to the network, Bluetooth, etc. Optionally, a communication network is created for multiple smart devices in the home scene, for example, each smart device is connected to the same local area network, and any two smart devices connected to the communication network are associated and can communicate and interact. By monitoring the access and exit of smart devices in the communication network, no matter where the user moves to in the home scene, he can query other relayable devices through the available devices in the user's current area (referred to as the current area), and select at least one target device from the relayable devices, and the target device sends its multimedia data to the devices in the current area for relay playback.

[0106] In some exemplary implementations, referring to FIG. 5( a ), the association may be reflected as any two smart devices communicating indirectly through a server, that is, the server is equivalent to a central device. Optionally, for example, a relay application is installed in each smart device, and the user registers and logs in to an account in the relay application. Each smart device can log in to the same or different accounts. The server monitors and records the account login status of each smart device, and when it is detected that device B initiates a relay, according to the preset communication association rules, automatically matches a relay-capable device associated with device B and capable of forwarding its own media data to device B for continuous playback.

[0107] In an exemplary implementation, the communication association rule is configured so that smart devices logged into the same Relay application account are associated with each other. For example, the living room TV is logged into account 1 in the Relay application, the master bedroom TV is logged into account 1 in the Relay application, the smart speaker is logged into account 2 in the Relay application, and the smart refrigerator is not logged into any account in the Relay application, then the server records that the living room TV and the master bedroom TV are currently associated, and the smart speaker and the smart refrigerator are not associated devices.

[0108] In an exemplary implementation, the communication association rules are configured so that smart devices whose relay application accounts are logged in are associated with each other. For example, the living room TV is logged in to account 1 in the relay application, the master bedroom TV is logged in to account 1 in the relay application, the smart speaker is logged in to account 2 in the relay application, and the smart refrigerator is not logged in to any account in the relay application. In this case, the living room TV, the master bedroom TV and the smart speaker are associated with each other, and the smart refrigerator has no associated device.

[0109] In an exemplary implementation, the communication association rules can also be customized by the user. For example, if the user selects the master bedroom TV and the smart door mirror as associated devices in the relay application of the living room TV, when the living room TV initiates the relay, the relayable devices are selected from the master bedroom TV and the smart door mirror associated with it, so that the user can select which device the media data to be played by the living room TV comes from. The communication association rules can be set according to actual requirements and are not limited to the examples of this application.

[0110] In an exemplary implementation, based on the communication architecture of FIG5(a), the user selects to initiate a relay to device A from the relayable device. Specifically, device B generates a data relay request for continuing to play the media data on device A. The server receives the data relay request through the relay application and forwards the data relay request to device A. Device A responds to the data relay request and sends the current media data and its playback progress to the server. The server then sends the received media data and playback progress to device B. Device B uses the location node specified by the playback progress as a reference to achieve continuous playback of the current media data on device A. That is, data, requests, instructions, etc. between any two smart devices in the home scene need to be conveyed to the other end via the server.

[0111] It can be seen that through the communication association between multiple smart devices in the home (such as network association, account association, etc.), it is possible to realize the sharing and relay of multimedia data between smart devices. Among them, the relayable device refers to a smart device that is associated with the currently available device and supports the media relay of the current device. For example, the user previously watched program A on the living room TV, and then moved from the living room to the bedroom. The current area is the bedroom, and the currently available device is the bedroom TV. The living room TV is one of the relayable devices of the bedroom TV, and the user can choose to have the bedroom TV relay the program A on the living room TV; for another example, there is only one smart device in the second bedroom, which is a smart speaker. The user moves from the second bedroom to the living room, then the current area is switched to the living room, and the currently available device is the living room TV. If the smart speaker is in the state of being turned off, standby, not connected to the Internet, or not logged into the relay application account, that is, the smart speaker currently has no audio data to be transmitted to the living room TV for continued playback, then the smart speaker does not belong to the relayable device of the living room TV.

[0112] In some exemplary implementations, a relay application may be configured in the smart device, and the relay application may be configured to run in the background when the device is started, or to be manually started by the user. When the user starts the relay application on device B, he can view other devices currently connected to the communication network or associated with device B and their media playback, and can also choose which target device's media data to relay playback. The relay application icon control can be displayed in a location such as the homepage or application center, so that the user can easily start the relay application.

[0113] In an exemplary implementation, FIG5(b) shows a relay distribution of smart devices in a home scene, and the communication mode between smart devices is not limited to end-to-end network communication, indirect communication via an intermediate server, etc. Assume that the home scene includes a smart TV 1 placed in the master bedroom, a smart refrigerator placed in the kitchen, a smart door mirror placed at the door, a smart speaker placed in the second bedroom, and a smart TV 2 placed in the living room. Among them, smart TV 1 is playing a movie, the smart refrigerator is equipped with a camera for collecting images of food in the storage room, the smart door mirror is equipped with a camera for collecting surveillance images outside the door, the smart speaker is currently playing music, and the user is currently moving to the living room area. The user can trigger the relay playback process through voice, remote control, touch, etc.

[0114] In an exemplary implementation, based on the above example, the user is currently in the living room area, and the available device is the smart TV 2. If the user wants to select a target device for docking, refer to Figure 6 For example, if the relay application icon control 61 is triggered in the application center, the smart device 2 will receive an access request to the relay application page. Taking the network association between devices as an example, the media information of the content currently being played by the relay-capable device in the local area network is queried. The media information includes but is not limited to the media type and media name, etc. The media type is but not limited to video, audio, image, text, etc. The media name is the resource name of the content being played. The role of the media information is to facilitate users to understand the playback content of each relay-capable device and thus select the relay target. Assuming that the smart device 2 inquires that the smart TV 1, smart door mirror and smart speaker in the current local area network are connected to the network, and the smart refrigerator is not connected to the network, then the relay-capable device set is obtained as {smart TV 1, smart door mirror, smart speaker}, and a play list is generated, and then the play list is displayed on the relay application page, refer to Figure 7 In the UI example, the play list 70 includes several relay options 71, and each relay option 71 includes media information 711 and device information 712 of a relayable device.

[0115] In an exemplary implementation, the device information 712 includes but is not limited to the device name / device ID, the location area of ​​the device, etc. The device name can default to the specific category of the device, such as refrigerator, TV, stereo, etc., or support users to customize any smart device in the home scene, such as personalizing the smart refrigerator to "My Food Warehouse". Regardless of how the device name is adjusted, the device ID is generally fixed. The device ID can be used to accurately identify the device and is generally globally unique, that is, each smart device has a different device ID. If the device category is used as the device name, consider that the same category of devices may be placed in multiple areas, such as TVs in the living room and bedroom. Optionally, the device name is set in the form of a combination of area + category, such as living room TV, bedroom TV, kitchen refrigerator.

[0116] For example, the relay option is "The smart TV (master bedroom) is playing the video "Coco the Movie", which expresses to the user that for the relayable device - the master bedroom TV, the media type currently playing is video, and the name of the video is Coco; for another example, the relay option is "The smart speaker (second bedroom) is playing the audio song "Seven Miles of Fragrance", which expresses to the user that for the relayable device - the second bedroom speaker, the media type currently playing is audio, and the name of the audio is Seven Miles of Fragrance; for another example, the relay option is "The smart door mirror is showing the video "The surveillance screen outside the door captured by the camera", which expresses to the user that for the relayable device - the smart door mirror, the media type currently playing is video, and the video is the surveillance screen outside the door captured by the door mirror camera.

[0117] In some exemplary implementations, the two ends of the relay interaction should maintain matching of playback capabilities. For example, device 1 is a TV in the living room, device 2 is a TV in the bedroom, device 3 is a stereo in the living room, and device 4 is a stereo in the second bedroom. Since device 1 has audio and video playback capabilities, devices 2, 3, and 4 may all become relay devices for device 1; since device 3 only has audio playback capabilities but not video playback capabilities, if device 1 and device 2 are playing media data such as videos and images that need to be displayed, device 3 cannot relay to device 1 and device 2. In this case, device 4 may be a relay device for device 3; if device 3 only plays audio content, such as application scenarios such as users listening to songs or books, then devices 1, 2, and 4 may all become relay devices for device 3. Therefore, when generating a playlist, it is necessary to determine which smart devices are relay devices for the relay initiator based on the playback capabilities of the relay initiator itself, as well as the networking status or account login status, playback capabilities, and currently played media types of other smart devices in the home scene.

[0118] In an exemplary implementation, for a smart device that supports a touch screen function, a user can directly touch and select a target relay option from the play list 70; or, the user can switch relay options by operating a remote control, and can choose to set the focus, and the focus is linked with the user's action of switching relay options. When the target relay option that the user expects to select obtains the focus, the user can press the confirmation / OK key of the remote control, and the target relay option is selected. Then, according to the target device pointed to by the target relay option, a data relay request is generated and sent, for example, the data relay request is directly sent to the target device or forwarded to the target device via a server. The focus presentation form is not limited, such as highlighting, or applying a focus frame outside the relay option, etc., as long as the focus option is prominently displayed so that the user can distinguish the difference between the focus and non-focus options with the naked eye.

[0119] In the above embodiment, for the case where the relayable device set is a non-empty set, if the smart device detects that there is no relayable device at present, that is, when the relayable device set is an empty set, refer to Figure 8 , a prompt message is displayed on the Relay application page. Taking the network association between devices as an example, the prompt message is in the form of "There is currently no device for relay playback, please try again later or check the device network connection". The user can reconnect the target device to be connected, or refresh the Relay application page, or exit and revisit the Relay application page to retry to obtain the playlist. If the devices are associated with accounts, the prompt message is in the form of "There is currently no device for relay playback, please try again later or re-login to the Relay account". In this case, the user can log in or switch the account of the Relay application to update the relayable device.

[0120] In an exemplary implementation, if the smart devices available in the current area are equipped with sound collection devices, such as microphones, etc., then users are supported to perform voice control on the smart devices. The sound collection devices of the smart devices collect voice information input by the users, and parse the voice information to clarify the user's intentions. For example, the user says to the smart device "I want to continue watching the movie on the bedroom TV", "Continue / relay playing the movie on the bedroom TV", and other semantically similar text messages. After parsing, the smart device learns that the user operation is relay, the target device for docking is the bedroom TV, and the target media for relay playback is the movie currently playing on the bedroom TV, thereby generating a data relay request and sending the data relay request to the target device or server.

[0121] In an exemplary implementation, the relay service at the bottom layer of each smart device can maintain a device list, which records the correspondence between the device names and device IDs of all smart devices in the home scene. The device ID can be a unique identification code such as a serial number. When the user modifies the device name on any device, the device needs to update the device list of the local device and synchronize the updated device list to other devices. After the smart device parses the target device name from the voice information, it queries the target device ID corresponding to the target device name in the device list, and generates and sends a data relay request based on the target device ID.

[0122] In an exemplary implementation, referring to Fig. 9 The voice relay control logic shown in the figure can configure a voice assistant in a smart device with voice collection capability. The voice assistant is associated with a sound collection device. The sound collection device returns the collected voice information to the voice assistant. The voice assistant sends the voice information to a voice processing system. The voice processing system performs parsing processing such as semantic analysis and keyword recognition on the voice information. The voice processing system can be configured on the smart device side or on the voice server side (such as semantic cloud, etc.) that communicates with the smart device to reduce the operation and memory load of the smart device side. After parsing the voice information, the voice processing system returns the parsing result to the voice assistant. The parsing result includes the action intention and the target device name. If the voice assistant recognizes that the action intention is "relay", the target device name is forwarded to the relay application in the smart device. The relay application obtains the device list from the relay service, and searches for the target device ID corresponding to the target device name from the device list, and initiates a relay to the relay service according to the target device ID. The relay service sends the generated data relay request to the server or the target device indicated by the ID.

[0123] In other possible implementations, after obtaining the target device name, the relay application can also directly forward the target device name to the relay service; the relay service retrieves the device list it maintains, matches the target device ID corresponding to the target device name, generates a data relay request based on the target device ID, and sends the data relay request to the target device or server. For the voice relay control mode, it can be combined with the device's voice low-power standby function, so that the user does not even need to actively turn on the device, but can issue a voice command in front of the standby device, and the standby device will be automatically awakened and execute the relay playback process in response to the voice command.

[0124] In an exemplary implementation, for the voice relay control mode, if the smart devices establish an association by accessing the same local area network (i.e., the network association), considering that the user may not be able to accurately know whether the target device that the user expects to relay is currently connected to the Internet before inputting the voice information, the smart device detects whether the target device is included in the relayable device set after obtaining the target device name or target device ID. If the target device is not included in the relayable device set, that is, the media data in the target device cannot be shared with the smart device end that initiates the relay, optionally, the smart device that initiates the relay request does not generate and send a data relay request, but instead makes a voice announcement to prompt the user that the relay playback for the target device has failed, for example, the announcement content is "Relay failed, please try again or check the device is connected to the Internet", and for example, "Relay failed, please re-specify the device to be relayed". If the target device is included in the relayable device set, it means that the target device has been connected to the communication network, and the current smart device and the target device can communicate, interact and share data, then a data relay request is generated and sent to the target device.

[0125] In an exemplary implementation, if smart devices are associated with each other through accounts, for example, smart devices logged in to the same relay application account are associated with each other, assuming that the user initiates a relay to device A through voice on device B, after voice analysis, device B generates a data relay request based on the ID of device A and sends the data relay request to the server; the server queries the current account login status and media playback status of device A based on the device ID carried in the data relay request. If device A and device B that initiates the relay are currently logged in to the same account, that is, device A is associated with device B (relay condition one), and device A currently has media content that can be played continuously by device B, that is, device A can serve as a relay device for device B (relay condition two), if the server determines that these two relay conditions are met at the same time, it can forward the data relay request to device A.

[0126] In an exemplary implementation, if at least one of the two relay conditions is not met, the server will not forward the data relay request to device A, and will feedback relay failure information to device B that initiated the relay. The relay failure information may indicate the specific reason for the failure. For example, if only relay condition one is not met, the relay failure information indicates that the relay failed because the device A and device B accounts are not associated; if only relay condition two is not met, the relay failure information indicates that device B currently has no media data that can be relayed and played; if both relay conditions are not met, the relay failure information indicates that device A and device B accounts are not associated, and device B currently has no media data to be relayed and played. After receiving the relay failure information, device B may give a voice prompt to the user that the relay failed to be initiated, as well as the specific reason for the relay failure. The way a user initiates a relay is not limited to voice and UI (remote control, touch) forms.

[0127] In an exemplary implementation, when each smart device plays multimedia resources such as video, audio, and images, it records the current media information so that the relay initiator can query the media information of the relayable device before generating a playlist. The media information includes the media type and the media name. For example, if the target device is playing the song "Seven-Li Fragrance", the recorded media type is audio and the media name is "Seven-Li Fragrance". When each smart device exits multimedia playback, the recorded media information is deleted.

[0128] In an exemplary implementation, the smart device can trigger a data relay request through any of the aforementioned remote control, touch control, and voice modes. When the target device receives the data relay request, it obtains the playback progress of the current media asset. The playback progress can be in the form of the current playback time node, or the percentage of the progress bar, etc. Then the target device directly sends the media data and the playback progress to the relay initiator or forwards it via the server, so that the relay initiator uses the playback progress as the relay turning point and automatically jumps to the playback progress to continue playing the media data. For example, the duration of the current video played by the master bedroom TV is 30 minutes. When the master bedroom TV receives the data relay request from the living room TV, the video is played to the 15th minute, and the playback progress can be reflected as 15' or 1 / 2. After receiving the media data and the playback progress, the living room TV automatically jumps to the playback node of the 15th minute, that is, skips the media data of the first 15 minutes, and continues to play the media data in the period from the 15th minute to the 30th minute, so as to realize the relay playback of the same media asset from the master bedroom TV to the living room TV.

[0129] In an exemplary implementation, the smart device that initiates the relay creates or starts a multimedia player of the corresponding type according to the type of media data sent by the target device. For example, if the target device sends video data, the smart device starts the video player, and the video player automatically seeks to the specified playback progress to relay the video data; if the target device sends audio data, the smart device starts the audio player, and the audio player automatically seeks to the specified playback progress to relay the audio data; if the target device sends image data, the smart device can start the image player and load and display the image data; if the target device sends text data, the smart device can start the text reader, and the text reader loads and displays the text data.

[0130] In an exemplary implementation, the image data may come from a designated album (e.g., a cloud album, a theme album, a local album, etc.), and the target device sorts the images in the album and switches the images in sequence, so the playback progress may be the sorting sequence number of the current image in the album. For example, if the target device currently displays the third image in the album, then after the target device sends the image data and playback progress '3' in the same album to the smart device, the smart device gives priority to displaying the third image in the album. If a forward cutting operation is received, the image with sequence number N-1 in the album is displayed. If a backward cutting operation is received, the image with sequence number N+1 in the album is displayed, where N represents the sequence number of the current image in the album.

[0131] In an exemplary implementation, if the image data is a network image, the target device can send the URL (Uniform Resource Locator) of the image to the smart device without sending the playback progress; after the smart device obtains the URL of the image, it accesses the URL and downloads the image, and then loads and displays the image in the image player.

[0132] In an exemplary implementation, some smart devices also support text reading functions, such as dedicated e-book reading tablets (such as Kindle, Palm Reader, etc.), or smart devices with e-book applications installed, etc. When the target device receives the data relay request, if the text content is currently being displayed, it obtains the current reading tag of the text, and the reading tag is used to record the current reading progress of the document (for example, in percentage), and then the target device sends the text data and its reading tag to the relay initiator; after receiving the text data and the reading tag, the relay initiator starts the text reader, and the text reader jumps to the position corresponding to the reading tag, and relays the subsequent text content.

[0133] In an exemplary implementation, in order to avoid playback conflicts, the user can select one or more target devices. For example, the living room TV is playing a movie, the second bedroom audio system is playing music, and the smart door mirror is displaying the surveillance image outside the door. If the user wants to initiate a relay on the master bedroom TV, and the master bedroom TV supports single-channel video playback, then only one of the living room TV and the smart door mirror can be selected as the target device. If the master bedroom TV supports multi-channel video synchronous playback, then the living room TV and the smart door mirror can be selected as the target device at the same time. For another example, if the video surveillance image of the smart door mirror is silent, then the smart door mirror and the second bedroom audio system can be selected as the target devices at the same time. Then the video player of the smart device relays the playback of the surveillance image taken by the smart door mirror, and at the same time, the audio player relays the playback of the music on the second bedroom audio system. It can be seen that in order to avoid playback conflicts, the smart device can support single-channel or multi-channel relay playback.

[0134] In an exemplary implementation, after the smart device initiates a data relay request through any of the aforementioned remote control, touch, and voice modes, if the target device ends playing the media asset C currently being played, for example, other family members exit the playing of the media asset C, or the playing process of the media asset C naturally reaches the end and is not switched to other media assets afterwards, the target device deletes the recorded media information of the media asset C, which is equivalent to the target device exiting the set of relayable devices. The target device may feedback indication information to the smart device that initiated the relay, or the server may feedback indication information to the relay initiating end when detecting that there is no media data to be played continuously on the target device; the smart device that initiated the relay receives the indication information, namely, it learns that the target device currently has no media content to be played continuously, and if the smart device has display capability, it refers to Fig.10 In the example of relay, a prompt message indicating that the relay cannot be carried out is displayed in the relay application page. If the smart device has the ability to speak, the prompt message can be broadcasted by voice. The prompt message in this embodiment is in the form of, for example, "The TV in the master bedroom currently has no media content to be relayed. Please try again later or reselect the device to be relayed."

[0135] In an exemplary implementation, the target device, in response to the data relay request, may implement an adaptive data transmission method according to the source and type of the media data when sending the media data to the relay initiator or server.

[0136] In an exemplary implementation, if the media data currently played by the target device originates from a network resource, the target device directly sends the URL address of the media data or forwards it to the smart device through a server. The smart device accesses the acquired URL address, downloads the media data, and then completes the relay playback of the media data according to the playback progress sent by the target device.

[0137] In an exemplary implementation, if the media data currently played by the target device comes from local storage, the local storage includes the target device's own storage space and external storage devices, and the external storage devices include but are not limited to USB flash drives, mobile hard drives, card readers, etc., then a network access path corresponding to the local area network is generated based on the local storage path, and the network access path and playback progress are returned to the smart device.

[0138] In an exemplary implementation, Fig.11 The logic of the relay initiator obtaining the relay media stream is shown, taking the example that device A and device B can communicate directly through the local area network. Fig.11In the example, the target device is device A, the relay initiator is device B, and device B receives the network access path returned by device A. Assuming the network access path is http: / / 192.168.1.104:8808 / relay / mnt / movie.mp4, device B initiates an access request to http: / / 192.168.1.104:8808 / relay / mnt / movie.mp4 to device A. Device A parses the local storage path from http: / / 192.168.1.104:8505 / relay / mnt / movie.mp4 as / mnt / ovie.mp4, and access the local file system corresponding to the local storage path, the local file system is the storage space of device A itself or an external storage device (such as a USB flash drive, a mobile hard disk, a card reader, etc.); the local file system converts the media file stored in / mnt / movie.mp4 into a local file stream, and sends the local file stream to the streaming media service of device A; the streaming media service of device A converts the local file stream into a network media stream (i.e., http stream), and returns the http stream to device B; device B pulls the http stream, jumps to the playback progress fed back by the target device, and relays the playback of the local movie file in device A. This embodiment converts the local media file in the target device into a media stream, which is convenient for the smart device to pull the stream, improves the transmission efficiency of the media data, and then improves the relay response speed, and realizes the fast relay playback of the media data.

[0139] In an exemplary implementation, device B initiates a relay to device A. When device A receives the data relay request, it sends the currently playing media data and its playback progress to device B or the server. Then, device A can maintain the current playback status and respond to the operation instructions input by the user.

[0140] In an exemplary implementation, in addition to being configured to request device B to obtain the relayed media data and its playback progress, the data relay request can also be configured to instruct device B to perform other preset actions, which are not limited to pausing playback, stopping playback, switching to standby, automatic shutdown, etc., so that when the user relays the viewing of media data on other devices, problems such as invalid playback and power consumption of the original device due to no one watching can be avoided.

[0141] In an exemplary implementation, device A is the target device to be relayed, and device B is the smart device that initiates the relay. Device A and device B can be connected to the same local area network for end-to-end communication, or device A and device B can also communicate indirectly through a server. The configuration of device A includes but is not limited to relay service A, streaming media service and playback terminal A. Playback terminal A may be a UI and / or an audio output channel. For example, if device A is a smart TV, playback terminal A includes a UI and an audio output channel; if device A is a smart speaker, playback terminal A may only include an audio output channel without a UI; if device A is an e-book reading tablet, playback terminal A may only include a UI without an audio output channel. The configuration of device B includes but is not limited to relay service B, and also includes playback end B and / or semantic service. For example, if device B only supports selecting the target device through remote control or touch, and is not configured with a microphone, the semantic service is not set; if device B has no UI display function but supports voice control, the semantic service needs to be set, and the playback end B only includes an audio output channel and no UI; if device B supports any relay operation mode of remote control / touch and voice control, the voice service can be set, and the playback end B includes a UI, and users can choose the relay operation mode according to their own usage habits.

[0142] In an exemplary implementation, referring to the voice relay interaction logic of the examples in FIG12 (a1) and FIG12 (a2), taking the end-to-end direct communication between device A and device B based on the network and performing video relay playback as an example, for other media types, the relay interaction logic is basically similar. After device A is started, relay service A is initialized first. Other devices in the local area network can know through the device discovery function that device A has been connected to the network and supports the relay of audio-visual content; then device B starts and initializes relay service B, and device A and device B synchronize the device status with each other, that is, device A knows that device B has been connected to the network and has the ability to play audio and video, and device B knows that device A has been connected to the network and has the ability to play audio and video. Since device B in the example of FIG12 (a1) supports the voice relay operation mode, device B can register the name of device A to the semantic service of this end, so as to perform device identification based on voice information later.

[0143] In an exemplary implementation, referring to FIG. 12 (a1), a user clicks on device A to watch the movie "Coco", and device A plays the movie "Coco" and records its media information, including the video type and the media name "Coco". A user inputs a voice message "I want to continue watching the movie on device A" on device B, and the semantic service parses the voice message, recognizes that the user's intention is "relay", and the target device for the relay is device A, and sends the parsing result to relay service B; after obtaining the parsing result, relay service B confirms that device A is a relayable device, and sends a data relay request to device A.

[0144] In an exemplary implementation, referring to FIG. 12 (a1), the relay service A in device A receives a data relay request and sends a first indication message to the playback terminal A; the playback terminal A obtains the current playback progress of "Coco" in response to the first indication message, and sends the playback progress and recorded media information to the relay service A, and the playback terminal A detects that the "Coco" movie originates from the local storage, and sends a second indication message to the streaming media service; the streaming media service generates a network access path corresponding to the current local area network according to the local storage path of the "Coco" movie file in response to the second indication message, and sends the network access path to the relay service A; the relay service A sends the network access path, playback progress and media information of the "Coco" movie to the device B.

[0145] In an exemplary implementation, referring to FIG12 (a1), the relay service B sends the received network access path, playback progress and media information to the playback terminal B. The playback terminal B detects that the relay media type is video, and creates or starts a video player. The video player requests the streaming media service of device A to obtain media data based on the network access path. The streaming media service of device A converts the local file stream of the "Coco" movie into an http video stream, and sends the http video stream to the video player of the playback terminal B. The video player automatically jumps to the playback progress to continue playing the pulled http video stream, and displays the media name "Coco" at the specified position of the playback interface, so that the user can know the name of the currently playing movie.

[0146] In an exemplary implementation, referring to FIG. 12 (a2), if the player A detects that the movie "Coco" is from a network resource, the device A does not need to create a streaming media service. The player A directly sends the URL address of "Coco" to the relay service A, and the relay service A sends the URL address, playback progress and media information of the movie "Coco" to the device B. The relay service B sends the received URL address, playback progress and media information to the player B. The player B detects that the relay media type is video, and then creates or starts a video player. The video player downloads the data packet of "Coco" according to the URL address, automatically jumps to the playback progress to continue to play "Coco", and displays the media name "Coco" at a specified position on the playback interface.

[0147] In an exemplary implementation, referring to FIG. 12 (b1) and FIG. 12 (b1), device B performs UI relay interaction based on a remote control or touch mode, and semantic services may or may not be set in device B. The user accesses the relay application page on device B, and playback terminal B sends a relay notification to relay service B; in response to the relay notification, relay service B queries the current relayable devices and their media information in the local area network, and uses M to represent the number of relayable devices. In this embodiment, it is assumed that M is greater than zero, and relay service B sends the media information and device information of M relayable devices to playback terminal B; playback terminal B generates a playlist including M relay options based on the media information and device information of M relayable devices, and displays the playlist in the relay application page. The user selects the target relay option from the playlist, and playback terminal B sends the target device information corresponding to the target relay option to relay service B; relay service B recognizes that the target device information is device A, and sends a data relay request to device A. Compared with Figure 12(a1), Figure 12(b1) and Figure 12(b2) compared with Figure 12(a2) only reflect the difference between voice relay interaction and UI relay interaction in generating data relay request and user operation method on device B side, while the subsequent device A transmits media data to device B and device B relays the playback of media data in basically similar ways, which will not be repeated here.

[0148] In an exemplary implementation, referring to Figure 12(a1) to Figure 12(b2) According to the relay interaction logic, the user exits the relay through the UI on device B, or enters a voice message to exit the relay, such as "end the relay" or "stop playing the movie on device A", etc., then the player B stops acquiring media data and closes the multimedia player. At the same time, relay service B notifies device A that the relay has been exited. If the media data currently played by device A comes from local storage, relay service A notifies the streaming media service that device B has exited the relay, and the streaming media service can be closed, thereby stopping the process of reading media files from the local storage path and converting and sending http streams, and the relay interaction process between device B and device A ends.

[0149] In each of the above embodiments, the relay playback is all existing media resources, such as media files that have been downloaded and saved locally, media files that have been recorded and saved locally, resources available on media application platforms (such as browsers, music applications, video applications, etc.), etc. In actual application scenarios, some smart devices can collect real-time media data, such as devices equipped with cameras have monitoring functions, which can collect and display video images in the shooting scene in real time, and devices equipped with microphones have recording / monitoring functions, which can collect or play sounds generated in the area in real time. This application also provides a relay playback solution for real-time collected media data.

[0150] In an exemplary implementation, Fig.13 The example shows the relay interaction logic in the scenario of real-time data collection during relay playback. Device B is the relay initiator, and device C is the target device being relayed. Device C has a collection device, which includes but is not limited to a camera, a microphone, etc. Device C can collect data of at least one media type in real time according to its own hardware configuration. After device C starts and initializes the relay service, other devices in the local area network can use the discovery function to learn that device C has been connected to the network and has the ability to collect specific types of media data in real time. Fig.13 The end-to-end communication between device B and device C based on a local area network is taken as an example. In actual implementation, device B and device C are not limited to other possible communication methods, such as Bluetooth communication, indirect communication through a server, etc.

[0151] In an exemplary implementation, if the UI relay mode is implemented, when the relay application page generates and displays the playlist, if device C has the ability to collect N types of media data, and N is greater than or equal to 1, then N relay options corresponding to device C are listed in the playlist. For example, if device C has both a camera and a microphone, when device A displays the playlist in the relay application page, if the camera and the microphone are turned on, the relay options are reflected, for example, as "Device C is displaying the video "Living Room Picture Collected by the Camera"", "Device C is playing the audio "Living Room Sound Collected by the Microphone""; if the camera and the microphone are turned off, the relay options are reflected, for example, as "Device C can collect video "Living Room Picture"", "Device C can collect audio "Living Room Sound"", so that the user can choose to relay the video data or audio data collected by device C.

[0152] In an exemplary implementation, in order to avoid playback conflicts, you can choose to relay multiple target devices, or relay the playback of different types of data collected by the same target device. For example, on device B, you can relay the playback of video data collected by device C, and at the same time, you can relay the playback of audio data collected by device C, and the audio and video data can be output synchronously.

[0153] In an exemplary implementation, the user initiates a relay on device B, triggering the relay service C to generate a data relay request and send the data relay request to device C. The implementation of this part can refer to Figure 12(a1) to Figure 12(b2) Examples and descriptions of Fig.13 The process starts with device B sending a data relay request to device C. Device B at least includes a playback terminal B and a relay service B, and device C at least includes a relay service C and a collection device. Optionally, device C also includes a Socket service.

[0154] In an exemplary implementation, taking the real-time media data collected by relay playback device C as a video, the user can input "I want to watch the surveillance video of device C" on device B, or the target relay option triggered by the user through the UI is "Device C is displaying the video "Real-time surveillance image outside the door collected by the camera", then device B sends a data relay request to device C. Fig.13 , the relay service C of device C receives the data relay request and sends a relay notification to the acquisition device; the acquisition device responds to the relay notification and acquires real-time video data. Optionally, the acquisition device is also configured with an encoder for encoding the acquired real-time video data to facilitate video transmission.

[0155] In an exemplary implementation, referring to Fig.13 The acquisition device outputs the encoded video data to the Socket service, and sends the encoded video data and its media information to device B through the Socket connection between device C and device B created in the Socket service, where the media information includes the media type "video" and the media name "real-time monitoring screen outside the door". Since the data collected by device C is real-time, device C does not need to send the playback progress to device B.

[0156] In an exemplary implementation, referring to Fig.13 , the relay service B of device B receives the media information and forwards the media information to the playback end B. When the playback end B detects that the media type of the relay playback is a video, it creates or starts a video player; the video player receives the video data collected and encoded by device C through the Socket connection with device C, and then calls the decoder to decode the video data, and displays the decoded video data in the playback interface of the video player, and displays the media name at the specified position of the interface. Fig.13 The example above takes the real-time video data collected by relay playback as an example. For other types of real-time collected media data, such as audio data, please refer to Fig.13 The logic of relay interaction is realized.

[0157] In an exemplary implementation, referring to Fig.13 When the user exits the relay through voice or UI, the relay service B generates a termination instruction and sends the termination instruction to device C. The relay service C of device C forwards the termination instruction to the acquisition device, and the acquisition device responds to the termination instruction to stop acquiring and encoding video data, thereby interrupting the transmission of the video stream.

[0158] In an exemplary implementation, when the collection device receives a relay notification, if the collection device is in an on state, it can directly collect and encode data. In this case, if the collection device receives a termination instruction, it only needs to stop collecting and encoding data and keep the collection device in an on state.

[0159] In an exemplary implementation, when the acquisition device receives the relay notification, if the acquisition device is in the off state, it needs to be turned on first, and then collect and encode data. In this case, if the acquisition device receives a termination instruction, it needs to stop collecting and encoding data, and control the acquisition device to return to the off state. That is, after device B exits the relay, the acquisition device of device C returns to the state before the relay.

[0160] In an exemplary implementation, when the relay service C receives a data relay request, it sends a relay notification to the Socket service; when the Socket service receives the relay notification, it actively creates a Socket connection between device C and device B, and the acquisition device inputs the encoded video stream into the data transmission channel corresponding to the Socket connection. The relay service C sends the Socket connection address in the local area network to device B; the relay service B forwards the Socket connection address to the video player started by the playback terminal B, and the video player pulls the video stream from the corresponding data transmission channel according to the Socket connection address, and decodes and plays the video stream.

[0161] In an exemplary implementation, the relay service C may also forward the termination instruction to the Socket service, and the Socket service disconnects the Socket connection between the device C and the device B in response to the termination instruction.

[0162] In some exemplary implementations, Fig.14 A data relay playback method executed by a relay initiating end device is provided, that is, the execution subject of the method is the aforementioned device B, and the method includes the following program steps:

[0163] Step S141 : when receiving an operation instruction to continue playing the content currently being played by the first target device, sending a first data relay request.

[0164] The first target device corresponds to Figure 12(a1) to Figure 12(b2) Device A in the video has the ability to play media data, and the first data relay request is used to obtain the media data and playback progress of the currently playing content from the first target device. The relay initiator can directly send the first data relay request to the first target device; or, the relay initiator sends the first data relay request to the server, which forwards it to the first target device.

[0165] Step S142, receiving the media data and the playing progress sent by the first target device. The first target device can send the data directly to the relay initiating end, or forward the data to the relay initiating end via the server.

[0166] Step S143: automatically jump to the playback progress and continue playing the media data.

[0167] Optionally, if the relay initiating end is equipped with a display, the UI relay interaction can be performed by a remote controller or touch control, and the method further includes generating the first data relay request according to the following steps:

[0168] When receiving an access request to the relay application page, querying the media information of the content currently being played by the relay-capable device associated with the smart device, the media information including the media type and the media name;

[0169] Displaying the generated playlist in the relay application page; wherein the playlist includes a number of relay options, and the relay options include media information and device information of the relayable device;

[0170] When a selection operation of a target relay option in the playlist is received, the first data relay request is generated according to the first target device corresponding to the target relay option. The device association may be based on a network association or an account association.

[0171] Optionally, if the relay initiating end is equipped with a sound collection device (such as a microphone), a voice relay interaction may be performed, and the method further includes generating the first data relay request according to the following steps:

[0172] Acquire voice information input by a user collected by a sound collection device, and analyze the voice information;

[0173] If it is analyzed that the user intends to continue playing the media data of the first target device, query the target device ID corresponding to the name of the first target device according to the device list; wherein the device list records the correspondence between the name of the currently relayable device and the device ID;

[0174] The first data relay request is generated according to the target device ID.

[0175] Optionally, relay playback of real-time collected media data is also supported, and the method further includes:

[0176] Upon receiving an operation instruction to play the real-time media data collected by the second target device, sending a second data relay request; the second data relay request is used to instruct the second target device to collect the real-time media data, and encode the real-time media data and send it to the smart device;

[0177] The real-time media data is received and decoded, and the playback device is controlled to play the decoded real-time media data.

[0178] The relay initiator may send the second data relay request directly to the second target device; or, the relay initiator may send the second data relay request to the server, which forwards the request to the second target device.

[0179] Optionally, for the relay playback scheme of real-time media data, the method further includes: upon receiving an operation instruction to exit the relay, sending a termination instruction, wherein the termination instruction is used to instruct the second target device to stop collecting and encoding the real-time media data. The relay initiator may directly send the termination instruction to the second target device; or, the relay initiator sends the termination instruction to the server, which then forwards it to the second target device.

[0180] Wherein the second target device corresponds to Fig.13 Device C in the example has the capability of collecting media data in real time.

[0181] In some exemplary implementations, Fig.15 A data relay playback method executed by a target device end of a relay is provided. The execution subject of the method is a first target device (i.e., device A). The method includes the following program steps:

[0182] Step S151, when receiving a data relay request, query the source of the currently playing media data. The data relay request may be sent by the first device, or forwarded by the first device via a server. The first device is the relay initiator. The data relay request in this embodiment corresponds to the aforementioned first data relay request.

[0183] Step S152, determine whether the media data is from local storage. If the media data is from network resources rather than local storage, execute step S153; if the media data is from local storage, execute steps S154 and S155.

[0184] Step S153: Send the URL address and the playback progress of the media data to the first device or the server. If sent to the server, the server forwards the URL address and the playback progress of the media data to the first device.

[0185] Step S154, generate a network access path corresponding to the local storage path, and send the network access path and playback progress of the media data to the first device or server. If sent to the server, the server needs to forward the network access path and playback progress to the first device.

[0186] Step S155: upon receiving the access request of the first device to the network access path, the local storage path is parsed from the network access path, the media data in the local storage path is converted into a network media stream, and the network media stream is sent to the first device or server. If sent to the server, the server needs to forward the network media stream to the first device.

[0187] In some exemplary implementations, Fig.16 Another data relay playback method executed by the relayed target device is provided. The execution subject of the method is the second target device (i.e., device C). The second target device is equipped with a collection device for real-time collection of specific types of media data (including video, audio, image, etc.). The method includes the following program steps:

[0188] Step S161, when receiving a data relay request, controlling the collection device to collect real-time media data.

[0189] The data relay request may be sent by the first device, or forwarded by the first device via the server. The first device is the relay initiator. The data relay request in this embodiment corresponds to the aforementioned second data relay request. When receiving the data relay request, if the collection device is in the open state, data can be collected directly; if the collection device is in the closed state, the collection device needs to be opened first and then the data is collected.

[0190] Step S162: Encode the real-time media data and send the encoded real-time media data to the first device or the server. If the data relay request is sent by the first device, the encoded real-time media data is directly sent to the first device; if the data relay request is forwarded by the first device via the server, the encoded real-time media data is sent to the server, and the server forwards the encoded data to the first device.

[0191] Optionally, refer to Fig.13 For example, a Socke connection can be established to build a data transmission channel between the two ends of the relay, so that the first device can pull the stream from the data transmission channel and decode and play it.

[0192] Step S163, when receiving the termination instruction generated by the first device when exiting the relay, stop collecting and encoding the real-time media data. When the first device receives the user's operation instruction to exit the relay, it generates a termination instruction, and the first device can directly send the termination instruction to the smart device end that collects data, or the first device can also send the termination instruction to the server, and the server forwards the termination instruction to the smart device end that collects data.

[0193] Figure 14 to Figure 16The provided method embodiments can all be referenced and verified with the aforementioned relay interaction logic between two-end devices, and will not be described in detail here.

[0194] As can be seen from the above technical solutions, this application can realize the sharing of media data among multiple smart devices. Users can view the media content played on other devices without operating separately on different devices, breaking the information islands between devices. When users switch areas and devices, they only need to indicate the target device object of the desired relay to the smart device in the new area to quickly complete the relay playback, without the need for users to perform tedious operations, and the playback method is flexible and fast, which improves the efficiency of media relay playback and user experience. Whether it is inherent media resources or real-time collected media data, relay playback can be achieved, improving scene adaptability and relay diversity, and users have a better experience of the relay function.

[0195] The data relay scheme between devices is not limited to the examples of the embodiments of the present application. Specifically, the data relay scheme and interaction logic between devices can be flexibly adjusted, modified and expanded according to the distribution of smart devices in the home scene, device categories, playback / collection capabilities, media types played and data sources, communication methods between devices, input methods for initiating relay instructions, user activity areas, etc. The UI display effects and UI-based user operations provided in this application are only exemplary, and should be based on actual product design and application. The hardware and software configurations and underlying architecture of smart devices that support relay interaction are also not limited to the examples of this application, and should specifically adapt to the requirements of device functions and operating systems. In addition, the communication and media data transmission methods between smart devices are not limited.

[0196] In an exemplary implementation, the present invention further provides a computer storage medium, which may store a program, and when the program is executed, the program steps involved in the data relay playback method executed by the aforementioned single-ended smart devices may be included, and the single-ended smart devices include the relay initiating end device B and the relayed target device end (including device A and device C). Among them, the computer storage medium may be a disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM), etc.

Claims

1. A smart device, characterized in that: include: A playback device, used for playing media data; A communicator, used for communicating with a server or other intelligent devices; The controller is configured to execute: After the smart device is started, synchronizing the device status with other smart devices, wherein the device status includes the connection status, relay capability and media playback capability of the device; receiving an operation instruction on the smart device to continue playing the content currently being played by the first target device, and if the first target device is determined to be a relayable device according to the device state, sending a first data relay request to the first target device; the first data relay request is used to obtain media data and playback progress of the content currently being played by the first target device; the first data relay request is also used to instruct the first target device to perform a preset action, the preset action including one of maintaining the current playback status, pausing playback, stopping playback, switching to a standby state, and shutting down; The media data and the playing progress are received, and the playing device is controlled to automatically jump to the playing progress to continue playing the media data.

2. The smart device according to claim 1, characterized in that: The playback device includes a display, and the controller is further configured to execute: Upon receiving an access request to a relay application page, querying the media information of the content currently being played by the relay-capable device associated with the smart device, the media information including the media type and the media name; Controlling the display to display the generated playlist in the relay application page; wherein the playlist includes a number of relay options, and the relay options include media information and device information of the relayable device; A selection operation of a target relay option in the playlist is received, and the first data relay request is generated according to a first target device corresponding to the target relay option.

3. The smart device according to claim 1, characterized in that: The smart device further includes a sound collection device, and the controller is further configured to execute: Acquire voice information input by a user collected by a sound collection device, and analyze the voice information; If it is analyzed that the user intends to continue playing the media data of the first target device, query the target device ID corresponding to the name of the first target device according to the device list; wherein the device list records the correspondence between the name of the currently relayable device and the device ID; The first data relay request is generated according to the target device ID.

4. The smart device according to claim 1, characterized in that: The controller is further configured to perform: Upon receiving an operation instruction to play the real-time media data collected by the second target device, a second data relay request is sent; the second data relay request is used to instruct the second target device to collect the real-time media data, and encode the real-time media data and send it to the smart device; The real-time media data is received and decoded, and the playback device is controlled to play the decoded real-time media data.

5. The smart device according to claim 4, characterized in that: The controller is further configured to perform: An operation instruction to exit the relay is received, and a termination instruction is sent, where the termination instruction is used to instruct the second target device to stop collecting and encoding the real-time media data.

6. The smart device according to claim 5, characterized in that: The controller is configured to: Sending the first data relay request to the first target device; or sending the first data relay request to the server, wherein the first data relay request is further used to instruct the server to forward the first data relay request to the first target device, and to instruct the server to send the media data and playback progress received from the first target device to the smart device; and / or, Sending the second data relay request and the termination instruction to the second target device; or, sending the second data relay request and the termination instruction to the server, wherein the second data relay request is used to instruct the server to forward the second data relay request to the second target device, and to instruct the server to send the real-time media data received from the second target device to the smart device, and the termination instruction is also used to instruct the server to forward the termination instruction to the second target device.

7. A smart device, characterized in that: include: A playback device, used for playing media data; A communicator, used for communicating with a server or other intelligent devices; The controller is configured to execute: After the smart device is started, synchronizing the device status with other smart devices, wherein the device status includes the connection status, relay capability and media playback capability of the device; Receiving a data relay request sent by the first device after determining that the smart device is a relayable device based on the device status, sending the currently playing media data and its playback progress to the first device, and executing a preset action indicated by the data relay request; wherein the playback progress is used to enable the first device to jump to the playback progress and continue playing the media data; the preset action includes one of maintaining the playback status, pausing playback, stopping playback, switching to standby mode, and shutting down.

8. The smart device according to claim 7, characterized in that: The controller is configured to send the media data in the following manner: Upon receiving the data relay request, querying the source of the currently playing media data; If the media data is from local storage, a network access path corresponding to the local storage path is generated, and the network access path is sent to the first device, or the network access path is forwarded to the first device via a server; receiving an access request from the first device to the network access path, parsing the local storage path from the network access path, converting media data in the local storage path into a network media stream, and sending the network media stream to the first device, or forwarding the network media stream to the first device via a server; Alternatively, if the media data originates from a network resource, the URL address of the media data is sent to the first device, or the URL address is forwarded to the first device via a server.

9. A data relay playback method, characterized in that: include: After the smart device is started, synchronize the device status with other smart devices, wherein the device status includes the connection status, relay capability and media playback capability of the device; receiving an operation instruction to continue playing the content currently being played by the first target device on the smart device, and if the first target device is determined to be a relayable device according to the device state, sending a first data relay request to the first target device; the first data relay request is used to obtain media data and playback progress of the content currently being played by the first target device; the first data relay request is also used to instruct the first target device to perform a preset action, the preset action including one of maintaining the playback status, pausing playback, stopping playback, switching to standby mode, and shutting down; The media data and the playback progress are received, and the playback progress is automatically jumped to continue playing the media data.