A media playing method, device and electronic device

By building media proxy services on client devices, cache media resources of server devices and implementing incremental transmission, the problem of real-time media playback between thin devices is solved, and the playback fluency and user experience are improved.

CN115134420BActive Publication Date: 2024-11-22HUAWEI TECH CO LTD
View PDF 16 Cites 0 Cited by

Patent Information

Application Number
CN202110312246.2
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-03-24
Publication Date
2024-11-22
Estimated Expiration
2041-03-24

AI Technical Summary

Technical Problem

The prior art is difficult to realize cross-device real-time playback of media between thin devices. The real-time streaming playback method requires high capabilities for server devices and is sensitive to network quality, resulting in poor user experience.

Method used

Build a media proxy service on the client device, cache the media resources of the server device to the local area, allowing applications to obtain and play directly from the local area, reducing the capability requirements of the server device, and real-time playback through incremental transmission.

Benefits of technology

Real-time media playback between thin devices is realized, which improves the fluency of streaming media playback, reduces the capability requirements for server devices, and improves user experience.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN115134420B_ABST
    Figure CN115134420B_ABST
Patent Text Reader

Abstract

The embodiments of the present application provide a media playback method, device, and electronic device. The method is used to implement the local media service function of the client, including: receiving a first playback request from an application, the first playback request including a first identifier; sending a first address to the application, the first address being a local uniform resource locator URL address; caching the target media resource corresponding to the first identifier of the server from the server device to the client device; when receiving a second playback request sent by the application to the first address, sending the cached target media resource to the application for playback. In this way, the server device only needs to provide ordinary file transfer functions to the electronic device, does not participate in and perceive the playback process, and therefore does not need to deploy playback-related services, which reduces the capability requirements for the server, thereby achieving real-time cross-device playback of media between thin devices.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of media playback technology, and in particular to a media playback method, device and electronic device. Background Art

[0002] In a distributed scenario, a client device can connect to multiple server devices through a network to play media resources in the server device or control the server device. Currently, the client device can play media resources in the server device in a local way and in a real-time streaming way.

[0003] Local playback means that when a user requests to play a media resource, the client device first pulls the complete audio / video file from the server device to the local device, and then starts playing. Since audio / video files are usually large, it takes some time from the user's request to play to the completion of the audio / video file pulling. Therefore, the user will feel obvious playback lag, and real-time playback cannot be achieved, resulting in a poor user experience.

[0004] Real-time streaming playback refers to the deployment of real-time streaming, content distribution and / or encoding and decoding services on the server device based on real-time streaming protocols or screen projection protocols, such as real-time streaming protocol (RTSP), real-time streaming protocol (HTTP live streaming, HLS) based on hypertext transfer protocol (HTTP), etc. When the client device requests to play the media resource, the server device sends the media stream to the client device based on the deployed service, and the client device plays the media stream while caching. However, the real-time streaming playback method requires the deployment of services on the server device, so it has high requirements on the capabilities of the server device and cannot be implemented in thin devices. In addition, the real-time streaming playback method also has certain requirements on the network quality. If the caching speed of the media stream is too low, the playback will be stuck.

[0005] It can be seen that there is currently no feasible solution for achieving cross-device real-time playback of media between thin devices. Summary of the invention

[0006] The embodiments of the present application provide a media playback method, apparatus, and electronic device to implement real-time cross-device playback of media between thin devices.

[0007] In the first aspect, an embodiment of the present application provides a media playback method. The method includes: a media proxy service on a client device receives a first playback request from an application on the client device, the first playback request includes a first identifier, and the first identifier corresponds to a target media resource in a server device; the media proxy service determines whether a cache transmission process of the target media resource has been started according to the first identifier; if the cache transmission process is not started, the media proxy service starts the cache transmission process to cache the target media resource from the server device to the client device, and sends a first address to the application, the first address is a local uniform resource locator URL address of the client device for accessing the cached target media resource; when the media proxy service receives a second playback request sent by the application to the first address, the cached target media resource is sent to the application for playback.

[0008] The media playback method provided in the embodiment of the present application builds a media proxy service on the client side. When the client device requests to play the media resources of the server device, the media proxy service can first cache the media resources locally, so that the application can directly request the media proxy service to obtain the media resources locally and play them. In this way, the server device only needs to provide the client device with ordinary file transfer functions, and does not participate in and perceive the playback process. Therefore, it is not necessary to deploy playback-related services, which reduces the capability requirements of the server device, thereby realizing real-time cross-device media playback between thin devices.

[0009] In one implementation, after the media proxy service determines whether the cache transmission process of the target media resource has been started, it also includes: if the cache transmission process has been started, but the cache transmission process is suspended, the media proxy service continues the cache transmission process to continue caching the target media resource from the server device to the client device according to the cache progress of the target media resource, and sends the second address to the application, the second address is the local URL address of the client device for accessing the cached target media resource; when the media proxy service receives the third play request sent by the application to the second address, the cached target media resource is sent to the application for playback. In this way, when the cache transmission is resumed from the suspended state, the media proxy service can continue to cache the target media resource in an incremental transmission manner without the need for re-caching.

[0010] In one implementation, after the media proxy service determines whether the cache transmission process of the target media resource has been started, the process further includes: if the target media resource has been completely cached in the client device, the media proxy service sends a third address to the application program, so that the application program obtains the target media resource from the third address for playback, and the third address is the file path of the target media resource cached in the client device. In this way, after the target media resource has been cached locally, the application program can directly access the locally stored target media resource file for playback, and the media proxy service is no longer required to generate a media stream.

[0011] In one implementation, after the media proxy service determines whether the cache transmission process of the target media resource has been started, the process further includes: if the target media resource has been completely cached to the client device, the media proxy service sends a fourth address to the application, where the fourth address is a local URL address of the client device for accessing the cached target media resource; when the media proxy service receives a fourth play request sent by the application to the fourth address, the cached target media resource is sent to the application for playback. In this way, after the target media resource has been cached locally, the media proxy service can continue to generate a media stream for the application, and the application can receive and play the media stream sent by the media proxy service.

[0012] In one implementation, the first address includes a local Internet Protocol IP address and a first port of the client device; and the second play request is a Hypertext Transfer Protocol HTTP request.

[0013] In one implementation, when the media proxy service receives the second play request sent by the application to the first address, the cached target media resource is sent to the application, including: the media proxy service monitors the first port of the local IP address; when the media proxy service monitors the second play request, the media proxy service generates a first response message, the first response message is an HTTP response message, and the first response message includes a media stream generated according to the cached target media resource. In this way, data transmission can be performed between the media proxy service and the application based on the HTTP protocol, such as transmitting media streams.

[0014] In one implementation, the method further includes: when the media proxy service receives the stop playback request from the application, if the cache transfer process is in progress, suspending the cache transfer process and recording the cache progress of the target media resource. In this way, when the user plays the video subsequently, the media proxy service can continue to execute the cache transfer process based on the cache progress.

[0015] In the second aspect, the embodiment of the present application provides a media playback method. The method includes: an application on a client device sends a first playback request to a media proxy service on the client device, the first playback request includes a first identifier, and the first identifier corresponds to a target media resource in a server device; the application receives a target address sent by the media proxy service; if the target address is a local URL address of the client device, the application sends a second playback request to the local URL address; the application receives the target media resource sent by the media proxy service in response to the second playback request and plays it, and the target media resource is cached from the server device to the client device by the media proxy service. The media playback method provided in the embodiment of the present application constructs a media proxy service on the client device. When the client device requests to play the media resource of the server device by the application, the media proxy service can first cache the media resource locally, so that the application can directly request the media proxy service to obtain the media resource from the local and play it. In this way, the server device only needs to provide a normal file transfer function to the client device, does not participate in and perceive the playback process, so there is no need to deploy playback-related services, which reduces the capability requirements for the server device, thereby realizing real-time cross-device playback of media between thin devices.

[0016] In one implementation, after the application receives the target address sent by the media proxy service, the application further includes: if the target address is a file path address in the client device, the application obtains the target media resource from the file path address for playback. In this way, after the target media resource has been cached locally, the application can directly access the locally stored target media resource file for playback, and the media proxy service is no longer required to generate a media stream.

[0017] In one implementation, the local URL address includes the local IP address and the first port of the client device; and the second play request is a Hypertext Transfer Protocol HTTP request.

[0018] In one implementation, the application sends the second play request to the local URL address, including: the application sends the second play request to the first port of the local IP address.

[0019] In one implementation, the application receives the target media resource sent by the media proxy service in response to the second play request, including: the application receives a first response message sent by the media proxy service in response to the second play request, the first response message is an HTTP response message, and the first response message includes a media stream generated by the media proxy service according to the cached target media resource. In this way, data transmission, such as transmission of media streams, etc., can be performed between the media proxy service and the application based on the HTTP protocol.

[0020] In the third aspect, an embodiment of the present application provides a media player device. The device includes: a virtual server, which is used to receive a first playback request from an application on a client device, the first playback request includes a first identifier, and the first identifier corresponds to a target media resource in a server device; the virtual server is also used to determine whether the cache transmission process of the target media resource has been started according to the first identifier; a file cache unit is used to start the cache transmission process to cache the target media resource from the server device to the client device if the cache transmission process is not started; the virtual server is also used to send a first address to the application, the first address is a local uniform resource locator URL address of the client device for accessing the cached target media resource; the virtual server is also used to send the cached target media resource to the application for playback when receiving a second playback request sent by the application to the first address.

[0021] In one implementation, the file cache unit is also used to, if the cache transfer process has been started but the cache transfer process is suspended, continue the cache transfer process to continue caching the target media resource from the server device to the client device according to the cache progress of the target media resource; the virtual server is also used to send a second address to the application, where the second address is a local URL address of the client device for accessing the cached target media resource; the virtual server is also used to send the cached target media resource to the application for playback when receiving a third playback request sent by the application to the second address.

[0022] In one implementation, the virtual server is also used to send a third address to the application if the target media resource has been completely cached in the client device, so that the application can obtain the target media resource from the third address for playback, and the third address is the file path of the target media resource cached in the client device.

[0023] In one implementation, the virtual server is further used to send a fourth address to the application if the target media resource has been completely cached to the client device, where the fourth address is a local URL address of the client device for accessing the cached target media resource; the virtual server is further used to send the cached target media resource to the application for playback upon receiving a fourth playback request sent by the application to the fourth address.

[0024] In one implementation, the first address includes a local Internet Protocol IP address and a first port of the client device; and the second play request is a Hypertext Transfer Protocol HTTP request.

[0025] In one implementation, the virtual server is used to send the cached target media resources to the application when receiving the second playback request sent by the application to the first address, including: the virtual server is used to listen to the first port of the local IP address; the virtual server is also used to generate a first response message when listening to the second playback request, the first response message is an HTTP response message, and the first response message includes a media stream generated according to the cached target media resources.

[0026] In one implementation, the virtual server is further configured to, upon receiving a request to stop playback from an application, suspend the cache transmission process if a cache transmission process is in progress, and record the cache progress of the target media resource.

[0027] In a fourth aspect, an embodiment of the present application provides a media player device. The device includes: a first sending unit, used to send a first play request to a media proxy service on a client device, the first play request includes a first identifier, and the first identifier corresponds to a target media resource in a server device; an address receiving unit, used to receive a target address sent by the media proxy service; a second sending unit, used to send a second play request to the local URL address if the target address is a local URL address of the client device; a playing unit, used to receive and play the target media resource sent by the media proxy service in response to the second play request, the target media resource being cached from the server device to the client device by the media proxy service.

[0028] In one implementation, the playback unit is further configured to obtain the target media resource from the file path address for playback if the target address is a file path address in the client device.

[0029] In one implementation, the local URL address includes the local IP address and the first port of the client device; and the second play request is a Hypertext Transfer Protocol HTTP request.

[0030] In one implementation, the second sending unit is used to send the second play request to the local URL address, including: sending the second play request to the first port of the local IP address.

[0031] In one implementation, the playback unit is used to receive a target media resource sent by a media proxy service in response to a second playback request, including: receiving a first response message sent by the media proxy service in response to the second playback request, the first response message being an HTTP response message, the first response message including a media stream generated by the media proxy service based on the cached target media resource.

[0032] In a fifth aspect, an embodiment of the present application provides an electronic device, comprising: a media proxy service module and an application module; the media proxy service module is used to execute the method of the first aspect and its various implementation methods; the application module is used to execute the method of the second aspect and its various implementation methods.

[0033] In a sixth aspect, an embodiment of the present application further provides a computer-readable storage medium, in which instructions are stored, and when the computer-readable storage medium is run on a computer, the computer executes the methods in the above aspects.

[0034] In a seventh aspect, an embodiment of the present application further provides a computer program product comprising instructions, which, when executed on a computer, enables the computer to execute the above-mentioned methods.

[0035] In an eighth aspect, an embodiment of the present application further provides a chip system, which includes a processor for supporting the above-mentioned device or system to implement the functions involved in the above-mentioned aspects, for example, generating or processing the information involved in the above-mentioned method. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 This is a schematic diagram of the architecture of a current distributed scenario;

[0037] Figure 2 This is a schematic diagram of the architecture of another distributed scenario.

[0038] Figure 3 is a schematic diagram of a user playing media resources in a server device on a client device according to an embodiment of the present application;

[0039] Figure 4 is a schematic diagram of a sub-page of a media resource page shown in an embodiment of the present application;

[0040] Figure 5 It is a schematic diagram of real-time streaming transmission and playback based on the real-time streaming protocol RTSP;

[0041] Figure 6 It is a schematic diagram of real-time streaming transmission and playback based on the real-time streaming protocol HLS;

[0042] Figure 7 is a schematic diagram of the hardware structure of a terminal device provided in an embodiment of the present application;

[0043] Figure 8 It is a schematic diagram of the software architecture provided by the embodiment of the present application;

[0044] Fig. 9 is a flow chart of a media playback method provided by an embodiment of the present application;

[0045] Fig.10It is a scene graph of the application generating the first play request shown in the embodiment of the present application;

[0046] Fig.11 It is a scene graph of the application generating a second play request shown in the embodiment of the present application;

[0047] Fig.12 is a flow chart of another media playback method shown in an embodiment of the present application;

[0048] Fig.13 is a flowchart showing the processing of the local media service in the start-playing scenario in an embodiment of the present application;

[0049] Fig.14 is a flowchart showing the processing of the local media service in the scenario of terminating playback in an embodiment of the present application;

[0050] Fig.15 is a hardware architecture diagram of another media playback method provided in an embodiment of the present application;

[0051] Fig.16 is a structural schematic diagram of a media playback device provided in an embodiment of the present application;

[0052] Fig.17 It is a schematic diagram of the structure of the chip system provided in the embodiment of the present application. DETAILED DESCRIPTION

[0053] A distributed scenario refers to a scenario in which multiple devices are connected through a network and interact with each other through information and data to jointly complete a task goal.

[0054] Figure 1 This is a schematic diagram of the architecture of a current distributed scenario. Figure 1 As shown, the distributed scene architecture may include one or more client devices 11, and a server 12. Among them, one or more client devices 11 are connected to a server 12 in a wide area network (WAN). The client device 11 may include, for example, a mobile phone, a laptop computer, a tablet computer, a large-screen display device, a virtual / mixed / augmented reality device, a local server, etc., which is not limited in the embodiments of the present application. Depending on the tasks performed by the distributed scene, the server 12 may include, for example, a media server, an application server, a storage server, a web server, a data center, a content distribution node, etc., which is not limited in the embodiments of the present application.

[0055] Figure 2 This is another architecture diagram of a distributed scenario. Figure 2As shown, the distributed scene may include multiple terminal devices 21, and the multiple terminal devices 21 may establish a connection through a wide area network WAN or a local area network (LAN). For example, when multiple terminal devices 21 establish a connection through a local area network WAN, the multiple terminal devices 21 may be connected to a wireless local area network (WLAN) created by a wireless access point (WAP) 22, and data transmission between each other is achieved through forwarding of the wireless access point 22. The terminal devices 21 here may include, for example, mobile phones, laptops, tablet computers, large-screen display devices, virtual / mixed / augmented reality devices, network attached storage (NAS), routers with storage functions, etc., wherein the router may be used as a wireless access point in the distributed scene.

[0056] As an implementation method, distributed scenarios can be used to implement the cross-device playback function of media objects, that is, one device plays the media resources in another device. For the sake of ease of description, the device that plays the media resources is called the client device, and the device that provides the media content is called the server device. Then, the way in which the client device plays the media content can generally include local playback and real-time streaming playback. Among them, local playback means that when the user requests to play the media resource, the client device first pulls the complete media file from the server device to the local, and then starts playing; real-time streaming playback refers to the deployment of real-time streaming, content distribution and / or encoding and decoding services on the server device based on the real-time streaming protocol or the screen projection protocol. When the client device requests to play the media resource, the server device sends the media stream to the client device based on the deployed service, and the client device caches and plays the media stream at the same time.

[0057] Figure 3 This is a schematic diagram of an embodiment of the present application showing a user performing a gesture operation in a client device to play a media resource in a server device.

[0058] Combine the following Figure 3 The local playback process is illustrated below.

[0059] like Figure 3As shown, the user can operate the client device 30 to enter the distributed entry interface 31 corresponding to the media playback provided by the client device 30, which may include, for example, a server device list 32, and the server device list 32 may include information 33 of some or all server devices in the distributed scenario. Among them, the server device information 33 may include, for example, one or more of the following information: device model (such as Mate 30pro, Huawei Smart Screen, etc.), device name (user-defined name, such as: Xiao Ming's Huawei mobile phone, etc.), device type (such as: mobile phone, tablet computer, notebook, large-screen display device), device status (online / offline), device occupancy status (such as whether it is occupied by other user devices, etc.).

[0060] Further Figure 3 As shown, when the user clicks 41 on the information 33 of any server device in the server device list 32, the client device 31 may display the media resource page 34 of the server device. The media resource page 34 may display the media files stored in the server device in the form of icons and / or text, wherein the icons may be, for example, thumbnails of videos or pictures, and the text may be, for example, the names of the media files.

[0061] In one implementation, the media resource page 34 may include multiple layers of pages. For example, when a user enters the media resource page 34, the media resource page may display a first layer page 35, the first layer page 35 includes at least one folder 36, and displays the name of each folder and the number of files contained in each folder. These folders may be created by the user, may be automatically created locally, or may be created by an application in a local program, and are used to classify media files according to sources, etc. The folder 36 may use thumbnails of one or more files contained in it as icons. When the user clicks 42 on any folder 36 on the first layer page 35, the client device 31 displays the second layer page 37 corresponding to the folder 36, and the second layer page 37 may include information such as icons 38 and names of each media file in the folder 36. It is understandable that the media resource page 34 may include fewer layers of pages (for example, only the first layer page 35 or only the second layer page 37) or more layers of pages when implemented, which may be determined according to the storage path of the media files in the server device, and the embodiment of the present application does not limit this.

[0062] Next, the user can click 43 on the icon 38 of any media file to request to play, and a file transfer can be established between the client device and the server device, and the media file in the server device is transferred to the client device. After receiving the complete media file, the client device starts to play the media file. The access and transmission process of the media file can be implemented through the file transfer protocol (FTP) service, the Web-based distributed authoring and versioning (WebDAV) service, etc., which is not limited in the embodiments of the present application.

[0063] It is understandable that local playback is similar to downloading media resources from the network and then playing them. It is necessary to first pull the media file from the server device to the client device and then play it. Therefore, there will be a certain delay from the user's request to play to the start of the media file playing, causing the user to feel a sense of lag in the experience. This delay is usually determined by the file transfer speed and the size of the media file. For example, if the media file is a movie, the transfer time may be as long as tens of seconds, which seriously affects the user experience.

[0064] In one implementation, Figure 4 As shown, when the media files in the server device include multiple types, the media resource page 34 may also include multiple sub-pages, each sub-page corresponding to one type, and the user may switch between different sub-pages by swiping 44 or clicking 45 the sub-page name. The above multiple types may include, for example, pictures, videos, audios, documents, others, recent, etc., which is not limited in the present embodiment.

[0065] Figure 5 This is a schematic diagram of real-time streaming playback based on the real-time streaming protocol RTSP. The real-time streaming protocol RTSP is a network application protocol designed for use in entertainment and communication systems to control the server. This protocol is used to create and control media sessions between terminals. The client connected to the server can issue commands such as play, record, and pause to facilitate real-time control of the media stream from the server to the client or from the client to the server. Figure 4 As shown, the real-time streaming transmission and playback based on the real-time streaming protocol RTSP can be achieved by the following steps:

[0066] Step S1, OPTIONS request process. Specifically includes:

[0067] Step S11: The client sends an OPTIONS request message (OPTIONS request) to the server to request the server for acceptable request types.

[0068] Step S12: The server sends an OPTIONS response message (OPTIONS response) to the client, so as to send all acceptable request types to the client.

[0069] Step S2, DESCRIBE request process. Specifically includes:

[0070] Step S21: the client sends a DESCRIBE request message (DESCRIBE request) to the server to request to obtain media initialization description information from the server.

[0071] Step S22: the server sends a DESCRIBE response message (DESCRIBE response) to the client to send the media initialization description information to the client.

[0072] Step S3, SETUP request process. Specifically includes:

[0073] Step S31: The client sends a SETUP request message (SETUP request) to the server to set the attributes and transmission mode of the media session and remind the server to establish the session.

[0074] Step S32: the server establishes a session and sends a SETUP response message (SETUP response) to the client to send the session identifier and related information to the client.

[0075] Step S4, PLAY playback request process, including:

[0076] Step S41: the client sends a play request message (PLAY request) to the server to request to play a media resource.

[0077] Step S42: the server sends streaming media data (PLAYresponse) to the client in response to the PLAY request.

[0078] Step S5, TEARDOWN request process, including:

[0079] Step S51: The client sends a TEARDOWN request message (TEARDOWN request) to the server to request closing the media session.

[0080] Step S52: The server responds to the TEARDOWN request and closes the session (TEARDOWN response).

[0081] It can be seen that real-time streaming transmission and playback based on RTSP involves a large number of interactive processes and has high requirements on device resources. In addition, RTSP requires the deployment of RTSP services in server devices. Generally speaking, for fat devices such as servers, it is feasible to deploy RTSP services, but for thin devices such as mobile phones and tablets, there is a performance bottleneck in deploying RTSP services. Therefore, real-time streaming transmission and playback based on RTSP is difficult to achieve on the Internet. Figure 2 In addition, RTSP streaming media transmission and playback are performed simultaneously, so the smoothness of playback depends on the network quality. When the network quality is poor, playback may be stuck.

[0082] Figure 6 This is a schematic diagram of real-time streaming playback based on the real-time streaming protocol HLS. HLS is a streaming network transmission protocol based on the hypertext transfer protocol (HTTP). It works by dividing the entire stream into small HTTP-based files for download, downloading only some at a time. When the streaming media is playing, the client can choose to download the same resource from many different alternative sources at different rates, allowing the streaming session to adapt to different data rates. In order to implement the corresponding functions of the above HLS, such as Figure 6 As shown in the figure, HLS includes not only the server and client, but also the content distribution end Distribution. Figure 5 , the specific process of HLS to achieve real-time streaming transmission and playback may include:

[0083] On the server side: The server side obtains media resources, encodes the media resources to obtain a media stream, then divides the media stream through a stream divider to obtain multiple small video files (.ts), and generates playlist files (Index file) corresponding to these video files, and finally sends the obtained video files (.ts) and playlist files to the content distribution end.

[0084] On the content distribution end and the client: The content distribution end is used to store video files (.ts) and can send the playlist file (Index file) to the client; the client can obtain the address of each video file (.ts) stored on the content distribution end according to the list file (Index file), and then obtain the video files (.ts) according to these addresses and play them.

[0085] It can be seen that the real-time streaming transmission and playback based on HLS requires the deployment of file segmentation services and content distribution services in the server-side device, which has high requirements on device resources and is also difficult to implement on the server-side device. Figure 2In addition, HLS streaming media transmission and playback are also carried out simultaneously, so the smoothness of playback depends on the network quality. When the network quality is poor, playback may also be stuck.

[0086] In addition, there are some screen projection solutions that may be used to achieve real-time streaming playback in distributed scenarios, but these solutions also require the deployment of media decoding and encoding services on the server side, and streaming media transmission and playback are also carried out simultaneously, so the same problems exist in RTSP and HLS solutions.

[0087] In order to solve the above problems, realize real-time streaming transmission and playback between thin devices, and improve the smoothness of streaming media playback, an embodiment of the present application provides a media playback method.

[0088] The method provided in the embodiment of the present application can be applied to Figure 1 or Figure 2 The distributed scenario shown can preferably be applied to Figure 2 Provides a distributed scenario consisting of thin devices.

[0089] Figure 7 Schematic diagram of the hardware structure of the terminal device provided in the embodiment of the present application. Figure 7 As shown, the terminal device 100 may include a processor 110, a memory 120, a universal serial bus (USB) interface 130, a radio frequency circuit 140, a mobile communication module 150, a wireless communication module 160, a camera 170, a display screen 180, a touch sensor 190, an air pressure sensor 210 and a button 220, etc.

[0090] The processor 110 may include one or more processing units, for example: the processor 110 may include an application processor (application processor, AP), a modem processor, a graphics processor (graphics processing unit, GPU), an image signal processor (image signal processor, ISP), a video codec, a digital signal processor (digital signal processor, DSP), a baseband processor, and / or a neural-network processing unit (neural-network processing unit, NPU), etc. Among them, different processing units can be independent devices, or they can be integrated into one or more processors, for example, integrated into a system on a chip (system on a chip, SoC). A memory can also be set in the processor 110 for storing instructions and data. In some embodiments, the memory in the processor 110 is a high-speed cache memory. The memory can save instructions or data that the processor 110 has just used or circulated.

[0091] In some embodiments, the processor 110 may include one or more interfaces. The interface may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0092] The memory 120 can be used to store computer executable program codes, and the executable program codes include instructions. The memory 120 may include a program storage area and a data storage area. Among them, the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc. The data storage area may store data created during the use of the terminal device 100 (such as audio data, a phone book, etc.), etc. In addition, the memory 120 may include one or more storage units, for example, it may include a volatile memory (volatile memory), such as: a dynamic random access memory (DRAM), a static random access memory (SRAM), etc.; it may also include a non-volatile memory (NVM), such as: a read-only memory (ROM), a flash memory, etc. The processor 110 executes various functional applications and data processing of the terminal device 100 by running instructions stored in the memory 120 and / or instructions stored in a memory provided in the processor.

[0093] The wireless communication function of the terminal device 100 can be implemented through the radio frequency circuit 140, the mobile communication module 150, the wireless communication module 160, the modem processor and the baseband processor.

[0094] The radio frequency circuit 140 may include at least one antenna 141 for transmitting and receiving electromagnetic wave signals. Each antenna in the terminal device 100 may be used to cover a single or multiple communication frequency bands. In some embodiments, the antenna may be used in combination with a tuning switch.

[0095] The mobile communication module 150 can provide solutions for wireless communications including 2G / 3G / 4G / 5G applied to the terminal device 100. The mobile communication module 150 may include at least one filter, a switch, a power amplifier, a low noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves from the antenna 141, and filter, amplify, and process the received electromagnetic waves, and transmit them to the modulation and demodulation processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modulation and demodulation processor, and convert it into electromagnetic waves for radiation through the antenna 141. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the processor 110. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the same device as at least some of the modules of the processor 110.

[0096] The modem processor may include a modulator and a demodulator. Among them, the modulator is used to modulate the low-frequency baseband signal to be sent into a medium-high frequency signal. The demodulator is used to demodulate the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After the low-frequency baseband signal is processed by the baseband processor, it is passed to the application processor. The application processor outputs a sound signal through an audio device (including but not limited to a speaker, a receiver, etc.), or displays an image or video through a display screen 180. In some embodiments, the modem processor may be an independent device. In other embodiments, the modem processor may be independent of the processor 110 and be set in the same device as the mobile communication module 150 or other functional modules.

[0097] The wireless communication module 160 may include a wireless fidelity (Wi-Fi) module, a Bluetooth (BT) module, a GNSS module, a near field communication (NFC) module, an infrared (IR) module, etc. The wireless communication module 160 may be one or more devices integrating at least one of the above modules. The wireless communication module 160 receives electromagnetic waves via the antenna 141, modulates and filters the electromagnetic wave signals, and sends the processed signals to the processor 110. The wireless communication module 160 may also receive the signal to be sent from the processor 110, modulate the frequency, amplify it, and convert it into electromagnetic waves for radiation via the antenna 141.

[0098] In the embodiment of the present application, the wireless communication function of the terminal device 100 may include, for example, global system for mobile communications (GSM), general packet radio service (GPRS), code division multiple access (CDMA), wideband code division multiple access (WCDMA), time-division code division multiple access (TD-SCDMA), long term evolution (LTE), fifth generation mobile networks new radio (5G NR), BT, GNSS, WLAN, NFC, FM, and / or IR. GNSS may include global positioning system (GPS), global navigation satellite system (GLONASS), Beidou navigation satellite system (BDS), quasi-zenith satellite system (QZSS) and / or satellite based augmentation system (SBAS).

[0099] The camera 170 is used to capture still images or videos. The camera 170 includes a lens and a photosensitive element. The object generates an optical image through the lens and is projected onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, and then passes the electrical signal to the ISP for conversion into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV, RYYB or other format. In some embodiments, the terminal device 100 may include 1 or N cameras 170, where N is a positive integer greater than 1.

[0100] NPU is a neural network (NN) computing processor. By drawing on the structure of biological neural networks, such as the transmission mode between neurons in the human brain, it can quickly process input information and can also continuously self-learn. Through NPU, applications such as intelligent cognition of the terminal device 100 can be realized, such as image recognition, face recognition, voice recognition, text understanding, etc.

[0101] The display screen 180 is used to display images, videos, etc. The display screen 180 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode or an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), MiniLED, MicroLED, Micro-OLED, quantum dot light-emitting diodes (QLED), etc. In some embodiments, the terminal device 100 may include 1 or N display screens 180, where N is a positive integer greater than 1.

[0102] The touch sensor 190 is also called a "touch control device". The touch sensor 190 can be arranged on the display screen 180. The touch sensor 190 and the display screen 180 form a touch screen, also called a "touch control screen". The touch sensor 190 is used to detect touch operations acting on or near it. The touch sensor can pass the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through the display screen 180. In other embodiments, the touch sensor 190 can also be arranged on the surface of the terminal device 100, which is different from the position of the display screen 180.

[0103] The air pressure sensor 210 is used to measure air pressure. In some embodiments, the terminal device 100 calculates the altitude through the air pressure value measured by the air pressure sensor 210 to assist positioning and navigation.

[0104] The key 220 includes a power key, a volume key, etc. The key 220 may be a mechanical key or a touch key. The terminal device 100 may receive key input and generate key signal input related to user settings and function control of the terminal device 100.

[0105] It is understood that the structure illustrated in the embodiment of the present application does not constitute a specific limitation on the terminal device 100. In other embodiments of the present application, the terminal device may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0106] exist Figure 2 In the distributed scenario shown, any terminal device can be used as a server device or a client device in real-time streaming playback. Whether a terminal device is a server device or a client device depends on whether it is a provider or a player of media resources. For example, for terminal device A and terminal device B, if terminal device A plays a media file in terminal device B across devices, then terminal device A is a client device and terminal device B is a server device; conversely, if terminal device B plays a media file in terminal device A across devices, then terminal device B is a client device and terminal device A is a server device.

[0107] The various steps of the media playback method provided in the embodiment of the present application can be implemented in the client device and the server device, mainly in the client device. The embodiment of the present application also provides a software architecture of a terminal device. The software architecture can be used for the client device and the server device to implement the various steps of the media playback method.

[0108] Figure 8 Schematic diagram of the software architecture provided by the embodiment of the present application. Figure 8 As shown, the software architecture includes a client part and a server part. The client part may include an application and a local media service (also referred to as a media proxy service, a local proxy service, etc., which is not limited in the present embodiment of the application), and the server part may include a file management function.

[0109] The application program can generally be an application program with file browsing and playing functions, such as a file manager application, a player application, or other media applications, etc., which is not limited in the embodiments of the present application. The application program can be a system application of the client device or a third-party application, which is not limited in the embodiments of the present application.

[0110] The local media service is a functional module newly added in the client device in the embodiment of the present application. The local media service may, for example, include a virtual server, a file caching function, and a file transfer function. Among them, the local media service can be regarded as a proxy service set up between the application and the server device, which isolates the original streaming media transmission behavior from the server device to the application, receives and caches the media resources from the server device, and provides the application with access to the media resources. The specific functions of the local media service will be further explained in the following embodiments.

[0111] The file management function of the server device is only required to implement conventional file storage and transmission functions, so there is no need to deploy any real-time streaming protocol related services in the server device.

[0112] Fig. 9 is a flow chart of a media playback method provided by an embodiment of the present application. The method can be applied to Figure 8 Client device with the software architecture shown. Fig. 9 As shown, the method may include the following steps:

[0113] Step S101: The local media service obtains media resource information of the server device.

[0114] The media resource information of the server device may include, for example, the file identifier of the media resource stored in the server device. In the server device, each media file has a file identifier, and different media files have different file identifiers. Therefore, the file identifier can be used by the client device and the server device to identify the unique identity of a media file.

[0115] In one implementation, the file identifier may be a file descriptor (FD), which may be in the form of a non-negative integer and may be generated by the server device when creating a media file, which is not limited in this embodiment of the present application.

[0116] In one implementation, the file identifier may be a path of a media file, such as an HTTP path of an audio file in a local area network in a distributed scenario, etc. This embodiment of the present application does not limit this.

[0117] It is understandable that, in addition to the FD and path shown above, the file identifier may also include other forms of file identifiers, which will not be described here.

[0118] In one implementation, the media resource information may also include one or more of the file name, file type, file size, thumbnail, and creation time of each media file.

[0119] It should be noted that, in the embodiment of the present application, step S101, i.e., the action of the local media service obtaining the media resource information of the server device, can be triggered in a variety of ways, and the embodiment of the present application does not limit this, as long as the media resource information can be obtained before the application triggers the first application request. For example: during the operation of the local media service, the local media service can obtain the media resource information of the server device at a regular interval; or, when the server device detects that the media resource information has changed (for example, when a file is added, a file is deleted, or the file attributes have changed), it actively sends the latest media resource information to the local media service; or, the local media service can trigger the behavior of obtaining the media resource information of the server device when the application is started.

[0120] Step S102: The application sends a first play request to the local media service, where the first play request includes a first identifier.

[0121] The implementation of step S102 is described in detail below in conjunction with a usage scenario.

[0122] Fig.10 This is a scene graph of the application generating the first playback request shown in the embodiment of the present application.

[0123] like Fig.10 As shown, when the file management application in the client device is opened and the “other devices” option is clicked 46 , the file management application can display the server device list 32 to the user.

[0124] Next, if the user wants to access the media resources in a server device, he can click on the server device. When the application detects that the user clicks 47 on a server device, it can obtain the media resource information of the server device from the local media service and generate a media resource page 35 of the server device based on the media resource information.

[0125] Next, if the user wants to play a certain media resource, hereinafter referred to as a target media resource, the user can click 48 on the icon 39 of the target media resource. When the application detects that the user clicks 48 on the icon 39 of the target media resource, the application sends a first play request to the local media service. It can be understood that the first identifier in the first play request is the media identifier of the target media resource.

[0126] It should be noted here that in addition to generating a first playback request in response to the user's touch input, the application can also generate a first playback request in other ways, such as responding to the user's voice input, responding to remote control input, etc., which is not limited to the embodiments of the present application.

[0127] In one implementation, the communication between the application and the local media service can be implemented through a virtual server in the local media service. The virtual server can be, for example, an HTTP virtual server, so that messages and data transmission can be established between the application and the HTTP virtual server based on the HTTP protocol.

[0128] In a specific implementation, the HTTP virtual server can listen to the application's request on the local IP address and a specific port. The port can be agreed upon in advance by the application and the local media service. For example, the local IP address can generally be 127.0.0.1, and the port can be any one or more of 1 to 65536. For ease of description, the port is represented by xx below. Accordingly, the first play request can be an HTTP request sent to 127.0.0.1:xx. In this way, the local media service can receive the first play request by listening to 127.0.0.1:xx.

[0129] In the embodiment of the present application, after receiving the first identifier, the local media service can first determine whether it has started the cache transmission process of the target media resource corresponding to the first identifier. If the local media service has not started the cache transmission process, it means that the application has not played the target media resource before, and the application is sending the first play request to the local media service for the first time. In this case, the local media service and the application can perform steps S103-S107.

[0130] It should be noted that, in the embodiment of the present application, "the cache transmission process is not started" means that the local media service has not started the target media resource transmission process from the server device to the client device. At this time, the client device has not cached any file fragments of the target media resource locally. This state generally corresponds to the scenario where the user requests to play the target media resource for the first time.

[0131] Step S103: The local media service sets a first address.

[0132] The first address may be an address of the client device for accessing the cached target media resource, and may be used by the application program to subsequently obtain the media stream corresponding to the target media resource.

[0133] In one implementation, the first address may be a local uniform resource locator URL address. If the application and the virtual server establish message and data transmission based on the HTTP protocol, the first address is also an HTTP address, for example: http: / / 127.0.0.1:xx / y, where y represents the file path of the target media resource to be cached in the client device.

[0134] Step S104 : The local media service starts a cache transmission process to cache the target media resource of the server device to the client device.

[0135] Step S105: The local media service sends the first address to the application program, and the application program receives the first address.

[0136] In a specific implementation, step S105 may be executed by a virtual server of the local media service. When the virtual server is an HTTP virtual server, the HTTP virtual server may send the first address to an application via an HTTP protocol message.

[0137] In a specific implementation, step S104 and step S105 can be performed synchronously, or step S105 can be performed before step S104, or step S105 can be performed after step S104. This means that the local media service can start caching the target media resource at the same time as sending the first address to the application, and the application has not yet received the user's request to start playing the media resource. It can be seen that in the embodiment of the present application, the caching process and the playback process of the media resource are independent and decoupled from each other, and the caching process of the media resource can be performed before the playback process.

[0138] In one implementation, step S105 can be implemented by the file cache function and file transfer function of the local media service. In a specific implementation, the file cache function can first obtain the storage location of the target media resource based on the distributed file system formed between the client device and the server device, and the storage location can also be a URL.

[0139] by Figure 2 Taking the distributed scenario shown as an example, assuming that the LAN IP address of the client device in the distributed scenario is 192.168.1.105, the LAN IP address of the server device is 192.168.1.107, and the storage path of the target media resource in the server device is a / b / c, then when the server device and the client device use the FTP protocol (only as an example, other protocols can also be used, which is not limited here) to transmit data, the storage location of the target media resource can be: ftp: / / 192.168.1.107 / a / b / c.

[0140] Next, the file caching function can determine the caching strategy based on some set strategy factors. The strategy factors that can be used here may include but are not limited to one or more of the following: the network capability of the server device, the network capability of the client device, the protocol rate between the server device and the client device, the disk I / O performance of the server device, the disk I / O performance of the client device, the disk I / O load of the server device, the disk I / O load of the client device, the processor load of the server device, the processor load of the client device, etc.

[0141] In addition, the cache strategies that can be used here may include but are not limited to one or more of the following: cache speed, the size of each file segment when using file segment transmission, buffer health (referring to the time difference between the cached progress and the progress played by the application), etc.

[0142] The embodiment of the present application does not specifically limit the relationship between the policy factor and the cache policy, and those skilled in the art can reasonably determine it based on their knowledge in the field. For example: the higher the network capability of the server device and the client device, the faster the cache speed, the larger the size of each file fragment, and the lower the buffer health; the lower the network capability of the server device and the client device, the smaller the size of each file fragment and the higher the buffer health.

[0143] Next, the file caching function can initiate cross-device transmission of the target media resource according to the caching strategy, for example: obtaining the target media resource from the storage location of the target media resource based on the determined caching speed, and caching it in segments at the first address until the cached target media resource meets the buffer health requirement.

[0144] Step S106: The application sends a second play request to the first address.

[0145] The second play request may be a request from a user to start playing a media resource.

[0146] Fig.11 This is a scene graph of the application generating a second playback request shown in an embodiment of the present application.

[0147] Fig.11 The scenario shown could be Fig.10 Continuation of the scene shown. Fig.11As shown, when the user wants to play the target media resource and clicks the icon of the target media resource, the application can enter the play interface 51 of the target media resource. In the play interface 51, the application can start playing the target media resource in at least two ways, one way is "not automatically play" and the other way is "automatically play". The "not automatically play" and "automatically play" modes can be set by the application as a default. It can also be set by the user.

[0148] In the "no automatic play" mode, the play interface 51 may include a start play button 52, and the play starts when the application detects that the user clicks 49 the start play button 52. Then, if the application adopts the "no automatic play" mode, the application may send a second play request to the first address when the user 49 clicks the start play button 52.

[0149] In the "auto play" mode, when the application detects that the user clicks the icon of the target media resource, it enters the play interface 51 of the target media resource and starts playing directly. Then, if the application adopts the "auto play" mode, the application can immediately send a second play request to the first address as long as the first address is obtained after detecting that the user clicks the icon of the target media resource. Among them, if the application is playing the target media resource for the first time, the application may not have obtained the first address in advance, and the second play request can be sent to the first address immediately after the local media service feedbacks the first address; if the application is not playing the target media resource for the first time, the application may have obtained the first address during the previous play, and the second play request can be sent to the first address immediately.

[0150] In one implementation, the second play request may be an HTTP request, which may be sent to a first port of the first address, and the first port may be a port agreed upon in advance by the application and the local media service. The application and the local media service may use the same port agreed upon by the first play request and the second play request, or may use different ports agreed upon by the first play request and the second play request, which is not limited in the embodiments of the present application.

[0151] Step S107: When receiving the second play request, the local media service sends the cached target media resource to the application program for play.

[0152] In a specific implementation, step S107 may be performed by a virtual server of a local media service. The virtual server may be, for example, an HTTP virtual server. The virtual server may create a service socket ServerSocket for the local IP address and the first port, monitor the HTTP request received at the first port of the local IP address through ServerSocket, and if the second play request is monitored, send the cached target media resource to the application for play.

[0153] In one implementation, the local media service sends the cached target media resource to the application for playback, which can be specifically implemented in the following manner: after monitoring the second playback request, the local media service can read the file segments of the segmented cached target media resource from the first address, process the file segments into media streams according to the order of the audio / video time corresponding to the file segments, and then send the media streams to the application for playback. The media stream can be a media stream based on the HTTP protocol, which is sent to the application in the form of an HTTP response message.

[0154] Furthermore, after receiving the media stream, the application may use its configured audio / video decoder, renderer, etc. to decode and render the media stream, thereby generating sound or image.

[0155] It can be seen from the above technical solutions that the media playback method provided in the embodiment of the present application builds a local media service on the client side. When the client device requests to play the media resources of the server device through the application, the media resources can be cached locally through the local media service first, so that the application can directly request the local media service to obtain the media resources locally and play them. In this way, the server device only needs to provide the client device with ordinary file transfer functions, and does not participate in and perceive the playback process. Therefore, there is no need to deploy playback-related services, which reduces the capability requirements of the server device, thereby realizing real-time cross-device media playback between thin devices.

[0156] It is understandable that in the scenario of real-time cross-device media playback, for the same target media resource in the server device, the application may or may not be playing it for the first time, depending on the user's wishes. When the application plays the target media resource for the first time, the local media service does not start the cache transmission process, so the client device can fully execute the method of steps S101-S107; when the application does not play the target media resource for the first time, the local media service may have already started the cache transmission process in the previous playback process. In this case, the client can also perform other method steps according to the cache status of the target media resource.

[0157] Specifically, in the scenario of real-time playback of media across devices, users may perform playback control operations such as starting playback, pausing playback, and stopping playback, or repeatedly play the target media resource. In this case, the target media resource can have different cache states. Taking the "start playback" scenario as an example, the cache state of the target media resource can include "cache transfer process not started" and "cache transfer process started". "cache transfer process started" can further include "cache transfer suspended", "cache transfer in progress", "cache transfer completed" and other states, among which:

[0158] "Cache transfer suspension" means that the local media service has previously started the target media resource transmission process from the server device to the client device, but due to some reasons (such as the user clicks on pause playback or temporarily closes playback), the transmission process is suspended, but the connection and pre-applied system resources used to transmit the target media resource between the server device and the local media service are not released. At this time, the local media service records the cache location of the target media resource on the client, the overall length of the target media resource, and the current transmission length of the target media resource (i.e., the transmission progress).

[0159] "Cache transfer in progress" means that the local media service has previously started the target media resource transmission process from the server device to the client device, and data transmission is in progress. This state generally corresponds to the user playing the target media resource, or pausing the target media resource, but due to some reasons (for example, the cached target media resource does not meet the buffer health requirements), the caching action of the target media resource is still in progress.

[0160] “Cache transfer completed” means that the cross-device transmission of the target media resource has been completed, and the client device has completely cached each file segment of the target media resource.

[0161] Further Fig. 9 As shown, when the local media service has started the cache transmission process, but the cache transmission process is suspended, the local media service and the application can perform the following steps S108-S112 after step S102:

[0162] Step S108: The local media service sets a second address.

[0163] The second address may be an address of a client device for accessing a cached target media resource, and the second address may be the same as or different from the first address. When both the first address and the second address are HTTP addresses, the difference between the first address and the second address may be only in the ports, for example, the first address may include a local IP address and a first port, and the second address may include a local IP address and a second port.

[0164] It should be noted that both the first address and the second address can be a temporary address. The local media service can generate such a temporary address each time it receives a first play request, so that the application can subsequently obtain the media stream corresponding to the target media resource from the temporary address.

[0165] Step S109 : The local media service continues the cache transmission process to continue caching the target media resource from the server device to the client device according to the cache progress of the target media resource.

[0166] Step S110: The local media service sends the second address to the application program, and the application program receives the second address.

[0167] Step S111: The application sends a third play request to the second address.

[0168] The third play request may be a request from the user to continue playing the media resource.

[0169] Step S112: When receiving the third play request, the local media service sends the cached target media resource to the application program for play.

[0170] In a specific implementation, step S112 may be performed by a virtual server of a local media service. The virtual server may be, for example, an HTTP virtual server. The virtual server may create a service socket ServerSocket for the local IP address and the second port, monitor HTTP requests received at the second port of the local IP address through ServerSocket, and send the cached target media resource to the application for playback if a third play request is monitored.

[0171] Fig.12 This is a flow chart of another media playback method shown in an embodiment of the present application. This method can be applied to the situation where the local media service has cached the target media resource locally.

[0172] In one implementation, Fig.12 As shown, the method may include the following steps 201-S203:

[0173] Step S201: The application sends a first play request to a first address.

[0174] Step S202: The local media service sends a third address to the application program in response to the first play request.

[0175] The third address may be a file path of the target media resource cached locally.

[0176] The difference between the first address and the third address is explained below: the first address is a local URL address, which is composed of the local IP address and port http: / / 127.0.0.1:xx and the file path y of the target media resource cached in the client device; unlike the first address, after the target media resource is cached, the local media service no longer needs to transmit data from the server device, and the target media resource no longer needs to be occupied by the HTTP virtual server, so the target media resource no longer needs to use the URL address, so the third address can directly be the file path y of the target media resource cached in the client device.

[0177] For example, y may be in the following form: Storage (root directory of the client device) / Movies / ShareMedia / 1.mov.

[0178] Step S203: the application reads the target media resource from the third address for playing.

[0179] The application can access the target media resource file located at the third address, such as 1.mov, and play the file. This process is the same as the application playing local media and does not involve the streaming process.

[0180] In another implementation, Fig.12 As shown, the method may include the following steps 201, S204-S207:

[0181] Step S201: The application sends a first play request to a first address.

[0182] Step S204: The local media service sets a fourth address.

[0183] The second address may be an address of a client device for accessing a cached target media resource, and the fourth address may be the same as or different from the first address and the second address. When the first address, the second address, and the fourth address are all HTTP addresses, the difference between the first address, the second address, and the fourth address may be only in the port. For example, the first address may include a local IP address and a first port, the second address may include a local IP address and a second port, and the fourth address may include a local IP address and a fourth port.

[0184] Step S205: The local media service sends the fourth address to the application program, and the application program receives the fourth address.

[0185] Step S206: The application sends a fourth play request to a fourth address.

[0186] The fourth play request may be a request from the user to continue playing the media resource.

[0187] Step S207: When receiving the third or fourth play request, the local media service sends the cached target media resource to the application for play.

[0188] In a specific implementation, step S207 may be performed by a virtual server of a local media service. The virtual server may be, for example, an HTTP virtual server. The virtual server may create a service socket ServerSocket for the local IP address and the fourth port, monitor HTTP requests received at the fourth port of the local IP address through ServerSocket, and if the fourth play request is monitored, send the cached target media resource to the application for play.

[0189] Fig.13 This is a complete processing flow chart of the local media service in the start playback scenario according to an embodiment of the present application. Fig.13 As shown, the complete processing flow may include the following steps:

[0190] Step S301: The application sends a request for a media stream resource corresponding to a target media resource to a local media service.

[0191] In one implementation, when the application detects that the user clicks the icon of the target media resource on the media resource page, the application sends a play request to the local media service, and the request is a request for applying for the media stream resource.

[0192] In one implementation, when the application detects that the user clicks a resume button when the target media resource is paused, the application sends a resume request to the local media service, which is a request for applying for a media stream resource.

[0193] Step S302: The local media service obtains the cache status of the target media resource according to the request of the application.

[0194] Depending on the cache status, the local media service can then perform different processing flows.

[0195] In the "Cache transfer not started" state, the local media service can perform the following steps:

[0196] Step S303a: The local media service obtains the storage location of the target media resource in the server device.

[0197] Step S303b: The local media service starts a cross-device transmission process and caches the target media resources.

[0198] During the cross-device transmission of the target media resource, the local media service is responsible for updating and managing the various file segments of the target media resource cached in the client device in real time.

[0199] Step S303c: The local media service generates media stream data based on the cached target media resource, and sends the media stream data to the client device for playback.

[0200] In the "Cached Transfer Pending" state, Local Media Services can perform the following steps:

[0201] Step S304a: The local media service releases the cache transmission suspension state and continues to cache the target media resource.

[0202] In a specific implementation, the local media service can continue to cache the target media resource in an incremental transmission manner according to the cache progress of the target media resource, thereby avoiding repeated transmission of the cached target media resource. During the cross-device transmission of the target media resource, the local media service is responsible for real-time updating and managing each file segment of the target media resource cached in the client device.

[0203] Step S304b: The local media service generates media stream data based on the cached target media resource and sends the media stream data to the client device for playback.

[0204] If step S301 is triggered when the user clicks the Continue Play button when the target media resource is in a paused state, the local media service can select a file segment of a corresponding time to generate a media stream according to the previous playback progress of the target media resource. For example, when the target media resource was last played to 10 minutes and 20 seconds, if the user clicks "Continue Play", the local media service can start generating a media stream from the file segment corresponding to 10 minutes and 20 seconds.

[0205] In some implementations, the application allows the user to choose whether to "continue playing from the last playback progress" or "play from the beginning" when the target media resource is not played for the first time. If the user selects "continue playing from the last playback progress", the local media service can immediately start caching the target media resource from the server device, and select the file segment of the corresponding time to generate the media stream according to the previous playback progress of the target media resource. If the user selects "play from the beginning", the local media service may not continue caching the target media resource from the server device, but generate the media stream of the target media resource from the beginning, and start caching the target media resource again when the buffer health is insufficient.

[0206] In the "cache transfer in progress" state, the local media service can maintain the current cache transfer action, and generate media stream data based on the cached target media resources, and send it to the client device for playback.

[0207] In the "cache transfer completed" state, the local media service can directly generate media stream data based on the cached target media resources and send it to the client device for playback, without having to re-cache the target media resources from the server device.

[0208] According to the above scheme, in the "start playing" scenario, the local media service can execute different processing flows according to the different cache states of the target media resources to achieve incremental transmission and continuous caching of the target media resources across devices, avoid repeated transmission, reduce the occupancy of transmission bandwidth between devices, and reduce the occupancy of client device storage space.

[0209] Next, taking the "end playback" scenario as an example, the cache status of the target media resource may also include "cache transfer not started", "cache transfer suspended", "cache transfer in progress", and "cache transfer completed".

[0210] Fig.14 FIG. 1 is a flowchart of an embodiment of the present application showing the processing of a local media service in a scenario where playback is terminated. Fig.14 As shown, the processing flow may include the following steps:

[0211] Step S401: The application sends a request to terminate playback to the local media service.

[0212] In one implementation, when the application detects that the user clicks the stop play button, the pause play button, or closes the play interface, the application sends a request to terminate the play to the local media service.

[0213] Step S402: The local media service obtains the cache status of the target media resource according to the request of the application.

[0214] Depending on the cache status, the local media service can then perform different processing flows.

[0215] In the "Cached transfer not started" and "Cached transfer pending" states, local media services can perform the following steps:

[0216] Step S403: the local media service releases the resources pre-applied for the cross-device transmission process.

[0217] These resources include, for example: storage space pre-allocated for caching target media resources, network resources pre-allocated for transmitting target media resources, processor resources pre-allocated for local media service related processes, etc.

[0218] In the "Cache transfer in progress" state, the local media service can perform the following steps:

[0219] Step S404: The local media service updates the cache state to a "cache transmission suspended" state.

[0220] In the "Cache Transfer Complete" state, the local media service can perform the following steps:

[0221] Step S405: The local media service clears the cache usage flag of the target media resource.

[0222] Among them, the cache usage flag can be used to identify the cache status of the target media resource. When the user stops playing and the cache transfer is completed, the cache usage flag can be cleared. At this time, the local media service can merge the various file fragments of the target media resource into a complete media file for subsequent local playback by the application.

[0223] According to the above scheme, in the "termination of playback" scenario, the local media service can execute different processing flows according to the different cache states of the target media resources, such as changing the cache state to facilitate the user to perform other playback actions later, or clearing the cache usage flag to facilitate the application's subsequent local playback, etc.

[0224] It can be understood that the technical solution provided in the embodiments of the present application can not only be used to realize the real-time transmission and playback of media resources across devices, but can also be used for the cross-device transmission of any other data in a distributed scenario, such as distributed file collaborative editing, distributed file storage, etc., which is not limited to the embodiments of the present application.

[0225] The media playback method provided in the above embodiments of the present application constructs a local media service on the client, and decouples the process of caching media resources and media streaming playback through the local media service, so that the server device does not need to deploy streaming media services and does not participate in the perception and playback process, thereby reducing the capability requirements of the server device, thereby realizing real-time cross-device media playback between thin devices. However, it is understandable that in order to achieve the same technical effect as the above embodiments, the local media service can also be constructed on other devices between the server device and the client device link, such as in a wireless access point.

[0226] Fig.15 FIG. 1 is a hardware architecture diagram of another media playback method provided in an embodiment of the present application. Fig.15 As shown, the client device 61 and the server device 62 are located in the same local area network through a router 63. The local media service can be constructed in the router 63, corresponding to a local area network address.

[0227] When the application of the client device 61 detects that the user wants to play the target media resource, the application may send a first play request including a first identifier to the local area network address of the local media service.

[0228] Next, the local media service located in the router 63 associates the first address with the first identifier in response to the first playback request. At this time, the first address can be a local area network address, such as http: / / 192.168.1.101:xx / y, wherein xx represents the port and y represents the file path of the target media resource to be cached in the router. In one implementation, in order to achieve the ability to cache media, the router can include a built-in or external memory 64. The router 63 can assign a local area network IP address to the memory 64, such as 192.168.1.101. It can be seen that the first address can be determined based on the local area network IP address of the memory 64.

[0229] Next, the local media service of the router 63 sends the first address to the application of the client device 61 and starts a process of caching the first address of the target media resource.

[0230] Next, the application of the client device 61 can send a second play request to the first address. When the local media service of the router 63 receives the second play request, it generates a media stream based on the cached target media resource and sends it to the application of the client device 61 for play.

[0231] Fig.15 The media playback method shown configures the local media service in the router and separates it from the client device. In this way, the client device no longer executes the process of caching media resources and generating media streams, further reducing the capability requirements of the client device. In addition, since the router generally supports data transmission with multiple client devices, this method can enable multiple client devices to play the media resources of the same server device across devices at the same time. For example, when the router is connected to a large-screen display device and multiple audio devices, the local media service of the router can send the image stream to the large-screen display device for display, and send the audio stream to different audio devices according to different channels to achieve a stereo playback effect.

[0232] In the embodiments provided by the present application, the various schemes of the media playback method provided by the present application are introduced from the perspective of the client device itself and from the perspective of the interaction between the client device and the server device. It is understandable that each device, such as the above-mentioned client device and access point device, includes a hardware structure and / or software module corresponding to each function in order to realize the above-mentioned functions. It should be easily appreciated by those skilled in the art that, in combination with the units and algorithm steps of each example described in the embodiments disclosed herein, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the present application.

[0233] Fig.16 is a schematic diagram of a structure of a media playback device provided in an embodiment of the present application. In one embodiment, the client device can Fig.16 The software device shown in the figure realizes the corresponding functions. Fig.16 As shown, the media playing device may include: a local media service module and an application module.

[0234] Among them, the local service module may include: a virtual server 701, which is used to receive a first playback request from an application on a client device, the first playback request includes a first identifier, and the first identifier corresponds to a target media resource in the server device; the virtual server 701 is also used to determine whether the cache transmission process of the target media resource has been started according to the first identifier; a file cache unit 702 is used to start the cache transmission process to cache the target media resource from the server device to the client device if the cache transmission process has not been started; the virtual server 701 is also used to send a first address to the application, the first address is a local uniform resource locator URL address of the client device for accessing the cached target media resource; the virtual server 701 is also used to send the cached target media resource to the application for playback when receiving a second playback request sent by the application to the first address.

[0235] In one implementation, the file cache unit 702 is also used to continue the cache transmission process if the cache transmission process has been started but the cache transmission process is suspended, so as to continue to cache the target media resources from the server device to the client device according to the cache progress of the target media resources; the virtual server 701 is also used to send a second address to the application, where the second address is a local URL address of the client device for accessing the cached target media resources; the virtual server 701 is also used to send the cached target media resources to the application for playback when receiving a third playback request sent by the application to the second address.

[0236] In one implementation, the virtual server 701 is also used to send a third address to the application if the target media resource has been completely cached in the client device, so that the application can obtain the target media resource from the third address for playback, and the third address is the file path of the target media resource cached in the client device.

[0237] In one implementation, the virtual server 701 is further used to send a fourth address to the application if the target media resource has been completely cached to the client device, where the fourth address is a local URL address of the client device for accessing the cached target media resource; the virtual server 701 is further used to send the cached target media resource to the application for playback upon receiving a fourth playback request sent by the application to the fourth address.

[0238] In one implementation, the first address includes a local Internet Protocol IP address and a first port of the client device; and the second play request is a Hypertext Transfer Protocol HTTP request.

[0239] In one implementation, the virtual server 701 is used to send the cached target media resources to the application when receiving the second playback request sent by the application to the first address, including: the virtual server 701 is used to listen to the first port of the local IP address; the virtual server 701 is also used to generate a first response message when listening to the second playback request, the first response message is an HTTP response message, and the first response message includes a media stream generated according to the cached target media resources.

[0240] In one implementation, the virtual server 701 is further configured to, upon receiving a request to stop playing from an application, suspend the cache transmission process if a cache transmission process is in progress, and record the cache progress of the target media resource.

[0241] Among them, the application module may include: a first sending unit 703, used to send a first playback request to the media proxy service on the client device, the first playback request includes a first identifier, and the first identifier corresponds to a target media resource in the server device; an address receiving unit 704, used to receive a target address sent by the media proxy service; a second sending unit 705, used to send a second playback request to the local URL address if the target address is a local URL address of the client device; a playback unit 706, used to receive and play the target media resource sent by the media proxy service in response to the second playback request, and the target media resource is cached from the server device to the client device by the media proxy service.

[0242] In one implementation, the playing unit 706 is further configured to obtain the target media resource from the file path address for playing if the target address is a file path address in the client device.

[0243] In one implementation, the local URL address includes the local IP address and the first port of the client device; and the second play request is a Hypertext Transfer Protocol HTTP request.

[0244] In one implementation, the second sending unit 705 is used to send the second play request to the local URL address, including: sending the second play request to the first port of the local IP address.

[0245] In one implementation, the playback unit 706 is used to receive a target media resource sent by the media proxy service in response to a second playback request, including: receiving a first response message sent by the media proxy service in response to the second playback request, the first response message being an HTTP response message, the first response message including a media stream generated by the media proxy service based on the cached target media resource.

[0246] The embodiment of the present application also provides a computer-readable storage medium, in which instructions are stored. When the computer-readable storage medium is run on a computer, the computer executes the above-mentioned methods.

[0247] The embodiment of the present application also provides a computer program product including instructions, which, when executed on a computer, enables the computer to execute the above-mentioned methods.

[0248] The embodiment of the present application also provides a chip system, Fig.17Schematic diagram of the structure of the chip system. The chip system includes a processor 801, which is used to support the above-mentioned device to implement the functions involved in the above-mentioned aspects, for example, to generate or process the information involved in the above-mentioned method. In one possible design, the chip system also includes a memory 802, which is used to store computer instructions 803 and data necessary for the long connection device. The chip system can be composed of chips, or it can include chips and other discrete devices.

[0249] The above specific implementation methods further explain in detail the purpose, technical solutions and beneficial effects of the embodiments of the present application. It should be understood that the above are only specific implementation methods of the embodiments of the present application and are not used to limit the protection scope of the embodiments of the present application. Any modifications, equivalent substitutions, improvements, etc. made on the basis of the technical solutions of the embodiments of the present application should be included in the protection scope of the embodiments of the present application.

Claims

1. A media playback method, It is characterized in that include: The media proxy service on the client device receives a first play request from an application on the client device, where the first play request includes a first identifier, and the first identifier corresponds to a target media resource in the server device; The media proxy service determines, according to the first identifier, whether a cache transmission process of the target media resource has been started; If the cache transmission process is not started, the media proxy service sets a first address, starts the cache transmission process to cache the complete target media resource from the server device to the first address in segments, and sends the first address to the application, where the first address is a local uniform resource locator URL address of the client device for accessing the cached target media resource; If the cache transmission process has been started, but the cache transmission process is suspended, the media proxy service continues the cache transmission process to continue caching the complete target media resource from the server device to the client device according to the cache progress of the target media resource, and sends a second address to the application, where the second address is set before the cache transmission process is suspended, and the second address is a local URL address of the client device for accessing the cached target media resource; When the media proxy service receives a second playback request sent by the application to the first address, or when the media proxy service receives a third playback request sent by the application to the second address, it processes the segmented cached target media resource into a media stream and sends the media stream to the application for playback, wherein the media stream corresponds to the complete target media resource.

2. The method according to claim 1, It is characterized in that After the media proxy service determines whether the cache transmission process of the target media resource has been started, the method further includes: If the target media resource has been completely cached to the client device, the media proxy service sends a third address to the application so that the application obtains the target media resource from the third address for playback, and the third address is the file path of the target media resource cached in the client device.

3. The method according to claim 1, It is characterized in that After the media proxy service determines whether the cache transmission process of the target media resource has been started, the method further includes: If the target media resource has been completely cached to the client device, the media proxy service sends a fourth address to the application, where the fourth address is a local URL address of the client device for accessing the cached target media resource; When the media proxy service receives the fourth play request sent by the application to the fourth address, the media proxy service sends the cached target media resource to the application for play.

4. The method according to claim 1, It is characterized in that The first address includes a local Internet Protocol IP address and a first port of the client device; The second play request is a Hypertext Transfer Protocol HTTP request.

5. The method according to claim 4, It is characterized in that When receiving the second play request sent by the application to the first address, the media proxy service sends the cached target media resource to the application, including: The media proxy service monitors the first port of the local IP address; When the media proxy service monitors the second play request, it generates a first response message, where the first response message is an HTTP response message, and the first response message includes a media stream generated according to the cached target media resource.

6. The method according to claim 1, It is characterized in that Also includes: When the media proxy service receives the stop playback request from the application, if the cache transmission process is in progress, it suspends the cache transmission process and records the cache progress of the target media resource.

7. A media playback method, It is characterized in that An application on a client device sends a first play request to a media proxy service on the client device, wherein the first play request includes a first identifier, and the first identifier corresponds to a target media resource in a server device; wherein, if the media proxy service does not start a cache transmission process, the media proxy service sets a first address, starts the cache transmission process to cache the complete target media resource from the server device to the first address in segments, and the first address is a local uniform resource locator URL address of the client device for accessing the cached target media resource; If the cache transmission process has been started, but the cache transmission process is suspended, the media proxy service continues the cache transmission process to continue caching the complete target media resource from the server device to the client device according to the cache progress of the target media resource, and sends a second address to the application, where the second address is set before the cache transmission process is suspended, and the second address is a local URL address of the client device for accessing the cached target media resource; The application receives the target address sent by the media proxy service; If the target address is the first address, the application sends a second play request to the first address; If the target address is the second address, the application sends a third play request to the second address; The application receives and plays the media stream sent by the media proxy service in response to the second playback request or the third playback request, and the target media resource is cached from the server device to the client device by the media proxy service, wherein the media stream corresponds to the complete target media resource.

8. The method according to claim 7, It is characterized in that After the application receives the target address sent by the media proxy service, the application further includes: If the target address is a file path address in the client device, the application program obtains the target media resource from the file path address for playing.

9. The method according to claim 7, It is characterized in that The first address includes a local IP address and a first port of the client device; The second play request is a Hypertext Transfer Protocol HTTP request.

10. The method according to claim 9, It is characterized in that The application sends a second play request to the first address, including: The application sends the second play request to the first port of the local IP address.

11. The method according to claim 9 or 10, It is characterized in that The application receives the target media resource sent by the media proxy service in response to the second play request, including: The application receives a first response message sent by the media proxy service in response to the second playback request, where the first response message is an HTTP response message, and the first response message includes a media stream generated by the media proxy service based on the cached target media resource.

12. A media playback device, It is characterized in that include: The virtual server is configured to receive a first playback request from an application on a client device, wherein the first playback request includes a first identifier, and the first identifier corresponds to a target media resource in the server device; The virtual server is further used to determine whether the cache transmission process of the target media resource has been started according to the first identifier; A file cache unit, configured to set a first address if the cache transmission process is not started, and start the cache transmission process to cache the complete target media resource from the server device to the first address in segments; The file cache unit is further configured to, if the cache transmission process has been started but the cache transmission process is suspended, continue the cache transmission process to continue caching the complete target media resource from the server device to the client device according to the cache progress of the target media resource; The virtual server is further used to send a first address to the application, where the first address is a local uniform resource locator URL address of the client device for accessing the cached target media resource; The virtual server is further used to send a second address to the application, where the second address is set before the cache transmission process is suspended, and the second address is a local URL address of the client device for accessing the cached target media resource; The virtual server is further configured to, upon receiving a second playback request sent by the application to the first address, process the cached target media resource into a media stream, and send the media stream to the application for playback, wherein the media stream corresponds to the complete target media resource; The virtual server is further configured to process the cached target media resource into the media stream and send the media stream to the application for playback when receiving a third playback request sent by the application to the second address.

13. The device according to claim 12, It is characterized in that The virtual server is also used to send a third address to the application if the target media resource has been completely cached to the client device, so that the application can obtain the target media resource from the third address for playback, and the third address is the file path of the target media resource cached in the client device.

14. The device according to claim 12, It is characterized in that The virtual server is further configured to send a fourth address to the application program if the target media resource has been completely cached to the client device, wherein the fourth address is a local URL address of the client device for accessing the cached target media resource; The virtual server is further configured to send the cached target media resource to the application for playback upon receiving a fourth playback request sent by the application to the fourth address.

15. The device according to claim 12, It is characterized in that The first address includes a local Internet Protocol IP address and a first port of the client device; The second play request is a Hypertext Transfer Protocol HTTP request.

16. The device according to claim 15, It is characterized in that The virtual server is used to send the cached target media resource to the application program when receiving the second play request sent by the application program to the first address, including: The virtual server is used to monitor the first port of the local IP address; The virtual server is further used to generate a first response message when monitoring the second playback request, wherein the first response message is an HTTP response message, and the first response message includes a media stream generated according to the cached target media resource.

17. The device according to claim 12, It is characterized in that The virtual server is further configured to, upon receiving a request to stop playing from the application, suspend the cache transmission process if the cache transmission process is in progress, and record the cache progress of the target media resource.

18. A media playback device, It is characterized in that include: A first sending unit is used to send a first playback request to a media proxy service on a client device, wherein the first playback request includes a first identifier, and the first identifier corresponds to a target media resource in a server device; wherein, if the media proxy service has not started a cache transmission process, the media proxy service sets a first address, starts the cache transmission process to cache the target media resource from the server device in segments to the first address, and the first address is a local uniform resource locator URL address of the client device for accessing the cached target media resource; if the cache transmission process has been started, but the cache transmission process is suspended, the media proxy service continues the cache transmission process to continue caching the complete target media resource from the server device to the client device according to the cache progress of the target media resource, and sends a second address to the application, wherein the second address is set before the cache transmission process is suspended, and the second address is a local URL address of the client device for accessing the cached target media resource; An address receiving unit, configured to receive a target address sent by the media proxy service; a second sending unit, configured to send a second play request to the first address if the target address is the first address; or If the target address is the second address, the application sends a third play request to the second address; A playback unit is used to receive and play the media stream sent by the media proxy service in response to the second playback request or the third playback request, wherein the media stream corresponds to the complete target media resource, and the target media resource is cached by the media proxy service from the server device to the client device.

19. The device according to claim 18, It is characterized in that The playback unit is further configured to obtain the target media resource from the file path address for playback if the target address is a file path address in the client device.

20. The device according to claim 18, It is characterized in that The first address includes a local IP address and a first port of the client device; The second play request is a Hypertext Transfer Protocol HTTP request.

21. The device according to claim 20, It is characterized in that The second sending unit is used to send a second play request to the first address, including: sending the second play request to the first port of the local IP address.

22. The device according to claim 19 or 20, It is characterized in that The playback unit is used to receive the target media resource sent by the media proxy service in response to the second playback request, including: receiving a first response message sent by the media proxy service in response to the second playback request, the first response message being an HTTP response message, and the first response message including a media stream generated by the media proxy service based on the cached target media resource.

23. An electronic device, It is characterized in that include: Media proxy service module and application module; The media proxy service module is used to execute the method according to any one of claims 1 to 6; The application module is used to execute the method described in any one of claims 7-11.

Citation Information

Patent Citations

  • Network media service method

    CN101309203A

  • Method, system and network equipment for realizing streaming media service based on hyper text transport protocol (HTTP)

    CN102055789A

  • Thin client streaming media-based service system running method

    CN102497359A

  • Online streaming media service method based on embedded terminal

    CN102497452A

  • Stream media playing method, device and system

    CN103348657A