An audio and video playing method and system for streaming media on demand

Abstract

The application discloses an audio and video playing method and system for streaming media on-demand, and the method comprises the following steps: when a user orders a media stream, creating a websocket service, establishing a websocket connection between a streaming media server and a client through the websocket service, and listening to the port of the websocket service to process the request data from the client; obtaining and analyzing the on-demand information to obtain the playing request information, and querying whether the playing request information contains legal data in the database; if yes, encapsulating the audio and video data into the client in real time through the websocket connection according to the state of the media stream, the playing request information and the private stream format, and the client performs format analysis and decoding playing on the real-time received audio and video data based on the private stream format; wherein the state of the media stream comprises a real-time stream and a playback stream; if no, returning the request failure information. The embodiment realizes the audio and video playing for streaming media on-demand, reduces the complexity of on-demand streaming, reduces the playing delay, and improves the experience of the user.

Description

Technical Field

[0001] This invention relates to the field of streaming media data playback, and in particular to a method and system for on-demand streaming audio and video playback. Background Technology

[0002] With the development of modern technology, the Internet brings people diverse forms of information. Streaming media technology uses streaming transmission technology to connect media formats for real-time playback on the Internet, allowing users to upload and watch anytime. Streaming media technology has, to a certain extent, broken through the limitations of bandwidth and other factors on media transmission, and is therefore widely used in various fields such as security, medical care, video conferencing, and e-commerce.

[0003] In streaming media, on-demand streaming plays a crucial role in modern digital media transmission. It not only saves resources but also meets personalized needs. On-demand streaming dynamically adjusts the transmission rate based on the user's network performance, ensuring playback requirements are met under different circumstances. Existing on-demand streaming technologies include: adaptive bitrate, which divides the video into multiple segments with different bitrates and dynamically selects appropriate segments for playback based on network conditions; dynamic GOP size, which adjusts the GOP size to balance encoding efficiency and transmission quality, avoiding frame loss due to excessively large GOPs; and dynamic resolution adjustment, which dynamically adjusts the streaming resolution based on the receiver's resolution capabilities, ensuring on-demand streaming quality and smooth playback.

[0004] On-demand streaming technology dynamically adjusts the transmission rate of push and pull streams based on the receiver's capabilities and network conditions, thereby effectively utilizing bandwidth to provide a smooth playback experience. In existing technologies, on-demand streaming of media involves dynamically adjusting parameters such as bitrate and resolution, requiring settings in both the encoder and decoder, increasing system complexity. Furthermore, adjusting encoding parameters may affect encoding efficiency, requiring higher computing resources, and excessive parameter adjustments under low bandwidth conditions can lead to image quality loss. Changes in image quality and resolution experienced during playback may negatively impact the user's viewing experience. Summary of the Invention

[0005] This invention provides a method and system for on-demand audio and video playback of streaming media, which realizes on-demand audio and video playback of streaming media, reduces the complexity of on-demand streaming, reduces playback latency, and improves the user experience.

[0006] To address the aforementioned technical problems, embodiments of the present invention provide a method for on-demand audio and video playback of streaming media, including:

[0007] When a client user requests the current media stream, a WebSocket service is created to establish a WebSocket connection between the streaming media server and the client. The WebSocket service listens on the port of the WebSocket service and processes the request data from the client.

[0008] The system acquires and parses the client's on-demand information to obtain playback request information. It then queries the database to check if the playback request information contains valid data. The on-demand information includes the device ID of the on-demand streaming media service, the device's subordinate video channel number, the main and sub-stream types of the channel number, and the playback time period. Valid data includes a valid device ID and valid channel information.

[0009] If included, the encapsulated audio and video data is sent to the client in real time via a WebSocket connection, based on the media stream status, playback request information, and private stream format. This allows the client to parse and decode the received audio and video data in real time based on the private stream format. The media stream status includes live stream and playback stream.

[0010] If not included, return a first request failure message to the client.

[0011] In implementing this embodiment of the invention, when a client user requests the current media stream, a WebSocket service is created to establish a WebSocket connection between the streaming media server and the client. The WebSocket service port is listened to, and request data from the client is processed. The client's request information is acquired and parsed to obtain playback request information. The database is then queried to determine if the playback request information contains valid data. The request information includes the device ID of the streaming media service, the device's subordinate video channel number, the main and sub-stream types of the channel number, and the playback time period. Valid data includes a valid device ID and valid channel information. If valid data is present, the encapsulated audio and video data is transmitted to the client in real-time via the WebSocket connection, based on the media stream's status, playback request information, and private stream format. This allows the client to perform format parsing and decoding playback of the received audio and video data based on the private stream format. The media stream's status includes live stream and playback stream. If the request is not valid, a first request failure message is returned to the client. By establishing a WebSocket connection, the private audio and video stream format is transmitted using the WebSocket transport protocol and played on the player. The server and player work together to achieve on-demand audio and video data playback. This enables on-demand streaming using a simple private bitstream transmission protocol, reducing the complexity of on-demand streaming. It eliminates the need for complex operations and calculations, achieving a simple, low-latency, and smooth streaming media service. On-demand streaming is a solution that balances bandwidth utilization and playback experience, while also meeting personalized needs, reducing playback latency, and improving the user experience.

[0012] As a preferred option, the proprietary streaming format includes a header and audio / video payload data;

[0013] The data header includes audio / video identifiers, keyframes, and timestamps for displaying the audio / video payload data;

[0014] When the audio / video payload data is video payload data, the video payload data is encoded and encapsulated into AVCC format; the AVCC format is obtained by adding a big-endian prefix of a specified length to each video encoding packet.

[0015] As a preferred solution, based on the media stream status, playback request information, and private stream format, the encapsulated audio and video data is transmitted to the client in real time via a WebSocket connection, specifically:

[0016] When the media stream is in the live stream state, the existence status of the current live stream publisher is checked according to the playback request information. If it exists, the stream data information published by the current live stream publisher is subscribed and the reference count of the current live stream publisher is incremented by 1. If it does not exist, the current live stream publisher is created, the websocket connection between the streaming media server and the client is maintained, and the encapsulated first audio and video data is sent to the client in real time through the websocket connection based on the client's read command request and private stream format.

[0017] When the media stream is in playback mode, the system checks whether the device's channel contains recordings within the playback time period based on the playback request information. If no recordings are found, playback fails, and a second request failure message is returned to the client. If recordings are found, the system constructs the current playback stream publisher based on the playback request information, maintains the WebSocket connection between the streaming media server and the client, and, based on the client's read command request, seek command request, and private stream format, transmits the encapsulated second audio and video data to the client in real time via the WebSocket connection.

[0018] As a preferred solution, based on the client's read command request and the private stream format, the encapsulated first audio and video data is sent to the client in real time via a WebSocket connection, specifically as follows:

[0019] The streaming media server pulls the stream based on the playback request information, obtains the first data packet and the current first audio and video data, and sends the first data packet to the client so that the client can receive the first data packet, initialize the player information based on the first data packet, dynamically adjust the current read command request according to the time variable, and send the current read command request to the streaming media server. The first data packet includes the video encoding type, audio encoding type, and audio sampling rate information; the current read command request includes specifying the number of GOPs for the current read command request as needed.

[0020] The system receives the current read command request from the streaming media server, obtains the number of GOPs corresponding to the current read command request, and then encapsulates the current first audio and video data according to the private stream format and the current number of GOPs. The encapsulated first audio and video data is then sent to the client in real time via a WebSocket connection.

[0021] As a preferred solution, based on the client's read command requests, seek command requests, and private stream format, the encapsulated second audio and video data is transmitted to the client in real time via a WebSocket connection, specifically as follows:

[0022] The streaming media server pulls the stream based on the playback request information to obtain the second data packet and the current second audio and video data. It then sends the second data packet to the client so that the client can receive it, initialize the player information based on the second data packet, dynamically adjust the current read command request according to the time variable, and send the current read command request to the streaming media server. It also sends the current seek command request to the streaming media server according to the specified timestamp in the playback stream; the seek command request is used to jump to the specified timestamp in the playback stream to continue playback. The second data packet includes the video encoding type, audio encoding type, and audio sampling rate information.

[0023] The system receives seek command requests from the streaming media server, clears existing read command request operations, finds the first keyframe after the seek operation based on the seek command request, and sends a seek response to the client so that the client can receive the seek response, clear the data buffer, dynamically adjust the current read command request according to the time variable, and send the current read command request to the streaming media server; wherein, the current read command request includes specifying the number of GOPs for the current read command request as needed;

[0024] The system receives the current read command request from the streaming media server, obtains the number of GOPs corresponding to the current read command request, and encapsulates the current second audio and video data according to the private stream format and the current number of GOPs. The encapsulated second audio and video data is then sent to the client in real time via a WebSocket connection.

[0025] As a preferred solution, the current read command request is dynamically adjusted based on the time variable, and the current read command request is sent to the streaming media server, specifically as follows:

[0026] The client's time variables include a first time variable, a second time variable, and a third time variable; where the first time variable is the number of read commands that have been sent, the second time variable is the remaining time to be received, and the third time variable is the remaining time of the received commands to be played.

[0027] When the client's time variable meets the first adjustment condition, the client pauses sending the current read command request to the streaming media server; wherein, the first adjustment condition is that the third time variable is greater than the first preset threshold;

[0028] When the client's time variable meets the second and first adjustment conditions, the client increases the number of gop in the current read command request and sends the current read command request to the streaming media server; wherein, the second and first adjustment conditions are that the second time variable is less than the second preset threshold, the third time variable is less than the third preset threshold, and the first time variable is less than the fourth preset threshold.

[0029] When the client's time variable does not meet the first adjustment condition and the second adjustment condition, the client sets the current read command request's GOP number based on the preset GOP number and sends the current read command request to the streaming media server.

[0030] As a preferred approach, based on the playback request information, the current status of the live stream publisher is checked, specifically as follows:

[0031] Query whether the key-value information corresponding to the playback request information exists in the map table; where the key-value information in the map table is composed of the ID of the on-demand device, the channel number of the on-demand device, and the main and sub-stream types of the channel number of the on-demand device.

[0032] If it exists, then the current live stream publisher already exists;

[0033] If it does not exist, then the current live stream publisher does not exist.

[0034] As a preferred option, the publisher of the current real-time stream is created as follows:

[0035] Use the device ID and channel number of the on-demand streaming media service as the main and sub-stream types as the current key, use the current real-time stream publisher object as the value corresponding to the current key, create a publisher object, and store the current key and the value corresponding to the current key in a map table;

[0036] The real-time stream is obtained by retrieving the current key value through the publisher object. If no real-time stream data is obtained within the preset time threshold, a third request failure message is returned to the client, the created publisher object is removed from the map table, and the resources of the publisher of the current path are released.

[0037] If real-time stream data is obtained within the preset time threshold, the current real-time stream publisher is successfully created.

[0038] To address the same technical problem, this invention also provides an audio and video playback system for on-demand streaming media, comprising: a connection establishment module, a valid query module, an on-demand playback module, and a playback failure module;

[0039] The connection establishment module is used to create a WebSocket service when a client user requests the current media stream. The WebSocket service establishes a WebSocket connection between the streaming media server and the client, listens on the WebSocket service port, and processes request data from the client.

[0040] The valid query module is used to obtain and parse the client's on-demand information, obtain playback request information, and query the database to see if the playback request information contains valid data. The on-demand information includes the device ID of the on-demand streaming media service, the device's subordinate video channel number, the main and sub-stream types of the channel number, and the playback time period. Valid data includes valid device IDs and valid channel information.

[0041] The on-demand playback module is used to send encapsulated audio and video data to the client in real time via a WebSocket connection if the playback request information contains valid data, based on the media stream status, playback request information, and private stream format. This allows the client to parse and decode the received audio and video data in real time based on the private stream format for playback. The media stream status includes live stream and playback stream.

[0042] The on-demand failure module is used to return a first request failure message to the client if the playback request information does not contain valid data.

[0043] As a preferred option, the on-demand playback module includes a real-time streaming playback unit and a playback streaming playback unit;

[0044] The real-time streaming unit is used to check the existence status of the current real-time stream publisher based on the playback request information when the media stream is in the real-time stream state. If it already exists, it subscribes to the stream data information published by the current real-time stream publisher and increments the reference count of the current real-time stream publisher by 1. If it does not exist, it creates the current real-time stream publisher, maintains the websocket connection between the streaming media server and the client, and, based on the client's read command request and private stream format, transmits the encapsulated first audio and video data to the client in real time through the websocket connection.

[0045] The playback unit is used to check whether the device's channel contains recordings within the playback time period based on the playback request information when the media stream is in the playback stream state. If no recordings are found, playback fails and a second request failure message is returned to the client. If recordings are found, the current playback stream publisher is constructed based on the playback request information, the websocket connection between the streaming media server and the client is maintained, and the encapsulated second audio and video data is sent to the client in real time through the websocket connection based on the client's read command request, seek command request, and private stream format. Attached Figure Description

[0046] Figure 1 This is a flowchart illustrating an embodiment of an on-demand audio and video playback method for streaming media provided by the present invention.

[0047] Figure 2 : A schematic diagram of an embodiment of an audio and video playback system for on-demand streaming provided by the present invention. Detailed Implementation

[0048] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0049] Example 1

[0050] Please refer to Figure 1 This is a flowchart illustrating a method for on-demand audio and video playback using streaming media, provided by an embodiment of the present invention. This embodiment establishes a WebSocket connection, uses the WebSocket transport protocol to transmit private audio and video stream formats, and plays them on the player, achieving on-demand audio and video playback. This reduces the complexity of on-demand streaming, lowers playback latency, and improves the user experience. The audio and video playback method includes steps 101 to 104, each step as follows:

[0051] Step 101: When a user on the client requests the current media stream, a WebSocket service is created. The WebSocket service is used to establish a WebSocket connection between the streaming media server and the client, and the WebSocket service port is listened to to process the request data from the client.

[0052] In this embodiment, the on-demand audio and video playback method of the present invention is implemented through a streaming media server. When a client requests a media stream, it may be a live stream or a playback stream. When a client requests a live stream, a WebSocket server is created on the streaming media server and listens on the port to handle connections from the client. When a client requests a playback stream, a WebSocket server is created on the streaming media server similarly to the live stream and listens on the port to handle connections from the client.

[0053] Step 102: Obtain and parse the client's on-demand information to get the playback request information, and query the database to see if the playback request information contains valid data; wherein, the on-demand information includes the device ID of the on-demand streaming media service, the device's subordinate video channel number, the main and sub-stream types of the channel number, and the playback time period; valid data includes valid device ID and valid channel information.

[0054] In this embodiment, when a client requests a live stream, the streaming media server parses the information sent by the client, which mainly includes the ID of a device under the streaming media service, the device's subordinate video channel number, and the bitrate type (main stream or sub-stream) of the requested channel number. The server then queries its database to determine if the playback request contains a valid device ID and channel information. Similarly, when a client requests a playback stream, the server parses the information sent by the client, which mainly includes the ID of a device under the streaming media service, the device's subordinate video channel number, and the playback time period (start and end times of the playback stream). The server then queries its database to determine if the request contains a valid device ID and channel information.

[0055] Step 103: If the playback request information contains valid data, then based on the media stream status, playback request information, and private stream format, the encapsulated audio and video data is sent to the client in real time via a WebSocket connection, so that the client can perform format parsing and decoding playback on the real-time received audio and video data based on the private stream format; wherein, the media stream status includes real-time stream and playback stream.

[0056] Optionally, the proprietary streaming format includes a header and audio / video payload data;

[0057] The data header includes audio / video identifiers, keyframes, and timestamps for displaying the audio / video payload data;

[0058] When the audio / video payload data is video payload data, the video payload data is encoded and encapsulated into AVCC format; the AVCC format is obtained by adding a big-endian prefix of a specified length to each video encoding packet.

[0059] In this embodiment, the private stream format consists of two parts: a data header and audio / video payload data. The data header is 6 bytes in total. The first byte is the audio / video identifier, indicating whether the payload data is audio or video (0 for video, 1 for audio). The second byte is the keyframe identifier, indicating whether the payload data is a keyframe (1 for keyframe, 0 for non-keyframe). Bytes 3 to 6 are the display timestamps of the audio / video payload data, indicating when the data is displayed to the user. For the video data in the audio / video payload, the video payload data also needs to be encoded and encapsulated into AVCC format for transmission. AVCC format adds a big-endian prefix (NALU packet size) to each NALU packet (video encoded packet), making this format packet parsing very easy. It should be noted that NALU refers to the video encoded packet, which is the basic unit for segmenting and encapsulating video encoded data.

[0060] Optionally, based on the media stream status, playback request information, and private stream format, the encapsulated audio and video data is transmitted to the client in real time via a WebSocket connection. This includes steps 1031 to 1032, each step as follows:

[0061] Step 1031: When the media stream is in the state of live stream, check the existence status of the current live stream publisher based on the playback request information. If it already exists, subscribe to the stream data information published by the current live stream publisher and increment the reference count of the current live stream publisher by 1. If it does not exist, create the current live stream publisher, maintain the websocket connection between the streaming media server and the client, and based on the client's read command request and private stream format, send the encapsulated first audio and video data to the client in real time through the websocket connection.

[0062] In this embodiment, when a client requests a live stream, if valid data can be found, the system checks whether the publisher of that live stream already exists based on the incoming playback request information. If the publisher already exists, there is no need to create a new one; simply increment the publisher reference count. This ensures that when multiple clients request the same stream, only one publisher is created. Different clients subscribing to the live stream only need to distribute the stream from the publisher to each client. When the publisher reference count is 0, it indicates that no client has subscribed to the live stream, and the publisher is closed.

[0063] Optionally, based on the playback request information, check the current existence status of the live stream publisher, specifically:

[0064] Query whether the key-value information corresponding to the playback request information exists in the map table; where the key-value information in the map table is composed of the ID of the on-demand device, the channel number of the on-demand device, and the main and sub-stream types of the channel number of the on-demand device.

[0065] If it exists, then the current live stream publisher already exists;

[0066] If it does not exist, then the current live stream publisher does not exist.

[0067] In this embodiment, the process of determining whether a real-time stream already exists is performed by querying a map table. During stream creation, a map table is used to store stream information, employing a key-value pair mapping method. The map table key is constructed by concatenating the device ID, the channel number for that device, and the main and sub-streams of that channel. The corresponding value stores the publisher object. When a new client (subscriber) subscribes to the stream, it first queries this map table using the key. If the map table contains the corresponding key, it proves that a real-time stream for that channel has already been published, indicating that the publisher for that real-time stream already exists.

[0068] Optionally, create the publisher of the current live stream, specifically:

[0069] Use the device ID and channel number of the on-demand streaming media service as the main and sub-stream types as the current key, use the current real-time stream publisher object as the value corresponding to the current key, create a publisher object, and store the current key and the value corresponding to the current key in a map table;

[0070] The real-time stream is obtained by retrieving the current key value through the publisher object. If no real-time stream data is obtained within the preset time threshold, a third request failure message is returned to the client, the created publisher object is removed from the map table, and the resources of the publisher of the current path are released.

[0071] If real-time stream data is obtained within the preset time threshold, the current real-time stream publisher is successfully created.

[0072] In this embodiment, when a client requests a stream from the streaming media server, the necessary information the client sends to the server is: the device ID, the channel number, and whether it's the main stream or a sub-stream. Based on this information, the server checks the publisher's map table to see if the stream has already been published. If so, it subscribes directly and forwards the stream information to different subscribers. If not, a live stream publisher needs to be created. Creating a live stream publisher simply requires using the device ID, channel number, and main / sub-stream type as keys, and the publisher object as the value, recorded as `pub`, and stored. The publisher object contains the object that needs to pull the stream from the specified device, channel, and stream type, recorded as `avin`. This `avin` object attempts to retrieve the live stream from the specified device, channel, and stream type. The object waits for a timeout. If the stream data is not retrieved within a preset time threshold (e.g., 5 seconds), the request fails, a third request failure message is returned to the client, the newly created publisher information is removed from the map table, and the publisher's resources are released. If the streaming data can be successfully obtained within 5 seconds, it proves that the publisher has successfully published the stream.

[0073] Optionally, based on the client's read command request and private stream format, the encapsulated first audio and video data is sent to the client in real time via a WebSocket connection. Specific steps S11-S12 are as follows:

[0074] S11: The streaming media server pulls the stream according to the playback request information, obtains the first packet of data and the current first audio and video data, and sends the first packet of data to the client so that the client can receive the first packet of data, initialize the player information according to the first packet of data, dynamically adjust the current read command request according to the time variable, and send the current read command request to the streaming media server; wherein, the first packet of data includes the video encoding type, the audio encoding type, and the audio sampling rate information; the current read command request includes specifying the number of GOPs for the current read command request as needed.

[0075] In this embodiment, after the streaming media server successfully creates a real-time stream publisher, the client maintains a connection with the server. The server sends a first data packet to the client, which specifies the audio and video encoding types to be transmitted to the client, including the video encoding type, audio encoding type, and audio sampling rate information. After receiving this information, the client initializes the player's relevant information, preparing to decode and display the audio and video data sent in real-time by the streaming media service.

[0076] In this embodiment, after the client establishes a WebSocket connection with the streaming media server, the streaming media server pulls the stream to the streaming media server according to the playback request information of the on-demand service. After the streaming media server sends out the first packet (the first packet of data), it waits for the client's instruction. When the client sends a read instruction request to the streaming media server (the read instruction specifies the number of GOPs to be sent), the streaming media server receives the client's request instruction and begins to send the specified number of GOPs of data to the client.

[0077] It's important to note that the data in the first packet (the first data packet) contains the encoding information for the audio and video data that will be sent subsequently from this real-time stream. For example, it specifies whether the audio will be in AAC or G711A format, and whether the video will be encoded in H.264 or H.265. This information is used to inform the client to initialize the player. The audio and video data includes both audio and video payload data. When transmitting the audio and video data to the client, the video payload data is encapsulated in AVCC format, while the audio payload data is encapsulated and sent to the client in its original format.

[0078] S12: Receive the current read command request from the streaming media server, obtain the number of GOPs corresponding to the current read command request, get the current number of GOPs, encapsulate the current first audio and video data according to the private stream format and the current number of GOPs, and send the encapsulated first audio and video data to the client in real time through a WebSocket connection.

[0079] In this embodiment, starting from the second packet, the streaming data from the streaming media server is encapsulated according to a private stream format and sent to the client via WebSocket. The client parses and decodes the data according to the private stream format for playback. For video streams, the video payload data also needs to be encoded and encapsulated into AVCC format for transmission. AVCC format adds a big-endian prefix (NALU packet size) to each NALU packet, making parsing this format very easy. NALU refers to the video encoded packet, which is the basic unit for segmenting and encapsulating video encoded data. For real-time streams, playback is achieved by transmitting the private bitstream through the established WebSocket connection.

[0080] Step 1032: When the media stream is in playback mode, check whether the device's channel contains recordings within the playback time period based on the playback request information. If no recordings are found, playback fails, and a second request failure message is returned to the client. If recordings are found, construct the current playback stream publisher based on the playback request information, maintain the websocket connection between the streaming media server and the client, and, based on the client's read command request, seek command request, and private stream format, transmit the encapsulated second audio and video data to the client in real time via the websocket connection.

[0081] In this embodiment, when the media stream is in the playback stream state, if valid data can be found, the system then checks whether the device's channel contains recordings within the playback time period. If not, playback fails, a request failure response is sent to the client, and a second request failure message (no recordings) is returned. If recordings exist within the time period, the server establishes a publisher for the playback stream, maintains a WebSocket connection between the streaming media server and the client, and, based on the client's read command request, seek command request, and private stream format, transmits the encapsulated second audio and video data to the client in real time via the WebSocket connection.

[0082] Optionally, based on the client's read command request, seek command request, and private stream format, the encapsulated second audio and video data is transmitted to the client in real time via a WebSocket connection. Specific steps S21-S23 are as follows:

[0083] S21: The streaming media server pulls the stream according to the playback request information to obtain the second packet data and the current second audio and video data. It then sends the second packet data to the client so that the client can receive it, initialize the player information based on the second packet data, dynamically adjust the current read command request according to the time variable, and send the current read command request to the streaming media server. It also sends the current seek command request to the streaming media server according to the specified timestamp in the playback stream. The seek command request is used to jump to the specified timestamp in the playback stream to continue playback. The second packet data includes the video encoding type, audio encoding type, and audio sampling rate information.

[0084] In this embodiment, if recording occurs within the specified time period, the server establishes a publisher for the playback stream and simultaneously sends the audio and video information of that playback stream, including the video encoding type, audio encoding type, and audio sampling rate information, to the client. Upon receiving this information, the client parses it and initializes its player resources. Starting from the second packet, the server sends proprietary audio and video format data to the client. Unlike real-time streaming, playback requires operations such as seek, thus necessitating different processing. As needed (using a specified timestamp in the playback stream), a seek command request is sent to the streaming media server. This seek command request is used to jump to the specified timestamp for continued playback.

[0085] S22: Receive seek command requests from the streaming media server, clear existing read command request operations, find the first keyframe after the seek operation based on the seek command request, and send a seek response to the client so that the client receives the seek response, clears the data buffer, dynamically adjusts the current read command request according to the time variable, and sends the current read command request to the streaming media server; wherein, the current read command request includes specifying the number of GOPs for the current read command request as needed;

[0086] In this embodiment, the seek operation is also implemented through on-demand streaming. Specifically, when a seek operation is needed, the client sends a seek command to the server, specifying the timestamp of the playback stream to seek to. Upon receiving the seek request, the server first clears any remaining read command requests, then performs the seek operation on the playback stream, finds the first keyframe (I-frame) after the seek operation, and sends a seek response to the client. The response message format is two bytes: the first byte is 2, indicating a seek operation; the second byte indicates whether the server's seek operation was successful (1 for failure, 0 for success). Upon receiving the seek response, the client immediately clears its data buffer and resends the read request to the server, thus enabling continued playback of the data after the seek operation and implementing the on-demand streaming method. It should be noted that for seek operations, the streaming media server always sends a successful seek response to the client. Upon receiving the successful response, the client immediately clears its buffered data and resends the read request command to the streaming media server.

[0087] S23: Receive the current read command request from the streaming media server, obtain the number of GOPs corresponding to the current read command request, get the current number of GOPs, encapsulate the current second audio and video data according to the private stream format and the current number of GOPs, and send the encapsulated second audio and video data to the client in real time through a WebSocket connection.

[0088] In this embodiment, the streaming media server provides two commands for the client: "read" and "seek," which are for reading and jumping, respectively. The read command is based on GOPs, which refers to the frame interval between two I-frames. The client dynamically adjusts the read command requests sent to the server according to the actual time variables. For example, under normal playback conditions, the client can dynamically send a read command and specify the number of GOPs requested in this command to the server. After receiving the command, the server sends the specified number of GOPs of stream data to the client for playback as needed.

[0089] Optionally, the current read command request can be dynamically adjusted based on the time variable, and the current read command request can be sent to the streaming media server, specifically as follows:

[0090] The client's time variables include a first time variable, a second time variable, and a third time variable; where the first time variable is the number of read commands that have been sent, the second time variable is the remaining time to be received, and the third time variable is the remaining time of the received commands to be played.

[0091] When the client's time variable meets the first adjustment condition, the client pauses sending the current read command request to the streaming media server; wherein, the first adjustment condition is that the third time variable is greater than the first preset threshold;

[0092] When the client's time variable meets the second and first adjustment conditions, the client increases the number of gop in the current read command request and sends the current read command request to the streaming media server; wherein, the second and first adjustment conditions are that the second time variable is less than the second preset threshold, the third time variable is less than the third preset threshold, and the first time variable is less than the fourth preset threshold.

[0093] When the client's time variable does not meet the first adjustment condition and the second adjustment condition, the client sets the current read command request's GOP number based on the preset GOP number and sends the current read command request to the streaming media server.

[0094] In this embodiment, the client has three time variables. The first time variable records the number of read commands sent, denoted as t1. The second time variable is the remaining time to be received, denoted as t2. The third time variable is the remaining time to be played after receiving data, denoted as t3. If t3 is relatively large, it indicates that the buffer is sufficient for playback for a period of time, and read requests are not sent to the server temporarily to avoid excessive data buffering caused by frequent sending. When t2, t3, and t1 are all relatively small, read requests are sent promptly to retrieve data from the media server in a timely manner, ensuring that the client has a certain amount of data buffering. In this way, by dynamically adjusting the time variables and retrieving streams on demand, the client's data can be played normally and smoothly.

[0095] Step 104: If the playback request information does not contain valid data, return the first request failure information to the client.

[0096] In this embodiment, if no valid data is found in the playback request information, the request stream is returned as failed, and the first request failure information is returned to the client.

[0097] In implementing this embodiment of the invention, when a client user requests the current media stream, a WebSocket service is created to establish a WebSocket connection between the streaming media server and the client. The WebSocket service port is listened to, and request data from the client is processed. The client's request information is acquired and parsed to obtain playback request information. The database is then queried to determine if the playback request information contains valid data. The request information includes the device ID of the streaming media service, the device's subordinate video channel number, the main and sub-stream types of the channel number, and the playback time period. Valid data includes a valid device ID and valid channel information. If valid data is present, the encapsulated audio and video data is transmitted to the client in real-time via the WebSocket connection, based on the media stream's status, playback request information, and private stream format. This allows the client to perform format parsing and decoding playback of the received audio and video data based on the private stream format. The media stream's status includes live stream and playback stream. If the request is not valid, a first request failure message is returned to the client. By establishing a WebSocket connection, the private audio and video stream format is transmitted using the WebSocket transport protocol and played on the player. The server and player work together to achieve on-demand audio and video data playback. This enables on-demand streaming using a simple private bitstream transmission protocol, reducing the complexity of on-demand streaming. It eliminates the need for complex operations and calculations, achieving a simple, low-latency, and smooth streaming media service. On-demand streaming is a solution that balances bandwidth utilization and playback experience, while also meeting personalized needs, reducing playback latency, and improving the user experience.

[0098] Example 2

[0099] Accordingly, see Figure 2 , Figure 2 This is a schematic diagram of a second embodiment of the streaming media on-demand audio and video playback system provided by the present invention. Figure 2 As shown, the streaming media on-demand audio and video playback system includes a connection establishment module 201, a valid query module 202, an on-demand playback module 203, and a playback failure module 204;

[0100] The connection establishment module 201 is used to create a WebSocket service when a user on the client requests the current media stream, establish a WebSocket connection between the streaming media server and the client through the WebSocket service, listen to the port of the WebSocket service, and process the request data from the client.

[0101] The legitimate query module 202 is used to obtain and parse the client's on-demand information, obtain playback request information, and query the database to see if the playback request information contains legitimate data. The on-demand information includes the device ID of the on-demand streaming media service, the device's subordinate video channel number, the main and sub-stream types of the channel number, and the playback time period. Legitimate data includes a legitimate device ID and legitimate channel information.

[0102] The on-demand playback module 203 is used to, if the playback request information contains valid data, send the encapsulated audio and video data to the client in real time via a WebSocket connection, based on the media stream status, playback request information, and private stream format, so that the client can perform format parsing and decoding playback on the real-time received audio and video data based on the private stream format; wherein, the media stream status includes real-time stream and playback stream;

[0103] The on-demand playback module 203 includes a real-time streaming playback unit 2031 and a playback streaming playback unit 2032;

[0104] The real-time streaming playback unit 2031 is used to check the existence status of the current real-time streaming publisher based on the playback request information when the media stream is in the real-time streaming state. If it already exists, it subscribes to the streaming data information published by the current real-time streaming publisher and increments the reference count of the current real-time streaming publisher by 1. If it does not exist, it creates the current real-time streaming publisher, maintains the websocket connection between the streaming media server and the client, and, based on the client's read command request and private stream format, transmits the encapsulated first audio and video data to the client in real time through the websocket connection.

[0105] The playback unit 2032 is used to query whether the device's channel contains recordings within the playback time period based on the playback request information when the media stream is in the playback stream state. If no recordings are found, the playback fails and a second request failure message is returned to the client. If recordings are found, the current playback stream publisher is constructed based on the playback request information, the websocket connection between the streaming media server and the client is maintained, and the encapsulated second audio and video data is sent to the client in real time through the websocket connection based on the client's read command request, seek command request, and private stream format.

[0106] The on-demand failure module 204 is used to return a first request failure message to the client if the playback request information does not contain valid data.

[0107] Implementing the embodiments of the present invention, on-demand streaming is a solution to balance bandwidth utilization and playback experience, while also meeting personalized needs. In order to improve the user experience and reduce the complexity of algorithm design, the present invention adopts a simple private streaming media protocol transmission method to realize on-demand streaming. It is simple to implement, requires no complicated operations and calculations, and the server and client work together to achieve a simple, low-latency, and smooth streaming media service.

[0108] The above-described streaming media on-demand audio and video playback system can implement one of the streaming media on-demand audio and video playback methods described in the above method embodiments. The options in the above method embodiments are also applicable to this embodiment, and will not be detailed here. The remaining content of this application's embodiments can be referred to the content of the above method embodiments, and will not be repeated in this embodiment.

[0109] The above specific embodiments further illustrate the purpose, technical solution, and beneficial effects of the present invention in detail. It should be understood that the above are merely specific embodiments of the present invention and are not intended to limit the scope of protection of the present invention. In particular, it should be noted that any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention for those skilled in the art.

Claims

1. A method for on-demand audio and video playback using streaming media, characterized in that, include: When a client user requests the current media stream, a WebSocket service is created to establish a WebSocket connection between the streaming media server and the client. The server listens on the port of the WebSocket service and processes the request data from the client. The client's on-demand information is acquired and parsed to obtain playback request information. The database is then queried to determine whether the playback request information contains valid data. The on-demand information includes the device ID of the on-demand streaming media service, the device's subordinate video channel number, the main and sub-stream types of the channel number, and the playback time period. The valid data includes a valid device ID and valid channel information. If the media stream is present, its state is determined to be either a live stream or a playback stream. When the media stream is a live stream, the existence of the current live stream publisher is checked based on the playback request information. If it already exists, the stream data information published by the current live stream publisher is subscribed to, and the reference count of the current live stream publisher is incremented by 1. If it does not exist, the current live stream publisher is created, the websocket connection between the streaming media server and the client is maintained, and the streaming media server pulls the stream according to the playback request information to obtain the first packet of data and the current first audio and video data. The first packet of data is then sent to the client so that the client can receive it. The first packet of data is used to initialize player information and dynamically adjust the current read command request based on a time variable. The current read command request is then sent to the streaming media server. The first packet of data includes video encoding type, audio encoding type, and audio sampling rate information. The current read command request includes specifying the number of GOPs for the current read command request as needed. The streaming media server, based on the number of GOPs corresponding to the read command request, sends the encapsulated first audio and video data to the client in real time, enabling the client to perform format parsing and decoding playback of the received audio and video data based on a private streaming format. If not included, a first request failure message is returned to the client.

2. The streaming media on-demand audio and video playback method as described in claim 1, characterized in that, The proprietary stream format includes a header and audio / video payload data; The data header includes audio / video identifiers, keyframes, and display timestamps for audio / video payload data. When the audio / video payload data is video payload data, the video payload data is encoded and encapsulated into AVCC format; wherein, the AVCC format is obtained by adding a big-endian prefix of a specified length to each video encoding packet.

3. The streaming media on-demand audio and video playback method as described in claim 1, characterized in that, Determining whether the media stream is a real-time stream or a playback stream includes: When the media stream is in the playback stream state, based on the playback request information, it queries whether the device channel contains recordings within the playback time period. If no recordings are found, playback fails, and a second request failure message is returned to the client. If the recordings are found, based on the playback request information, the current playback stream publisher is constructed, the websocket connection between the streaming media server and the client is maintained, and based on the client's read command request, seek command request, and the private stream format, the encapsulated second audio and video data is sent to the client in real time through the websocket connection, so that the client can perform format parsing and decoding playback of the real-time received audio and video data based on the private stream format.

4. The streaming media on-demand audio and video playback method as described in claim 1, characterized in that, The streaming media server, based on the number of GOPs corresponding to the read command request, sends the encapsulated first audio and video data to the client in real time, specifically as follows: The streaming media server receives the current read command request, obtains the number of GOPs corresponding to the current read command request, and obtains the current number of GOPs. Based on the private stream format and the current number of GOPs, the current first audio and video data is encapsulated, and the encapsulated first audio and video data is sent to the client in real time through the websocket connection.

5. The streaming media on-demand audio and video playback method as described in claim 3, characterized in that, Based on the client's read command request, seek command request, and the private stream format, the encapsulated second audio and video data is transmitted to the client in real time via the WebSocket connection, specifically as follows: The streaming media server pulls the stream according to the playback request information to obtain the second packet data and the current second audio and video data. It then sends the second packet data to the client, enabling the client to receive it, initialize player information based on the second packet data, dynamically adjust the current read command request according to a time variable, and send the current read command request to the streaming media server. Additionally, it sends a current seek command request to the streaming media server based on a specified timestamp in the playback stream. The seek command request is used for the playback stream to jump to the specified timestamp to continue playback. The second packet data includes the video encoding type, audio encoding type, and audio sampling rate information. The streaming media server receives the seek command request, clears the existing read command request operations, finds the first keyframe after the seek operation based on the seek command request, and sends a seek response to the client so that the client receives the seek response, clears the data cache, dynamically adjusts the current read command request according to the time variable, and sends the current read command request to the streaming media server; wherein, the current read command request includes specifying the number of GOPs for the current read command request as needed; The streaming media server receives the current read command request, obtains the number of GOPs corresponding to the current read command request, and then encapsulates the current second audio and video data according to the private stream format and the current number of GOPs. The encapsulated second audio and video data is then sent to the client in real time through the websocket connection.

6. The streaming media on-demand audio and video playback method as described in claim 1 or 5, characterized in that, The step of dynamically adjusting the current read command request based on the time variable and sending the current read command request to the streaming media server specifically involves: The client's time variables include a first time variable, a second time variable, and a third time variable; wherein, the first time variable is the number of read commands that have been sent, the second time variable is the remaining time to be received, and the third time variable is the remaining time of the received commands to be played. When the client's time variable meets the first adjustment condition, the client pauses sending the current read command request to the streaming media server; wherein, the first adjustment condition is that the third time variable is greater than a first preset threshold; When the client's time variable meets the second adjustment condition, the client increases the number of gop in the current read command request and sends the current read command request to the streaming media server; wherein, the second adjustment condition is that the second time variable is less than the second preset threshold, the third time variable is less than the third preset threshold, and the first time variable is less than the fourth preset threshold. When the client's time variable does not meet the first adjustment condition and does not meet the second adjustment condition, the client sets the number of GOPs for the current read command request based on a preset number of GOPs, and sends the current read command request to the streaming media server.

7. The streaming media on-demand audio and video playback method as described in claim 1, characterized in that, The step of checking the current existence status of the live stream publisher based on the playback request information specifically involves: Query whether the key-value information corresponding to the playback request information exists in the map table; wherein, the key-value information in the map table is constructed by concatenating the ID of the on-demand device, the channel number of the on-demand device, and the main and sub-stream types of the channel number of the on-demand device. If it exists, then the current real-time stream publisher already exists; If it does not exist, then the current real-time stream publisher does not exist.

8. The streaming media on-demand audio and video playback method as described in claim 7, characterized in that, The publisher that creates the current real-time stream is specifically: The device ID and channel number of the on-demand streaming media service are used as the main and sub-stream types as the current key, and the publisher object of the current real-time stream is used as the value corresponding to the current key. A publisher object is created, and the current key and the value corresponding to the current key are stored in the map table. The publisher object retrieves the real-time stream based on the current key value. If no real-time stream data is retrieved within a preset time threshold, a third request failure message is returned to the client, the created publisher object is removed from the map table, and the resources of the publisher of the current path are released. If the real-time stream data is obtained within the preset time threshold, then the current real-time stream publisher is successfully created.

9. A streaming media on-demand audio and video playback system, characterized in that, The streaming media on-demand audio and video playback system includes: a connection establishment module, a valid query module, an on-demand playback module, and a playback failure module; the streaming media on-demand audio and video playback system is used to implement the streaming media on-demand audio and video playback method as described in any one of claims 1-8; The connection establishment module is used to create a WebSocket service when the user of the client requests the current media stream, establish a WebSocket connection between the streaming media server and the client through the WebSocket service, listen to the port of the WebSocket service, and process the request data from the client. The legal query module is used to obtain and parse the client's on-demand information, obtain playback request information, and query the database to see if the playback request information contains legal data; wherein, the on-demand information includes the device ID of the on-demand streaming media service, the device's subordinate video channel number, the main and sub-stream types of the channel number, and the playback time period; the legal data includes a legal device ID and legal channel information; The on-demand playback module is used to, if the playback request information contains valid data, send the encapsulated audio and video data to the client in real time via the WebSocket connection, based on the status of the media stream, the playback request information, and the private stream format, so that the client can perform format parsing and decoding playback on the real-time received audio and video data based on the private stream format; wherein, the status of the media stream includes real-time stream and playback stream; The on-demand failure module is used to return a first request failure message to the client if the playback request information does not contain valid data.

10. The streaming media on-demand audio and video playback system as described in claim 9, characterized in that, The on-demand playback module includes a real-time streaming playback unit and a replay streaming playback unit; The real-time streaming unit is used to check the existence status of the current real-time streaming publisher according to the playback request information when the media stream is in the state of real-time streaming. If it already exists, it subscribes to the streaming data information published by the current real-time streaming publisher and increments the reference count of the current real-time streaming publisher by 1. If it does not exist, then create the publisher of the current real-time stream, maintain the websocket connection between the streaming media server and the client, and based on the client's read command request and the private stream format, send the encapsulated first audio and video data to the client in real time through the websocket connection; The playback stream unit is used to query whether the device channel contains recordings within the playback time period according to the playback request information when the media stream is in the state of playback stream. If the recordings are not contained, the playback fails and a second request failure message is returned to the client. If the video recording is present, the current playback stream publisher is constructed based on the playback request information. The websocket connection between the streaming media server and the client is maintained. Based on the client's read command request, seek command request, and the private stream format, the encapsulated second audio and video data is transmitted to the client in real time through the websocket connection.

Images