A streaming media processing method and related apparatus
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- BEIJING SOHU NEW MOMENTUM INFORMATION TECH CO LTD
- Filing Date
- 2026-04-14
- Publication Date
- 2026-06-23
AI Technical Summary
The massive number of independent files generated during streaming media playback leads to low storage efficiency, high index node resource consumption, and difficulty in cache management.
The virtual storage engine creates a unique, fixed-length single physical storage file for each video stream and uses a virtual storage index to record the physical offset of media data segments. Media data segments are read directly from the single physical storage file, reducing the resource consumption of inodes. Fragmented tasks are merged through a delayed waiting queue and a timing window mechanism to optimize network requests.
It reduces index node resource consumption, improves streaming media storage efficiency and data response speed, reduces the problem of low disk input/output read/write efficiency, and optimizes cache management and network request processing.
Smart Images
Figure CN122269054A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of computer technology, and more specifically, to a streaming media processing method and related apparatus. Background Technology
[0002] With the widespread adoption of short and long video applications, HLS (HTTP Live Streaming) has become the most mainstream streaming media protocol on the iOS platform. HLS works by dividing the video into numerous TS (Transport Stream) segments, each 2-0 seconds long. When playing HLS format videos, the iOS system's built-in player, AVPlayer, automatically manages the downloading of these TS segments.
[0003] In traditional HLS local caching, each TS (Time Series) segment is stored as an independent file. Since a long video may contain thousands of independent files, this consumes a large amount of file system inode resources, reducing disk I / O read / write efficiency and posing significant challenges to cache management. For example, cleaning up expired cache or calculating cache size requires traversing a large number of independent files, generating substantial I / O pressure. Therefore, the massive number of independent files generated during streaming media playback leads to low streaming media storage efficiency and high inode resource consumption. Summary of the Invention
[0004] In view of this, the present invention discloses a streaming media processing method and related apparatus to solve the problem of low streaming media storage efficiency and high index node resource consumption caused by the massive number of independent files generated during streaming media playback.
[0005] A streaming media processing method, comprising:
[0006] In response to a streaming media playback request sent by the player, obtain the original streaming media link;
[0007] The protocol header of the original streaming media link is modified to a custom protocol header, and the loading request based on the custom protocol header is intercepted using the resource loading delegate protocol interface;
[0008] The data request range contained in the loading request is parsed;
[0009] The local virtual storage index is searched according to the data request range, and the streaming media data segment corresponding to the data request range is determined according to the search results. It is determined whether the streaming media data segment corresponding to the data request range has been cached in a single physical storage file on the local disk. For each video stream, the virtual storage engine creates only one corresponding fixed-length single physical storage file on the disk.
[0010] If the search results indicate that a media data segment within the data request range already exists, the media data segment is read directly from the single physical storage file based on the physical offset of the media data segment recorded in the virtual storage index, and then encapsulated and returned to the player.
[0011] Optionally, it also includes:
[0012] If the search results indicate that the streaming media data segment corresponding to the data request range does not exist, the loading request is generated into a fragmented task and added to the delayed waiting queue, and a timer window of a preset duration is opened;
[0013] Within the timing window, if there are multiple fragmented tasks that are consecutive or adjacent on logical addresses in the delay waiting queue, the multiple fragmented tasks are merged into one aggregated task.
[0014] The request range of the aggregation task is aligned according to the page size of the file system to generate the final target network request;
[0015] Based on the target network request, a network download is initiated to the remote server to obtain the streaming media data stream.
[0016] Optionally, it also includes:
[0017] The received streaming media data stream is populated into the player's content information request and data request in real time to support the normal operation of the playback process.
[0018] Optionally, it also includes:
[0019] After receiving the streaming media data stream, the physical write position of the streaming media data stream in the single physical storage file is calculated based on the alignment range of the target network request.
[0020] Based on the physical write location, the streaming media data stream is written to the corresponding location in the single physical storage file, and the mapping relationship between the logical offset, physical offset, and the completion status of the streaming media data stream download and storage to the disk in the virtual storage index is updated synchronously.
[0021] Optionally, it also includes:
[0022] When it is detected that the user drags from the first time point to the second time point on the player's timeline, it is determined that the intermediate streaming media data between the first time point and the second time point has not yet been downloaded.
[0023] The virtual storage engine is used to mark two non-contiguous storage intervals in the virtual storage index, namely a cached start segment containing the first time point and a cached end segment containing the second time point, wherein the start segment and the end segment are uncached intermediate missing segments;
[0024] When the player requests the target streaming media data of the missing intermediate segment, the virtual storage engine is used to quickly retrieve the index and query whether the target streaming media data requested by the player is cached locally.
[0025] If the target streaming media data is already cached locally, it is read directly from the corresponding single physical storage file locally.
[0026] If the target streaming media data is not cached locally, a network request is triggered to download it from a remote server.
[0027] Optionally, it also includes:
[0028] When the network request fails, the scheduler intercepts the error response and initiates an internal retry algorithm to perform an internal retry.
[0029] When the number of retries reaches a preset threshold and the network request still fails, a network request failure message is sent to the player.
[0030] A streaming media processing device, comprising:
[0031] The response unit is used to respond to the streaming media playback request sent by the player and obtain the original streaming media link;
[0032] The protocol header modification unit is used to modify the protocol header of the original streaming media link to a custom protocol header, and to intercept loading requests based on the custom protocol header using the resource loading delegate protocol interface;
[0033] The parsing unit is used to parse the data request range contained in the loading request;
[0034] The judgment unit is used to search the local virtual storage index library according to the data request range, and to determine whether the streaming media data segment corresponding to the data request range has been cached in a local single physical storage file according to the search results. For each video stream, the virtual storage engine creates only one corresponding fixed-length single physical storage file on the disk.
[0035] The media data reading unit is used to read the media data segment directly from the single physical storage file according to the physical offset of the media data segment recorded in the virtual storage index library if the search result indicates that the media data segment within the data request range already exists, and then encapsulate and return it to the player.
[0036] Optionally, it also includes:
[0037] The delay waiting unit is used to add the loading request to the delay waiting queue and open a timer window of preset duration if the search result shows that the streaming media data segment corresponding to the data request range does not exist.
[0038] The task merging unit is used to merge multiple fragmented tasks that are consecutive or adjacent on logical addresses into an aggregated task within the timing window.
[0039] The alignment unit is used to align the request range of the aggregation task according to the page size of the file system to generate the final target network request.
[0040] The network download unit is used to initiate a network download to a remote server based on the target network request and obtain the streaming media data stream.
[0041] A computer storage medium storing at least one instruction that, when executed by a processor, implements the streaming media processing method described above.
[0042] An electronic device, comprising: a memory and a processor;
[0043] The memory is used to store at least one instruction;
[0044] The processor is used to execute the at least one instruction to implement the streaming media processing method described above.
[0045] As can be seen from the above technical solution, the present invention discloses a streaming media processing method and related apparatus. In response to a streaming media playback request sent by a player, the method obtains the original streaming media link, modifies the protocol header of the original streaming media link to a custom protocol header, intercepts the loading request based on the custom protocol header using a resource loading delegation protocol interface, parses the data request range contained in the loading request, searches in the local virtual storage index based on the data request range, and determines whether the streaming media data segment corresponding to the data request range has been cached in a local single physical storage file based on the search results. If the search results show that the media data segment within the data request range already exists, the method directly reads the media data segment from the single physical storage file according to the physical offset of the media data segment recorded in the virtual storage index, encapsulates it, and returns it to the player. Compared to traditional caching schemes that store each TS segment as an independent file, resulting in thousands of independent files for a single long video, this invention creates a unique, fixed-length single physical storage file for each video stream using a virtual storage engine. It utilizes a virtual storage index to record the physical offset of media data segments, allowing the file system to maintain only one file per video stream. This reduces inode resource consumption and solves the problem of low disk I / O read / write efficiency caused by frequent access to individual files in traditional schemes. Furthermore, when cleaning expired caches or calculating cache space, this invention allows the system to quickly locate data segments directly from a single physical storage file using physical offsets, eliminating the need to traverse massive numbers of independent files. This improves data response speed and streaming media storage efficiency. Attached Figure Description
[0046] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the published drawings without creative effort.
[0047] Figure 1 This is a flowchart of a streaming media processing method disclosed in an embodiment of the present invention;
[0048] Figure 2 This is a flowchart of another streaming media processing method disclosed in an embodiment of the present invention;
[0049] Figure 3 This is a schematic diagram of the structure of a streaming media processing device disclosed in an embodiment of the present invention;
[0050] Figure 4 This is a schematic diagram of another streaming media processing device disclosed in an embodiment of the present invention;
[0051] Figure 5 This is a schematic diagram of the structure of an electronic device disclosed in an embodiment of the present invention. Detailed Implementation
[0052] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0053] Explanation of relevant terms:
[0054] AVAssetResourceLoaderDelegate: A delegate protocol in the AVFoundation framework of the iOS system used to handle media resource loading requests, allowing developers to take over the network data loading process of the player.
[0055] Range request: A technique in the HTTP protocol that allows a client to request only a portion of a resource (such as bytes 0-1024).
[0056] Sparse cache: refers to the state where downloaded data segments are discontinuous and scattered in different locations of the file when the media stream has not been fully downloaded.
[0057] IOPS (Input / Output Operations Per Second): The number of read / write (I / O) operations per second.
[0058] An inode (index node) is the "identity card" and "metadata file" of a file system.
[0059] This invention discloses a streaming media processing method and related apparatus. It creates a unique, fixed-length single physical storage file for each video stream using a virtual storage engine and records the physical offsets of media data segments using a virtual storage index. This allows the file system to maintain only one file for each video stream, reducing inode resource consumption and solving the problem of low disk I / O read / write efficiency caused by frequent access to individual files in traditional solutions. Furthermore, when cleaning expired caches or calculating cache space, the system can directly and quickly locate data segments from a single physical storage file using physical offsets, without traversing a massive number of individual files, thereby improving data response speed and streaming media storage efficiency.
[0060] See Figure 1The present invention discloses a streaming media processing method flowchart, the method comprising the following steps:
[0061] Step S101: In response to the streaming media playback request sent by the player, obtain the original streaming media link.
[0062] In practical applications, the streaming media playback request can carry a unique identifier of the original streaming media link, and the corresponding original streaming media link can be obtained based on this unique identifier.
[0063] Step S102: Modify the protocol header of the original streaming media link to a custom protocol header, and use the resource loading delegation protocol interface to intercept loading requests based on the custom protocol header.
[0064] In this embodiment, before outputting the playback address to the player AVPlayer, the protocol header Scheme, which serves as the URL (Uniform Resource Locator), is modified from http: / / or https: / / to a custom protocol header, such as custom-hls: / / .
[0065] When AVPlayer detects an unprocessable protocol header, it triggers the shouldWaitForLoadingOfRequestedResource method of AVAssetResourceLoaderDelegate (resource loading delegation protocol).
[0066] This embodiment uses the resource loading delegation protocol interface to intercept loading requests based on a custom protocol header. The loading request can be an AVAssetResourceLoadingRequest (resource loading request object), which is then encapsulated as an internal task object.
[0067] Therefore, this embodiment solves the problem of the inability to intervene in the system's native AVPlayer black-box requests through the following technologies: 1) Protocol header replacement: Modify the protocol header (such as http or https) of the original streaming media link to a custom protocol header (such as custom-hls); 2) Request interception: Intercept loading requests based on the custom protocol header using the resource loading delegation protocol (AVAssetResourceLoaderDelegate) interface; 3) Network control takeover: Forcefully obtain the underlying network data loading permission through the custom protocol header, providing a prerequisite for the subsequent execution of scheduling strategies.
[0068] This invention is based entirely on the official proxy interface provided by AVFoundation, without modifying any system private APIs (Application Programming Interface), and without using unstable global Swizzle or interception, resulting in extremely high system stability and compatibility with future iOS versions. Furthermore, this invention not only supports native apps, but can also be reused for video components in cross-platform frameworks such as React Native or Flutter through bridging.
[0069] Step S103: Parse the data request range contained in the loading request.
[0070] Step S104: Search the local virtual storage index library according to the data request range, and determine whether the streaming media data segment corresponding to the data request range has been cached in a single local physical storage file based on the search results.
[0071] For each video stream, the virtual storage engine creates only one corresponding fixed-length physical storage file on the disk, along with a lightweight virtual storage index library, SQLite Map.
[0072] The virtual storage index contains a mapping table of {logical offset, length, physical offset, completion status}, where completion status refers to the completion status of the streaming media data stream download and storage to the disk.
[0073] To illustrate logical offsets, consider this example: the logical offset for the first TS segment of a video is 0-5000, and the logical offset for the second TS segment is 5001-10000. Within a single physical storage file, the logical offsets corresponding to each TS segment are written consecutively.
[0074] Step S105: If the search result indicates that the media data segment within the data request range already exists, the media data segment is directly read from the single physical storage file according to the physical offset of the media data segment recorded in the virtual storage index library, and then encapsulated and returned to the player.
[0075] This embodiment uses the physical offset of media data segments recorded in the virtual storage index to directly locate and read media data segments from a single physical storage file, achieving extremely fast response. At the same time, it solves the data reuse conflict problem in traditional caching mechanisms through precise physical address mapping.
[0076] In summary, this invention discloses a streaming media processing method. In response to a streaming media playback request sent by a player, the method obtains the original streaming media link, modifies the protocol header of the original streaming media link to a custom protocol header, intercepts loading requests based on the custom protocol header using a resource loading delegation protocol interface, parses the data request range contained in the loading request, searches the local virtual storage index based on the data request range, and determines whether the streaming media data segment corresponding to the data request range has been cached in a single local physical storage file based on the search results. If the search results indicate that the media data segment within the data request range already exists, the method directly reads the media data segment from the single physical storage file based on the physical offset of the media data segment recorded in the virtual storage index, encapsulates it, and returns it to the player. Compared to traditional caching schemes that store each TS segment as an independent file, resulting in thousands of independent files for a single long video, this invention creates a unique, fixed-length single physical storage file for each video stream using a virtual storage engine. It utilizes a virtual storage index to record the physical offset of media data segments, allowing the file system to maintain only one file per video stream. This reduces inode resource consumption and solves the problem of low disk I / O read / write efficiency caused by frequent access to individual files in traditional schemes. Furthermore, when cleaning expired caches or calculating cache space, this invention allows the system to quickly locate data segments directly from a single physical storage file using physical offsets, eliminating the need to traverse massive numbers of independent files. This improves data response speed and streaming media storage efficiency.
[0077] In weak network environments or when implementing features like "downloading while playing" or "offline caching," developers typically need to intervene in the download process. Existing solutions often utilize URL loading protocols for global URL interception or run a small local HTTP (Hypertext Transfer Protocol) server (e.g., localhost:8080) to forward traffic. However, with the increased security of iOS systems, the interception capabilities of URL loading protocols in WKWebView and AVPlayer are limited, and local server solutions suffer from port congestion and background persistence issues.
[0078] To solve the above problems, see [link to relevant documentation]. Figure 2 The flowchart of another streaming media processing method disclosed in this embodiment of the invention is as follows: Figure 1 Based on the illustrated embodiment, after step S104, the following may also be included:
[0079] Step S106: If the search result indicates that the streaming media data segment corresponding to the data request range does not exist, the loading request fragmentation task is added to the delayed waiting queue, and a timer window of preset duration is opened.
[0080] The value of the timing window is determined according to actual needs. For example, the timing window is 50ms. This invention does not limit this value.
[0081] To enable simultaneous download and playback, native media players frequently initiate fine-grained range requests. If these requests were directly passed through to the network layer, it would lead to repeated execution of the TCP (Transmission Control Protocol) three-way handshake and the TLS (Transport Layer Security) handshake, significantly consuming bandwidth and power in weak network environments. This solution introduces a timing window mechanism to merge the multiple fragmented tasks that would otherwise need to be split into a single aggregated task, thereby reducing the number of network handshakes.
[0082] To address the server concurrency overload issue caused by native media players initiating a large number of fine-grained range requests (such as hundreds of 16KB segment requests per second) when playing streaming media, this invention introduces a delayed waiting queue (LatentQueue) and a timing window mechanism to shape request traffic and transform request concurrency into a controllable queue.
[0083] Step S107: Within the timing window, if there are multiple fragmented tasks that are consecutive or adjacent on logical addresses in the delay waiting queue, merge the multiple fragmented tasks into one aggregated task.
[0084] For example, when this embodiment receives request A, it does not immediately initiate a network download. Instead, it generates a fragmented task from the loading request and adds it to a delayed waiting queue, and opens a timer window of a preset duration.
[0085] If request B is received within the timing window, and the start position of request B is consecutive to the end position of request A or the difference is within a threshold (e.g., 1KB), requests A and B are merged into an aggregation task. The request range of this aggregation task is then aligned downwards / upwards according to the file system page size (e.g., 4KB) to generate the final target network request. This solves the problems of high network TCP connection overhead and disk I / O fragmentation. For example, when the player requests data ranging from 100 to 200 bytes, the system automatically expands the actual download range to a complete block (e.g., 0-4096 bytes) and pre-stores the excess data in the cache to improve the hit rate of subsequent requests.
[0086] Step S108: Align the request range of the aggregation task according to the page size of the file system to generate the final target network request.
[0087] This embodiment significantly reduces the dual overhead of network transmission and disk storage by merging multiple fragmented tasks into a single aggregate task and strictly aligning the request range of the aggregate task with the file system page size (e.g., 4KB). Specifically, this scheme achieves performance optimization through the following mechanisms:
[0088] 1. Reduce TCP connection overhead: The merged aggregation task only needs to establish a single TCP connection, avoiding multiple three-way handshakes and TLS security handshakes caused by fragmented tasks, which can greatly save network handshake latency in weak network environments.
[0089] 2. Eliminate disk I / O fragmentation: Page-aligned request ranges ensure that data writing conforms to the physical storage characteristics of the file system, reducing frequent addressing and block reallocation operations caused by random writes and improving disk I / O efficiency.
[0090] 3. Synergistic optimization effect: The dual optimization of the network layer and storage layer creates a superimposed effect, which can shorten the first frame loading time and reduce the playback stuttering rate, especially in video streaming scenarios.
[0091] In practical applications, the request range of the aggregation task can be aligned up or down according to the page size of the file system to generate the final target network request.
[0092] The page size is determined based on actual needs, for example, 4KB.
[0093] To improve disk write efficiency, network range requests are automatically aligned downwards / upwards by 4KB (the file system page size). For example, a request for 100-200 might actually be adjusted to download 0-4096, with the extra data being cached in advance to improve the hit rate in subsequent downloads.
[0094] Step S109: Initiate a network download to the remote server based on the target network request to obtain the streaming media data stream.
[0095] In summary, when the streaming media data segment corresponding to the data request range does not exist, this invention adds the fragmented task generated by the loading request to a delayed waiting queue and opens a preset timer window. Within the timer window, if multiple fragmented tasks with consecutive or adjacent logical addresses exist in the delayed waiting queue, these multiple fragmented tasks are merged into a single aggregate task. The request range of the aggregate task is aligned according to the page size of the file system to generate the final target network request. Based on the target network request, a network download is initiated to the remote server to obtain the streaming media data stream. This invention reduces the number of network handshakes by introducing a timer window mechanism to dynamically merge multiple fragmented tasks that originally needed to be split into a single aggregate task. Specifically, the system forces data alignment according to the page size of the file system when executing the aggregate task. This design not only effectively reduces the number of TCP handshakes, especially the first frame loading time (Time To First Byte, TTFB), but also achieves a hidden preloading effect by downloading edge data multiple times. Because the preloaded data is strictly aligned with the file system page boundary, the cache hit rate is significantly improved during subsequent playback, and network jitter can be effectively resisted even in weak network environments, greatly reducing video stuttering.
[0096] In one embodiment, the streaming media processing method may further include:
[0097] The received streaming media data stream is populated into the player's content information request and data request in real time to support the normal operation of the playback process.
[0098] This embodiment ensures that the playback engine can continuously read data from the dual queues without blocking by parsing the received streaming media data stream in real time and accurately filling it into the player's ContentInformationRequest and DataRequest.
[0099] In one embodiment, the streaming media processing method may further include:
[0100] After receiving the streaming media data stream, the physical write position of the streaming media data stream in the single physical storage file is calculated based on the alignment range of the target network request.
[0101] Based on the physical write location, the streaming media data stream is written to the corresponding location in the single physical storage file, and the mapping relationship between the logical offset, physical offset, and the completion status of the streaming media data stream download and storage to the disk in the virtual storage index is updated synchronously.
[0102] In practical applications, the virtual storage index records a mapping table of {logical offset, length, physical offset, completion status}.
[0103] In summary, upon receiving streaming media data, this invention performs a bidirectional distribution operation, namely, real-time filling of the streaming media data stream into the player's content information request and data request. Through a collaborative scheduling mechanism between the content information request and data request, the normal operation of the playback process is supported. Based on the alignment interval of the target network request, the physical write position of the streaming media data stream in a single physical storage file is calculated, and the streaming media data stream is written to the corresponding position in the single physical storage file. Simultaneously, the mapping relationship of "logical offset - physical offset - completion status" in the virtual storage index is updated, effectively solving the I / O performance bottleneck problem of writing small files with high concurrency.
[0104] In practical applications, when a user drags the progress bar on the player's timeline, for example, from 00:00 to 10:00, the streaming media data in between has not been downloaded. This results in a large amount of non-continuous cached data (i.e., gaps), and existing technologies cannot achieve efficient reuse of this "sparse data".
[0105] To achieve efficient reuse of "sparse data", streaming media processing methods may also include:
[0106] (1) When it is detected that the user drags from the first time point to the second time point on the player timeline, it is determined that the intermediate streaming media data between the first time point and the second time point has not been downloaded;
[0107] (2) Using the virtual storage engine, mark two non-contiguous storage intervals in the virtual storage index, namely a cached start segment containing the first time point and a cached end segment containing the second time point, wherein the start segment and the end segment are uncached intermediate missing segments;
[0108] (3) When the player requests the target streaming media data of the missing intermediate segment, the virtual storage engine is used to quickly retrieve the index and query whether the target streaming media data requested by the player is cached locally;
[0109] (4) If the target streaming media data is already cached locally, it is read directly from the corresponding single physical storage file locally;
[0110] (5) If the local cache is not available, a network request is triggered to download the target streaming media data from the remote server.
[0111] This invention constructs a virtual storage index library to record the logical and physical offsets of each video data segment (whether stored continuously or not) in a single physical storage file. This ensures that when a user drags the player's timeline, causing data to be downloaded discontinuously (creating "holes"), the file system maintains only one physical storage file for each video stream. Therefore, it can effectively avoid the problem of index node resource exhaustion caused by a large number of small files on mobile devices.
[0112] This invention adopts a storage structure for a single physical storage file, which effectively avoids the creation and closing of thousands of small files, greatly reducing the metadata operations of the file system. When performing cache cleanup and querying cache space size, the speed is increased by 1-2 orders of magnitude compared with the existing technology, and it does not cause the main thread to be stuck.
[0113] In one embodiment, the streaming media processing method may further include:
[0114] When the network request fails, the scheduler intercepts the error response and initiates an internal retry algorithm to perform an internal retry.
[0115] When the number of retries reaches a preset threshold and the network request still fails, a network request failure message is sent to the player.
[0116] The value of the preset number of times threshold is determined according to actual needs, such as 3 times.
[0117] In summary, this invention designs an anomaly tolerance mechanism. When a network request fails, the scheduler immediately intercepts the error response and does not directly send a failure notification to the player, thus avoiding playback interruption. At this time, the system starts a backoff retry algorithm and enters a silent retry phase. When the number of retries reaches a preset threshold and the network request is still recognized, a network request failure message is then sent to the player, thereby improving resource utilization.
[0118] Corresponding to the above method embodiments, the present invention also provides a streaming media processing apparatus.
[0119] See Figure 3 The present invention discloses a schematic diagram of a streaming media processing device, which may include:
[0120] The response unit 201 is used to obtain the original streaming media link in response to the streaming media playback request sent by the player.
[0121] In practical applications, the streaming media playback request can carry a unique identifier of the original streaming media link, and the corresponding original streaming media link can be obtained based on this unique identifier.
[0122] The protocol header modification unit 202 is used to modify the protocol header of the original streaming media link to a custom protocol header, and to intercept loading requests based on the custom protocol header using the resource loading delegation protocol interface.
[0123] In this embodiment, before outputting the playback address to the player AVPlayer, the protocol header Scheme, which serves as the URL (Uniform Resource Locator), is modified from http: / / or https: / / to a custom protocol header, such as custom-hls: / / .
[0124] When AVPlayer detects an unprocessable protocol header, it triggers the shouldWaitForLoadingOfRequestedResource method of AVAssetResourceLoaderDelegate (resource loading delegation protocol).
[0125] This embodiment uses the resource loading delegation protocol interface to intercept loading requests based on a custom protocol header. The loading request can be an AVAssetResourceLoadingRequest (resource loading request object), which is then encapsulated as an internal task object.
[0126] Therefore, this embodiment solves the problem of the inability to intervene in the system's native AVPlayer black-box requests through the following technologies: 1) Protocol header replacement: Modify the protocol header (such as http or https) of the original streaming media link to a custom protocol header (such as custom-hls); 2) Request interception: Intercept loading requests based on the custom protocol header using the resource loading delegation protocol (AVAssetResourceLoaderDelegate) interface; 3) Network control takeover: Forcefully obtain the underlying network data loading permission through the custom protocol header, providing a prerequisite for the subsequent execution of scheduling strategies.
[0127] This invention is based entirely on the official proxy interface provided by AVFoundation, without modifying any system private APIs (Application Programming Interface), and without using unstable global Swizzle or interception, resulting in extremely high system stability and compatibility with future iOS versions. Furthermore, this invention not only supports native apps, but can also be reused for video components in cross-platform frameworks such as React Native or Flutter through bridging.
[0128] The parsing unit 203 is used to parse the data request range contained in the loading request.
[0129] The judgment unit 204 is used to search in the local virtual storage index library according to the data request range, and to determine whether the streaming media data segment corresponding to the data request range has been cached in a local single physical storage file according to the search results.
[0130] For each video stream, the virtual storage engine creates only one corresponding fixed-length single physical storage file on the disk.
[0131] The virtual storage index contains a mapping table of {logical offset, length, physical offset, completion status}, where completion status refers to the completion status of the streaming media data stream download and storage to the disk.
[0132] To illustrate logical offsets, consider this example: the logical offset for the first TS segment of a video is 0-5000, and the logical offset for the second TS segment is 5001-10000. Within a single physical storage file, the logical offsets corresponding to each TS segment are written consecutively.
[0133] The media data reading unit 205 is used to read the media data segment directly from the single physical storage file according to the physical offset of the media data segment recorded in the virtual storage index library if the search result shows that the media data segment within the data request range already exists, and then encapsulate and return it to the player.
[0134] This embodiment uses the physical offset of media data segments recorded in the virtual storage index to directly locate and read media data segments from a single physical storage file, achieving extremely fast response. At the same time, it solves the data reuse conflict problem in traditional caching mechanisms through precise physical address mapping.
[0135] In summary, this invention discloses a streaming media processing device that, in response to a streaming media playback request sent by a player, obtains the original streaming media link, modifies the protocol header of the original streaming media link to a custom protocol header, intercepts loading requests based on the custom protocol header using a resource loading delegation protocol interface, parses the data request range contained in the loading request, searches the local virtual storage index based on the data request range, and determines whether the streaming media data segment corresponding to the data request range has been cached in a local single physical storage file based on the search results. If the search results indicate that the media data segment within the data request range already exists, the media data segment is directly read from the single physical storage file based on the physical offset of the media data segment recorded in the virtual storage index, encapsulated, and returned to the player. Compared to traditional caching schemes that store each TS segment as an independent file, resulting in thousands of independent files for a single long video, this invention creates a unique, fixed-length single physical storage file for each video stream using a virtual storage engine. It utilizes a virtual storage index to record the physical offset of media data segments, allowing the file system to maintain only one file per video stream. This reduces inode resource consumption and solves the problem of low disk I / O read / write efficiency caused by frequent access to individual files in traditional schemes. Furthermore, when cleaning expired caches or calculating cache space, this invention allows the system to quickly locate data segments directly from a single physical storage file using physical offsets, eliminating the need to traverse massive numbers of independent files. This improves data response speed and streaming media storage efficiency.
[0136] In weak network environments or when implementing features like "downloading while playing" or "offline caching," developers typically need to intervene in the download process. Existing solutions often utilize URL loading protocols for global URL interception or run a small local HTTP (Hypertext Transfer Protocol) server (e.g., localhost:8080) to forward traffic. However, with the increased security of iOS systems, the interception capabilities of URL loading protocols in WKWebView and AVPlayer are limited, and local server solutions suffer from port congestion and background persistence issues.
[0137] To solve the above problems, see [link to relevant documentation]. Figure 4 A schematic diagram of another streaming media processing device disclosed in an embodiment of the present invention is shown. Figure 3 Based on the embodiment shown, after the determination unit 204, it may further include:
[0138] The delay waiting unit 206 is used to add the loading request to the delay waiting queue and open a timer window of preset duration if the search result shows that the streaming media data segment corresponding to the data request range does not exist.
[0139] The value of the timing window is determined according to actual needs. For example, the timing window is 50ms. This invention does not limit this value.
[0140] To enable simultaneous download and playback, native media players frequently initiate fine-grained range requests. If these requests were directly passed through to the network layer, it would lead to repeated execution of the TCP (Transmission Control Protocol) three-way handshake and the TLS (Transport Layer Security) handshake, significantly consuming bandwidth and power in weak network environments. This solution introduces a timing window mechanism to merge the multiple fragmented tasks that would otherwise need to be split into a single aggregated task, thereby reducing the number of network handshakes.
[0141] To address the server concurrency overload issue caused by native media players initiating a large number of fine-grained range requests (such as hundreds of 16KB segment requests per second) when playing streaming media, this invention introduces a delayed waiting queue (LatentQueue) and a timing window mechanism to shape request traffic and transform request concurrency into a controllable queue.
[0142] The task merging unit 207 is used to merge multiple fragmented tasks that are consecutive or adjacent on logical addresses into an aggregated task within the timing window.
[0143] Alignment unit 208 is used to align the request range of the aggregation task according to the page size of the file system to generate the final target network request.
[0144] This embodiment significantly reduces the dual overhead of network transmission and disk storage by merging multiple fragmented tasks into a single aggregate task and strictly aligning the request range of the aggregate task with the file system page size (e.g., 4KB). Specifically, this scheme achieves performance optimization through the following mechanisms:
[0145] 1. Reduce TCP connection overhead: The merged aggregation task only needs to establish a single TCP connection, avoiding multiple three-way handshakes and TLS security handshakes caused by fragmented tasks, which can greatly save network handshake latency in weak network environments.
[0146] 2. Eliminate disk I / O fragmentation: Page-aligned request ranges ensure that data writing conforms to the physical storage characteristics of the file system, reducing frequent addressing and block reallocation operations caused by random writes and improving disk I / O efficiency.
[0147] 3. Synergistic optimization effect: The dual optimization of the network layer and storage layer creates a superimposed effect, which can shorten the first frame loading time and reduce the playback stuttering rate, especially in video streaming scenarios.
[0148] In practical applications, the request range of the aggregation task can be aligned up or down according to the page size of the file system to generate the final target network request.
[0149] The page size is determined based on actual needs, for example, 4KB.
[0150] To improve disk write efficiency, network range requests are automatically aligned downwards / upwards by 4KB (the file system page size). For example, a request for 100-200 might actually be adjusted to download 0-4096, with the extra data being cached in advance to improve the hit rate in subsequent downloads.
[0151] The network download unit 209 is used to initiate a network download to a remote server based on the target network request to obtain streaming media data streams.
[0152] In summary, when the streaming media data segment corresponding to the data request range does not exist, this invention adds the fragmented task generated by the loading request to a delayed waiting queue and opens a preset timer window. Within the timer window, if multiple fragmented tasks with consecutive or adjacent logical addresses exist in the delayed waiting queue, these multiple fragmented tasks are merged into a single aggregate task. The request range of the aggregate task is aligned according to the page size of the file system to generate the final target network request. Based on the target network request, a network download is initiated to the remote server to obtain the streaming media data stream. This invention reduces the number of network handshakes by introducing a timer window mechanism to dynamically merge multiple fragmented tasks that originally needed to be split into a single aggregate task. Specifically, the system forces data alignment according to the page size of the file system when executing the aggregate task. This design not only effectively reduces the number of TCP handshakes, especially the first frame loading time (Time To First Byte, TTFB), but also achieves a hidden preloading effect by downloading edge data multiple times. Because the preloaded data is strictly aligned with the file system page boundary, the cache hit rate is significantly improved during subsequent playback, and network jitter can be effectively resisted even in weak network environments, greatly reducing video stuttering.
[0153] In one embodiment, the streaming media processing apparatus may further include:
[0154] The filling unit is used to fill the received streaming media data stream into the content information request and data request of the player in real time to support the normal operation of the playback process.
[0155] In one embodiment, the streaming media processing apparatus may further include:
[0156] An alignment unit is used to calculate the physical write position of the streaming media data stream in the single physical storage file based on the alignment interval requested by the target network after receiving the streaming media data stream.
[0157] The update unit is used to write the streaming media data stream to the corresponding position in the single physical storage file according to the physical write position, and synchronously update the mapping relationship of logical offset, physical offset and completion status of the streaming media data stream download and storage to disk in the virtual storage index library.
[0158] In one embodiment, the streaming media processing apparatus may further include:
[0159] The drag unit is used to determine that the intermediate streaming media data between the first and second time points has not been downloaded when it detects that the user is dragging the media from the first time point to the second time point on the player's timeline.
[0160] The marking unit is used to mark two non-contiguous storage intervals in the virtual storage index using the virtual storage engine. These intervals are a cached start segment containing the first time point and a cached end segment containing the second time point, wherein the start segment and the end segment are an uncached intermediate missing segment.
[0161] The query unit is used to quickly retrieve the index using the virtual storage engine when the player requests the target streaming media data of the missing intermediate segment, and query whether the target streaming media data requested by the player is cached locally.
[0162] The reading unit is used to read the target streaming media data directly from the corresponding single physical storage file if it is already cached locally.
[0163] The remote download unit is used to trigger a network request to download the target streaming media data from a remote server if the data is not cached locally.
[0164] In one embodiment, the streaming media processing apparatus may further include:
[0165] The initiation unit is used to intercept the error response through the scheduler and initiate an internal retry algorithm when the network request fails.
[0166] The information sending unit is used to send a network request failure message to the player when the number of retries reaches a preset threshold and the network request still fails.
[0167] It should be noted that for the specific working principles of each component in the device embodiment, please refer to the corresponding section of the method embodiment, which will not be repeated here.
[0168] Corresponding to the above method embodiments, the present invention also discloses a computer storage medium that stores at least one instruction, which, when executed by a processor, implements the steps shown in the streaming media processing method embodiments.
[0169] Computer storage media can be tangible media that may contain or store programs for use by or in conjunction with an instruction execution system, apparatus, or device. Computer storage media can be machine-readable signal media or machine-readable storage media. Computer storage media can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.
[0170] Corresponding to the above embodiments, such as Figure 5 As shown, the present invention also provides a schematic diagram of the structure of an electronic device, which may include: a processor 1 and a memory 2;
[0171] The processor 1 and memory 2 communicate with each other via communication bus 3.
[0172] Processor 1, for executing at least one instruction;
[0173] Memory 2 is used to store at least one instruction;
[0174] Processor 1 may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement embodiments of the present invention.
[0175] Memory 2 may include high-speed RAM memory, and may also include non-volatile memory, such as at least one disk storage device.
[0176] The processor executes at least one instruction to implement the steps shown in the embodiment of the streaming media processing method.
[0177] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.
[0178] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.
[0179] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. A streaming media processing method, characterized in that, include: In response to a streaming media playback request sent by the player, obtain the original streaming media link; The protocol header of the original streaming media link is modified to a custom protocol header, and the loading request based on the custom protocol header is intercepted using the resource loading delegate protocol interface; The data request range contained in the loading request is parsed; The local virtual storage index is searched according to the data request range, and the streaming media data segment corresponding to the data request range is determined according to the search results. It is determined whether the streaming media data segment corresponding to the data request range has been cached in a single physical storage file on the local disk. For each video stream, the virtual storage engine creates only one corresponding fixed-length single physical storage file on the disk. If the search results indicate that a media data segment within the data request range already exists, the media data segment is read directly from the single physical storage file based on the physical offset of the media data segment recorded in the virtual storage index, and then encapsulated and returned to the player.
2. The streaming media processing method according to claim 1, characterized in that, Also includes: If the search results indicate that the streaming media data segment corresponding to the data request range does not exist, the loading request is generated into a fragmented task and added to the delayed waiting queue, and a timer window of a preset duration is opened; Within the timing window, if there are multiple fragmented tasks that are consecutive or adjacent on logical addresses in the delay waiting queue, the multiple fragmented tasks are merged into one aggregated task. The request range of the aggregation task is aligned according to the page size of the file system to generate the final target network request; Based on the target network request, a network download is initiated to the remote server to obtain the streaming media data stream.
3. The streaming media processing method according to claim 2, characterized in that, Also includes: The received streaming media data stream is populated into the player's content information request and data request in real time to support the normal operation of the playback process.
4. The streaming media processing method according to claim 2 or 3, characterized in that, Also includes: After receiving the streaming media data stream, the physical write position of the streaming media data stream in the single physical storage file is calculated based on the alignment range of the target network request. Based on the physical write location, the streaming media data stream is written to the corresponding location in the single physical storage file, and the mapping relationship between the logical offset, physical offset, and the completion status of the streaming media data stream download and storage to the disk in the virtual storage index is updated synchronously.
5. The streaming media processing method according to claim 1, characterized in that, Also includes: When it is detected that the user drags from the first time point to the second time point on the player's timeline, it is determined that the intermediate streaming media data between the first time point and the second time point has not yet been downloaded. The virtual storage engine is used to mark two non-contiguous storage intervals in the virtual storage index, namely a cached start segment containing the first time point and a cached end segment containing the second time point, wherein the start segment and the end segment are uncached intermediate missing segments; When the player requests the target streaming media data of the missing intermediate segment, the virtual storage engine is used to quickly retrieve the index and query whether the target streaming media data requested by the player is cached locally. If the target streaming media data is already cached locally, it is read directly from the corresponding single physical storage file locally. If the target streaming media data is not cached locally, a network request is triggered to download it from a remote server.
6. The streaming media processing method according to claim 5, characterized in that, Also includes: When the network request fails, the scheduler intercepts the error response and initiates an internal retry algorithm to perform an internal retry. When the number of retries reaches a preset threshold and the network request still fails, a network request failure message is sent to the player.
7. A streaming media processing device, characterized in that, include: The response unit is used to respond to the streaming media playback request sent by the player and obtain the original streaming media link; The protocol header modification unit is used to modify the protocol header of the original streaming media link to a custom protocol header, and to intercept loading requests based on the custom protocol header using the resource loading delegate protocol interface; The parsing unit is used to parse the data request range contained in the loading request; The judgment unit is used to search the local virtual storage index library according to the data request range, and to determine whether the streaming media data segment corresponding to the data request range has been cached in a local single physical storage file according to the search results. For each video stream, the virtual storage engine creates only one corresponding fixed-length single physical storage file on the disk. The media data reading unit is used to read the media data segment directly from the single physical storage file according to the physical offset of the media data segment recorded in the virtual storage index library if the search result indicates that the media data segment within the data request range already exists, and then encapsulate and return it to the player.
8. The streaming media processing apparatus according to claim 7, characterized in that, Also includes: The delay waiting unit is used to add the loading request to the delay waiting queue and open a timer window of preset duration if the search result shows that the streaming media data segment corresponding to the data request range does not exist. The task merging unit is used to merge multiple fragmented tasks that are consecutive or adjacent on logical addresses into an aggregated task within the timing window. The alignment unit is used to align the request range of the aggregation task according to the page size of the file system to generate the final target network request. The network download unit is used to initiate a network download to a remote server based on the target network request and obtain the streaming media data stream.
9. A computer storage medium, characterized in that, The computer storage medium stores at least one instruction, which, when executed by a processor, implements the streaming media processing method as described in any one of claims 1 to 6.
10. An electronic device, characterized in that, The electronic device includes: a memory and a processor; The memory is used to store at least one instruction; The processor is used to execute the at least one instruction to implement the streaming media processing method as described in any one of claims 1 to 6.