Video high-performance storage system

Abstract

The invention relates to the technical field of video storage, and discloses a video high-performance storage system, which comprises a producer end, a transfer end and a consumer end, the producer end and the consumer end are respectively communicated with the transfer end, the producer end is configured with a pre-partitioning strategy, a pre-selected partition of video data is selected according to the pre-partitioning strategy, and the pre-selected partition of the video data is stored in the pre-selected partition. Video data is serialized to generate a transmission signal to the transfer end, the transfer end receives the transmission signal from the producer end and adds the serialized video data to the file of the preselected partition, the transfer end allocates partitions according to a consumption request initiated by the consumer end, and each partition is consumed by one consumer. And the consumer end pulls the video data according to the allocated partitions, and submits offset to the transfer end after consumption is completed.

Description

Technical Field

[0001] The present invention relates to the technical field of video storage, and in particular to a high-performance video storage system. Background Art

[0002] The current mainstream video storage systems are divided into the following categories: GFS (Google File System), which is an early distributed file system designed for Google search engine, with the characteristics of large file storage, high throughput and low cost. It adopts a master-slave architecture, with the master node managing metadata and the slave node storing data, and has good scalability. However, it has a single point of failure problem, that is, the failure of the master node will cause the system to be unavailable, and lacks efficient support for small files; HDFS (Hadoop Distributed File System), suitable for the storage and processing of large-scale data sets, is based on the GFS design and is the core component of the Hadoop ecosystem. It also adopts a master-slave architecture, with the master node NameNode managing the file system metadata and the slave node DataNode storing data, and has good fault tolerance and scalability. However, the random read and write performance is poor, suitable for sequential reading and writing of large files, and the NameNode has a single point of failure problem; S3 (Simple Storage Service), used for the storage of massive data (such as videos, pictures, etc.), is an object storage service provided by Amazon, with high reliability and high scalability, providing a RESTful interface for easy use. However, its cost is high, especially when storing large-scale data, and its efficiency in storing small files is low. Ceph is an open source unified storage platform that provides object storage, block storage, and file storage. It has the advantages of an open source unified storage platform, such as flexibility, scalability, high performance, and support for multiple storage interfaces. However, the system is highly complex, and the configuration and management are relatively complex. MinIO is a high-performance, open source object storage that is compatible with the S3 protocol. It is lightweight, easy to deploy, and has excellent performance, and is suitable for scenarios with high performance requirements. However, its functions are relatively simple and may not meet all requirements.

[0003] In the above video storage systems, GFS and HDFS have single point failure problems, GFS, S3, and HDFS have insufficient support for small files, and Ceph and MinIO systems are relatively complex. Summary of the invention

[0004] In view of the deficiencies in the prior art, an object of the present invention is to provide a high-performance video storage system.

[0005] To achieve the above object, the present invention provides the following technical solution: a high-performance video storage system, comprising a producer end, a transfer end and a consumer end, wherein the producer end and the consumer end communicate with the transfer end respectively. The producer end is configured with a pre-partitioning strategy, and a pre-selected partition of the video data is selected according to the pre-partitioning strategy, and the video data is serialized to generate a transmission signal to the transfer end. The transfer end receives the transmission signal from the producer end and adds the serialized video data to the file of the pre-selected partition. The transfer end allocates partitions according to the consumption request initiated by the consumer end. Each partition will be consumed by a consumer. The transfer end schedules between partitions after the partitions are consumed. The consumer end pulls the video data according to the allocated partition and submits the offset to the transfer end after the consumption is completed.

[0006] Furthermore, the consumer end is configured with a consumption record module, which obtains the consumer's consumption data on each video data in the partition, and the consumption data includes viewing time, number of replays, and progress adjustment. When the consumer finishes consumption, the consumer end sends the consumption data to the transfer end, and the transfer end is configured with a re-partitioning strategy. The re-partitioning strategy determines whether there is video data that meets the cross-interest conditions based on the consumption data and the interest tags corresponding to the consumer. If there is video data that meets the cross-interest conditions, the video data is defined as allocated video data and sent to the corresponding transfer end for storage.

[0007] Furthermore, the re-partitioning strategy specifically includes Obtain the consumption data submitted by each consumer in the current partition and the interest tags corresponding to the consumers, define the interest tag corresponding to the current partition as the current tag, and define the remaining interest tags as reference tags. The video with a playback volume exceeding a preset playback value is regarded as a to-be-processed video, and a first interest parameter and a second interest parameter are calculated according to the interest calculation sub-strategy. The first interest parameter is specifically the interest level of consumers with to-be-processed tags in the to-be-processed video, and the second interest parameter is specifically the interest level of consumers without processing tags in the to-be-processed video. The difference between the first interest level and the second interest level is calculated as a difference parameter. If the difference parameter exceeds a preset interest difference threshold, it is considered that the cross-interest condition is met.

[0008] Furthermore, the expression of the interest calculation sub-strategy is specifically: + , Where S represents the level of interest, The ratio parameter representing the interest level, T represents the ratio of the video viewing time to the total video time, The ratio parameter representing the ratio of the video viewing time to the total video time. N represents the number of times the corresponding consumer consumes the video. F represents the number of fast-forward times during playback. It indicates the ratio parameter of the number of fast forward times during playback, and R indicates the number of rewind times during playback. Indicates the ratio parameter of the number of rewinds during playback. represents the number of likes, Z represents the number of reposts, and C represents the number of comments. , , They represent the proportional coefficients of likes, reposts and comments respectively.

[0009] Furthermore, the transfer end is configured with an interest verification strategy. When the transfer end receives the deployed video data, the interest verification strategy is executed. The interest verification strategy includes The verification playback data of the allocated video within the set verification time is obtained and the verification data value is calculated, wherein the verification playback data includes the number of video clicks, video playback time and the number of video players. If the verification data value exceeds the preset verification threshold, the allocated video data is retained in the corresponding partition; if the verification data value does not exceed the preset verification threshold, the allocated video data is deleted in the corresponding partition and a clear signal is generated to the transfer end that sent the allocated video data.

[0010] Furthermore, the transfer end is configured with an interest difference threshold correction strategy. When the transfer end receives the clear signal, the interest difference threshold correction strategy is executed to obtain a new interest difference threshold. The expression of the dynamic impact factor correction sub-strategy of the interest difference threshold correction strategy is specifically: , in represents the new difference threshold of interest, represents the difference threshold of interest, and p represents the correction parameter.

[0011] Furthermore, the producer end is configured with a pre-partitioning strategy, which includes pre-setting partitions or selecting the pre-selected partitions through a custom partitioner.

[0012] Furthermore, the producer end compresses the volume of the video data, packages multiple pieces of video data into a batch for sending, and returns immediately after sending the video data.

[0013] Furthermore, the consumer end asynchronously processes the video data and consumes the video data in batches.

[0014] Beneficial effects of the present invention: 1. Sequential read and write and zero copy function: This is to ensure high-performance writing. Through sequential read and write, data transmission efficiency is greatly improved, and the zero copy function avoids repeated copying of data in memory, further improving performance; 2. Linux-based mmap file mapping: This technology reduces the intermediate links of the response. When processing large files, mmap file mapping can map the file directly into the memory, reducing unnecessary intermediate steps in the file reading and writing process, thereby improving the system's response speed.

[0015] 3. Optimize video acquisition efficiency: In the process of video storage and processing, efficient acquisition of video data is the key. This system ensures that video data can be acquired and transmitted quickly and stably through optimized storage structure and algorithm. This invention not only significantly improves performance, but also overcomes the shortcomings of existing systems in bandwidth control, providing a reliable solution for applications with strict bandwidth requirements. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 It is a schematic diagram of the architecture of the video high-performance storage system in the present invention; Figure 2 It is a flow chart of the re-partitioning strategy in the present invention; Figure 3 It is a flowchart of the interest verification strategy in the present invention. DETAILED DESCRIPTION

[0017] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0018] It should be noted that when a component is referred to as being "fixed to" another component, it may be directly on the other component or there may also be a component centered. When a component is considered to be "connected to" another component, it may be directly connected to the other component or there may also be a component centered. When a component is considered to be "set on" another component, it may be directly set on the other component or there may also be a component centered. The terms "vertical", "horizontal", "left", "right" and similar expressions used herein are for illustrative purposes only.

[0019] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which the present invention belongs. The terms used herein in the specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. The term "and / or" used herein includes any and all combinations of one or more of the related listed items.

[0020] like Figure 1-Figure 3 As shown, a high-performance video storage system of this embodiment includes a producer end, a transfer end and a consumer end, and the producer end and the consumer end communicate with the transfer end respectively. The high-performance video storage system involved in this application needs to have the following hardware conditions: 1. Sufficient memory is needed to cache messages and indexes to reduce disk I / O; 2. Using SSD or NVMe disk can significantly improve the reading and writing speed; 3. High-speed network card can improve network transmission speed; 4. When deploying a cluster, partitions should be reasonably divided according to the data volume and access mode to enhance parallel processing capabilities; 5. Replica configuration: Set the number of replicas reasonably according to fault tolerance requirements to balance performance and reliability.

[0021] The following introduces each end separately: Producer (video data producer), which is the producer end in this article, has the following characteristics: 1. Batch sending: Pack multiple video data into one batch and send them to reduce the number of network requests.

[0022] 2. Asynchronous sending: Return immediately after sending video data to improve producer performance.

[0023] 3. Compression: compress the video data volume to reduce network transmission overhead.

[0024] The producer end is configured with a pre-partitioning strategy, and a pre-selected partition of the video data is selected according to the pre-partitioning strategy, and the video data is serialized to generate a transmission signal to the transfer end.

[0025] Broker (video data transfer station), which is the transfer end in this article, has the following characteristics: 1. Sequential writing: Append messages to the data file in sequence.

[0026] 2. Zero copy: reduce the number of data copies.

[0027] 3. Multi-threaded processing: Process network requests and disk I / O through multi-threading.

[0028] Broker is the core component of the video high-performance storage cluster, responsible for receiving, storing and processing video data from producers and providing messages to consumers. It acts as a message transfer station to ensure reliable delivery and persistence of messages.

[0029] The transfer end receives the transmission signal from the producer end and adds the serialized video data to the file of the pre-selected partition. The transfer end allocates partitions according to the consumption request initiated by the consumer end, and each partition will be consumed by a consumer. The transfer end performs scheduling between partitions after the partitions are consumed.

[0030] The main functions of the Broker are as follows: Message reception and storage: Broker receives video data from producers and persists it to disk.

[0031] Message distribution: Broker distributes messages to consumers who have subscribed to the corresponding topics.

[0032] Replica management: Broker is responsible for maintaining replicas of topic partitions to ensure data reliability.

[0033] Cluster management: Broker participates in cluster election and management to ensure the normal operation of the cluster.

[0034] Consumer (video data consumer end), that is, the consumer end in this article, has the following characteristics: 1. Batch consumption: Consumers consume video data in batches to reduce the number of network requests.

[0035] 2. Asynchronous consumption: Consumers process video data asynchronously to increase consumption speed.

[0036] 3. Offset management: Consumers maintain offsets by themselves to ensure that video data is not lost.

[0037] The consumer end pulls the video data according to the allocated partition and submits the offset to the transfer end after the consumption is completed.

[0038] Furthermore, the producer end is configured with a pre-partitioning strategy, which includes pre-setting partitions or selecting the pre-selected partitions through a custom partitioner.

[0039] Furthermore, the consumer end is configured with a consumption record module, which obtains the consumer's consumption data on each video data in the partition, and the consumption data includes viewing time, number of replays, and progress adjustment. When the consumer finishes consumption, the consumer end sends the consumption data to the transfer end, and the transfer end is configured with a re-partitioning strategy. The re-partitioning strategy determines whether there is video data that meets the cross-interest conditions based on the consumption data and the interest tags corresponding to the consumer. If there is video data that meets the cross-interest conditions, the video data is defined as allocated video data and sent to the corresponding transfer end for storage.

[0040] Furthermore, the re-partitioning strategy specifically includes Obtain the consumption data submitted by each consumer in the current partition and the interest tags corresponding to the consumers, define the interest tag corresponding to the current partition as the current tag, and define the remaining interest tags as reference tags. Videos with playback volume exceeding a preset playback value are treated as videos to be processed, and the first interest parameter and the second interest parameter are calculated according to the interest calculation sub-strategy. The first interest parameter is specifically the interest level of consumers with the to-be-processed label in the video to be processed, and the second interest parameter is specifically the interest level of consumers without the processing label in the video to be processed.

[0041] For example: Consumer 1's interest tags are A, B, and C. Consumer 1 consumed Video 1, Video 2, Video 3, and Video 4 in area a. Consumer 2's interest tags are A and B. The consumer consumed Video 1, Video 2, and Video 3 in area a. Consumer 3’s interest tags are A and C. The consumer consumed Video 1 and Video 4 in area a. Pre-partitioning can be selected by the producer or partitioned through a pre-partitioning strategy. The partitioning may have errors or be incomplete. There may be a video that meets the requirements of zone a, zone b, or zone c but is only marked with interest tag A. At this time, we hope to judge whether video 2 and video 3 meet interest tag B and whether video 4 meets interest tag C based on actual consumption, so as to improve the accuracy of the partitioning. As the number of video plays increases, the accuracy of the partitioning will also increase, and the confidence will also increase accordingly.

[0042] It is determined whether video 2 and video 3 conform to partition B. At this time, interest parameters are introduced, including a first interest parameter and a second interest parameter.

[0043] Furthermore, the expression of the interest calculation sub-strategy is specifically: + , Where S represents the level of interest, The ratio parameter representing the interest level, T represents the ratio of the video viewing time to the total video time, The ratio parameter representing the ratio of the video viewing time to the total video time. N represents the number of times the corresponding consumer consumes the video. F represents the number of fast-forward times during playback. It indicates the ratio parameter of the number of fast forward times during playback, and R indicates the number of rewind times during playback. Indicates the ratio parameter of the number of rewinds during playback. represents the number of likes, Z represents the number of reposts, and C represents the number of comments. , , They represent the proportional coefficients of likes, reposts and comments respectively.

[0044] The first interest parameter and the second interest parameter can be calculated according to the interest calculation sub-strategy. The difference between the first interest level and the second interest level is calculated as a difference parameter. If the difference parameter exceeds a preset interest difference threshold, it is considered that the cross interest condition is met.

[0045] Furthermore, the transfer end is configured with an interest verification strategy. When the transfer end receives the deployed video data, the interest verification strategy is executed. The interest verification strategy includes The verification playback data of the allocated video within the set verification time is obtained and the verification data value is calculated. The verification playback data includes the number of video clicks, the video playback time, and the number of video players. If the verification data value exceeds the preset verification threshold, the allocated video data is retained in the corresponding partition. If the verification data value does not exceed the preset verification threshold, the allocated video data is deleted in the corresponding partition and a clear signal is generated to the transfer end that sends the allocated video data. In this embodiment, the verification data value is positively correlated with the number of video clicks, the video playback time, and the number of video players, which can reflect the consumption situation after the allocated video is placed in other partitions.

[0046] Furthermore, the transfer end is configured with an interest difference threshold correction strategy. When the transfer end receives the clear signal, the interest difference threshold correction strategy is executed to obtain a new interest difference threshold. The expression of the dynamic impact factor correction sub-strategy of the interest difference threshold correction strategy is specifically: , in represents the new difference threshold of interest, represents the difference threshold of interest, and p represents the correction parameter.

[0047] Working principle: The producer side selects a partition based on the key and partition strategy (such as Round Robin, custom partitioner) of the message, serializes the video data for storage and transmission, and then sends the serialized message to the corresponding transfer end. The transfer end receives the message from the producer and appends it to the specified partition file. At the same time, it synchronizes the message to other replicas to ensure data redundancy and creates an index for the video data for quick search. Consumers subscribe to the categories of interest. Consumers in the consumer group will be assigned partitions through the coordinator transfer end. Each partition will be consumed by a consumer. The consumer pulls the video data from the assigned partition. After consuming the video data, the consumer side will submit the offset to inform the transfer end which video data has been consumed.

[0048] The above are only preferred embodiments of the present invention. The protection scope of the present invention is not limited to the above embodiments. All technical solutions under the concept of the present invention belong to the protection scope of the present invention. It should be pointed out that for ordinary technicians in this technical field, some improvements and modifications without departing from the principle of the present invention should also be regarded as the protection scope of the present invention.

Claims

1. A high-performance video storage system, characterized in that: It includes a producer end, a transfer end and a consumer end, wherein the producer end and the consumer end communicate with the transfer end respectively. The producer end is configured with a pre-partitioning strategy, and a pre-selected partition of the video data is selected according to the pre-partitioning strategy, and the video data is serialized to generate a transmission signal to the transfer end. The transfer end receives the transmission signal from the producer end and adds the serialized video data to the file of the pre-selected partition. The transfer end allocates partitions according to the consumption request initiated by the consumer end. Each partition will be consumed by a consumer. The transfer end schedules between partitions after the partitions are consumed. The consumer end pulls the video data according to the allocated partition and submits the offset to the transfer end after the consumption is completed.

2. The high-performance video storage system according to claim 1, characterized in that: The consumer end is configured with a consumption record module, which obtains the consumer's consumption data on each video data in the partition, and the consumption data includes viewing time, number of replays, and progress adjustment. When the consumer finishes consumption, the consumer end sends the consumption data to the transfer end, and the transfer end is configured with a re-partitioning strategy. The re-partitioning strategy determines whether there is video data that meets the cross-interest conditions based on the consumption data and the interest tags corresponding to the consumer. If there is video data that meets the cross-interest conditions, the video data is defined as allocated video data and sent to the corresponding transfer end for storage.

3. The high-performance video storage system according to claim 2, characterized in that: The repartitioning strategy specifically includes Obtain the consumption data submitted by each consumer in the current partition and the interest tags corresponding to the consumers, define the interest tag corresponding to the current partition as the current tag, and define the remaining interest tags as reference tags. The video with a playback volume exceeding a preset playback value is regarded as a to-be-processed video, and a first interest parameter and a second interest parameter are calculated according to the interest calculation sub-strategy. The first interest parameter is specifically the interest level of consumers with to-be-processed tags in the to-be-processed video, and the second interest parameter is specifically the interest level of consumers without processing tags in the to-be-processed video. The difference between the first interest level and the second interest level is calculated as a difference parameter. If the difference parameter exceeds a preset interest difference threshold, it is considered that the cross-interest condition is met.

4. The high-performance video storage system according to claim 3, characterized in that: The expression of the interest calculation sub-strategy is specifically: + ， Where S represents the level of interest, The ratio parameter representing the interest level, T represents the ratio of the video viewing time to the total video time, The ratio parameter representing the ratio of the video viewing time to the total video time. N represents the number of times the corresponding consumer consumes the video. F represents the number of fast-forward times during playback. It indicates the ratio parameter of the number of fast forward times during playback, and R indicates the number of rewind times during playback. Indicates the ratio parameter of the number of rewinds during playback. represents the number of likes, Z represents the number of reposts, and C represents the number of comments. , , They represent the proportional coefficients of likes, reposts and comments respectively.

5. The high-performance video storage system according to claim 2, characterized in that: The transfer end is configured with an interest verification strategy. When the transfer end receives the allocated video data, the interest verification strategy is executed. The interest verification strategy includes The verification playback data of the allocated video within the set verification time is obtained and the verification data value is calculated, wherein the verification playback data includes the number of video clicks, video playback time and the number of video players. If the verification data value exceeds the preset verification threshold, the allocated video data is retained in the corresponding partition; if the verification data value does not exceed the preset verification threshold, the allocated video data is deleted in the corresponding partition and a clear signal is generated to the transfer end that sent the allocated video data.

6. The high-performance video storage system according to claim 5, characterized in that: The transfer end is configured with an interest difference threshold correction strategy. When the transfer end receives the clear signal, the interest difference threshold correction strategy is executed to obtain a new interest difference threshold. The expression of the dynamic impact factor correction sub-strategy of the interest difference threshold correction strategy is specifically: ， in represents the new difference threshold of interest, represents the difference threshold of interest, and p represents the correction parameter.

7. The high-performance video storage system according to claim 1, characterized in that: The producer end is configured with a pre-partitioning strategy, which includes pre-setting partitions or selecting the pre-selected partitions through a custom partitioner.

8. The high-performance video storage system according to claim 1, characterized in that: The producer end compresses the volume of the video data, packages multiple video data into a batch for sending, and returns immediately after sending the video data.

9. The high-performance video storage system according to claim 1, characterized in that: The consumer end asynchronously processes the video data and consumes the video data in batches.

Images