A video high-performance storage system

Abstract

The application relates to the technical field of video storage, and discloses a high-performance video storage system, which comprises a producer end, a transfer end and a consumer end, the producer end and the consumer end are in communication with the transfer end, the producer end is configured with a pre-partition strategy, the pre-selected partition of video data is selected according to the pre-partition strategy, and a transmission signal is generated after the video data is serialized to the transfer end, the transfer end receives the transmission signal from the producer end and adds the serialized video data to a file of the pre-selected partition, the transfer end allocates the partition according to a consumption request initiated by the consumer end, each partition is consumed by one consumer, the consumer end pulls the video data according to the allocated partition, and submits an offset to the transfer end after consumption is completed.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of video storage, and particularly relates to a high-performance video storage system. BACKGROUND

[0002] The current mainstream video storage systems are divided into the following categories: GFS (Google FileSystem), which is an early distributed file system designed for Google search engine, has the characteristics of large file storage, high throughput and low cost. It adopts a master-slave architecture, the master node manages metadata, and the slave node stores data, and has good scalability. However, it has a single point of failure problem, that is, the master node failure will cause the system to be unavailable, and it lacks efficient support for small files; HDFS (Hadoop Distributed FileSystem), suitable for storage and processing of large-scale data sets, based on GFS design, is the core component of the Hadoop ecosystem. It also adopts a master-slave architecture, the master node NameNode manages file system metadata, and the slave node DataNode stores data, and has good fault tolerance and scalability. However, the random read-write performance is poor, suitable for large file sequential read-write, and the NameNode has a single point of failure problem; S3 (Simple Storage Service) is used for storage of massive data (such as video, picture, etc.), is an object storage service provided by Amazon, has high reliability and high scalability, provides RESTful interface, and is convenient to use. However, its cost is relatively high, especially for large-scale storage, and the storage efficiency for small files is low; Ceph is an open source unified storage platform, providing object storage, block storage and file storage. It has the advantages of open source unified storage platform, such as flexibility, scalability, high performance, and supports multiple storage interfaces. However, the system complexity is relatively high, and the configuration and management are relatively complex. MinIO, a high-performance, open-source object storage, compatible with S3 protocol. It has the characteristics of lightweight, easy to deploy, excellent performance, and is suitable for scenarios with high performance requirements. However, the function is relatively simple, and it may not meet all needs.

[0003] In the above video storage systems, GFS and HDFS have a single point of failure problem, GFS, S3 and HDFS have insufficient support for small files, and Ceph and MinIO systems have high complexity. SUMMARY

[0004] In view of the deficiencies in the prior art, the purpose of the present application is to provide a high-performance video storage system.

[0005] To achieve the above purpose, the present application provides the following technical scheme: a high-performance video storage system, comprising a producer end, a transfer end and a consumer end, the producer end and the consumer end are in communication with the transfer end, respectively,

[0006] The producer end is configured with a pre-partition strategy, according to which a preselected partition of video data is selected, and after the video data is serialized, a transmission signal is generated to the transit end,

[0007] The transit end receives the transmission signal from the producer end and adds the serialized video data to the preselected partition file, and according to the consumption request initiated by the consumer end, the transit end allocates partitions, each of which is consumed by a consumer, and the transit end schedules between partitions after consumption,

[0008] The consumer end pulls video data according to the allocated partition, and after consumption is completed, submits an offset to the transit end.

[0009] Further, the consumer end is configured with a consumption record module, which obtains consumption data of each video data in the partition by the consumer, the consumption data including viewing time, review times, progress adjustment, and when the consumer ends the consumption, the consumer end sends the consumption data to the transit end, and the transit end is configured with a repartition strategy, which judges whether there is video data meeting the cross-interest condition according to the consumption data and the interest label corresponding to the consumer, if there is video data meeting the cross-interest condition, the video data is defined as a deployment video data and is sent to the corresponding transit end for storage.

[0010] Further, the repartition strategy specifically includes

[0011] Obtain the consumption data submitted by each consumer end in the current partition and the interest label corresponding to the consumer, define the interest label corresponding to the current partition as the current label, and define the remaining interest labels as reference labels,

[0012] The video with a play count exceeding a preset play value is treated as a to-be-processed video, and a first interest parameter and a second interest parameter are calculated according to an interest calculation sub-strategy, the first interest parameter specifically being the interest degree of the consumer with the to-be-processed label to the to-be-processed video, and the second interest parameter specifically being the interest degree of the consumer without the to-be-processed label to the to-be-processed video,

[0013] The difference between the first interest degree and the second interest degree is calculated as a difference parameter, and if the difference parameter exceeds a preset interest difference threshold, it is considered to meet the cross-interest condition.

[0014] Further, the expression of the interest calculation sub-strategy is specifically:

[0015] + ,

[0016] wherein S represents the degree of interest, a proportional parameter representing the degree of interest, T represents the ratio of the video watching time length to the total time length of the video, a proportional parameter representing the ratio of the video watching time length to the total time length of the video, N represents the number of consumption times of the corresponding consumer to the video, and F represents the number of fast-forward times in the playing process, a proportional parameter representing the number of fast-forward times in the playing process, and R represents the number of backtracking times in the playing process, a proportional parameter representing the number of backtracking times in the playing process, representing the number of likes, Z represents the number of forwards, and C represents the number of comments, respectively represent the proportional coefficients of the number of likes, the number of forwards and the number of comments.

[0017] Further, the relay end is configured with an interest verification strategy, when the relay end receives the deployment video data, the interest verification strategy is executed, the interest verification strategy includes

[0018] acquiring verification playing data of the deployment video within a set verification time length and calculating a verification data value, the verification playing data includes the number of video clicks, the video playing time length and the number of video playing people, if the verification data value exceeds a preset verification threshold value, the deployment video data is retained in the corresponding partition, if the verification data value does not exceed the preset verification threshold value, the deployment video data is deleted in the corresponding partition and a cleaning signal is generated to the relay end sending the deployment video data.

[0019] Further, the relay end is configured with an interest difference threshold value correction strategy, when the relay end receives the cleaning signal, the interest difference threshold value correction strategy is executed to acquire a new interest difference threshold value, the expression of the interest difference threshold value correction strategy is specifically:

[0020]

[0021] wherein represents the new interest difference threshold value, represents the interest difference threshold value, and p represents a correction parameter.

[0022] Further, the producer end is configured with a pre-partition strategy, the pre-partition strategy includes pre-setting a partition or selecting the pre-selected partition through a self-defined partitioner.

[0023] Further, the producer end compresses the volume of the video data, packs multiple pieces of video data into a batch for sending, and returns immediately after sending the video data.

[0024] ​​​Further, the consumer end asynchronously processes video data and consumes video data in batches.

[0025] Advantages of the present application:

[0026] 1. Sequential read-write and zero-copy function: This is to ensure high-performance writing. Through sequential read-write, the data transmission efficiency is greatly improved, and the zero-copy function avoids repeated copying of data in memory, further improving performance.

[0027] 2. Linux-based mmap file mapping: This technology reduces the intermediate steps of response. When processing large files, mmap file mapping can map files directly to memory, reducing unnecessary intermediate steps in the file read-write process, thereby improving system response speed.

[0028] 3. Optimized video acquisition efficiency: In the process of video storage and processing, efficient acquisition of video data is crucial. This system ensures that video data can be quickly and stably acquired and transmitted through optimized storage structure and algorithm, which not only significantly improves performance, but also overcomes the shortcomings of existing systems in bandwidth control, providing a reliable solution for applications with strict bandwidth requirements. BRIEF DESCRIPTION OF DRAWINGS

[0029] Fig. 1 is the architecture diagram of the video high-performance storage system in the present application;

[0030] Fig. 2 is the flowchart of the re-partitioning strategy in the present application;

[0031] Fig. 3 is the flowchart of the interest verification strategy in the present application. DETAILED DESCRIPTION

[0032] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor are within the scope of protection of the present application.

[0033] It should be understood that when an element, referred to as being "on" another element, it can be directly on the other element or intervening elements can also be present. In addition, it should be understood that when an element is referred to as being "connected", "coupled", "attached", or "disposed" to another element, it can be directly connected, coupled, attached, or disposed to the other element or intervening elements can also be present. As used herein, the terms "vertical", "horizontal", "left", "right", and the like are merely used for the purpose of illustration and are not intended to be limiting.

[0034] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used in the description herein is for describing particular embodiments only and is not intended to be limiting. As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items.

[0035] As shown in FIG. 1, the video high-performance storage system of the present embodiment includes a producer end, a transfer end, and a consumer end. The producer end and the consumer end are in communication with the transfer end. For the video high-performance storage system of the present application, the following hardware conditions are required: Figs. 1-3 1. Sufficient memory is required to cache messages and indexes, thereby reducing disk I / O;

[0036] 2. The use of SSD or NVMe disks can significantly improve read and write speeds;

[0037] 3. High-speed network cards can improve network transmission speed;

[0038] 4. When deploying in a cluster, the partitions should be reasonably divided according to the data volume and access mode to enhance parallel processing capability;

[0039] 5. Replication configuration: the number of replicas should be reasonably set according to the fault tolerance requirements to balance performance and reliability.

[0040] The following will be introduced respectively by each end:

[0041] Producer (video data producer), also referred to as the producer end herein, has the following characteristics:

[0042] 1. Batch sending: multiple video data are packaged into a batch for sending, thereby reducing the number of network requests.

[0043] 2. Asynchronous sending: returns immediately after sending video data, thereby improving the performance of the producer.

[0044]

[0045] ​3. Compression: Compress the video data volume to reduce network transmission overhead.

[0046] The producer end is configured with a pre-partitioning strategy, and the pre-selected partition of the video data is selected according to the pre-partitioning strategy. After the video data is serialized, a transmission signal is generated to the relay end.

[0047] Broker (video data relay station), also known as the relay end in this paper, has the following characteristics:

[0048] 1. Sequential writing: sequentially append messages to data files.

[0049] 2. Zero-copy: reduce the number of data copies.

[0050] 3. Multi-thread processing: process network requests and disk I / O through multi-threading.

[0051] Broker is the core component of the video high-performance storage cluster, responsible for receiving, storing and processing video data from producers, and providing messages to consumers. It acts as a message relay station, ensuring reliable delivery and persistence of messages.

[0052] The relay end receives the transmission signal from the producer end and adds the serialized video data to the pre-selected partition file. The relay end allocates partitions according to the consumption request initiated by the consumer end, and each partition is consumed by a consumer. The relay end schedules between partitions after partition consumption.

[0053] The main functions of the Broker end are as follows:

[0054] Message reception and storage: Broker receives video data from producers and persists it to disk.

[0055] Message distribution: Broker distributes messages to consumers who subscribe to the corresponding topic.

[0056] Replica management: Broker is responsible for maintaining replicas of topic partitions to ensure data reliability.

[0057] Cluster management: Broker participates in cluster election and management to ensure normal operation of the cluster.

[0058] Consumer (video data consumer end), also known as the consumer end in this paper, has the following characteristics:

[0059] 1. Batch consumption: consumers consume video data in batches to reduce the number of network requests.

[0060] 2. Asynchronous consumption: consumers asynchronously process video data to improve consumption speed.

[0061] 3. Offset management: the consumer maintains the offset by itself, ensuring that the video data is not lost.

[0062] The consumer end pulls video data according to the assigned partition, and submits the offset to the relay end after consumption is completed.

[0063] Further, the producer end is configured with a pre-partition strategy, which includes pre-setting a partition or selecting the pre-selected partition through a custom partitioner.

[0064] Further, the consumer end is configured with a consumption record module, which obtains consumption data of each video data in the partition by the consumer, including viewing time, review times, progress adjustment, and sends the consumption data to the relay end when the consumer ends consumption. The relay end is configured with a re-partition strategy, which determines whether there is video data meeting the cross-interest condition according to the consumption data and the interest label corresponding to the consumer. If there is video data meeting the cross-interest condition, the video data is defined as a distribution video data and is sent to the corresponding relay end for storage.

[0065] Further, the re-partition strategy specifically includes

[0066] obtaining consumption data submitted by each consumer end in the current partition and the interest label corresponding to the consumer, defining the interest label corresponding to the current partition as a current label, and defining the remaining interest labels as reference labels,

[0067] taking the video with a playback value exceeding a preset playback value as a to-be-processed video and calculating a first interest parameter and a second interest parameter according to an interest calculation sub-strategy. The first interest parameter is specifically the interest degree of the consumer with the to-be-processed label to the to-be-processed video, and the second interest parameter is specifically the interest degree of the consumer without the to-be-processed label to the to-be-processed video.

[0068] For example:

[0069] The interest label of consumer 1 is A, B and C, and consumer 1 consumes video 1, video 2, video 3 and video 4 in a zone;

[0070] The interest label of consumer 2 is A and B, and consumer consumes video 1, video 2 and video 3 in a zone;

[0071] The interest label of consumer 3 is A and C, and consumer consumes video 1 and video 4 in a zone;

[0072] The pre-partition can be selected by the producer or partitioned by a pre-partition strategy, which can have errors or be incomplete. For example, a video can meet the a, b or c partition conditions and only be marked with interest label A. In this case, we want to determine whether video 2 and video 3 meet the interest label B and whether video 4 meets the interest label C through actual consumption, so as to improve the accuracy of the partition, and as the video plays, the accuracy of the partition will also increase, and the confidence will also increase.

[0073] Determine whether video 2 and video 3 meet partition B. At this time, the interest parameter is introduced, including the first interest parameter and the second interest parameter.

[0074] Further, the expression of the interest calculation sub-strategy is specifically:

[0075]

[0076] where S represents the interest degree, T represents the ratio of the video viewing time to the total video time, N represents the number of times of consumption of the corresponding consumer to the video, F represents the number of times of fast-forwarding in the playing process, R represents the number of times of rewinding in the playing process, Z represents the number of forwarding, and C represents the number of comments, respectively represent the proportionality coefficient of the number of likes, the number of forwarding and the number of comments.

[0077] The first interest parameter and the second interest parameter can be calculated according to the interest calculation sub-strategy. The difference between the first interest degree and the second interest degree is calculated as a difference parameter. If the difference parameter exceeds a preset interest difference threshold, it is considered to meet the cross-interest condition.

[0078] Further, the intermediate end is configured with an interest verification strategy. When the intermediate end receives the deployment video data, the interest verification strategy is executed. The interest verification strategy includes

[0079] ​​​​​Obtaining verification play data of the deployment video within a verification duration and calculating a verification data value, the verification play data including video click times, video play duration and video play number, if the verification data value exceeds a preset verification threshold, retaining the deployment video data in a corresponding partition, if the verification data value does not exceed the preset verification threshold, deleting the deployment video data in the corresponding partition and generating a clearing signal to a transit end sending the deployment video data.

[0080] Further, the transit end is configured with an interest difference threshold correction strategy, when the transit end receives the clearing signal, executing the interest difference threshold correction strategy to obtain a new interest difference threshold, an expression of the interest difference threshold correction strategy being specifically:

[0081] ,

[0082] Wherein represents the new interest difference threshold, represents the interest difference threshold, and p represents a correction parameter.

[0083] Working principle:

[0084] The producer end selects a partition according to the key of the message and a partition strategy (such as Round Robin, a custom partitioner), serializes the video data for storage and transmission, and then sends the serialized message to the corresponding transit end. The transit end receives the message from the producer and appends it to the specified partition file, while synchronizing the message to other replicas to ensure data redundancy, and also creates an index for the video data for fast lookup. The consumer subscribes to the classification of interest, and the consumers in the consumer group will be assigned partitions through the coordinator transit end, each partition will be consumed by a consumer, and the consumer pulls the video data from the assigned partition. After consuming the video data, the consumer end submits the offset to inform the transit end which video data has been consumed.

[0085] The above is only a preferred embodiment of the present application, the protection scope of the present application is not limited to the above-mentioned embodiments, any technical solution belonging to the idea of the present application is within the protection scope of the present application. It should be noted that for ordinary skilled in the art, some improvements and refinements without departing from the principles of the present application, these improvements and refinements should also be considered as the protection scope of the present application.

Claims

1. A high-performance video storage system, characterized by: It includes a producer end, a transfer end and a consumer end, wherein the producer end and the consumer end communicate with the transfer end respectively. The producer end is configured with a pre-partitioning strategy, and pre-selected partitions of the video data are selected according to the pre-partitioning strategy, and the video data is serialized to generate a transmission signal to the transfer end. The transfer end receives the transmission signal from the producer end and adds the serialized video data to the file of the pre-selected partition. The transfer end allocates partitions according to the consumption request initiated by the consumer end. Each partition will be consumed by a consumer. The transfer end schedules between partitions after the partition is consumed. The consumer end pulls the video data according to the allocated partition and submits the offset to the transfer end after consumption is completed; The consumer end is configured with a consumption record module, which obtains the consumer's consumption data on each video data in the partition, the consumption data including viewing time, number of replays, and progress adjustment. When the consumer ends consumption, the consumer end sends the consumption data to the transfer end. The transfer end is configured with a re-partitioning strategy, which determines whether there is video data that meets the cross-interest condition based on the consumption data and the interest tags corresponding to the consumer. If there is video data that meets the cross-interest condition, the video data is defined as allocated video data and sent to the corresponding transfer end for storage; The repartitioning strategy specifically includes Obtain the consumption data submitted by each consumer in the current partition and the corresponding interest tags of the consumers, define the interest tag corresponding to the current partition as the current tag, and define the remaining interest tags as reference tags. Videos with a playback volume exceeding a preset playback value are treated as videos to be processed and a first interest parameter and a second interest parameter are calculated according to the interest calculation sub-strategy. The first interest parameter is specifically the interest level of consumers with a to-be-processed tag in the video to be processed, and the second interest parameter is specifically the interest level of consumers without a processing tag in the video to be processed. The difference between the first interest level and the second interest level is calculated as a difference parameter. If the difference parameter exceeds a preset interest difference threshold, it is considered that the cross-interest condition is met.

2. The high-performance video storage system according to claim 1, wherein: The expression of the interest calculation sub-strategy is specifically: + ， Where S represents the level of interest, The ratio parameter representing the degree of interest, T represents the ratio of video viewing time to the total video time, The parameter representing the ratio of the video viewing time to the total video time, N represents the number of times the corresponding consumer consumes the video, and F represents the number of fast-forward times during playback. Indicates the ratio parameter of the number of fast forwards during playback, R indicates the number of rewinds during playback, Indicates the ratio parameter of the number of rewinds during playback. represents the number of likes, Z represents the number of reposts, and C represents the number of comments. 、 、 Represent the proportional coefficients of the number of likes, reposts and comments respectively.

3. The high-performance video storage system according to claim 1, wherein: The transfer end is configured with an interest verification strategy. When the transfer end receives the allocated video data, the interest verification strategy is executed. The interest verification strategy includes Obtain the verification playback data of the allocated video within the set verification time and calculate the verification data value, the verification playback data including the number of video clicks, video playback time and number of video players. If the verification data value exceeds the preset verification threshold, the allocated video data is retained in the corresponding partition. If the verification data value does not exceed the preset verification threshold, the allocated video data is deleted in the corresponding partition and a clear signal is generated to the transfer end that sent the allocated video data.

4. The high-performance video storage system according to claim 3, wherein: The transfer end is configured with an interest difference threshold correction strategy. When the transfer end receives the clear signal, the interest difference threshold correction strategy is executed to obtain a new interest difference threshold. The expression of the interest difference threshold correction strategy is specifically: ， in represents the new difference threshold of interest, represents the difference threshold of interest, and p represents the correction parameter.

5. The high-performance video storage system according to claim 1, wherein: The producer end is configured with a pre-partitioning strategy, which includes pre-setting partitions or selecting the pre-selected partitions through a custom partitioner.

6. The high-performance video storage system according to claim 1, wherein: The producer end compresses the volume of the video data, packages multiple video data into a batch for sending, and returns immediately after sending the video data.

7. The high-performance video storage system according to claim 1, wherein: The consumer end asynchronously processes the video data and consumes the video data in batches.

Images