Hierarchical compression storage data processing method, device and equipment and readable storage medium

Abstract

The invention relates to a hierarchical compression storage data processing method and device, equipment and a readable storage medium. The method comprises the following steps: acquiring real-time original code stream data, determining a first storage medium from a plurality of storage mediums of storage equipment, and writing the real-time original code stream data into the first storage medium; determining a second storage medium from the plurality of storage media, and reading historical code stream data corresponding to the real-time original code stream data from the second storage medium; the second storage medium is different from the first storage medium; and compressing the historical code stream data to obtain compressed code stream data, and writing the compressed code stream data into a first storage medium. By adopting the method, the storage performance of the existing storage medium can be utilized to the maximum extent under the condition that the storage space is not additionally increased, and the bottleneck problem of the storage performance is solved.

Description

Technical Field

[0001] This application relates to the field of video storage technology, and in particular to a hierarchical compression storage data processing method, apparatus, device, and readable storage medium. Background Technology

[0002] With the rapid development of information technology, the amount of data has exploded, leading to the emergence of compressed data storage and processing technologies. This technology compresses data, optimizing data storage costs, access efficiency, and management complexity, and is widely used in various fields, such as security and healthcare.

[0003] Taking the security industry as an example, in security storage devices, due to the large amount of video data, backend storage devices utilize local encoding and decoding capabilities to re-encode the incoming frontend bitstream to improve storage efficiency, thereby reducing storage (hard disk) space requirements and lowering storage solution costs. However, this method leads to problems such as loss of detailed features in the re-encoded video stream data. Therefore, tiered compression storage technology has emerged.

[0004] In related technologies, hierarchical compression storage compresses bitstream data from different time periods; however, this method places high demands on storage performance. Summary of the Invention

[0005] Therefore, it is necessary to provide a hierarchical compression storage data processing method, apparatus, computer equipment, computer-readable storage medium, and computer program product that can solve the storage performance bottleneck, in order to address the above-mentioned technical problems.

[0006] In a first aspect, this application provides a hierarchical compression storage data processing method, including:

[0007] Acquire real-time raw bitstream data, determine a first storage medium from multiple storage media of the storage device, and write the real-time raw bitstream data into the first storage medium;

[0008] A second storage medium is determined from the plurality of said storage media, and historical bitstream data corresponding to the real-time raw bitstream data is read from the second storage medium; the second storage medium is different from the first storage medium;

[0009] The historical bitstream data is compressed to obtain compressed bitstream data, and the compressed bitstream data is written to the first storage medium.

[0010] In one embodiment, acquiring real-time raw bitstream data includes:

[0011] Obtain the real-time raw bitstream data of the first channel;

[0012] The step of reading historical bitstream data corresponding to the real-time raw bitstream data from the second storage medium includes:

[0013] Historical bitstream data of the second channel corresponding to the real-time raw bitstream data is read from the second storage medium, wherein the first channel and the second channel are different channels.

[0014] In one embodiment, determining a first storage medium from a plurality of storage media of the storage device and writing the real-time raw bitstream data into the first storage medium includes:

[0015] A recycling queue is determined for the storage device, the recycling queue including at least one storage medium that has reclaimed historical bitstream data storage space, the storage medium being used for new bitstream data writing and allocation;

[0016] The first storage medium is determined from the recycling queue, and the real-time raw bitstream data is written into the first storage medium.

[0017] In one embodiment, the method further includes:

[0018] Once the real-time raw bitstream data has been written, the first storage medium is updated to the overwrite queue of the storage device.

[0019] In one embodiment, determining the second storage medium from the plurality of storage media includes:

[0020] Obtain the overlay queue of the storage device, and determine a second storage medium from the overlay queue, wherein the overlay queue includes at least one storage medium for historical storage space to be overwritten and reclaimed, and the storage medium is used for historical bitstream data read allocation.

[0021] In one embodiment, reading historical bitstream data corresponding to the real-time raw bitstream data from a second storage medium among the plurality of storage media includes:

[0022] Determine the current time of the real-time raw bitstream data;

[0023] The reading time is determined based on the current time, preset coverage granularity, and preset coverage period;

[0024] Determine the storage space corresponding to the reading time from the second storage medium, and read the historical bitstream data corresponding to the real-time raw bitstream data from the storage space corresponding to the reading time.

[0025] In one embodiment, the method further includes:

[0026] Upon completion of reading the historical bitstream data, the second storage medium is updated to the recycling queue of the storage device.

[0027] Secondly, this application also provides a hierarchical compressed storage data processing apparatus, comprising:

[0028] The bitstream acquisition module is used to acquire real-time raw bitstream data.

[0029] A stream writing module is used to determine a first storage medium from multiple storage media of a storage device and write the real-time raw stream data into the first storage medium.

[0030] A stream reading module is used to determine a second storage medium from a plurality of storage media, and to read historical stream data corresponding to the real-time raw stream data from the second storage medium; the second storage medium is different from the first storage medium.

[0031] A compression module is used to compress the historical bitstream data to obtain compressed bitstream data, and the bitstream writing module writes the compressed bitstream data to the first storage medium.

[0032] Thirdly, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform the following steps:

[0033] Acquire real-time raw bitstream data, determine a first storage medium from multiple storage media of the storage device, and write the real-time raw bitstream data into the first storage medium;

[0034] A second storage medium is determined from the plurality of said storage media, and historical bitstream data corresponding to the real-time raw bitstream data is read from the second storage medium; the second storage medium is different from the first storage medium;

[0035] The historical bitstream data is compressed to obtain compressed bitstream data, and the compressed bitstream data is written to the first storage medium.

[0036] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, performs the following steps:

[0037] Acquire real-time raw bitstream data, determine a first storage medium from multiple storage media of the storage device, and write the real-time raw bitstream data into the first storage medium;

[0038] A second storage medium is determined from the plurality of said storage media, and historical bitstream data corresponding to the real-time raw bitstream data is read from the second storage medium; the second storage medium is different from the first storage medium;

[0039] The historical bitstream data is compressed to obtain compressed bitstream data, and the compressed bitstream data is written to the first storage medium.

[0040] Fifthly, this application also provides a computer program product, including a computer program that, when executed by a processor, performs the following steps:

[0041] Acquire real-time raw bitstream data, determine a first storage medium from multiple storage media of the storage device, and write the real-time raw bitstream data into the first storage medium;

[0042] A second storage medium is determined from the plurality of said storage media, and historical bitstream data corresponding to the real-time raw bitstream data is read from the second storage medium; the second storage medium is different from the first storage medium;

[0043] The historical bitstream data is compressed to obtain compressed bitstream data, and the compressed bitstream data is written to the first storage medium.

[0044] The aforementioned hierarchical compression storage data processing method, apparatus, computer equipment, computer-readable storage medium, and computer program product, in the scenario of hierarchical compression storage, write the acquired real-time raw bitstream data to the first storage medium, and read historical bitstream data corresponding to the real-time raw bitstream data from another storage medium at the same time, and compress the read historical bitstream data before writing it to the first storage medium. This method separates real-time stream writing and historical stream reading, avoiding the simultaneous performance of three I / O operations—historical stream reading, real-time stream writing, and historical compressed stream writing—on the same storage medium. Without increasing storage space, it maximizes the utilization of the storage bandwidth (performance) of the existing storage medium, solving the bottleneck problem of storage performance. Attached Figure Description

[0045] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0046] Figure 1 This is an application environment diagram of a hierarchical compression storage data processing method in one embodiment;

[0047] Figure 2 This is a flowchart illustrating a hierarchical compression storage data processing method in one embodiment;

[0048] Figure 3 This is a schematic diagram illustrating the hierarchical storage historical stream read / write optimization in one embodiment;

[0049] Figure 4 This is a flowchart illustrating a method for determining historical bitstream data in one embodiment;

[0050] Figure 5 This is a schematic diagram of the hierarchical compression storage channel scheduling process in one embodiment;

[0051] Figure 6 This is a flowchart illustrating a hierarchical compression storage data processing method in another embodiment;

[0052] Figure 7 This is a schematic diagram of the structure of a hierarchical compressed storage data processing system in one embodiment;

[0053] Figure 8 This is a structural block diagram of a hierarchical compressed storage data processing device in one embodiment;

[0054] Figure 9 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0055] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0056] In security storage devices, to improve storage efficiency, backend storage devices utilize more powerful local encoding and decoding capabilities to perform secondary encoding on the front-end streams from connected front-end devices. Compared to the original front-end video stream bitrate, this can reduce the stream bandwidth, thereby reducing storage (hard disk) space requirements and lowering storage solution costs. For example, it can reduce the bandwidth to about 30% of the original stream. It should be noted that different compression storage technologies will result in different reductions in stream bandwidth.

[0057] Understandably, while compressed storage can reduce bitrate and storage costs, the re-encoded video stream data suffers from issues such as loss of detailed features. This data can only be used in scenarios where video quality requirements are not high. Tiered compressed storage mainly compresses data according to different time periods. For example, recent data (e.g., within the last 30 days) is more important and needs to be stored in its original bitrate, while historical data or even older data (e.g., from 30 to 90 days ago) has significantly reduced importance and can be compressed to reduce costs.

[0058] Tiered compression storage technology, a related technique, compresses and stores historical streams. For example, it reads real-time raw video data from 30 days ago, compresses it, and then stores it again. However, at any given moment, a storage medium may experience three I / O operations: historical stream read, real-time stream write, and historical compressed stream write. This approach places higher demands on read and write performance, requiring more storage hard drives or higher-performance, more expensive SAS hard drives or SSDs, thus increasing storage costs.

[0059] For example, in security storage scenarios, RAID storage solutions based on HDDs are mainly used. Recording for 30 days a month, assuming 256 video streams at 2Mbps, requires 165TB of storage space. A typical RAID array is between 50-300TB. 165TB is equivalent to the logical head and tail of a typical RAID array, which translates to the distance between the outer tracks of a single HDD. If a storage medium simultaneously performs historical read, real-time write, and historical compressed write operations, the HDD's seek latency typically accounts for 60-70% of the read / write latency. In practice, this can lead to a 50% drop in the overall RAID / HDD read / write bandwidth (performance). If a single storage medium (RAID, single disk, etc.) previously had an average performance of 256 read / write streams, it will now consistently operate under the worst-case scenario of random disk seeks, achieving only the overall tiered compressed storage performance of 128 mixed read / write streams.

[0060] To address the storage performance bottleneck in hierarchical compressed storage, a hierarchical compressed storage data processing method is proposed. This method comprehensively schedules different channels on multiple storage media in both time and space dimensions, so as to separate the reading of historical bitstream data and the writing of real-time raw bitstream data as much as possible, thereby maximizing the storage bandwidth (performance) of the storage device in the scenario of mixed real-time streaming and compressed streaming recording.

[0061] The hierarchical compression storage data processing method provided in this application embodiment can be applied to, for example... Figure 1 In the application environment shown, the first terminal 102 communicates with the second terminal 104 via a network. The first terminal 102 can be different types of front-end image acquisition devices, such as different types of cameras. The second terminal 104 can be different types of back-end storage devices, which are equipped with multiple storage media. Here, the type of storage media is not limited.

[0062] The data storage system can store the data that the second terminal 104 needs to process. The data storage system can be integrated into the second terminal 104 or placed in the cloud or on another network server. The second terminal acquires the real-time raw bitstream data collected by the first terminal. The second terminal selects a first storage medium from multiple storage media and writes the real-time raw bitstream data into the first storage medium. It then selects a second storage medium from the multiple storage media and reads historical bitstream data corresponding to the real-time raw bitstream data from the second storage medium. The second storage medium is different from the first storage medium. The historical bitstream data is compressed to obtain compressed bitstream data, which is then written to the first storage medium.

[0063] In one exemplary embodiment, such as Figure 2 As shown, a hierarchical compression storage data processing method is provided, which can be applied to... Figure 1 Taking the second terminal as an example, the explanation includes the following steps 202 to 206. Wherein:

[0064] Step 202: Obtain real-time raw bitstream data, determine the first storage medium from multiple storage media of the storage device, and write the real-time raw bitstream data into the first storage medium.

[0065] Understandably, a storage device (i.e., the second terminal) contains multiple storage media. To ensure normal read / write performance and bandwidth, and to ensure that real-time streaming recordings are spatially continuous, in security scenarios, recordings are often divided and packaged into logically multiple files of fixed size or time periods. By default, files are overwritten in chronological order based on their expiration time. If the real-time stream is overwritten every 30 days, with one file per day, meaning at least 31 days of file space are reserved, then the recording written on day T0 can overwrite the recording space of day T1 = T0 - 30 days.

[0066] In a tiered compression scenario, the speed of historical stream reading and real-time stream writing must be consistent, as they are dependent on each other. The real-time stream recordings that need to be covered must wait for the corresponding historical stream reading task to read and compress the real-time stream historical recordings of day T1 before they can be written to the corresponding storage space. Real-time stream historical recordings can also be called historical bitstream data or historical streams.

[0067] The first storage medium can be a specific storage unit used for writing real-time raw bitstream data. It can also be a target storage medium within a storage device that can store the real-time raw bitstream data. This target storage medium can be the storage medium containing historical bitstream data, determined based on the current time, preset coverage granularity, and preset coverage period corresponding to the real-time raw bitstream data. It should be noted that this historical bitstream data has been read and compressed. The target storage medium can also be a pre-specified storage medium. For example, to interleave historical stream reads and real-time stream writes on different hard drives, overall scheduling can be performed with a fixed number of channels and fixed time intervals. For instance, channels 1-256 might be written on storage medium A on days T0-T1, on storage medium B on days T1-T2, and on storage medium C on days T2-T3. The real-time raw bitstream data can be the video data stream captured at the current time point by the first terminal, i.e., the front-end video capture device (such as a camera). The current time corresponding to the real-time raw bitstream data can be the current day or the current point in time.

[0068] For example, when the storage device stores bitstream data of a preset duration, real-time raw bitstream data is obtained from the front-end video capture device. A first storage medium is determined from multiple storage media of the storage device, and the real-time raw bitstream data is written to the storage space corresponding to the first storage medium. The storage space can be a storage medium with a fixed logical address space, such as a logical volume, a single hard drive, RAID, or a storage pool. Furthermore, real-time raw bitstream data corresponding to multiple channels can be obtained simultaneously, and for each channel's real-time raw bitstream data, a corresponding first storage medium is determined from multiple storage media of the storage device.

[0069] It should be noted that the storage space corresponding to the first storage medium to which the real-time raw bitstream data is written can be the historical storage space corresponding to the same channel as the channel to which the real-time raw bitstream data belongs, or it can be the historical storage space corresponding to a channel different from the channel to which the real-time raw bitstream data belongs.

[0070] Step 204: Determine a second storage medium from multiple storage media, and read historical bitstream data corresponding to the real-time raw bitstream data from the second storage medium; the second storage medium is different from the first storage medium.

[0071] The second storage medium is a specific storage unit for storing historical bitstream data. The historical bitstream data corresponding to the real-time raw bitstream data can be determined by: real-time raw bitstream data for a historical period determined based on the current time of the real-time bitstream data and a preset coverage period. For example, if the real-time stream is covered in a 30-day cycle, the historical bitstream data for the current time T0 is the bitstream data for T0-30 days. It can be understood that the data for each historical time is real-time.

[0072] For example, real-time raw bitstream data is acquired, a first storage medium is determined from multiple storage media of the storage device, and a second storage medium different from the first storage medium is determined from multiple storage media, the real-time raw bitstream data is written to the first storage medium, and historical bitstream data corresponding to the real-time raw bitstream data is read from the second storage medium.

[0073] Step 206: Compress the historical bitstream data to obtain compressed bitstream data, and write the compressed bitstream data to the first storage medium.

[0074] The compressed bitstream data can be a video data stream processed by a compression algorithm, generated by compressing historical bitstream data. For example, historical bitstream data can be compressed using H.264, H.265, or other encoding standards. No specific limitations are placed on the implementation of the compression algorithm here.

[0075] For example, after reading the corresponding historical bitstream data in the second storage medium, the historical bitstream data is compressed using a compression algorithm to obtain compressed bitstream data, and the compressed bitstream data is written into the storage space in the first storage medium for storing the compressed bitstream data.

[0076] It is understandable that the hierarchical compression storage data processing method in this embodiment needs to ensure that the overall read and write operations of the storage device are interleaved as much as possible. For example, storage medium A and storage medium B may be interleaved during a certain time period. At time T0, real-time streaming writes are performed on storage medium A, while historical streaming reads are performed on storage medium B and compressed writes are performed on storage medium A. Figure 3 The diagram illustrates a tiered storage historical stream read / write optimization in an exemplary embodiment. On day N, the acquired real-time raw bitstream data, i.e., the new real-time stream, is written to the storage space to be overwritten (i.e., the original recording space) in storage medium B. Simultaneously, on day N, historical bitstream data corresponding to this real-time raw bitstream data is retrieved from storage medium A, i.e., historical stream read. On day N+1, the acquired real-time raw bitstream data, i.e., the new real-time stream, is written to storage medium A. Simultaneously, on day N+1, historical bitstream data corresponding to the real-time raw bitstream data of day N+1 is retrieved from storage medium B, i.e., historical stream read. Based on Figure 3 As shown in the diagram illustrating the hierarchical storage historical stream read / write optimization, this method improves read / write performance compared to reading and writing on the same storage medium. For example, if the traditional method originally had an average performance of 256 read / write operations on a single storage medium (RAID, single disk, etc.), the method in this example can achieve the same performance on two storage media simultaneously. Compared to the situation where a storage medium has three types of IO operations at the same time: historical stream read, real-time stream write, and historical compressed stream write, the overall global read / write performance is twice that of the corresponding read / write performance.

[0077] It should be noted that when writing real-time raw bitstream data to the storage space corresponding to the first storage medium, there may be a situation where the actual storage space required for the real-time raw bitstream data is greater than the storage space corresponding to the first storage medium. In this case, the redundancy mechanism of the storage device can be used to determine the actual storage space for writing the real-time raw bitstream data based on the preset redundant storage space and the storage space corresponding to the first storage medium, so as to ensure the integrity of the data.

[0078] The aforementioned hierarchical compression storage data processing method acquires real-time raw bitstream data, determines a first storage medium from multiple storage media and writes the real-time raw bitstream data; determines a second storage medium from multiple storage media and reads historical bitstream data; compresses the historical bitstream data to generate compressed bitstream data and writes it to the first storage medium. By separating the writing of real-time raw bitstream data from the reading of historical bitstream data, and by using compression technology to reprocess historical data, hierarchical management of data storage is achieved. Through reasonable scheduling of real-time and historical streams in both time and space dimensions, the read and write performance of the first and second storage media is ensured, and the storage space requirements are reduced, alleviating the performance bottleneck problem in traditional compressed storage methods. At the same time, by reasonably allocating tasks to different storage media, the overall system storage efficiency is improved. In other words, the above embodiment maximizes the utilization of the storage bandwidth (performance) of existing storage media without increasing the storage space.

[0079] To achieve comprehensive scheduling of different channels on multiple storage media, in an exemplary embodiment, acquiring real-time raw bitstream data includes: acquiring real-time raw bitstream data of a first channel; correspondingly, reading historical bitstream data corresponding to the real-time raw bitstream data from a second storage medium includes: reading historical bitstream data of a second channel corresponding to the real-time raw bitstream data from the second storage medium, wherein the first channel and the second channel are different channels.

[0080] In this context, a channel can be understood as a data transmission path or a classification of data sources. For example, in the field of video surveillance, each surveillance camera is an independent channel, and each channel generates corresponding video data.

[0081] It should be noted that, without adding additional storage space, in order to maximize the utilization of the storage bandwidth (performance) of the existing storage media, the storage media in the storage device need to be scheduled for storage time and space according to the globally optimal IO performance.

[0082] In one embodiment, determining a first storage medium from a plurality of storage media of a storage device and writing real-time raw bitstream data into the first storage medium includes: determining a recycling queue of the storage device, the recycling queue including at least one storage medium that has reclaimed historical bitstream data storage space, the storage medium being used for new bitstream data writing allocation; determining the first storage medium from the recycling queue and writing real-time raw bitstream data into the first storage medium.

[0083] The reclaim queue includes reclaimed historical video space that can be allocated for new video writing. The number of storage media can be one or more. Historical video space can also be called historical storage space, and new video can be called real-time raw bitstream data. The reclaim queue can be understood as a queue structure in a storage device used to manage released storage space; it records a list of storage media available for new data writing. The reclaim queue can improve the reuse efficiency of storage media and determine the best first storage media based on globally optimal IO performance, thus optimizing storage space allocation logic. For example, the storage media of the earliest reclaimed historical video space can be allocated from the reclaim queue for real-time raw bitstream data writing. The reclaim queue can be updated, but is not limited to, updates triggered by space release events.

[0084] The historical bitstream data storage space that has been reclaimed, i.e. the historical video recording space that has been reclaimed, is the storage space that has been released after the historical bitstream data stored in this space has been read and compressed.

[0085] For example, when acquiring real-time raw bitstream data, idle candidate storage media are determined from the recycling queue of the storage device. If multiple idle candidate storage media exist in the recycling queue, the first storage media can be determined by random selection, or the storage media of the earliest recycled historical recording space can be used as the first storage media. Furthermore, considering a tiered compression storage scenario, which involves different channels of multiple storage media, acquiring the real-time raw bitstream data of the first channel can be done by writing the real-time raw bitstream data into the historical recording space of the same channel in the first storage media, or by writing the real-time raw bitstream data into the historical recording space of the third channel in the first storage media.

[0086] For example, in the security field, a large amount of video data is written, compressed, and archived daily in security storage scenarios. After the compression of historical videos is completed, a large amount of storage space is released. The storage management module marks this space as reclaimed and adds it to the reclaim queue. When new real-time raw bitstream data is acquired, the first storage medium is determined from the reclaim queue for writing the real-time raw bitstream data.

[0087] In the above embodiments, storage media available for writing are determined from multiple reclaimed historical bitstream data storage spaces, and a first storage medium is selected from these for writing real-time raw bitstream data. Managing storage media using a reclamation queue allows for the determination of the optimal storage medium for new real-time stream writing, thereby improving storage media reuse efficiency. While ensuring real-time data writing performance, the scheduling logic of storage resources is also optimized, further alleviating storage performance bottlenecks.

[0088] In one exemplary embodiment, upon completion of the real-time raw bitstream data writing, the first storage medium is updated to the overwrite queue of the storage device. The overwrite queue includes storage media that can be used to allocate historical video readings and have their historical video space to be overwritten and reclaimed.

[0089] For example, once the real-time raw bitstream data has been written, an update is triggered by a write completion event, updating the first storage medium to the overwrite queue of the storage device. This approach, by updating the first storage medium to the overwrite queue after the real-time raw bitstream data has been written, can improve storage medium utilization efficiency and reduce performance bottlenecks caused by resource contention.

[0090] In one exemplary embodiment, such as Figure 4 As shown, a method for determining historical bitstream data is provided, including the following steps:

[0091] Step 402: Determine the current time of the real-time raw bitstream data.

[0092] The current time can be the system timestamp for real-time raw bitstream data acquisition or reception.

[0093] Step 404: Determine the reading time based on the current time, preset coverage granularity, and preset coverage period.

[0094] The preset coverage period is determined based on the actual application scenario. For example, it could be 30 days, 7 days, or 60 days as a coverage cycle; no specific limitation is made here. The preset coverage granularity can be any of hourly, daily, or weekly granularity. In this embodiment, the preset coverage granularity is set to daily granularity. The read time can be obtained by subtracting the coverage period from the current time and aligning it according to the coverage granularity.

[0095] Step 406: Determine the storage space corresponding to the reading time from the second storage medium, and read the historical bitstream data corresponding to the real-time raw bitstream data from the storage space corresponding to the reading time.

[0096] Taking the intelligent security video storage system as an example, the front-end camera continuously generates high-definition real-time raw bitstream data. The system calculates the historical data time point that needs to be read and compressed based on the current time, the set coverage granularity (such as daily) and the coverage period (such as 7 days), locates the historical bitstream data corresponding to the time point in the second storage medium and performs compression processing, and finally writes the compressed bitstream data into the first storage medium.

[0097] Furthermore, after completing the reading of historical bitstream data, the second storage medium is updated to the storage device's recycling queue.

[0098] In this processing method, historical stream data is read from a second storage medium different from the first storage medium, thus separating real-time stream writing and historical stream reading and alleviating storage performance bottlenecks.

[0099] In one exemplary embodiment, such as Figure 5 As shown, based on the above-described hierarchical compressed storage data processing method, a hierarchical compressed storage channel scheduling flowchart is provided, including the following:

[0100] The channel scheduling module of the storage device maintains two queues composed of different storage media: a reclaim queue and an overwrite queue. Both queues are initialized. The reclaim queue contains storage media for reclaimed historical video space that can be allocated for new video recordings, while the overwrite queue contains storage media for historical video space to be overwritten and reclaimed that can be allocated for historical video readings. Simultaneously, all channels are grouped and scheduled according to different batches.

[0101] On day N, storage medium A is allocated from the recycle queue for writing new recordings to channel X. After writing is completed on day N+1, storage medium A is added to the overwrite queue. On day N, storage medium B is allocated from the overwrite queue for reading historical recordings to channel Y, while simultaneously compressing and storing them. After reading and compressing all historical recordings on day N+1, storage medium B is added to the recycle queue. This process continues, performing global channel read / write cross-scheduling. The global channel read / write cross-scheduling ends when a preset termination condition is met. The preset termination condition can be set according to actual needs, for example, based on time periods.

[0102] This approach involves comprehensive scheduling of different channels on multiple storage media in both time and space dimensions, achieving maximum separation between historical video reading and real-time video writing, and ensuring that the storage device maximizes storage bandwidth (performance) in scenarios involving mixed real-time and compressed video recording.

[0103] In one exemplary embodiment, such as Figure 6 As shown, a hierarchical compression storage data processing method is provided, which can be applied to... Figure 1Taking the second terminal as an example, the explanation includes steps 602 to 610. Wherein:

[0104] Step 602: Obtain the real-time raw bitstream data of the first channel, determine the first storage medium from the recycling queue of the storage device, and write the real-time raw bitstream data into the first storage medium.

[0105] Step 604: Determine the second storage medium from the overlay queue of the storage device, and read the historical bitstream data of the second channel corresponding to the real-time raw bitstream data from the second storage medium.

[0106] Step 606: Compress the historical bitstream data to obtain compressed bitstream data, and write the compressed bitstream data to the first storage medium.

[0107] Step 608: After completing the real-time raw bitstream data writing, update the first storage medium to the overwrite queue of the storage device.

[0108] Step 610: After completing the reading of historical bitstream data, update the second storage medium to the recycling queue of the storage device.

[0109] It should be noted that the specific implementation of this embodiment can be achieved in the manner defined above, and will not be elaborated here.

[0110] In the above embodiments, without increasing storage space, the reading of historical bitstream data and the writing of real-time raw bitstream data are separated by comprehensively scheduling different channels on multiple storage media in both time and space dimensions. This avoids frequent jumps between physical tracks on the same storage medium where read and write operations are performed simultaneously, thereby ensuring that the storage bandwidth (performance) of the storage device is maximized in the scenario of mixed recording of real-time and compressed streams.

[0111] Based on the same inventive concept, this application also provides a hierarchical compression storage data processing system for implementing the hierarchical compression storage data processing method described above. The solution provided by this system is similar to the implementation scheme described in the above method; therefore, the specific limitations in one or more hierarchical compression storage data processing system embodiments provided below can be found in the limitations of the hierarchical compression storage data processing method described above, and will not be repeated here.

[0112] In one exemplary embodiment, such as Figure 7 The diagram illustrates the structure of a hierarchical compressed storage data processing system. This system includes a streaming module, a real-time streaming storage module, a storage scheduling module, a historical streaming read module, a compression module, and a compressed streaming storage module.

[0113] The streaming module is used to pull real-time raw bitstream data from the front-end device. The real-time stream storage module receives the raw bitstream data and sends it to the storage scheduling module for storage. For example, it determines a first storage medium from multiple storage media of the storage device and writes the real-time raw bitstream data to the first storage medium. The historical stream read module is used to read historical bitstream data corresponding to the real-time raw bitstream data. For example, it determines a second storage medium from multiple storage media and reads the historical bitstream data corresponding to the real-time raw bitstream data from the second storage medium. The compression module receives the historical bitstream data and compresses it into compressed bitstream data. The compressed stream storage module receives the compressed bitstream data and sends it to the storage scheduling module for storage. The storage scheduling module performs storage time and space scheduling according to the globally optimal IO performance.

[0114] In the above embodiments, the acquired real-time raw bitstream data is written to the first storage medium, and historical bitstream data corresponding to the real-time raw bitstream data is read from another second storage medium at the same time. The read historical bitstream data is compressed and then written to the first storage medium. This method separates real-time stream writing and historical stream reading, avoiding the simultaneous performance of three IO operations—historical stream reading, real-time stream writing, and historical compressed stream writing—on the same storage medium. Without increasing storage space, it maximizes the utilization of the storage bandwidth (performance) of the existing storage medium, thus solving the bottleneck problem of storage performance.

[0115] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0116] Based on the same inventive concept, this application also provides a hierarchical compression storage data processing apparatus for implementing the hierarchical compression storage data processing method described above. The solution provided by this apparatus is similar to the implementation scheme described in the above method; therefore, the specific limitations in one or more embodiments of the hierarchical compression storage data processing apparatus provided below can be found in the limitations of the hierarchical compression storage data processing method described above, and will not be repeated here.

[0117] In one exemplary embodiment, such as Figure 8As shown, a hierarchical compressed storage data processing device is provided, comprising: a bitstream acquisition module 802, a bitstream writing module 804, a bitstream reading module 806, and a compression module 808, wherein:

[0118] The bitstream acquisition module 802 is used to acquire real-time raw bitstream data.

[0119] The bitstream writing module 804 is used to determine the first storage medium from multiple storage media of the storage device and write the real-time raw bitstream data into the first storage medium.

[0120] The bitstream reading module 806 is used to determine a second storage medium from multiple storage media, and to read historical bitstream data corresponding to the real-time raw bitstream data from the second storage medium; the second storage medium is different from the first storage medium.

[0121] The compression module 808 is used to compress historical bitstream data to obtain compressed bitstream data, and the compressed bitstream data is written to the first storage medium through the bitstream writing module.

[0122] The aforementioned hierarchical compression storage data processing device writes the acquired real-time raw bitstream data to the first storage medium, and reads historical bitstream data corresponding to the real-time raw bitstream data from another second storage medium during the same time period. The read historical bitstream data is then compressed and written back to the first storage medium. This method separates real-time stream writing and historical stream reading, avoiding the simultaneous performance of three I / O operations—historical stream reading, real-time stream writing, and historical compressed stream writing—on the same storage medium. This reduces the demand on storage performance and maximizes the utilization of the storage bandwidth (performance) of the existing storage medium without adding additional storage space, thus solving the bottleneck problem of storage performance.

[0123] In one embodiment, the bitstream acquisition module 802 is also used to acquire real-time raw bitstream data of the first channel.

[0124] In one embodiment, the bitstream reading module 806 is used to read historical bitstream data of the second channel corresponding to the real-time raw bitstream data from the second storage medium, wherein the first channel and the second channel are different channels.

[0125] In one embodiment, the stream writing module 804 is used to determine the recycling queue of the storage device, the recycling queue including at least one storage medium that has reclaimed historical stream data storage space, the storage medium being used for new stream data writing allocation;

[0126] The first storage medium is determined from the recycling queue, and the real-time raw bitstream data is written to the first storage medium.

[0127] In one embodiment, a queue update module is also included, which is used to update the first storage medium to the overlay queue of the storage device when the real-time raw bitstream data has been written.

[0128] In one embodiment, the bitstream reading module 806 is used to obtain the overlay queue of the storage device and determine the second storage medium from the overlay queue, wherein the overlay queue includes at least one storage medium for historical storage space to be overwritten and reclaimed, and the storage medium is used for historical bitstream data reading allocation.

[0129] In one embodiment, the bitstream reading module 806 is used to determine the current time of the real-time raw bitstream data;

[0130] In one embodiment, the reading time is determined based on the current time, a preset coverage granularity, and a preset coverage period;

[0131] Determine the storage space corresponding to the reading time from the second storage medium, and read the historical bitstream data corresponding to the real-time raw bitstream data from the storage space corresponding to the reading time.

[0132] In one embodiment, the queue update module is used to update the second storage medium to the recycling queue of the storage device after the historical bitstream data reading is completed.

[0133] Each module in the aforementioned hierarchical compression storage data processing device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in the processor of a computer device in hardware form or independent of it, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0134] In one exemplary embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 9As shown, the computer device includes a processor, memory, input / output interfaces, a communication interface, a display unit, and an input device. The processor, memory, and input / output interfaces are connected via a system bus, and the communication interface, display unit, and input device are also connected to the system bus via the input / output interfaces. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The input / output interfaces are used for exchanging information between the processor and external devices. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, Near Field Communication (NFC), or other technologies. When the computer program is executed by the processor, it implements a hierarchical compression storage data processing method. The display unit is used to form a visually visible image and can be a display screen, a projection device, or a virtual reality imaging device. The display screen can be an LCD screen or an e-ink screen. The input device of the computer device can be a touch layer covering the display screen, or buttons, trackballs, or touchpads set on the casing of the computer device, or external keyboards, touchpads, or mice, etc.

[0135] Those skilled in the art will understand that Figure 9 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0136] In one embodiment, a computer device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above method embodiments.

[0137] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the steps in the above method embodiments.

[0138] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0139] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0140] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0141] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0142] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A hierarchical compression storage data processing method, characterized in that, The method includes: Acquire real-time raw bitstream data, determine a first storage medium from multiple storage media of the storage device, and write the real-time raw bitstream data into the first storage medium; A second storage medium is determined from the plurality of said storage media, and historical bitstream data corresponding to the real-time raw bitstream data is read from the second storage medium; the second storage medium is different from the first storage medium; The historical bitstream data is compressed to obtain compressed bitstream data, and the compressed bitstream data is written to the first storage medium.

2. The method according to claim 1, characterized in that, The acquisition of real-time raw bitstream data includes: Obtain the real-time raw bitstream data of the first channel; The step of reading historical bitstream data corresponding to the real-time raw bitstream data from the second storage medium includes: Historical bitstream data of the second channel corresponding to the real-time raw bitstream data is read from the second storage medium, wherein the first channel and the second channel are different channels.

3. The method according to claim 1 or 2, characterized in that, The step of determining a first storage medium from multiple storage media of the storage device and writing the real-time raw bitstream data into the first storage medium includes: A recycling queue is determined for the storage device, the recycling queue including at least one storage medium that has reclaimed historical bitstream data storage space, the storage medium being used for new bitstream data writing and allocation; The first storage medium is determined from the recycling queue, and the real-time raw bitstream data is written into the first storage medium.

4. The method according to claim 3, characterized in that, The method further includes: Once the real-time raw bitstream data has been written, the first storage medium is updated to the overwrite queue of the storage device.

5. The method according to claim 1, characterized in that, Determining the second storage medium from the plurality of storage media includes: Obtain the overlay queue of the storage device, and determine a second storage medium from the overlay queue, wherein the overlay queue includes at least one storage medium for historical storage space to be overwritten and reclaimed, and the storage medium is used for historical bitstream data read allocation.

6. The method according to claim 5, characterized in that, The step of reading historical bitstream data corresponding to the real-time raw bitstream data from the second storage medium among the plurality of storage media includes: Determine the current time of the real-time raw bitstream data; The reading time is determined based on the current time, preset coverage granularity, and preset coverage period; Determine the storage space corresponding to the reading time from the second storage medium, and read the historical bitstream data corresponding to the real-time raw bitstream data from the storage space corresponding to the reading time.

7. The method according to claim 5, characterized in that, The method further includes: Upon completion of reading the historical bitstream data, the second storage medium is updated to the recycling queue of the storage device.

8. A hierarchical compression storage data processing device, characterized in that, The device includes: The bitstream acquisition module is used to acquire real-time raw bitstream data; A stream writing module is used to determine a first storage medium from multiple storage media of a storage device and write the real-time raw stream data into the first storage medium. A stream reading module is used to determine a second storage medium from a plurality of storage media, and to read historical stream data corresponding to the real-time raw stream data from the second storage medium; the second storage medium is different from the first storage medium. A compression module is used to compress the historical bitstream data to obtain compressed bitstream data, and the bitstream writing module writes the compressed bitstream data to the first storage medium.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

11. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

Images