A method and system for stream management and persistence of files

Abstract

The application provides a method and system for stream storage management and persistence of files, and relates to the technical field of computer storage, which comprises the following steps: dividing a hard disk storage space into a plurality of file blocks with the same size to obtain a plurality of initial state unused file blocks; creating and managing a plurality of virtual domains based on the obtained plurality of initial state unused file blocks; allocating the plurality of initial state unused file blocks to different virtual domains for management; when a to-be-written file is received according to the established virtual domain, matching to the corresponding virtual domain based on the attribute of the to-be-written file, and allocating a target file block and a unique file serial number for the to-be-written file in the corresponding virtual domain; and generating a corresponding file ID based on the allocated target file block and unique file serial number. The application realizes low-latency, high-throughput and sequential read-write storage management of sequentially, massively and quickly continuously arriving stream files in application scenarios.

Description

Technical Field

[0001] This invention relates to the field of computer storage technology, and in particular to a method and system for streaming storage management and persistence of files. Background Technology

[0002] In real-time scenarios such as the Internet, various video surveillance, and AI computing, relevant files (such as security monitoring images and AI training samples) are mostly streaming data, which are sequential, large-scale, and rapidly arriving file sequences. Their storage and access must meet the rigid requirements of sequential read and write, low latency, and high throughput.

[0003] Mainstream storage solutions include NAS (Network Attached Storage) and object storage systems, but none are specifically designed for the core characteristics of sequentially arriving and sequentially processed streaming files. This leads to the following technical shortcomings: they cannot simultaneously meet the rigid requirements of sequential read / write operations in streaming file storage and the performance demands for low latency and high throughput. Existing solutions lack targeted aggregation and sequential storage mechanisms for file streams. They cannot orderly aggregate consecutively arriving files into batches for writing, nor do they construct a physical storage layout and index structure that matches the writing order, resulting in the inability to form efficient sequential I / O operations during file read / write processes.

[0004] The above defects directly lead to the following problems: When writing streaming files, the continuously arriving files are stored in a scattered manner, which cannot form a batch of continuous IO operations. Random IO is frequent, and the disk IO performance is limited. For example, in video surveillance scenarios, vehicle images captured by intersection cameras need to be stored in real time. The storage response latency of existing solutions cannot meet the low latency requirements of the scenario, affecting subsequent real-time early warning analysis. Moreover, when reading files, due to the lack of physical storage layout and index support that match the writing order, it is difficult to quickly locate the physical location of continuous streaming files, and high-throughput access cannot be achieved. Summary of the Invention

[0005] The technical problem to be solved by this invention is to provide a method and system for streaming storage management and persistence of files, which realizes sequential reading and writing of streaming files, low latency response and high throughput access, effectively adapting to the storage management needs of streaming files in real-time application scenarios.

[0006] To solve the above-mentioned technical problems, the technical solution of the present invention is as follows: A method for streaming storage management and persistence of files, the method comprising: Divide the hard disk storage space into multiple file blocks of the same size to obtain multiple file blocks that are initially unused. Based on the multiple file blocks that are initially unused, create and manage multiple virtual storage domains, denoted as virtual domains; allocate the multiple file blocks that are initially unused to different virtual domains for management. Based on the established virtual domain, when a file to be written is received, the corresponding virtual domain is matched according to the attributes of the file to be written, and a target file block is allocated for the file to be written in the corresponding virtual domain, and a unique file sequence number is assigned to the file to be written. Based on the allocated target file block and the unique file sequence number, a corresponding file ID is generated; at the same time, the data to be written to the file is cached, and the file ID is returned to the client to complete the write response. Based on the completion of the write response, for the cached data, when the preset flush conditions are met, the cached data is written in batches sequentially to the physical storage space of the target file block, and the offset information of each file in the target file block is recorded synchronously to form the index information of the target file block. Based on the index information of the target file blocks, an index structure for recording file position relationships is constructed and persistently stored, forming a persistently stored index structure; Upon receiving a file read request, the file ID contained in the request is parsed to obtain the corresponding target virtual domain, target file block, and target file sequence number. Based on the persistent storage index structure, the offset position of the target file sequence number in the target file block is determined, and the corresponding file data is read from the target file block accordingly.

[0007] Furthermore, the hard disk storage space is divided into multiple file blocks of the same size, resulting in multiple file blocks that are initially unused, including: Based on the preset file block size parameters, a space partitioning operation is performed on the physical storage space of the hard disk to form multiple storage units with the same logical capacity. For each storage unit with the same logical capacity obtained from the partitioning, a corresponding metadata control structure is created. The metadata control structure is used to record at least the physical location, logical capacity and current usage status of the storage unit. Each storage unit with the same logical capacity that has a metadata control structure is defined as a file block, and its current usage state is uniformly initialized to an unused state, thus obtaining multiple file blocks with an initial state of unused.

[0008] Furthermore, based on the multiple file blocks initially in an unused state, multiple virtual storage domains are created and managed, denoted as virtual domains; the multiple file blocks initially in an unused state are allocated to different virtual domains for management, including: Based on the business attributes and storage requirements of the files, multiple virtual storage domain structures are created, and a unique virtual domain identifier is assigned to each virtual storage domain structure to obtain the initial virtual domain structure. Based on the initially established virtual domain structure, configure corresponding file management attributes for each virtual domain; the file management attributes are at least used to define the business types and storage time requirements of files that are allowed to be stored in the virtual domain. Based on the file management attributes configured for each virtual domain, one or more file blocks are dynamically allocated to each virtual domain from multiple file blocks that are initially in an unused state, in order to establish a binding relationship between the virtual domain and the allocated file blocks; Based on the binding relationship between the established virtual domains and the allocated file blocks, a global virtual domain and file block mapping management table is constructed and maintained to record the identifiers and status information of all file blocks managed by each virtual domain, thereby completing the creation and resource initialization management of virtual storage domains.

[0009] Furthermore, based on the established virtual domain, upon receiving a file to be written, the system matches the corresponding virtual domain according to the file's attributes, allocates a target file block for the file to be written within the corresponding virtual domain, and assigns a unique file sequence number to the file to be written, including: Receive the file to be written and extract its attribute information; the attribute information includes at least the business type, storage time limit requirements, and spatial coordinates. Based on the extracted attribute information, and using the established and maintained global virtual domain and file block mapping management table, the target virtual domain corresponding to the attribute information is matched. Based on the matched target virtual domain, query the current status information of all file blocks within the target virtual domain, and select a file block that is in a usable state as the target file block for writing the file to be written. In the target file block used to write the file to be written, a unique file number is assigned to the file to be written, which is sequentially incremented relative to the file numbers already stored in the target file block.

[0010] Furthermore, based on the allocated target file block and the unique file sequence number, a corresponding file ID is generated; simultaneously, the data to be written to the file is cached, and the file ID is returned to the client to complete the write response, including: Based on the allocated target file block and the unique file sequence number, and in accordance with the predefined file ID generation rules, a unique file ID is generated; The data to be written to the file is stored in the memory cache area corresponding to the target file block; After storing the data to be written to the file, the generated file ID is returned to the client that initiated the write request, thus completing the write response.

[0011] Furthermore, based on the completed write response, for the cached data, when the preset flush conditions are met, the cached data is sequentially written in batches to the physical storage space of the target file block, and the offset information of each file within the target file block is recorded synchronously to form the index information of the target file block, including: By monitoring the memory cache area corresponding to the target file block, it is determined whether the amount of cached data or the cache duration has reached a preset threshold; when the preset threshold is reached, the batch persistence process for the target file block is triggered. In response to the triggered batch persistence process, all currently cached file data in the memory cache area are written sequentially in batches as a continuous data block to the physical storage space corresponding to the target file block. During the batch sequential write process, the starting physical offset and file length of each file to be written in the target file block are recorded synchronously to obtain the offset information of each file to be written in the target file block. The offset information of all files recorded in the target file block is summarized and organized in the order of file number to form a structured index information that describes the physical location relationship of files within the target file block.

[0012] Furthermore, based on the index information of the target file blocks, an index structure for recording file positional relationships is constructed and persistently stored, forming a persistently stored index structure, including: Obtain the structured index information of the resulting target file blocks; Based on the obtained structured index information, construct an index structure that uniquely corresponds to the target file block; The constructed index structure, which uniquely corresponds to the target file block, is persistently saved to a non-volatile storage medium as an independent index file, thereby forming a persistent index structure for the target file block.

[0013] Furthermore, upon receiving a file read request, the file ID contained in the request is parsed to obtain the corresponding target virtual domain, target file block, and target file sequence number; based on the persistent storage index structure, the offset position of the target file sequence number within the target file block is determined, and the corresponding file data is read from the target file block accordingly, including: Receive a file read request and extract the file ID corresponding to the file to be read from the received file read request; By parsing the extracted file ID, the target virtual domain identifier, target file block identifier, and target file sequence number contained therein can be obtained; Based on the obtained target file block identifier, query the physical offset position corresponding to the target file sequence number from the formed and persistently stored index structure; Based on the queried and determined physical offset position, data of the corresponding length is directly read from the target file block to obtain the requested file data.

[0014] The above-described solution of the present invention has at least the following beneficial effects: By employing techniques such as file block storage partitioning, virtual domain classification management, memory caching + batch sequential disk flushing, persistent index structure, and file ID parsing and positioning, this solution effectively overcomes the core shortcomings of existing storage solutions that cannot simultaneously meet the requirements of sequential reading and writing of streaming files with low latency and high throughput performance, as well as the problems of frequent random I / O during file writing and inefficient positioning during reading. This enables sequential storage and access of streaming files, achieving the technical effect of low latency write response and high throughput data reading and writing, effectively adapting to the streaming file storage management needs of real-time application scenarios. Attached Figure Description

[0015] Figure 1 This is a flowchart illustrating a method for streaming storage management and persistence of files provided by an embodiment of the present invention. Detailed Implementation

[0016] Exemplary embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.

[0017] like Figure 1 As shown, embodiments of the present invention propose a method for streaming storage management and persistence of files, the method comprising the following steps: Step 1: Divide the hard disk storage space into multiple file blocks of the same size, resulting in multiple file blocks that are initially unused; Step 2: Based on the multiple file blocks that are initially unused, create and manage multiple virtual storage domains, denoted as virtual domains; allocate the multiple file blocks that are initially unused to different virtual domains for management. Step 3: Based on the established virtual domain, when a file to be written is received, the corresponding virtual domain is matched according to the attributes of the file to be written, and a target file block is allocated for the file to be written in the corresponding virtual domain, and a unique file sequence number is assigned to the file to be written. Step 4: Based on the allocated target file block and the unique file sequence number, generate the corresponding file ID; at the same time, cache the data to be written to the file and return the file ID to the client to complete the write response; Step 5: Based on the completion of the write response, for the cached data, when the preset flush conditions are met, the cached data is written in batches sequentially to the physical storage space of the target file block, and the offset information of each file in the target file block is recorded synchronously to form the index information of the target file block. Step 6: Based on the index information of the target file blocks, construct an index structure to record file position relationships and persist it to form a persistent index structure. Step 7: When a file read request is received, parse the file ID contained in the request to obtain the corresponding target virtual domain, target file block and target file sequence number; determine the offset position of the target file sequence number in the target file block according to the persistent storage index structure, and read the corresponding file data from the target file block accordingly.

[0018] In this embodiment of the invention, standardized file block partitioning and virtual domain classification management enable streaming files to be accurately matched and stored according to attributes; unique file sequence number allocation combined with instant response of cached writes ensures low latency and sequentiality of writes; batch sequential disk flushing effectively reduces the impact of random I / O and significantly improves storage write efficiency; persistent index structure and file ID parsing and positioning enable fast locking of file reads; the overall process adapts to the continuous characteristics of streaming files, making storage management more efficient and read / write performance more stable and reliable.

[0019] In a preferred embodiment of the present invention, step 1 above may include: Step 1.1: Based on the preset file block size parameters, perform space partitioning on the physical storage space of the hard drive to form multiple storage units with the same logical capacity. Specifically, this includes: considering the core characteristics of streaming files arriving continuously and requiring batch sequential read / write operations, as well as the storage requirements of low latency and high throughput, first, based on the business scenario of streaming files (such as the typical size of surveillance images and AI training samples) and the characteristics of storage devices, SSDs are adapted for hot data, and HDDs are adapted for cold data, with a preset unified logical capacity. This capacity is set according to the principle of balancing read / write efficiency and resource utilization, with MB-level sizes adapted for frequently accessed hot data and larger sizes adapted for infrequently accessed cold data. Subsequently, perform initialization testing on the target hard drive, comprehensively scan the entire physical space of the hard drive, identify and mark unusable areas, such as bad sectors, occupied system partitions, and reserved maintenance space, and retain only continuous and writable valid physical space as the partitioning object. To avoid physical damage or space overlap issues in subsequent storage units, the space partitioning process is initiated using physical sectors of the hard drive as the smallest unit, following a linear traversal order from the starting sector to the ending sector. During the traversal, a preset uniform logical capacity is used as a fixed benchmark to segment effective physical space. Starting from the first available sector of the hard drive, the number of sectors is accumulated. When the physical capacity corresponding to the accumulated sector is exactly equal to the preset uniform logical capacity, the starting and ending sectors of that segment are immediately marked as the boundary of a storage unit. Then, the traversal continues from the ending sector as the new starting point, repeating the boundary marking operation until all available effective physical space has been traversed. Finally, each storage unit formed after partitioning has the same logical capacity, and the storage units are in a continuous and seamless state in physical space, providing a solid physical foundation for subsequent batch sequential writing of streaming files and reducing random I / O.

[0020] Step 1.2: Create a corresponding metadata control structure for each storage unit with the same logical capacity obtained after partitioning. The metadata control structure is used to record at least the physical location, logical capacity, and current usage status of the storage unit. Specifically, for each storage unit with the same logical capacity obtained after partitioning, an independent and unique metadata control structure is created for it in the SSD high-speed non-volatile storage medium. At the same time, a unique identifier ID is assigned to this structure, which is generated by concatenating the unique identifier of the hard disk and the starting sector number of the storage unit to ensure global uniqueness across hard disks and storage units, and to achieve precise binding of metadata to the corresponding storage unit. The recording of physical location information is achieved through the LBA (Logical Block Address Translation) interface provided by the control terminal: first, obtain the physical addresses of the starting and ending sectors of the storage unit, and then map them to standard LBA addresses through the interface. Simultaneously record the LBA start value, LBA length, and the mapping relationship table between physical address and LBA to ensure that the physical sector can be quickly located through LBA in the future.

[0021] In addition to specifying the preset standard value and unit, the logical capacity information also performs reverse verification: it reads the actual writable capacity of the storage unit and compares it with the specified value. If there is a discrepancy, the storage unit is marked as unavailable to prevent subsequent data overflow. The current usage status information is recorded through a 3-bit binary status code field, initially uniformly set to 00 to represent the pending allocation status. At the same time, it reserves extended code bits such as 01 (unused), 10 (in use), and 11 (full) to adapt to the subsequent full lifecycle status management of file blocks (chunks). Finally, a global metadata index table is created, which sorts the storage addresses of all metadata control structures in ascending order by file block (chunk) identifier ID. The index table is cached in memory to ensure that subsequent metadata queries do not require traversing the SSD, further reducing access latency and meeting the needs of high-concurrency management of streaming files.

[0022] Step 1.3 defines each storage unit with the same logical capacity that has a created metadata control structure as a file block (chunk), and initializes its current usage state to unused, thus obtaining multiple file blocks (chunks) with an initial unused state. Specifically, this involves: first, formally defining each storage unit with the same logical capacity that has had its dedicated metadata control structure created and verified as a file block (chunk); establishing a permanent binding relationship between the file block (chunk) and the dedicated identifier ID of the metadata control structure, which is synchronously written to the metadata index table to ensure global traceability; then, using an atomic lock mechanism to perform batch state update operations, atomically modifying the current usage state field of the metadata control structure corresponding to each file block (chunk), uniformly updating the status code from 00 pending allocation to 01 unused, avoiding state conflicts caused by multi-process concurrent operations, and ensuring state consistency; finally, all unused file blocks (chunks) are registered to the global file block (... In the chunk resource management list, the list is stored in partitions according to storage device type (SSD or HDD). Each partition is further categorized by logical capacity of file blocks. In addition to recording the unique identifier ID, metadata control structure storage address, logical capacity, and current status of each file block, the list also includes a free space statistics field to synchronize the available storage capacity of file blocks in real time. Finally, an initialization self-check is performed on each registered file block: test data is written to the file block and read for verification to confirm the read and write availability of the storage unit. At the same time, the integrity and consistency of all fields of metadata are verified. If an anomaly is detected, such as read / write failure or missing metadata fields, the status code of the file block is updated to 111 unavailable and it is removed from the resource pool. This ensures that all file blocks in the final basic storage resource pool have stable read and write capabilities, providing reliable support for subsequent on-demand allocation of virtual domains (Vzones) and sequential storage of streaming files.

[0023] In this embodiment of the invention, by dividing the hard disk space according to a preset standard to form storage units with the same logical capacity, the physical space fragmentation obstacle is removed for batch sequential writing of streaming files; a metadata control structure is created for each storage unit to record the physical location, logical capacity and usage status, so as to realize the rapid location and accurate management of storage units; the storage units that adapt to the metadata are defined as file blocks (chunks) and uniformly initialized to an unused state, thus constructing a regular and controllable basic storage resource pool. The three work together to lay a solid foundation for the classification and allocation of virtual domains (Vzones), low-latency file writing and high-throughput reading, effectively meeting the core requirements of continuous storage and efficient management of streaming files, and improving the stability and adaptability of the overall storage system.

[0024] In a preferred embodiment of the present invention, step 2 above may include: Step 2.1: Based on the business attributes and storage requirements of the files, create multiple virtual storage domain structures and assign a unique virtual domain (Vzone) identifier to each virtual storage domain structure. The initial establishment of the virtual domain (Vzone) structure includes: first, performing full-dimensional feature decomposition and quantification on all streaming files to be stored: business attributes are subdivided by monitoring, into subcategories such as vehicles, pedestrians, face images, and AI training samples, each assigned a 1-2 digit numeric code; in storage requirements, retention time is quantified into specific values ​​in days, such as 7 days, 30 days, and 90 days; access frequency is divided into high, medium, and low levels based on the average daily access count, corresponding to quantization values ​​of 1, 2, and 3; IO priority is assigned weights of 1 to 5 levels based on real-time requirements; spatial coordinates are converted into latitude and longitude combinations, and then mapped to storage node area codes, such as codes based on city administrative districts; these quantified dimensional values ​​are integrated into standardized multi-dimensional feature points, with the dimension order fixed as business code, retention time, access frequency, IO priority, and area code, forming a global feature point set.

[0025] The optimized Graham scan method is used to execute the convex hull algorithm: The first step is to carry out refined preprocessing. First, an outlier judgment system is established for each business. The retention time of monitoring files is reasonably defined as 1 to 180 days, and that of AI training samples is 7 to 365 days. The regional code must completely match the preset storage node code dictionary. The dictionary covers all legal administrative region codes and their corresponding storage node mapping relationships. Feature points that exceed the corresponding retention time range or have no matching items in the code dictionary are directly marked as outliers and removed. Next, full-dimensional deduplication verification is performed. Each pair of feature points is compared. When the values ​​of business code, retention time, access frequency, IO priority, and regional code are completely consistent, only the first feature point that appears is retained, and subsequent duplicate points are deleted. Finally, standardization and normalization processing is performed. The min-max normalization method is used to map the values ​​of each dimension to the interval between 0 and 1, eliminating the interference of different dimension scale differences on the accuracy of convex hull calculation, and obtaining a purified set of effective feature points.

[0026] The second step involves sorting feature points and determining the initial convex hull vertices. First, a fixed business priority dictionary is established, where monitoring class codes have higher priority than AI training sample classes. Within the same business category, vehicle image codes have higher priority than pedestrian and face image codes. Feature points are then sorted according to this dictionary. Within the same business code, a secondary sort is performed by ascending regional code values. After sorting, the feature point with the smallest retention time dimension value is selected as the initial candidate vertex. If multiple points have the smallest retention time value, the business code values ​​are further compared, and the point with the smallest code is selected as the unique initial convex hull vertex. Then, using the initial vertex as the origin and the horizontal direction to the right as the reference direction, the polar angles of all other feature points relative to the origin are calculated. When calculating the polar angles, the vertical height and horizontal length of the line segment formed by the feature point and the origin are first determined. The angle between this line segment and the reference direction is determined through their proportional relationship, thus obtaining the polar angle value for each feature point. The feature points are sorted three times from smallest to largest polar angle value. If feature points have the same polar angle, the straight-line distance between these points and the origin is calculated, and only the point with the farthest distance is retained. Finally, an ordered sequence of feature points to be iterated is determined.

[0027] The third step iteratively filters convex hull vertices and generates the convex hull region. It initializes the convex hull vertex stack, pushing initial vertices onto the stack. Then, it sequentially selects feature points from the iteration sequence as the current point. For each selected current point, it pops the last two vertices from the vertex stack, denoted as the previous and middle vertices, constructing two vectors: vector one pointing from the middle vertex to the previous vertex, and vector two pointing from the middle vertex to the current point. Next, it solves for the positional relationship between the two vectors. The cross product is calculated using the corresponding dimension values. If the result is greater than 0, it means the current point is in the counter-clockwise direction of vector one, i.e., outside the convex hull, and the current point is pushed onto the vertex stack. If the result is less than 0, it means the current point is inside the convex hull, and the current point is discarded. The process continues by popping the last vertex from the vertex stack and recalculating the vertices from the stack. If the result is 0, it means that the three points are collinear. The distance from the current point to the previous vertex to the initial vertex is calculated, and only the points that are farther away are kept, while the points that are closer are removed. This process is iterated until all feature points to be iterated are processed. The vertices in the stack are connected in sequence to form a closed convex hull boundary. Based on this boundary, multiple non-overlapping minimum convex hull regions are divided. The consistency of feature points in the region is further verified, and the standard deviation of the values ​​in each dimension is calculated to ensure that the difference between feature points in the region in terms of business type, storage timeliness, spatial distribution, etc. is less than the preset threshold of 15%, while ensuring that adjacent convex hull regions do not overlap or have gaps.

[0028] Based on the core features of each convex hull region, the median of each dimension of all feature points within the region is selected as the core value to create a corresponding virtual storage domain structure. A unique virtual domain (Vzone) identifier is assigned to each structure. The identifier consists of 6 characters: the first 2 characters are the sequential number of the convex hull region, the middle 2 characters are the core business attribute code, and the last 2 characters are the core storage timeliness level code, which is divided according to the median retention time to ensure global uniqueness. Finally, boundary verification is performed on the convex hull region corresponding to each virtual domain (Vzone), and the coverage rate of feature points within the region is calculated. It must reach 95% or above. If it does not meet the standard, the convex hull vertices are readjusted, and finally, the initially established virtual domain (Vzone) structure is obtained.

[0029] Step 2.2: Based on the initially established virtual zone (Vzone) structure, configure corresponding file management attributes for each virtual zone (Vzone). The file management attributes are used to define at least the business types and storage time requirements of files allowed to be stored in the virtual zone (Vzone). Specifically, this includes: firstly, extracting the full-dimensional boundary thresholds of the convex hull region corresponding to each virtual zone (Vzone): the business type value range is the minimum and maximum values ​​of the business codes of all feature points in the region, including the boundaries; the upper and lower limits of storage time are the minimum and maximum quantized values ​​of retention time; the access frequency threshold is the range of quantized access frequency values ​​in the region; the spatial coordinate range is the minimum to maximum combination of latitude and longitude value pairs; and the IO priority threshold is the range of weights in the region.

[0030] These boundary thresholds are transformed into structured file management attributes. The attribute set includes basic rules and verification logic: the basic rules clearly define the whitelist of allowed storage business types, all subclasses within the corresponding business code range, the longest and shortest retention periods for storage validity, the effective coverage range of spatial coordinates, and the IO priority adaptation range; the verification logic adopts a phased progressive mechanism. The first stage verifies whether the business code is within the value range, and if it fails, it is directly determined to be a mismatch; the second stage verifies whether the storage validity falls between the upper and lower limits, and if it fails, the match is rejected; the third stage verifies whether the spatial coordinates are within the valid range, and if it passes, it is finally determined that the file can be accessed to the virtual domain (Vzone).

[0031] Simultaneously, a dynamic calibration mechanism for file management attributes is established: every 7 days, the actual feature points of the files stored in the corresponding virtual domain (Vzone) are collected, the convex hull algorithm is re-executed to calculate the new boundary threshold, and compared with the original attribute threshold. If the difference exceeds 10%, the boundary parameters in the file management attributes are automatically updated according to the average deviation of the thresholds of each dimension, ensuring that the virtual domain (Vzone) always adapts to the changes in file characteristics and guarantees the high consistency of file characteristics within the virtual domain (Vzone).

[0032] Step 2.3: Based on the file management attributes configured for each virtual zone (Vzone), dynamically allocate one or more file blocks (chunks) from the multiple initially unused file blocks (chunks) to each Vzone to establish a binding relationship between the Vzone and the allocated file blocks (chunks). Specifically, this includes: performing full-dimensional quantization of resource characteristics for file blocks (chunks) with an initial global unused state: assigning quantization values ​​of 1 and 0 to storage media type (SSD and HDD respectively); converting logical capacity to corresponding values ​​according to preset standards, such as 100GB and 200GB; combining physical location with rack number and slot number of storage node into numerical pairs; taking the average rated continuous read / write throughput of the storage media model; and calculating the free space ratio as the ratio of current free capacity to logical capacity, rounded to two decimal places. Finally, a multi-dimensional resource feature point is formed for each file block (chunk), with the dimension order fixed as media type, logical capacity, physical location, IO throughput, and free space ratio.

[0033] For each virtual zone (Vzone), the boundary thresholds of its file management attributes are first converted into quantifiable resource requirements: I / O throughput requirements of file blocks (chunks) are matched according to I / O priority thresholds, with priority levels 1 and 2 corresponding to throughput of no less than 1GB / s, and levels 3 to 5 corresponding to throughput of no less than 500MB / s; storage media types are matched according to business access frequency, with high frequency corresponding to SSD (quantization value 1) and medium-low frequency corresponding to HDD (quantization value 0); the total required logical capacity is calculated based on the average file size and estimated storage volume. These quantifiable requirements are then integrated into resource requirement feature points for the Vzone, and the Andrew algorithm is used to construct the demand convex hull: The first step expands the resource requirement feature points into demand intervals, for example, the I / O throughput requirement of no less than 1GB / s is expanded to 1GB / s up to the maximum throughput of the media, mapping to a set of demand feature intervals; the second step projects the resource feature points of all unused file blocks (chunks) onto this demand interval set, constructing a minimum convex hull resource set that completely covers the demand interval, where all file blocks (chunks) satisfy the basic resource requirements of the Vzone.

[0034] Candidate file chunks are selected from the convex hull resource set: chunks with less than 30% free space are removed, and then sorted according to the following weighting rules: resource utilization (40%), physical location (30%), and IO throughput (30%). Resource utilization is calculated by subtracting the free space percentage from 1. Physical location is determined by the network latency between the storage node where the file chunk is located and the file acquisition node; lower latency results in a higher score. IO throughput is calculated as the ratio of the actual value to the required threshold. Based on the estimated storage capacity of the virtual domain (Vzone), a suitable number of chunks are selected from the sorted candidate file chunk set. File chunks are allocated at 1.2 times the estimated amount for high-frequency, long-duration virtual zones (Vzones) and at 1.0 times the estimated amount for low-frequency, short-duration virtual zones (Vzones). During allocation, an atomic lock mechanism is used to mark the allocation status of file chunks to avoid concurrent conflicts. After allocation, the file chunk status is updated to "allocated to a virtual zone (Vzone)," and the effective time of the binding relationship and the upper limit of resource consumption are recorded synchronously, not exceeding 90% of the logical capacity of the file chunk. A unique and exclusive binding relationship is established between the virtual zone (Vzone) and the allocated file chunk.

[0035] Step 2.4: Based on the established binding relationship between virtual zones and allocated file blocks, construct and maintain a global virtual zone to file block mapping management table to record the identifiers and status information of all file blocks managed by each virtual zone, completing the creation and resource initialization management of virtual storage zones. Specifically, this includes: constructing a multi-field structured global virtual zone to file block mapping management table with the binding relationship between virtual zones and file blocks as the core; the basic fields include the unique identifier of the virtual zone. The system stores the identification, convex hull region number, and convex hull full-dimensional boundary threshold in JSON format; the file block (chunk) association field includes a list of unique identifiers for the bound file blocks (chunks), the current status of each file block (chunk) (divided into five states: unused, allocated, in use, full, and faulty), allocation time, remaining free space, storage medium type, and real-time IO throughput value; the index field includes a primary index built by virtual domain (Vzone) identifier and an inverted index built by file block (chunk) identifier; the management table is stored on a dual storage medium of distributed cache and persistent database, with the cache used for high-frequency queries and the database used for data persistence to ensure that data is not lost.

[0036] Develop refined dynamic maintenance rules: Three conditions triggering updates are defined: First, the remaining free space of a file block (chunk) is less than 10% of its logical capacity. The threshold is adjusted according to the storage medium type of the file block (chunk): 8% for SSD and 12% for HDD. Second, the feature point coverage of the corresponding convex hull region of the virtual domain (Vzone) is less than 90%. Third, the average access frequency of files within the virtual domain (Vzone) changes by more than 30% over three consecutive days. When any condition is triggered, the resource reallocation process is initiated: First, unbind file blocks (chunks) that do not conform to the current convex hull resource set, such as storage medium mismatch or insufficient IO throughput. Before unbinding, verify whether the files within the file block (chunk) have been migrated to other file blocks (chunks) in the same virtual domain (Vzone). After migration, mark them as unused and return them to the global resource pool. Second, re-collect the current file feature points of the virtual domain (Vzone) and execute the convex hull algorithm to update the convex hull region boundary and resource requirements. Third, follow the process in step 2.3 to filter new candidate file block (chunk) sets and complete the allocation, synchronously updating the binding relationships.

[0037] Establish a consistency guarantee and query optimization mechanism for the management table: In a distributed scenario, a two-phase commit protocol is adopted to update the management table, ensuring data consistency across all storage nodes; a composite index is created for virtual zone (Vzone) identifiers, file block (chunk) identifiers, and convex hull region numbers to optimize query performance, keeping the latency for querying file block (chunk) information within a single virtual zone to the millisecond level; simultaneously, logs are recorded for each update, including update time, triggering conditions, and changed content, supporting data backtracking and troubleshooting, ultimately completing the creation and resource initialization management of virtual storage zones, providing reliable support for rapid matching and resource scheduling during subsequent file writes.

[0038] In this embodiment of the invention, by creating virtual domains (Vzones) based on file business attributes and storage requirements and assigning unique identifiers, the classification and aggregation management of streaming files is realized. By configuring file management attributes that define business types and storage timeliness for each virtual domain (Vzone), the high homogeneity of file characteristics within the virtual domain (Vzone) is ensured, laying the foundation for accurate file matching. By dynamically allocating unused file chunks according to management attributes and establishing binding relationships, on-demand scheduling of storage resources is realized, improving resource utilization and avoiding resource idleness or waste. By constructing and maintaining a global virtual domain (Vzone) and file chunk (chunk) mapping management table, the visual management and fast query of the binding relationship between virtual domains (Vzones) and file chunks (chunks) are realized, supporting efficient resource matching during subsequent file writing. The overall process works synergistically, providing orderly and controllable resource management support for low-latency, high-throughput storage of streaming files, effectively adapting to the storage needs of real-time scenarios.

[0039] In a preferred embodiment of the present invention, step 3 above may include: Step 3.1: Receive the file to be written and extract its attribute information. This attribute information includes at least the business type, storage timeliness requirements, and spatial coordinates. Specifically, it involves receiving files from different business terminals such as security monitoring terminals and AI training platforms according to a preset streaming data transmission protocol. This preset streaming data transmission protocol is pre-defined based on the data characteristics and transmission requirements of each business terminal, determining the data packet format, field encapsulation rules, and verification standards for file transmission. During the receiving process, data integrity verification is performed synchronously. A dual verification rule is set here: a 128-bit MD5 checksum embedded in the file header is used to verify the integrity of the main file data, and a 32-bit CRC cyclic redundancy checksum appended to the transmission tail is used to verify the continuity of the transmission process. The actual data integrity is calculated by comparing the received file with the data integrity information. The consistency between the actual verification code and the verification code at the preset location ensures that the file data has not been lost or tampered with during transmission. After the verification is successful, the metadata block at a fixed offset position in the file is located. This fixed offset position is uniformly set, and the start and end byte offset range of the metadata block in the file is determined in advance. At the same time, the fixed retrieval order of the business identification field, the timeliness requirement field, and the spatial coordinate field is preset. When extracting the business type, it matches the preset business type encoding dictionary. This dictionary is established after sorting out all the supported business categories and subcategories in advance. It contains the unique code and text description mapping relationship of each category. Based on this, specific types such as security monitoring - vehicle images and AI training samples - image type are parsed out. At the same time, it is verified whether the type is in the preset business whitelist. The whitelist is a set of compliant business types that have been screened in advance.

[0040] When extracting storage timeliness requirements, the timeliness field in the metadata is converted into a standard retention days value. If the field is a relative time expression, it is automatically converted to an absolute number of days. At the same time, it is verified whether the value is within a preset reasonable range of 1 to 365 days. This range is pre-defined based on business storage needs and resource carrying capacity. When extracting spatial coordinates, latitude and longitude value pairs are read and redundant format characters are removed. Then, they are matched with a set storage node area code dictionary. This dictionary is established in advance based on geographic zoning planning and contains a one-to-one mapping relationship between latitude and longitude value ranges and corresponding administrative region codes. Latitude and longitude are mapped to the corresponding administrative region codes. If the mapping fails, it is marked as a spatial attribute anomaly. Finally, the extracted business type, retention days, and area code are integrated into a standardized attribute set. The format of the standardized attribute set is preset, which determines a fixed field order and a unified data format. At the same time, attribute integrity verification is performed. If any core field is missing, a retransmission mechanism is triggered. The retransmission mechanism specifies the frequency of retransmission requests and the timeout threshold. If retransmission fails, subsequent processing is rejected.

[0041] Step 3.2: Based on the extracted attribute information, and using the established and maintained global virtual domain (Vzone) and file block (chunk) mapping management table, match the target virtual domain (Vzone) corresponding to the attribute information. Specifically, this includes: retrieving the basic information partition of the virtual domain (Vzone) in the global virtual domain (Vzone) and file block (chunk) mapping management table, obtaining the unique identifier of all virtual domains (Vzone) and their corresponding file management attributes, where the file management attributes include the full-dimensional boundary threshold determined based on the convex hull algorithm. A three-level progressive matching mechanism is initiated. The first level performs precise matching of business types, comparing the business category and subcategories of the files to be written with the business whitelist in the file management attributes of each virtual domain (Vzone), and retaining only the virtual domains (Vzones) whose whitelists contain the specific subcategory as the candidate set. The second level performs storage validity period matching, comparing the standard retention days of the files with the upper and lower validity period thresholds of each virtual domain (Vzone) in the candidate set, and eliminating virtual domains (Vzones) whose retention days exceed the threshold range. If the file retention days are equal to the threshold boundary, it is considered a successful match. The third level performs precise spatial coordinate adaptation, comparing the region code corresponding to the file with the spatial coverage code list of the virtual domains (Vzones) in the candidate set, and verifying whether the latitude and longitude values ​​fall within the spatial coordinate boundaries of the convex hull region of the corresponding virtual domain (Vzone). If both conditions are met, it is retained as the final candidate virtual domain (Vzone). If multiple candidate virtual domains (Vzones) exist, a priority ranking mechanism is activated. A comprehensive score is calculated based on "60% weight for business compatibility and 40% weight for spatial distance." Business compatibility is scored according to the accuracy of subclass matching, and spatial distance is calculated based on the network latency between the file acquisition node and the storage node to which the virtual domain (Vzone) belongs. The virtual domain (Vzone) with the highest comprehensive score is selected as the target virtual domain (Vzone). If no candidate virtual domain (Vzone) matches, an exception alarm is triggered, and information about file attribute mismatch is reported.

[0042] Step 3.3: Based on the matched target virtual domain (Vzone), query the current status information of all file blocks (chunks) within the target Vzone, and select a file block (chunk) in a usable state as the target file block (chunk) for writing the file to be written. Specifically, this includes: based on the unique identifier of the target Vzone, retrieving the complete information of all file blocks (chunks) bound to the Vzone from the file block (chunk) association partition of the global Vzone-to-file block (chunk) mapping management table, including file block (chunk) identifier, current status, remaining free space, storage media type, physical location, last write time, etc.; first, perform status filtering according to the preset status judgment rule: this rule is based on the division of the entire life cycle stage of the file block (chunk), determining that the file block (chunk) in use is bound to a virtual domain (Vzone) and has remaining free space and no fault alarms. Unused indicates that the file chunk has been allocated to the virtual domain (Vzone) but has not yet been written to a file and is in a normal state; Full indicates that the remaining free space is less than 1%; Fault indicates that a read / write error or hardware malfunction has been detected; Migrating indicates that the file chunk is in the data migration process. Based on this, file chunks marked as used or unused in the status field are included in the usable candidate set, while file chunks that are full, faulty, or migrating are removed. A secondary screening is performed on the file chunks in the candidate set. First, the file size information is extracted from the standardized attribute set and metadata of the file to be written, which is used as the basic estimated size. Then, the minimum free space threshold is calculated by multiplying the estimated size by 1.2: 20% space redundancy is reserved to cope with data expansion during the file writing process or subsequent append writing needs. Then, the remaining free space of each file chunk in the candidate set is compared with the threshold, and file chunks with remaining free space below the threshold are removed to avoid insufficient space during subsequent writing.

[0043] Subsequently, a multi-dimensional sorting mechanism is initiated. The first priority is descending order of remaining free space, ensuring that file chunks with sufficient space are selected first to reduce frequent switching. The second priority is storage media type: SSD file chunks are preferred for virtual zones corresponding to frequently accessed business files, while HDD file chunks are preferred for infrequently accessed files. The third priority is physical proximity, sorted in ascending order by network latency between the storage node to which the file chunk belongs and the file acquisition node, with lower latency ranking higher. Based on the sorting results, the first file chunk is selected as the target file chunk, and an atomic lock operation is performed to temporarily update the status of the file chunk to "allocation in progress." The lock duration is set to a preset file write preparation duration: this duration is based on the time consumption statistics of historical file write preparation processes, with a default setting of 5 seconds. It also supports dynamic fine-tuning according to the file processing complexity of different business types, ensuring that it can cover the entire process of preparation operations such as file sequence allocation and metadata synchronization, avoiding resource contention due to locks that are too short or resource idleness due to locks that are too long, and avoiding resource contention when multiple files are written concurrently.

[0044] Step 3.4: In the target file chunk used for writing the file to be written, allocate a unique file sequence number for the file to be written, which is sequentially incremented relative to the file sequence numbers already stored in the target file chunk. Specifically, this includes: retrieving the metadata control structure corresponding to the target file chunk. This metadata adopts a dual backup mechanism of caching and persistent storage. The sequence number management field is in structured storage form, containing two core subfields: an ordered list of valid sequence numbers and a cached item with the current maximum sequence number. The ordered list of valid sequence numbers is arranged in ascending order of values ​​and records all allocated sequence numbers for which files have been successfully written. The cached item with the current maximum sequence number is synchronized in real time. The maximum value in the list; Initiate the sequence number query and generation process: First, read the ordered list of valid sequence numbers, and according to the set invalid sequence number judgment rules, remove invalid sequence numbers marked as deleted or failed to write. If the list is empty, it is directly determined that there are no valid sequence numbers. Then, compare the maximum value of the valid sequence number list with the maximum sequence number cache item, and take the larger value as the baseline maximum sequence number. On this basis, perform an increment operation to generate a new sequence number. The new sequence number adopts a standardized format, consisting of the first 8 bits of the unique identifier of the file block (chunk) + 6 incrementing digits. If the 6 incrementing digits are less than 6 digits, zeros are added in front to ensure that the sequence number length is uniform and can be traced back to the file block (chunk).

[0045] After generating a new sequence number, a dual uniqueness check is initiated: the first check verifies the list of valid sequence numbers in the target file block (chunk) metadata to confirm that the new sequence number does not exist in the list; the second check verifies the associated record of the file block (chunk) in the global virtual domain (Vzone) and file block (chunk) mapping management table to ensure no duplicate allocation; after the verification passes, a three-position synchronous write is performed in the order of caching first and then persistence: first, the new sequence number is written to the extended field of the metadata of the file to be written and the file checksum is updated; then, the sequence number is added to the ordered list of valid sequence numbers, and the maximum sequence number cache entry is updated; finally, the most recently allocated sequence number field of the corresponding file block (chunk) in the global mapping management table is updated synchronously; the synchronization process adopts a two-phase commit mechanism. The first phase completes the writing of all cache layers and records logs, and the second phase completes the persistent storage writing. If any step fails, a rollback operation is triggered, the new sequence number is marked as invalid and removed from the written position to avoid sequence number confusion.

[0046] After synchronization is complete, the target file block (chunk) status update process is triggered: First, the atomic lock status during allocation is released and updated to "in use"; then, the operation log field of the metadata is supplemented with detailed information about this sequence number allocation, including the new sequence number, allocation time, and the associated identifier of the file to be written; finally, the sequence number continuity of the target file block (chunk) is checked. If an interval is detected in the list of valid sequence numbers, that is, there are unused intermediate sequence numbers, the interval sequence number is recorded in the sequence number pool to be reused, providing a sequence number reuse alternative for subsequent file writing, ensuring that the file sequence numbers in the target file block (chunk) maintain sequential increment and maximize the utilization of sequence number resources, laying an orderly foundation for subsequent fast file indexing and positioning.

[0047] In this embodiment of the invention, by accurately receiving the file to be written and extracting core attribute information including business type, storage timeliness requirements, and spatial coordinates, an accurate data foundation is laid for file classification and matching. Through an attribute matching mechanism based on a global virtual domain (Vzone) and file block (chunk) mapping management table, precise correspondence between files and target Vzones is achieved, ensuring the classification and aggregation of file storage. By filtering usable file blocks (chunks) within the target Vzone, the occupation of invalid resources such as faults and full blocks is avoided, and the stability of the writing process is ensured through space threshold settings, preventing space shortages. By assigning a unique, sequentially increasing sequence number to each file within the target file block, ordered file storage management is achieved, providing reliable support for fast file indexing, querying, and tracing. The entire process is interconnected, effectively improving the efficiency and accuracy of streaming file writing and optimizing the utilization of storage resources.

[0048] In a preferred embodiment of the present invention, step 4 above may include: Step 4.1: Based on the allocated target file chunk and the unique file sequence number, and in accordance with predefined file ID generation rules, a unique file ID is generated. Specifically, this includes retrieving the 6-digit unique identifier of the target virtual domain (Vzone), the 8-digit unique identifier of the target file chunk, and the allocated 6-digit file sequence number. A globally unique file ID is then generated using the predefined file ID generation rules. These rules specify that the file ID is composed of four fixed-length fields concatenated sequentially: the Vzone identifier, the file chunk identifier, the file sequence number, and a 13-digit millisecond-level timestamp. Hyphens are used as separators between the fields. The timestamp is taken from the system's real-time time after the sequence number allocation, ensuring that IDs generated within the same millisecond are unique due to different sequence numbers. Immediately after generating the file ID, a dual uniqueness verification mechanism is initiated. First, the file ID cache pool for the most recent 24 hours is queried from memory. This cache pool uses a hash table structure, with the file ID string as the key and the ID's generation status and associated information as the value. This is achieved by calculating the uniqueness of the newly generated file ID. The hash value of the file ID is used to quickly locate the corresponding hash bucket. The hash value is compared to see if there are any duplicate keys in the hash bucket to determine if the ID is duplicated, achieving millisecond-level fast verification. Then, a second verification is performed by querying the full file ID history pool in the persistent database. This record pool stores all historical file IDs in time shards. First, the millisecond-level timestamp field in the new file ID is extracted to locate the data table of the corresponding time shard. Then, a file ID exact match query is performed to confirm that there are no duplicate historical records. After both verifications pass, a structured association is established between the file ID and the target virtual domain (Vzone) identifier, the target file chunk (chunk) identifier, the file sequence number, and the file attribute information. A two-phase commit mechanism is used to synchronously write the metadata control structure of the target file chunk (chunk) and the corresponding record entries in the global virtual domain (Vzone) and file chunk (chunk) mapping management table. The first phase completes the cache layer writing and records the pre-commit log. The second phase completes the persistent storage writing. If either phase fails, a rollback operation is triggered to ensure the consistency of the associated data.

[0049] Step 4.2: Store the data to be written to the file in the memory cache area corresponding to the target file chunk. Specifically, this includes: locating a dedicated memory cache area corresponding one-to-one with the target file chunk. This area is a fixed-size space pre-divided according to 10% of the logical capacity of the file chunk. A read-write permission isolation mechanism is adopted, where the write thread has exclusive write permissions, and the query thread only has read permissions. At the same time, a data overflow protection threshold is configured. When the cached data reaches 95% of the threshold, an asynchronous persistence instruction is automatically triggered. First, the metadata of the file to be written is read to determine the total file size. Then, the file data is split into several contiguous data blocks according to the set standard 4KB data block size. Data less than 4KB at the end is discarded. The blocks are zero-padding for alignment. During the splitting process, each data block is assigned a unique block number that increments sequentially. The data blocks are written to the memory cache area in the order of their block numbers. During writing, the CRC32 checksum of each data block is calculated synchronously and bound to the block number and stored in the metadata sub-area of ​​the cache area. After all data blocks have been written, the total number of data blocks is counted and compared with the total file size. At the same time, the cumulative value of the checksums of all data blocks is calculated and compared with the original total checksum of the file to confirm that no data blocks are lost, redundant, or tampered with. Then, the status of the memory cache area is marked as the data has been completely stored. At the same time, the cache data record information in the target file block (chunk) metadata is updated, including the cache write time, the total number of data blocks, the total checksum, and the block number range.

[0050] Step 4.3: After storing the data to be written to the file, return the generated file ID to the client that initiated the write request to complete the write response. Specifically, this includes: after confirming that the data in the memory cache area has been written completely, the verification has passed, and the status flag is correct, encapsulating the generated unique file ID according to a standardized response format. The encapsulated content is arranged in a fixed order as follows: a three-digit write success status code, a unique file ID, the data cache completion time, the target file block (chunk) storage medium type, and a response serial number. The write success status code is fixed at 200, and the response serial number consists of 8 random characters for subsequent response traceability. Then, select the corresponding communication link according to the type of client that initiated the write request. For security monitoring terminal clients, a TCP link is used for transmission to ensure the response is handled correctly. For reliability, AI training platform clients use UDP links for transmission to improve response efficiency in high-concurrency scenarios. Timeout thresholds are set during link transmission: 3 seconds for TCP links and 1 second for UDP links. The encapsulated response information is pushed to the client, and detailed logs of this write response are recorded in a distributed log system. The log content includes the client IP address, port number, client identifier, file ID, response time, communication link type, and file size. If the client does not respond with a receipt confirmation within the timeout threshold, a response retransmission mechanism is automatically triggered. The retransmission interval is 500 milliseconds, with a maximum of 2 retransmissions. If the number of retransmissions is exceeded, the response is marked as abnormal, and the reason for the abnormality is recorded. After the response is completed, the asynchronous persistence instruction for cached data is triggered synchronously, thus completing the complete write response process.

[0051] In this embodiment of the invention, by combining the target file chunk with a unique file sequence number and generating a globally unique file ID according to predefined rules, accurate file identification and global uniqueness control are achieved, effectively avoiding the problem of duplicate file identifiers and laying a solid foundation for rapid file retrieval, tracing, and management. By storing the file data to be written in the memory cache area corresponding to the target file chunk, the high-speed read and write characteristics of memory are leveraged to significantly improve file writing efficiency. At the same time, the dedicated configuration of the cache area also ensures the security and orderliness of data storage. By promptly returning the unique file ID to the client after data storage is completed, the write response is quickly completed. This allows the client to confirm the file writing status in real time and provides a core basis for subsequent file query and access operations. The entire process is efficiently connected, further optimizing the end-to-end performance and user interaction experience of streaming file storage.

[0052] In a preferred embodiment of the present invention, step 5 above may include: Step 5.1 involves monitoring the memory cache area corresponding to the target file chunk to determine whether the amount of cached data or the cache duration has reached a preset threshold. When the preset threshold is reached, a batch persistence process for the target file chunk is triggered. This process includes continuously monitoring the dedicated memory cache area bound to the target file chunk at a fixed sampling frequency. The monitoring dimensions include the real-time cached data volume, the cumulative duration of cached data, and the number of cached data write batches. The preset trigger threshold uses a dual threshold + fallback judgment rule: the core threshold is that the cached data volume reaches 80% of the total capacity of the cache area; the secondary threshold is that the cache duration reaches 30 minutes, starting from the first file written to the cache; and the fallback threshold is that the cache write batches reach 100 batches. Meeting any of these thresholds triggers the batch persistence process for the target file chunk.

[0053] Once the process is initiated, write permissions for new files in the cache area are immediately suspended. The lock identifier is synchronously updated to the global virtual domain (Vzone) and file block (chunk) mapping management table to avoid layout chaos caused by the insertion of new data during persistence. Subsequently, all core attribute information of all files to be persisted in the cache area is extracted, including file ID, business type sub-classification code, standardized spatial coordinates after latitude and longitude conversion, file size, and write timestamp. A structured feature dataset of files to be persisted is constructed according to the rule of file ID as the primary key. The dataset adopts a fixed format with row and column alignment. Files with missing attribute fields are marked as needing to be completed and are filtered first. Simultaneously, the pre-configured Hilbert space-filling curve data physical layout optimization algorithm is loaded to complete the algorithm initialization: First, the order of the Hilbert curve is dynamically determined according to the number of files to be persisted. When the number of files is ≤1024, a 10th-order curve is used, and when the number of files is 1024 < ≤4096, a 12th-order curve is used. Then, the business type sub-classification code is mapped to 1-dimensional numerical features (assigned values ​​from 1 to 20 according to the business priority dictionary), and the latitude and longitude coordinates are mapped to 2-dimensional spatial features (latitude and longitude are normalized to the numerical range of 0 to 1000 respectively), forming a 3-dimensional feature space. The core mapping parameters of the algorithm are determined, laying the foundation for the subsequent conversion of multi-dimensional features to one-dimensional sorting values.

[0054] Step 5.2: Respond to the triggered batch persistence process by batch-writing all currently cached file data in the memory cache area as a contiguous data block into the physical storage space corresponding to the target file block (chunk). Specifically, this includes: responding to the triggered batch persistence process by first performing standardized preprocessing on the feature dataset: calibrating the 3D feature vectors of each file—business type value, longitude normalized value, and latitude normalized value—according to preset dimensional weights, i.e., business type 40%, longitude 30%, and latitude 30%, to eliminate the influence of differences in dimensional units on the mapping results; then... The core implementation of the mapping process from multidimensional features of a dynamic Hilbert space-filling curve to one-dimensional sorting values ​​is as follows: First, based on the selected Hilbert curve order (10th order for files ≤ 1024, 12th order for files < 1024 < 4096), determine the binary encoding length of each dimension feature: 10th order corresponds to 10 bits, and 12th order corresponds to 12 bits. Then, perform binary encoding standardization on the calibrated 3D feature vector: business type values ​​(1 to 20) are first padded with leading zeros to the set number of bits and then converted to equal-length binary encoding; longitude / latitude normalized values ​​(0 to 1000) are first processed using the following linear scaling formula: ; Mapped to The integer interval, where n Let Hilbert curve be the order. x These are the normalized original latitude and longitude values. x ′ is the scaled integer value. The process involves rounding down; then converting the result to a binary code of length n bits, forming three sets of equal-length, aligned binary code strings; finally, loading the Hilbert curve base path encoding table that matches the current order. This table is pre-configured structured data containing mapping rules between multi-dimensional binary bit combinations and one-dimensional path segment order at the corresponding order, defining the one-dimensional path sequence order corresponding to the multi-dimensional binary codes at different positions when the curve traverses the 3-dimensional feature space.

[0055] Next, the corresponding bits of the 3D feature binary code are extracted bit by bit according to the traversal priority of the Hilbert curve and embedded into the path segment of the base path encoding table. First, the highest bit is extracted and combined into a 3-bit temporary code to match the highest bit path segment rule. Then, the second highest bit is extracted for repeated matching and path segment splicing. After iteratively processing all binary bits, a complete one-dimensional path encoding string is formed. Finally, the path encoding string is converted into a decimal value as the initial Hilbert one-dimensional sort value. If the sort value is repeated, the lower 8 decimal bits of the file writing timestamp are extracted and superimposed on the end of the initial value to ensure the global uniqueness of the sort value and not destroy the sorting continuity of adjacent features.

[0056] After mapping is complete, all files to be persisted are sorted in ascending order based on their one-dimensional sorting value. If duplicate sorting values ​​occur during the sorting process, or in extreme cases where multi-dimensional features are completely identical, the file write timestamp is added as a secondary sorting criterion. The sorted file data is then split and integrated into a continuous large data block according to a preset 64KB standard data block size. Data blocks smaller than 64KB at the end are zero-padded for alignment. An independent CRC32 checksum is generated for each sub-block and stored bound to the data block. A high-efficiency write mode of read-ahead + sequential write is adopted. First, 10 consecutive sectors (512 bytes each) in the physical storage space corresponding to the target file block (chunk) are pre-loaded into the physical storage controller cache to reduce the number of head seeks. Then, the integrated continuous data block is sequentially written to the target physical storage space in batches. During the write process, the checksum of the sub-block is compared with the checksum after the physical storage write. If they are inconsistent, a rewrite mechanism is immediately triggered, with no more than 3 rewrites, ensuring the integrity and accuracy of the data write. Step 5.3: During the batch sequential write process, the starting physical offset and file length of each file written within the target file block (chunk) are recorded synchronously to obtain the offset information of each file written within the target file block (chunk). Specifically, during the batch sequential write process, a file-by-file + real-time synchronization offset information recording mechanism is initiated. Using the starting address of the physical storage area corresponding to the target file block (chunk) as the reference zero point, and using 512 bytes of physical sectors as the minimum unit of measurement, the starting physical offset of each file data write is collected in real time, i.e., the address of the sector where the first byte of the file is located. At the same time, the actual file length of each file is accurately recorded, counted by the number of bytes, and the zero bytes used for padding are deducted. During the recording process, a structured temporary offset record containing the file ID, Hilbert one-dimensional sort value, starting physical offset, file length, number of sub-data blocks, and checksum summary value is constructed. Each record is stored in a fixed-length 128-byte binary format and written to the temporary index area in memory. A memory snapshot is performed every time a record is generated to prevent record loss.

[0057] A real-time fault detection mechanism is integrated during the write process. It monitors the IO response status of the physical storage and the checksum comparison results to determine if a write interruption has occurred. If an interruption is detected, the interrupt handling process is immediately executed: First, all subsequent file write operations are paused, and the physical storage sector corresponding to the current write position is locked through atomic operations to prevent data overwriting. Second, the core information of the interruption event, including the interruption timestamp, the physical offset of the interruption position, the file ID being written, the number of bytes written, and the interruption type (e.g., IO timeout / checksum mismatch), is fully recorded in the local fault log and distributed alarm system. Third, the fault self-triggered... The inspection process automatically identifies interruption causes such as storage media failure, link interruption, and controller anomaly. If the failure is automatically recoverable, such as a temporary IO timeout, it attempts to restart the write link within 30 seconds and resume writing from the unwritten start position of the interrupted file, completing the offset record of the file. If the failure is not automatically recoverable, such as storage media damage, it immediately triggers an alarm to notify the operation and maintenance personnel. After manual intervention to repair the failure, it reloads the interruption information in the fault log, locates the unwritten start position of the interrupted file, and re-executes the write and offset recording operations for that file and subsequent files to ensure that the offset information of each file is recorded completely, without omissions or errors.

[0058] Step 5.4 summarizes the offset information of all files recorded in the target file chunk, organizes them in the order of file number, and forms a structured index information describing the physical location relationship of files within the target file chunk. Specifically, after the batch write is completed, first, all temporary offset records in the memory temporary index area are summarized, and a dual-dimensional deduplication mechanism is started: the first layer judges duplicate records by file ID + write batch number, and only retains the latest written valid record; the second layer checks the continuity of offsets. If offset overlap occurs, i.e., write anomalies in extreme scenarios, the correct offset is recalculated by combining the Hilbert sort value and file length, and the erroneous records are removed. Record; After deduplication, first sort the offset records in the natural ascending order of the file sequence number, and at the same time retain the Hilbert one-dimensional sort value corresponding to each record; Based on the sorting result, construct a three-level + double-index structured index information. The specific structure of this hierarchical index is as follows: The top level is the file sequence number main index area: adopts an array-style fixed-length structure, each index item occupies 32 bytes, arranged continuously in ascending order of file sequence number, and stores the file sequence number 8 bytes, Hilbert sort value pointer (8 bytes, pointing to the corresponding position in the middle index area), and offset information pointer (16 bytes, pointing to the corresponding position in the bottom storage area) in sequence, supporting fast retrieval by file sequence number in O(1) time complexity; The middle layer is the Hilbert spatial auxiliary index area: it adopts a skip list structure, arranged in ascending order according to the Hilbert one-dimensional sort value. Each index entry occupies 64 bytes, including the Hilbert sort value (8 bytes), a pointer to the associated file ID list (8 bytes, pointing to all file IDs within the sort value range), a physical offset range (16 bytes, recording the start and end physical offsets of the files in this range), the number of sub-ranges (8 bytes), and 24 bytes of reserved extension space. It utilizes the spatial proximity of the Hilbert curve to achieve batch fast retrieval of files with the same business and near spatial proximity. The bottom layer is the offset information storage area: it adopts key-value pair structured storage. The key is the file ID (16 bytes), and the value is a fixed 128-byte offset information structure, including the starting physical offset (8 bytes), the actual file length (8 bytes), the write timestamp (8 bytes), the sub-data block checksum list pointer (16 bytes), the data integrity flag (1 byte), the Hilbert sort value (8 bytes), and 79 bytes of reserved space, ensuring that the offset information is traceable and verifiable.

[0059] After the structured index information is generated, a two-phase commit mechanism is used to complete the persistence: In the first phase, the index information is written to the metadata control structure cache of the target file block (chunk) and a pre-commit log is recorded; in the second phase, the index information is synchronously written to the file block (chunk) index partition of the global virtual domain (Vzone) and file block (chunk) mapping management table, and the status of the target file block (chunk) is updated to persistence complete, releasing the write lock of the memory cache area; if either phase fails, a rollback operation is triggered to restore the index state before persistence, ensuring the consistency of the index information; this structured index, combined with the layout optimization of the Hilbert space filling curve, not only preserves the order of file sequence number sorting, but also optimizes the physical storage layout of business-related files through spatial proximity.

[0060] In a preferred embodiment of the present invention, step 6 above may include: Step 6.1: Obtain the structured index information of the formed target file chunk. Specifically, this includes: using the unique identifier of the target file chunk as the retrieval primary key, initiating parallel requests to obtain structured index information from two storage locations: the metadata control structure cache area of ​​the target file chunk and the file chunk index partition of the global virtual domain (Vzone) and the file chunk mapping management table. During the acquisition process, a dual mechanism of dual-source consistency verification and integrity verification is activated. First, the 256-bit SHA-256 digest value of the index information in the two storage locations is calculated, and the digest values ​​are compared to see if they are consistent. If they are inconsistent, the data in the metadata control structure is corrected based on the data in the global mapping management table. Then, the integrity of the index information fields is verified. The array length of the file sequence number main index area, the number of skip table nodes in the Hilbert space auxiliary index area, and the number of key-value pairs in the offset information storage area are checked to see if they match the total number of files persisted in the batch. If any are missing, the index information completion process is triggered.

[0061] The complete content obtained includes: the full array data of the main index area (including the file number, Hilbert sort value pointer, and offset information pointer for each 32-byte index entry), the full node data of the skip list in the Hilbert space auxiliary index area (including the Hilbert sort value of each level node, pointer to the associated file ID list, and physical offset range), all key-value pair data in the offset information storage area (including the starting physical offset corresponding to each file ID, the actual file length, and other complete information), and metadata such as field definitions, byte lengths, and storage rules for each level of index; at the same time, the unique identifier, the virtual domain identifier, the index generation timestamp, and the data batch number of the target file chunk are extracted, and bound to the structured index information in a fixed order of file chunk identifier, metadata, and index data to form a complete index information dataset containing traceability information. The dataset is encapsulated in a fixed format to ensure compatibility in subsequent processing.

[0062] Step 6.2: Based on the acquired structured index information, construct an index structure uniquely corresponding to the target file chunk. Specifically, this includes: constructing an index structure uniquely corresponding to the target file chunk based on the acquired complete index information dataset. This index structure adopts a logical layering + physical block design, fully aligned with the three-level logical structure defined in Step 5.4, while also adapting to the read / write characteristics of non-volatile storage media; first, process the file sequence number main index area: according to the preset memory mapping rules, load the array data into a contiguous memory address space, and set the array starting address. A base pointer is used to set an address offset that increments by the index number for each index item. The offset is equal to the index item number multiplied by 32 bytes. This ensures that when searching by file number, the target index item can be directly located by the base pointer plus the offset, achieving fast retrieval with a time complexity of O(1). O(1) is the representation of the algorithm's time complexity (Big O notation). The core meaning is that no matter how many files are to be searched, that is, how large the file number scale is in the index, the time taken to retrieve the corresponding index item by file number is fixed and does not change with the growth of data volume, which is the optimal retrieval efficiency level.

[0063] Next, the Hilbert space auxiliary index area is processed: the skip list data is split into three physical blocks for storage: top-level, middle-level, and bottom-level. A doubly linked list pointer is established for each level node, and fast jump pointers across levels are constructed. Frequently accessed sorted value range nodes are mapped to memory cache to optimize the path traversal length during spatial proximity retrieval and shorten retrieval time. Then, the offset information storage area is processed: key-value pair data is stored in fragments according to the hash value of the file ID. Each fragment corresponds to an independent hash bucket. A hash index table is established to record the starting address and data length of each fragment. At the same time, a data integrity flag is added to each key-value pair, and the flag is assigned according to the rule of write complete = 1, pending verification = 0, and exception = -1. Finally, a header information block is added to the entire index structure, which includes the unique identifier of the target file block (chunk), the index structure version number, the starting offset of each level data block, the total index length, and the generation timestamp, to ensure a one-to-one correspondence between the index structure and the target file block (chunk) and to support backward compatibility in subsequent index version iterations. Step 6.3: Persistently save the constructed index structure uniquely corresponding to the target file chunk to a non-volatile storage medium as an independent index file. This forms a persistent index structure for the target file chunk. Specifically, this includes: converting the constructed index structure according to preset binary serialization rules. The serialization process follows a fixed order: header information block, file sequence number main index area, Hilbert space auxiliary index area, and offset information storage area. Independent serialization rules are used for each level of data block: the header information block is written sequentially according to the byte length defined by the fields; the file sequence number main index area is written continuously in array order; the Hilbert space auxiliary index area is written from top to bottom according to the skip list level; and the offset information storage area is written in hash fragment order. After serialization, the byte stream is compressed at a compression level of [missing information]. Set to level 6 to balance compression ratio and compression speed, after compression, the 256-bit SHA-256 checksum of the entire byte stream is calculated and appended to the end of the byte stream as an integrity check segment. The processed byte stream is written to non-volatile storage media, such as SSD solid-state drives, in the form of an independent index file. The index file naming follows a globally unique rule: 6 bits for virtual domain (Vzone) identifier, 8 bits for file block (chunk) identifier, 4 bits for index version number, and 13 bits for generation timestamp .idx. During writing, a mechanism of block writing + block-by-block verification + overall verification is adopted: the index file is split into fixed-size 4MB data blocks, and after writing each block to the storage medium, the CRC32 checksum of the data block is immediately verified. After all data blocks are written, the SHA-256 checksum of the entire index file is recalculated and compared with the checksum generated in the serialization stage.

[0064] After successful verification, the global index file directory is updated. This directory uses a distributed key-value storage format to record information such as the storage path, file size, SHA-256 checksum, associated file chunk identifier, virtual domain identifier, generation timestamp, and last access time of the index file. At the same time, the storage path of the index file is synchronously written into the metadata control structure of the target file chunk. Based on this, a persistent index structure for the target file chunk is formed, ensuring that the index file can be quickly located through the global directory during subsequent file retrieval, and the validity of the index file can be quickly verified through the metadata control structure.

[0065] In a preferred embodiment of the present invention, step 7 above may include: Step 7.1: Receive file read requests and extract the file ID corresponding to the file to be read from the received file read requests. Specifically, this includes: receiving file read requests from clients through a set file read request receiving link. This link adapts to different transmission protocols according to the client type. Security monitoring terminal clients use a TCP link, while AI training platform clients use a UDP link. During the receiving process, the format integrity of the request is checked synchronously. The request header is checked to see if it contains required fields such as client identifier, request timestamp, and signature information. If missing, a format error response is returned. After successful verification, the file ID corresponding to the file to be read is extracted from the fixed field positions of the request body according to the preset request body parsing rules. The file ID is a string that conforms to the format of virtual domain (Vzone) identifier, file block (chunk) identifier, file sequence number, and millisecond-level timestamp. After extraction, the file ID is format-validated to confirm that the length of each field and the separator conform to the generation rules. If the verification fails, it is marked as an invalid request and an exception log is recorded.

[0066] Step 7.2 involves parsing the extracted file ID to obtain the target virtual domain (Vzone) identifier, target file chunk (chunk) identifier, and target file sequence number. Specifically, this includes: initiating a structured file ID parsing process; reversing the string according to the file ID generation rules; first, splitting the file ID into four subfields using hyphens as delimiters, corresponding to the virtual domain (Vzone) identifier, file chunk (chunk) identifier, file sequence number, and millisecond-level timestamp; and then performing validity checks on the split subfields to verify that the virtual domain (Vzone) identifier is valid. The system checks whether the target virtual domain (Vzone) identifier, target file chunk identifier, and target file sequence number are 6 characters, 8 characters, 6 digits, and 13 digits, respectively. After successful verification, it extracts the target Vzone identifier, target file chunk identifier, and target file sequence number. The extracted identifiers and sequence numbers are converted to a standardized format, while the Vzone identifier and file chunk identifier retain their character format. The file sequence number is converted to a decimal integer. The parsing results are then bound to the file ID and recorded in the read request processing log to ensure that the parsing process is traceable and verifiable.

[0067] Step 7.3: Based on the obtained target file chunk identifier, query the physical offset position corresponding to the target file sequence number from the formed and persistently stored index structure. Specifically, this includes: based on the obtained target file chunk identifier, first initiating a precise query on the global index file directory, using the target file chunk identifier as the search key, matching the associated file chunk identifier field recorded in the directory, and locating the complete storage path, file size, SHA-256 checksum, and other core information of the persistent index file corresponding to that file chunk; then loading the index file from the non-volatile storage medium to the index-dedicated cache area in memory according to the storage path. The process adopts a block loading mechanism, splitting the index file into 4MB blocks and loading them one by one. After each block is loaded, its CRC32 checksum is verified. After all blocks are loaded, the SHA-256 checksum of the entire file is verified again. The double verification ensures that the index data is complete and has not been tampered with. After loading, the header information block of the index structure is searched first to confirm that the index version is compatible with the current processing logic. Then, the main index area of ​​the file number is searched based on the target file number. Using the O(1) retrieval mechanism of the base pointer + offset, the index item with the corresponding number is directly located by the calculation method of base pointer address + target file number × 32 bytes (fixed length of index item). The Hilbert sort value pointer and offset information pointer are extracted from the index item.

[0068] Based on the offset information pointer, the system directly jumps to the corresponding key-value pair storage location in the offset information storage area, reads complete information including the starting physical offset of the file to be read, file length, sub-data block checksum list, and data integrity flag, thereby accurately determining the specific physical offset position and data range of the file to be read in the physical storage space of the target file block (chunk); if no corresponding file sequence number is matched during the search process, or the data integrity flag is abnormal, the search process is immediately terminated, marked as file not found or file data abnormal, and the abnormal information, including the target file block (chunk) identifier, target file sequence number, and search steps, is recorded in the search abnormal log.

[0069] Step 7.4: Based on the queried and determined physical offset position, directly read data of the corresponding length from the target file chunk to obtain the requested file data. Specifically, this includes: based on the queried starting physical offset position, file length, and sub-data block checksum list, first locating the physical storage device and specific storage partition corresponding to the target file chunk, and initiating a full-process reading mechanism of precise positioning, continuous reading, and block-by-block verification; first, precisely moving the storage controller's read pointer to the starting physical offset position, calculating the total number of bytes to be read according to the file length, and generating a clear read range instruction to avoid data redundancy or loss caused by cross-range reading; during the reading process, an intelligent pre-read optimization mechanism is enabled. The number of sectors to be pre-read is dynamically adjusted based on file length: 5 sectors are pre-read when file length is ≤64KB, 10 sectors are pre-read when file length is ≤1MB and 64KB < 1MB, and 20 sectors are pre-read when file length is >1MB. By pre-loading continuous sector data into the cache, the time spent on head seeking and disk rotation is significantly reduced. According to the sub-data block division rules, the read data is split into a corresponding number of sub-data blocks, and the CRC32 checksum is calculated for each block. The checksum is compared one by one with the sub-data block checksum list recorded in the offset information storage area to ensure the integrity of each data block. After all sub-data blocks pass the checksum, the checksum of the entire data is calculated and compared with the checksum summary value a second time. This double checksum ensures that the data has not been tampered with or damaged.

[0070] After successful verification, the read binary data is converted into the corresponding format according to the parsed client type. The data is then encapsulated according to the set read response protocol, including standardized file data, a 200 success status code, a unique file ID, file length, read time, and checksum information. Finally, the encapsulated data is returned to the client using the same transmission protocol as the request reception phase. During transmission, a data fragmentation mechanism is enabled. Data larger than 1MB is split for transmission, and the fragment sequence number and total number of fragments are recorded to ensure that the client can correctly assemble the data. After transmission is complete, the system waits for the client to return a receipt confirmation. If no confirmation is received within a timeout period, a retransmission mechanism is triggered, with a maximum of two retransmissions. If a retransmission fails, a transmission exception is recorded. At the same time, detailed information about this read operation (file ID, read time, starting offset, file length, read time, verification result, and transmission status) is written to the distributed read operation log, completing the entire file read process.

[0071] The above description represents the preferred embodiments of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A method for streaming storage management and persistence of files, characterized in that, The method includes: Divide the hard disk storage space into multiple file blocks of the same size to obtain multiple file blocks that are initially unused. Based on the multiple file blocks that are initially unused, create and manage multiple virtual domains; assign the multiple file blocks that are initially unused to different virtual domains for management. Based on the established virtual domain, when a file to be written is received, the corresponding virtual domain is matched according to the attributes of the file to be written, and a target file block and a unique file sequence number are allocated for the file to be written in the corresponding virtual domain. Based on the allocated target file block and the unique file sequence number, a corresponding file ID is generated; at the same time, the data to be written to the file is cached, and the file ID is returned to the client to complete the write response. Based on the completion of the write response, for the cached data, when the preset flush conditions are met, the cached data is written in batches sequentially to the physical storage space of the target file block, and the offset information of each file in the target file block is recorded synchronously to form the index information of the target file block. Based on the index information of the target file blocks, an index structure for recording file position relationships is constructed and persistently stored, forming a persistently stored index structure; Parse the file ID contained in the file read request to obtain the corresponding target virtual domain, target file block and target file number; determine the offset position of the target file number in the target file block according to the persistent storage index structure, and read the corresponding file data from the target file block accordingly.

2. The method for streaming storage management and persistence of files according to claim 1, characterized in that, The hard disk storage space is divided into multiple file blocks of the same size, resulting in multiple file blocks that are initially unused, including: Based on the preset file block size parameters, a space partitioning operation is performed on the physical storage space of the hard disk to form multiple storage units with the same logical capacity. A corresponding metadata control structure is created for each storage unit with the same logical capacity obtained from the partitioning. The metadata control structure is used to record at least the physical location, logical capacity and current usage status of the storage unit. Each storage unit with the same logical capacity that has a metadata control structure is defined as a file block, and its current usage state is uniformly initialized to an unused state, thus obtaining multiple file blocks with an initial state of unused.

3. The method for streaming storage management and persistence of files according to claim 2, characterized in that, Based on the multiple file blocks that are initially unused, create and manage multiple virtual storage domains, denoted as virtual domains; Multiple file blocks initially in an unused state are assigned to different virtual domains for management, including: Based on the business attributes and storage requirements of the files, multiple virtual storage domain structures are created, and a unique virtual domain identifier is assigned to each virtual storage domain structure to obtain the initial virtual domain structure. Based on the initially established virtual domain structure, configure corresponding file management attributes for each virtual domain; the file management attributes are at least used to define the business types and storage time requirements of files that are allowed to be stored in the virtual domain. Based on the file management attributes configured for each virtual domain, one or more file blocks are dynamically allocated to each virtual domain from multiple file blocks that are initially in an unused state, in order to establish a binding relationship between the virtual domain and the allocated file blocks; Based on the binding relationship between the established virtual domains and the allocated file blocks, a global virtual domain and file block mapping management table is constructed and maintained to record the identifiers and status information of all file blocks managed by each virtual domain, thereby completing the creation and resource initialization management of virtual storage domains.

4. The method for streaming storage management and persistence of files according to claim 3, characterized in that, Based on the established virtual domains, upon receiving a file to be written, the system matches the corresponding virtual domain according to the file's attributes, allocates a target file block for the file within that virtual domain, and assigns a unique file sequence number to the file, including: Receive the file to be written and extract its attribute information; the attribute information includes at least the business type, storage time limit requirements, and spatial coordinates. Based on the extracted attribute information, and using the established and maintained global virtual domain and file block mapping management table, the target virtual domain corresponding to the attribute information is matched. Based on the matched target virtual domain, query the current status information of all file blocks within the target virtual domain, and select a file block that is in a usable state as the target file block for writing the file to be written. In the target file block used to write the file to be written, a unique file number is assigned to the file to be written, which is sequentially incremented relative to the file numbers already stored in the target file block.

5. The method for streaming storage management and persistence of files according to claim 4, characterized in that, Based on the allocated target file block and the unique file sequence number, a corresponding file ID is generated; simultaneously, the data to be written to the file is cached, and the file ID is returned to the client to complete the write response, including: Based on the allocated target file block and the unique file sequence number, and in accordance with the predefined file ID generation rules, a unique file ID is generated; The data to be written to the file is stored in the memory cache area corresponding to the target file block; After storing the data to be written to the file, the generated file ID is returned to the client that initiated the write request, thus completing the write response.

6. The method for streaming storage management and persistence of files according to claim 5, characterized in that, Based on the completed write response, for the cached data, when the preset flush conditions are met, the cached data is sequentially written in batches to the physical storage space of the target file block, and the offset information of each file within the target file block is recorded synchronously to form the index information of the target file block, including: Monitor the memory cache area corresponding to the target file block to determine whether the amount of cached data or the cache duration has reached a preset threshold; when the preset threshold is reached, trigger the batch persistence process for the target file block. In response to the triggered batch persistence process, all currently cached file data in the memory cache area are written sequentially in batches as a continuous data block to the physical storage space corresponding to the target file block. During the batch sequential write process, the starting physical offset and file length of each file to be written in the target file block are recorded synchronously to obtain the offset information of each file to be written in the target file block. The offset information of all files recorded in the target file block is summarized and organized in the order of file number to form a structured index information that describes the physical location relationship of files within the target file block.

7. The method for streaming storage management and persistence of files according to claim 6, characterized in that, Based on the index information of the target file blocks, an index structure for recording file positional relationships is constructed and persistently stored, forming a persistently stored index structure, including: Obtain the structured index information of the resulting target file blocks; Based on the obtained structured index information, construct an index structure that uniquely corresponds to the target file block; The constructed index structure, which uniquely corresponds to the target file block, is persistently saved to a non-volatile storage medium as an independent index file, thereby forming a persistent index structure for the target file block.

8. The method for streaming storage management and persistence of files according to claim 7, characterized in that, Upon receiving a file read request, the file ID contained in the request is parsed to obtain the corresponding target virtual field, target file block, and target file sequence number. Based on the persistent storage index structure, the offset position of the target file sequence number within the target file block is determined, and the corresponding file data is read from the target file block accordingly, including: Receive a file read request and extract the file ID corresponding to the file to be read from the received file read request; By parsing the extracted file ID, the target virtual domain identifier, target file block identifier, and target file sequence number contained therein can be obtained; Based on the obtained target file block identifier, query the physical offset position corresponding to the target file sequence number from the formed and persistently stored index structure; Based on the queried and determined physical offset position, data of the corresponding length is directly read from the target file block to obtain the requested file data.

Images