Streamed data writing method, system, electronic device, and storage medium

Abstract

The application discloses a stream data writing method and system, electronic equipment and a storage medium, and belongs to the technical field of data storage. The method comprises the following steps: in response to a received writing instruction, allocating a storage resource and determining data attribute information; writing a transaction header containing an unfinished state identifier at a preset position of the storage resource and persisting, and temporarily suspending the establishment of an index of the storage resource in an index unit; receiving a data stream to be written and writing the data stream into the storage resource; in response to the completion of the writing of the data stream, updating the state identifier of the transaction header to completed and persisting, and completing the establishment of the index of the storage resource, so that a data object is visible externally. The atomicity of data writing can be effectively guaranteed, dirty reading can be prevented, stream storage of large objects is supported, and the data consistency and reliability of a storage system are improved.

Description

Technical Field

[0001] This application relates to the field of data storage technology, and in particular to a streaming data writing method, system, electronic device, and storage medium. Background Technology

[0002] With the explosive growth of data volume, the demand for streaming storage of unstructured data, such as emails, log files, and large binary objects, is increasing daily. Existing data storage systems for unstructured data face numerous challenges in handling this type of data. First, traditional writing methods typically require loading the entire data object into application-level memory before writing it to disk, which can easily lead to memory overflow when writing extremely large objects. Second, the underlying writing process lacks transaction-level atomicity control. If a network interruption or system power failure occurs during the writing process, invalid "half-finished data" can easily be generated. Furthermore, traditional file systems lack an application-level aware "incomplete is invisible" mechanism, which can easily lead to upper layers reading incomplete and dirty data. Finally, data fragments left by abnormal interruptions occupy storage space for a long time, and there is a lack of effective automatic identification and reclamation methods.

[0003] While existing technical solutions have made optimizations in some areas, they still have significant limitations. For example, some solutions propose separating append writes from data indexing and implementing garbage collection mechanisms, but their index updates rely on file encapsulation and do not provide atomic control and visibility isolation of transaction states during streaming writes. Furthermore, conventional garbage collection, based on file modification time or overwriting write flags, cannot accurately identify and safely clean up "half-finished data" caused by abnormal interruptions. Another example is the use of transaction header information to record start and end positions to identify long transactions; however, this is mainly applied to log parsing scenarios to avoid parsing delays and does not address the physical storage layer's streaming direct write anti-OOM mechanism, nor does it provide a fragmentation reclamation strategy based on transaction state bits combined with time thresholds to prevent accidental deletion. Therefore, the challenge lies in how to avoid memory overflow, ensure the atomicity of data writes, and prevent incomplete writes from being visible when writing extremely large data objects, while automatically and efficiently reclaiming abnormal fragments. Summary of the Invention

[0004] This application provides a streaming data writing method, system, electronic device, and storage medium to at least solve the problems in the prior art that data writing is prone to memory overflow, lack of atomicity control leading to dirty data, and difficulty in automatically identifying and recycling abnormal fragments.

[0005] In a first aspect, this application provides a streaming data writing method, the method comprising: In response to a received write command, storage resources are allocated and data attribute information is determined, wherein the data attribute information is used to represent the metadata of the data object to be written and the location information of the storage resources; The transaction header is persisted to a preset location of the storage resource, and the creation of the index of the storage resource in the index unit is temporarily suspended. The transaction header includes an initial incomplete status flag, the data attribute information, and a magic number used to identify the transaction data block. Receive the data stream to be written and write it to the storage resource; In response to the completion of the data stream writing, the status flag of the transaction header is updated to "completed", and the updated transaction header is persisted to complete the establishment of the index of the storage resource, so that the data object is visible to the outside world.

[0006] The above technical solution achieves atomic control of the data writing process by persisting a transaction header containing a status identifier at the beginning of the storage resource and delaying index creation until the write is complete. By checking the status identifier of the transaction header, it ensures that only fully written data objects are visible to the outside world. At the same time, it solves the problem of resource waste caused by abnormal interruption by identifying and reclaiming abnormal fragments through a scanning mechanism. Its beneficial effects are: avoiding memory overflow caused by fully loading large objects, ensuring the atomicity of data writing, effectively preventing the generation of dirty data, and providing automated abnormal fragment cleanup capabilities, thereby improving the resource utilization efficiency and operational stability of the storage system while ensuring data consistency and reliability.

[0007] Secondly, this application provides a streaming data writing system, comprising: The resource management module is used to allocate storage resources and determine data attribute information in response to the received write command. The data attribute information is used to represent the metadata of the data object to be written and the location information of the storage resource. The transaction control module is used to persist the transaction header to a preset location of the storage resource and postpone the creation of the index of the storage resource in the index unit. The transaction header includes an initialized incomplete status flag, the data attribute information, and a magic number used to identify the transaction data block. The creation of the index of the storage resource in the index unit is postponed. The data writing module is used to receive the data stream to be written and write it to the storage resource; The index management module is used to update the status flag of the transaction header to "completed" in response to the completion of the data stream writing, and persist the updated transaction header to complete the establishment of the index of the storage resource so that the data object is visible to the outside world.

[0008] Thirdly, this application provides an electronic device including one or more processors and one or more memories, wherein at least one piece of program code is stored in the one or more memories, the program code being loaded and executed by the one or more processors to implement the operations performed by the streaming data writing method.

[0009] Fourthly, this application also provides a computer-readable storage medium storing at least one piece of program code, which is loaded and executed by a processor to implement the operations performed by the streaming data writing method.

[0010] Fifthly, this application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of any of the above-described streaming data writing methods. Attached Figure Description

[0011] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0012] To more clearly illustrate the technical solutions in the embodiments of this disclosure or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0013] Figure 1 A flowchart illustrating the streaming data writing method provided in this application embodiment; Figure 2 A flowchart illustrating the abnormal fragment recovery method provided in this application embodiment; Figure 3 A flowchart illustrating the data truncation confirmation method provided in this application embodiment; Figure 4 A flowchart illustrating the index record deletion method provided in this application embodiment; Figure 5 A flowchart illustrating the fragmentation recycling strategy execution method provided in this application embodiment; Figure 6 A flowchart illustrating the adaptive write mode switching method provided in this application embodiment; Figure 7 A flowchart illustrating the breakpoint resume method provided in this application embodiment; Figure 8 A flowchart illustrating the transaction integrity verification method provided in this application embodiment; Figure 9 A flowchart illustrating the data writing method in page cache write mode provided in this application embodiment; Figure 10 A flowchart illustrating the index creation method provided in this application embodiment; Figure 11A flowchart illustrating the read request processing method provided in an embodiment of this application; Figure 12 This is a schematic diagram of the structure of the streaming data writing system provided in the embodiments of this application; Figure 13 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0014] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the protection scope of this application.

[0015] It should be noted that, in the description of this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. The terms "first," "second," etc., in this application are used to distinguish similar objects and are not used to describe a specific order or sequence.

[0016] In related technologies, with the explosive growth of data volume, existing data storage systems face numerous challenges in processing unstructured data. While existing technical solutions have partially solved these problems, they still have significant limitations: On the one hand, there is an existing "queue and storage integration" solution (refer to CN121458251A). The core of this solution is to directly write email data into data blocks at fixed locations and manage the state through index units, thereby reducing the I / O overhead caused by data movement between queues. However, this solution mainly focuses on optimizing the data movement path and does not explicitly address the risk of memory overflow when writing very large objects. When large objects transmitted over the network need to be fully loaded into application-level memory before being written to disk, if the object size exceeds the server's memory threshold, it can easily lead to service crashes. Furthermore, this solution does not describe in detail the data integrity guarantee mechanism when network interruptions or system power failures cause partial writes, which can easily result in invalid "half-finished data" occupying storage space, and lacks effective control over the atomicity of data writes.

[0017] On the other hand, existing technologies have proposed a "group storage" scheme based on user behavior attributes (refer to CN118612180A). This scheme groups and caches frequently accessed data to improve the hit rate. However, the underlying write process of this scheme still relies on traditional file system calls and lacks transaction-level atomicity control. If an exception occurs during the write process, it may lead to corruption of the group file structure. At the same time, this scheme cannot automatically identify and clean up invalid data fragments left by abnormal interruptions, resulting in long-term invalid occupation of storage space.

[0018] Furthermore, in common data storage technologies, database transactions (ACID, Atomicity, Consistency, Isolation, Durability) typically rely on Write-Ahead Logging (WAL) mechanisms to ensure consistency, but their log maintenance overhead is significant; message queues (such as Kafka) support streaming processing, but their offset-driven approach cannot directly control the atomic state of physical writes; traditional file systems (such as ext4) lack an application-layer-aware "incomplete is invisible" mechanism, making it difficult to effectively prevent upper-layer applications from reading dirty data that has not been fully written.

[0019] To address the aforementioned technical challenges, this application proposes a streaming data writing method based on transaction header state control and delayed index creation. By persisting a transaction header containing initialization state at the start of the write operation and deferring index creation until the transaction is complete, atomic semantics—data is not visible until completion—are achieved. Simultaneously, a periodic fragmentation scanning and reclamation mechanism accurately identifies residual data left due to abnormal interruptions and safely cleans or archives it. This mechanism ensures data consistency and the health of the storage system throughout the entire lifecycle of writing, indexing, and maintenance.

[0020] The application scenarios of the technical solutions provided in the embodiments of this application are described below.

[0021] The technical solutions provided in this application are mainly applied to unstructured data storage scenarios with extremely high requirements for the atomicity of data writing, memory security of large object transmission, and automated operation and maintenance of storage space. Specifically, they can include the following application scenarios: 1. Large attachment email server scenario In email systems, with business growth, it's increasingly common for single emails to contain extremely large attachments, such as high-definition videos, large design drawings, and database backup files. Traditional mail servers typically employ a full-reception, temporary storage, parsing, and disk writing model. When the attachment size exceeds the application's memory threshold, it can easily trigger an OutOfMemoryError (OOM), leading to service crashes. Furthermore, if the client disconnects or experiences network fluctuations during the receiving process, "half-finished emails" with only headers or partial data can easily be generated, resulting in corrupted files or garbled text received by the user. This application's technical solution can be applied to the email receiving module. Through a streaming direct-write mechanism, SMTP / HTTP data streams are directly written to disk, eliminating the memory pressure caused by large attachments. Simultaneously, a transaction header mechanism ensures that the email index is only updated and made visible to the user after complete reception and verification, thereby guaranteeing the data integrity and service stability of the email system.

[0022] 2. Distributed Log Collection and Storage System Scenario In big data or IoT platforms, log collection agents need to transmit massive amounts of application logs and runtime logs to backend storage in real time. Due to complex network environments or collection node failures, the write process is frequently interrupted, resulting in a large number of incomplete log fragment files remaining on the storage medium. These fragments not only consume a significant amount of disk space but may also interfere with the accuracy of log analysis tasks. The technical solution in this application can be applied to log storage nodes, ensuring high-speed disk writing of log data through streaming without blocking the data collection link; simultaneously, the built-in fragment recycling daemon can identify and clean up log fragments generated by interruptions based on transaction status, preventing storage space from being consumed by invalid data and significantly reducing manual maintenance costs.

[0023] 3. Cloud storage and object archiving system scenarios In scenarios such as cloud storage, video surveillance archiving, or scientific research data archiving, there is a need for long-term storage of petabyte-scale files, such as high-definition video streams and gene sequencing data. Writing this type of data is time-consuming and extremely sensitive to transmission interruptions. If a power outage occurs during upload or archiving, it is crucial to ensure that no unreadable or corrupted files appear in the archive repository. The technical solution presented in this application can be applied to gateway nodes of object storage. By combining pre-written transaction headers with delayed indexes, it achieves atomic control of file uploads, meaning either the file is complete and usable, or it completely does not exist. Furthermore, with fragment verification and breakpoint resume functionality, it supports rapid recovery of transmission from the breakpoint when abnormal fragments are detected, avoiding bandwidth waste caused by repeated uploads over long-distance networks and improving the reliability of massive data archiving.

[0024] 4. Edge computing and gateway device scenarios On edge computing gateways, such as industrial gateways and vehicle gateways, hardware resources (memory, CPU) are limited, but high-frequency sensor data or local video recordings need to be collected and uploaded. Traditional caching mechanisms are prone to exhausting gateway memory, leading to system crashes. The technical solution in this application can run in a lightweight storage system for edge devices, using a very small fixed memory buffer to process streaming data, ensuring constant memory usage even when storing terabyte-level video streams or logs. Simultaneously, the automatic fragmentation reclamation mechanism adapts to the unattended and difficult-to-maintain characteristics of edge devices, ensuring the storage health of the device under long-term harsh operating environments.

[0025] After introducing the implementation environment and application scenarios of the embodiments of this application, the technical solutions provided by the embodiments of this application are described below. (See also...) Figure 1 Taking a storage server as the execution subject as an example, the streaming data writing method includes the following steps.

[0026] Step S101: In response to the received write command, allocate storage resources and determine data attribute information.

[0027] A write command is used to request the allocation of space on the storage medium for a new data object and to prepare for receiving data. The sender of a write command can be a client, an edge gateway, or an upper-layer application. The receiver of the write command can be a storage server or a storage node.

[0028] Data attribute information is used to represent the metadata of the data object to be written and the location information of the storage resource. For example, the above data attribute information may include, but is not limited to: object unique identifier (Object ID), data object size, content type, creation time, physical path or logical address of the storage resource, etc.

[0029] Specifically, when the storage server receives a write command from the client, it first parses the command and extracts the metadata (such as the sender and estimated size). Based on the expected data size indicated in the write command or the system's default block allocation strategy, it pre-allocates corresponding storage resources on the storage medium (such as disk arrays or object storage buckets), such as physical storage blocks or file space, to ensure sufficient storage resources to accommodate the upcoming data flow. Simultaneously, it generates a globally unique object identifier for this write command and records the location of the allocated storage resources. This information is combined into data attribute information to prepare for subsequent transaction header construction and index record creation.

[0030] Step S102: Persist the transaction header to the preset location of the storage resource, and postpone the creation of the index of the storage resource in the index unit.

[0031] The transaction header is the metadata description of this write transaction. It includes an initial incomplete status flag, the data attribute information, a magic number to identify the transaction data blocks, and a write start timestamp. The write start timestamp records the write start time during transaction initialization. The incomplete status flag, for example, can be set to "INIT" or "0x01," indicating that the transaction is in progress but the data has not been completely written. The magic number is a special byte sequence used to quickly verify whether subsequent data blocks belong to the ongoing transaction and to verify the integrity of the transaction header itself. The default location is usually the starting position of the allocated storage resource, such as a file.

[0032] Persistence means writing the transaction header data stably to a non-volatile storage medium, such as a disk, rather than just keeping it in memory.

[0033] An index unit is a logical or physical structure in a storage system used to manage metadata of data objects. It is responsible for maintaining the mapping relationship between the identifier of a data object (such as an object ID) and the physical location of the storage resource. An index unit can be an in-memory index tree, a hash table, or an external metadata database. The index unit determines the visibility and addressability of data; only data objects successfully registered in the index unit can be queried, retrieved, and accessed by upper-layer applications.

[0034] Delaying the creation of an index in an index cell means that, at this point, although storage resources have been allocated and transaction headers have been written, there are no records of this new data object in the metadata index cell, such as in the database table or the in-memory index tree.

[0035] Specifically, immediately after allocating storage resources, a formatted transaction header is written at the beginning of the storage resource (Offset=0). This operation calls system interfaces such as fsync to ensure that the transaction header has been physically written to disk, preventing the storage resource from being unrecognizable in the event of a system crash due to subsequent operations. Crucially, the step of registering the object in the global index is deliberately skipped at this stage. This design ensures that even if the write process fails midway, the incomplete data object is "invisible" to any external read requests, thus avoiding the generation of dirty data and achieving atomic isolation in the write process.

[0036] Step S103: Receive the data stream to be written and write it to the storage resource.

[0037] A data stream refers to data received in the form of a continuous sequence of bytes, which carries the actual content of the data object to be written. Unlike the method of receiving the complete data object at once and then writing it, this step adopts a streaming write mechanism, that is, receiving and writing simultaneously: while receiving the data stream from the client or application, the storage system writes the received data to the storage resources in real time, thereby reducing the occupation of the memory buffer.

[0038] During the writing process, the optimal writing mode is selected based on the characteristics of the data stream. The specific characteristics of the data stream can be determined based on the expected data size obtained in the preceding steps. The writing modes include direct write mode and page cache write mode. The direct write mode is suitable for extremely large objects, such as those with expected data sizes exceeding a preset threshold, like 100MB. In direct write mode, the data stream bypasses the operating system's page cache and is directly transferred to the disk from the network interface. This mechanism reduces the number of memory copies between kernel and user space, lowering the memory pressure on the page cache and effectively avoiding the risk of insufficient memory when processing large files. The page cache write mode, on the other hand, is suitable for objects of normal size. In page cache write mode, the operating system's page cache mechanism is utilized to optimize I / O scheduling through read-ahead and delayed write-back strategies, merging write operations to improve overall data throughput and write efficiency.

[0039] Regardless of the mode used, the written data is sequentially appended to the storage space following the transaction header in the storage resource.

[0040] Step S104: In response to the completion of data stream writing, update the status flag of the transaction header to complete, persist the updated transaction header, and complete the establishment of the index of the storage resource so that the data object is visible to the outside world.

[0041] A write completion indicates that the amount of data received matches the expected amount, or that a data stream end signal has been received. The status flag is updated to complete, for example, it can be set to "COMMITTED" or "0x00".

[0042] Specifically, the write transaction succeeds once all data is received and successfully written to the storage medium. At this point, the transaction header at the beginning of the storage resource is read, its status flag is changed from "incomplete" to "complete," and the modified transaction header is persisted back to disk. This "update status" operation is an atomic commit point. Once this status update is successfully written to disk, a complete index record is created in the metadata index unit. This record contains the data object's data attribute information, such as path, ID, size, and checksum. At this point, the data object changes from an "invisible" state to an "externally visible" state, allowing subsequent read requests to find and access it through the index. If the system crashes before the status update, and after recovery it still reads an "incomplete" state, the object will be treated as invalid fragment.

[0043] This embodiment persists a transaction header containing an incomplete status flag at the beginning of the storage resource and postpones index building. The index is only built after the data stream is written, the status flag is atomically updated to complete, and persisted. Combined with streaming write technology that bypasses page caching, this achieves atomic control of the data writing process, visibility isolation of incomplete data, and low-memory writing of large objects. Its beneficial effects are that it ensures that data objects are not visible to the outside world when the writing process is interrupted or the system crashes, effectively reducing the risk of upper-layer applications reading incomplete and dirty data. At the same time, it effectively avoids memory overflow caused by writing all large data objects to disk, ensuring data consistency, integrity, and system stability.

[0044] It should be noted that the above steps S101-S104 are a simplified description of the embodiments provided in this application.

[0045] The methods provided in the embodiments of this application will be described in more detail below with some examples. See also... Figure 2 The method also includes an automatic recycling process for abnormal fragments, specifically comprising the following steps: Step S201: In response to the arrival of the preset scan cycle, scan the transaction header of the storage resource and determine the status identifier of the transaction header.

[0046] The preset scan cycle can be a fixed time interval set by the system, such as 24 hours. Storage resources refer to all possible data blocks or file regions.

[0047] Specifically, a fragmentation reclamation daemon is set up. This task starts periodically, traversing or sampling all data resources within the storage space. For each inspected storage resource, the transaction header information at its starting position is read first, and the status flags within are checked.

[0048] Step S202: Verify the magic number in the transaction header and obtain the verification result.

[0049] Magic number verification is a fast integrity check. It compares the magic number field in the transaction header with a preset magic number agreed upon by the system.

[0050] Specifically, if the magic number read does not conform to the convention, or if the transaction header itself is corrupted and the magic number cannot be read, it indicates that the storage resource header is corrupted, or that it is not a valid transaction block created by this system at all.

[0051] Step S203: When the verification result is failed, the storage resource is determined to be an abnormal fragment.

[0052] Specifically, if the magic number check fails, it indicates that the storage resource is not a valid data object. Therefore, it is identified as an abnormal fragment and enters the cleanup process.

[0053] Step S204: If the verification result is successful and the status of the transaction header is incomplete, obtain the time difference.

[0054] The time difference value represents the difference between the current system time and the write start timestamp recorded in the transaction header. The transaction header typically records the write start timestamp during initialization, as described in step S102.

[0055] Specifically, if the magic number check passes but the status indicator shows "incomplete," it means the object represents an interrupted transaction. However, it's necessary to distinguish between "just started writing but not yet completed" and "interrupted and left over from a long time ago." Therefore, it's necessary to calculate the difference between the current time and the start time recorded in the transaction header.

[0056] Step S205: When the time difference exceeds the preset time threshold, the storage resource is determined to be an abnormal fragment.

[0057] The preset time threshold is a reasonable upper limit for write time set according to business requirements, such as 5 minutes or 1 hour.

[0058] Specifically, if a transaction is in an "incomplete" state and has exceeded a preset time threshold, it is highly likely that an abnormal interruption has occurred, and the transaction can no longer be completed normally. Therefore, it is also classified as an abnormal fragment. This time threshold-based judgment mechanism effectively avoids mistakenly classifying ongoing large file writes as fragments and deleting them.

[0059] Step S206: When the storage resource is abnormally fragmented, release the storage resource occupied by the abnormal fragment and delete the index record corresponding to the abnormal fragment in the index unit.

[0060] Specifically, once a storage resource is identified as an abnormal fragment, a reclamation operation will be performed. Physically, the fragmented file will be deleted or its space will be marked as available; logically, the index cells will be checked for any residual index records due to some reason (such as concurrent creation). If any are found, they will be deleted to ensure the consistency between the index and the physical storage and to remove garbage data.

[0061] This embodiment achieves accurate identification and automatic cleanup of abnormal fragments through a multi-dimensional strategy that combines periodic scanning with magic number verification, status identification, and time difference judgment. Its beneficial effects are: it can safely and automatically reclaim residual data generated by various abnormal interruptions (network failures, process crashes, power outages, etc.), prevent storage space from being occupied by invalid data for a long time, and avoids accidental deletion of long-running transactions through a time threshold mechanism, effectively improving the space utilization efficiency and automated operation and maintenance capabilities of the storage system.

[0062] This embodiment further optimizes the accuracy of abnormal fragment identification. See also... Figure 3 After determining that the storage resource is an abnormal fragment, and before releasing it, the method further includes: Step S301: Read the expected data volume in the transaction header and obtain the actual data volume of the storage resource.

[0063] The expected data size refers to the total size of the data object declared by the client during the write instruction phase, and this value is persistently stored in the transaction header.

[0064] The actual data volume refers to the physical byte size currently occupied by the fragment file in the file system.

[0065] Specifically, the expected amount of data is extracted from the specified fields in the transaction header. Simultaneously, the actual disk space occupied by the fragmented file is obtained through a file status query interface provided by the operating system (such as fstat).

[0066] Step S302: When the actual data volume is less than the expected data volume, confirm that the data object is truncated.

[0067] Specifically, the expected amount of data read is compared with the actual amount of data read. If the actual amount of data is significantly less than the expected amount of data, it means that the data stream was forced to stop before the writing process was completed, resulting in incomplete file content and data truncation.

[0068] Step S303: Based on the existence of data object truncation, release the storage resources occupied by abnormal fragments and delete the index record corresponding to the abnormal fragment in the index unit.

[0069] Specifically, after confirming data truncation, the fragment is deemed invalid and deleted. The disk space occupied by the fragment is released, and the relevant index records are retrieved and deleted from the metadata index unit to ensure that there are no logical links pointing to the incomplete file in the system.

[0070] This embodiment confirms the data truncation status by comparing the actual data volume with the expected data volume, thus achieving secondary confirmation of the fragment type. Its beneficial effect is that it provides a more reliable and objective basis for judging abnormal fragments, ensures the accuracy of the recycling operation, prevents the loss of valid data due to misjudgment, and further protects the data integrity of the storage system.

[0071] This example describes the specific logic for deleting index records, particularly for handling concurrent scenarios. See also... Figure 4 The step of deleting the index record corresponding to the abnormal fragment in the index unit includes: Step S401: Confirm that there are index records with abnormal fragmentation in the index unit.

[0072] Specifically, the search is performed using the unique identifier of the fragmented file (such as a file path or ObjectID) through the metadata index unit. If the search finds a match, it indicates that an indexed record exists, and the process proceeds to the next step; if the search fails, the index cleanup process ends directly.

[0073] Step S402: If the index record of the abnormal fragment was partially created due to concurrent writes, delete the index record of the abnormal fragment.

[0074] Concurrent write refers to the possibility that there is a very short time window misalignment between index creation and file writing during the write process.

[0075] Specifically, if an index record exists, but its associated physical file is determined to be fragmented or nonexistent, this is usually due to a crash that occurred after the index creation process was completed but before the file status was updated to complete. In this case, the index record is considered "partially created" dirty data. The index record must be deleted to prevent upper-layer applications from accessing invalid or nonexistent data through the index, thereby eliminating retrieval errors that may be caused by inconsistencies between physical storage and logical indexes.

[0076] This embodiment achieves the maintenance of metadata consistency in concurrent scenarios by confirming the existence of index records and the reasons for their creation before index deletion. Its beneficial effect is that it can accurately clean up indexes generated due to abnormal interruptions, ensuring the integrity of the data structure of the storage system and avoiding query anomalies.

[0077] This embodiment provides a flexible configuration of the fragmentation recycling strategy. See also... Figure 5The fragmentation recycling strategy is dynamically switched by system configuration parameters.

[0078] After confirming that the data object is truncated, the streaming data writing method also includes: Step S501: The preset fragmentation recycling strategy is implemented by performing resource cleanup operations.

[0079] Specifically, the storage system provides a configuration interface that allows administrators to dynamically select fragmentation reclamation strategies, i.e., the specific execution methods of resource cleanup, based on the data's security level and auditing requirements.

[0080] Step S502: Configure the policy as a forced recycling policy. The forced recycling policy includes deleting the files corresponding to the abnormal fragments.

[0081] Specifically, when the policy is configured as a forced reclamation policy, after confirming fragmentation, the underlying file deletion command is directly invoked to remove the fragmented files from the storage medium, immediately releasing all occupied storage resources. This policy is suitable for scenarios with high requirements for storage space reclamation efficiency and where fragmentation auditing is not required.

[0082] Step S503: Configure the policy as an archive retention policy. The archive retention policy includes: moving the files corresponding to the abnormal fragments to the recycle bin directory and setting a preset retention period.

[0083] Specifically, when the policy is configured for archive retention, fragments are not immediately physically deleted; instead, they are moved to a dedicated recycle bin directory. Simultaneously, an extended attribute or scheduled task is set for the file to record its preset retention period (e.g., 7 days). Within the retention period, the file still occupies space but cannot be indexed or retrieved; after the retention period expires, it is automatically and permanently deleted. This policy is suitable for scenarios requiring data auditing or potentially needing data recovery.

[0084] This embodiment provides two strategies—forced recycling and archive retention—to achieve flexible configuration of the fragmentation recycling mechanism. Its advantage lies in the ability to select the most suitable fragmentation processing method according to different business needs and data sensitivity requirements, ensuring system security while also taking into account the needs of data auditing and potential recovery.

[0085] This embodiment implements adaptive write mode switching based on data size. See also Figure 6 The write instruction also carries the expected data size of the data object. Receiving the data stream to be written and writing it to the storage resource includes: Step S601: When the expected data size is greater than the preset size threshold, the received data stream is written to the storage resource through direct write mode.

[0086] The direct write mode allows the data stream to bypass the operating system's page cache and be written directly to disk without allocating a user-space memory buffer. A preset size threshold can be set, for example, to 100MB.

[0087] Specifically, for very large files, the O_DIRECT flag is set when opening the file, and large memory buffers are avoided in user space. After the data stream is received from the network, it is directly transmitted to the disk controller for writing in small, fixed chunks via direct memory access. This mode avoids multiple copies of data between the user-space buffer and the kernel page cache, greatly reducing memory usage and preventing OutOfMemoryError (OOM).

[0088] Step S602: When the expected data size does not exceed the preset size threshold, the received data stream is written to the storage resource through the page cache write mode.

[0089] Specifically, for files of normal size, the operating system's page caching mechanism is utilized. Data is first written to the kernel's page cache, and the operating system determines when to write it back to disk based on an algorithm (such as LRU, Least Recently Used). This mode can leverage the operating system's read-ahead and write-merge optimizations to reduce disk I / O operations and improve the write throughput of small files.

[0090] This embodiment achieves a balance between storage performance and system stability by adaptively switching between direct write and page cache write modes based on data size. Its beneficial effects are that it adopts the optimal write path for data objects of different sizes, effectively avoids memory bottlenecks when processing very large objects, and makes full use of the system cache advantages when processing ordinary objects, thereby improving the overall processing capacity and robustness of the storage system.

[0091] This embodiment introduces a fragmentation verification area in the transaction header, enabling fine-grained recording and verification of write progress, and supporting breakpoint resumption in abnormal fragmentation scenarios. See also... Figure 7 In addition to the original fields, the transaction header data structure also reserves a sharding verification area to store the checksums of multiple shards. The specific method steps are as follows: Step S701: During the process of receiving the data stream to be written and writing it to the storage resource, when the written data meets the preset data size, update the fragment checksum in the fragment check area.

[0092] Specifically, the transaction header data structure reserves an area for storing multiple fragment checksums. During streaming data writing, whenever the amount of streaming data written reaches a preset fragment size, the checksum of the current fragment needs to be calculated and updated to the corresponding position in the fragment checksum area of ​​the transaction header. For example, the preset fragment size can be 10MB.

[0093] Step S702: If the storage resource is determined to be an abnormal fragment, read the last updated valid fragment checksum in the fragment check area.

[0094] Specifically, when abnormal fragmentation of storage resources is detected, the fragment checksum area in the transaction header is traversed to find the last non-empty checksum value. This checksum value represents the last successfully written fragment marker. This valid checksum will be used to determine the breakpoint location later.

[0095] Step S703: Determine the breakpoint offset based on the valid fragment checksum.

[0096] The breakpoint offset is used for subsequent resume transmission based on the breakpoint offset.

[0097] Specifically, based on the position of the valid fragment checksum within the fragment checksum area and the fragment size, the physical offset address at the end of the fragment is calculated. This physical offset address is the exact location where the data stream is interrupted. This breakpoint offset is communicated to the client, allowing the client to continue uploading the remaining data from this position without having to retransmit from the beginning. For example, if the valid fragment checksum corresponds to the second fragment and the fragment size is 10MB, then the breakpoint offset is 20MB.

[0098] This embodiment introduces a fragment checksum mechanism in the transaction header to achieve fine-grained recording and verification of the write progress. Its advantages are that it can not only quickly locate the last valid data block when the write fails, supporting breakpoint resumption to save network bandwidth and rewrite time, but also further verify the integrity of the written data by verifying the checksum of each fragment, providing more possibilities for data recovery.

[0099] This embodiment adds an integrity check before the transaction is committed. See also Figure 8 Before updating the status flag of the transaction header to indicate completion, the process also includes: Step S801: Obtain the overall checksum of the data stream after writing is complete.

[0100] Specifically, after the data stream has been received, a hash calculation is performed on all data objects written to the entire storage resource (excluding the transaction header itself) to generate an overall checksum. This checksum can be a digest value of a strong checksum algorithm such as MD5, SHA-256, or CRC32.

[0101] Step S802: Write the overall checksum into the checksum reservation area in the transaction header.

[0102] Specifically, the transaction header reserves space for storing the checksum during initialization. The calculated overall checksum is filled into this area, and the transaction header is written back to disk.

[0103] Step S803: After the overall checksum is written, update the status flag of the transaction header to "completed" and persist the updated transaction header to disk.

[0104] Specifically, the status flag is changed from "incomplete" to "completed" only after the overall checksum is successfully written to the transaction header, and fsync is called again to ensure the update is written to disk. This order ensures that the "completed" status immediately follows the data integrity commitment. This ensures that externally visible data objects are not only data-complete but also include integrity verification information, facilitating verification during subsequent reads.

[0105] This embodiment achieves an integrity commitment to the complete data object by calculating and writing the overall checksum before transaction commit. Its advantage lies in adding a layer of content-based verification protection to the data object. During subsequent reading or verification, the overall checksum can be recalculated and compared to quickly determine whether data corruption occurred during storage, thus enhancing the reliability of data storage.

[0106] This embodiment is a further optimization of the page cache write mode. See also... Figure 9 In the page cache write mode, receiving the data stream to be written and writing it to the storage resource includes: Step S901: When the expected data size is not greater than the preset size threshold, the received data stream is segmented to obtain data fragments.

[0107] Specifically, for ordinary objects written using page caching mode, the entire data stream is not treated as a whole. Instead, the data stream is divided into several smaller data fragments according to the convenience of network transmission or internal processing, in order to adapt to the network packet size or memory page size.

[0108] Step S902: Transfer data fragments from the network kernel buffer to the file system page cache via the kernel-level zero-copy interface.

[0109] The file system page cache is used to cache data to be written to disk in kernel mode.

[0110] Specifically, zero-copy system calls provided by the operating system, such as sendfile or splice in Linux, are utilized. These interfaces allow data to be transferred directly between different buffers in kernel space, such as from the network card's receive buffer to the file system's page cache, without needing to be copied through user-space memory. This effectively reduces the overhead of CPU context switching and data copying.

[0111] This embodiment optimizes the data transmission path by employing a kernel-level zero-copy interface in page cache write mode. Its advantages include reducing CPU load and data copy latency, and improving the performance of ordinary-scale data streaming writes and the overall system throughput.

[0112] This embodiment details how to create an index for storage resources. See also... Figure 10 The process of establishing the index for the storage resource includes: Step S1001: Extract data attribute information and overall checksum from the transaction header.

[0113] Specifically, after a transaction is committed, the index management module reads the transaction header from the beginning of the storage resource. From the parsed header information, it extracts key metadata, including previously written data attribute information (object ID, size, type, etc.) and the final calculated overall checksum.

[0114] Step S1002: Insert an index record into the metadata index unit.

[0115] The index record contains the data block path, object identifier, overall checksum, and creation time.

[0116] Specifically, the extracted information is organized into a complete index record according to a preset format. This record is then inserted into the system's metadata index unit. The metadata index unit can be an in-memory B+ tree, a hash table, or an external database (such as MySQL or Redis). After successful insertion, the data object can be discovered and accessed by the retrieval system.

[0117] This embodiment establishes the foundation for the retrievability of data objects by registering complete transaction metadata and verification information in the index unit. Its beneficial effect is that it ensures that data objects are not only physically stored intact, but their metadata is also accurately and completely reflected in the system, providing solid support for subsequent data access, management, verification, and lifecycle management.

[0118] This embodiment describes the processing logic for read requests, serving as a reverse security check for the write process. See also... Figure 11 The method further includes: Step S1101: In response to the received data read request, retrieve the index unit.

[0119] Specifically, when the storage server receives a data read request from a client, it first searches for the object ID in the metadata index unit based on the object ID in the request.

[0120] Step S1102: When a corresponding index record exists in the index unit, read the status identifier in the transaction header of the corresponding storage resource.

[0121] Specifically, if a corresponding index record is found in the index unit, it means that the object logically exists. Instead of directly reading the data, the corresponding storage resource file is located and opened based on the path in the index record, and the status flag in its transaction header is read.

[0122] Step S1103: When the status indicator is completed, respond to the data read request and return the corresponding data object.

[0123] Specifically, if the status indicator read is "completed", it means that the data has been fully submitted. At this point, the data stream in the file is read normally and encapsulated into a response to be returned to the client.

[0124] Step S1104: When the status indicator is incomplete, intercept the data read request.

[0125] Specifically, if the status is marked "incomplete," it means the data object is being written or the write process has been interrupted, and the data is incomplete. In this case, the read request is directly intercepted, and a "data does not exist" or "unavailable" flag is returned to the client. This secondary verification mechanism, as the last line of defense for atomicity control, ensures that even if the underlying file system or cache delays cause the file to be physically visible, the application layer can still adhere to the atomicity contract of "incomplete means invisible," and will not read dirty data even in extreme cases, such as when index creation precedes status updates.

[0126] This embodiment achieves access control for incomplete data by introducing a secondary verification of the transaction header status identifier during the reading process. Its beneficial effect is that even if the index record exists in advance due to certain extreme situations, the read operation will be intercepted due to the status check, thereby eliminating the possibility of reading dirty data at the final gate and building an end-to-end data consistency guarantee system.

[0127] This embodiment provides a streaming data writing system. See also... Figure 12 The streaming data writing system 12 includes: The resource management module 1201 is used to allocate storage resources and determine data attribute information in response to received write commands. This module is responsible for parsing commands, pre-allocating space on the storage medium, and generating unique identifiers and location information for objects.

[0128] The transaction control module 1202 is used to persist the transaction header to a preset location of the storage resource and postpone the creation of the index of the storage resource in the index unit. This module is responsible for initializing the transaction state and ensuring that it is not visible to the outside world before the data is completed.

[0129] The data writing module 1203 is used to receive the data stream to be written and write it to the storage resource. This module is responsible for receiving network data and selecting direct write or page cache mode according to the data size to perform the actual disk write operation.

[0130] The index management module 1204 is used to update the status flag of the transaction header to "completed" in response to the completion of the data stream writing, and persist the updated transaction header to complete the establishment of the index for the storage resource. This module is responsible for the final commit of the transaction, ensuring that the data is complete and visible.

[0131] Specifically, when creating an index, the index management module 1204 is used to: extract the data attribute information and overall checksum from the transaction header; and insert an index record into the metadata index unit, wherein the index record includes the data block path, object identifier, overall checksum, and creation time.

[0132] The fragmentation recycling daemon process 1205 is used to scan the transaction header of the storage resources in response to the arrival of the preset scan cycle, determine abnormal fragments based on the status identifier, magic number verification and time threshold, and perform resource release and index cleanup operations.

[0133] This embodiment relates to the system's abnormal fragmentation recovery function. The system also includes a fragmentation recovery daemon process 1205.

[0134] The fragment recycling daemon process 1205 is used for: Scanning and Judgment: In response to the arrival of a preset scan cycle, the transaction header of the storage resource is scanned to determine the status identifier of the transaction header; the magic number in the transaction header is verified to obtain a verification result. When the verification result fails, the storage resource is determined to be abnormally fragmented. When the verification result passes and the status identifier of the transaction header is incomplete, the time difference value is obtained, that is, the difference between the current system time and the write start time. When the time difference value exceeds a preset time threshold, the storage resource is determined to be abnormally fragmented.

[0135] Truncation Confirmation: After determining that the storage resource is an abnormal fragment, and before releasing the storage resource occupied by the abnormal fragment, read the expected data volume in the transaction header and obtain the actual data volume of the storage resource; if the actual data volume is less than the expected data volume, confirm that the data object is truncated; based on the fact that the data object is truncated, release the storage resource occupied by the abnormal fragment.

[0136] Index cleanup: After releasing resources, the index management module 1204 is instructed to perform a cleanup operation. Specifically, the index management module 1204 is used to: confirm the existence of index records with abnormal fragments in the index unit; and delete the index records with abnormal fragments if the abnormal fragments were partially created due to concurrent writes.

[0137] Strategy Execution: The fragmentation recycling daemon 1205 is also used to dynamically switch the fragmentation recycling strategy according to system configuration parameters. After confirming that a data object is truncated, a resource cleanup operation is performed to implement the preset fragmentation recycling strategy. Specifically, when executing the forced recycling strategy, the files corresponding to the abnormal fragments are deleted; when executing the archive retention strategy, the files corresponding to the abnormal fragments are moved to the recycle bin directory, and a preset retention period is set.

[0138] This embodiment relates to the system's adaptive write mode. The write command also carries the expected data size of the data object.

[0139] The data writing module 1203 is specifically used for: Mode Selection: When the expected data size exceeds a preset size threshold (e.g., 100MB), the received data stream is written to the storage resource via direct write mode. Direct write mode allows the data stream to bypass the operating system's page cache and be written directly to disk without allocating a user-space memory buffer. When the expected data size does not exceed the preset size threshold, the received data stream is written to the storage resource via page cache write mode.

[0140] Zero-copy execution: In the page cache write mode, when the expected data size is not greater than a preset size threshold, the received data stream is segmented to obtain data fragments; the data fragments are transferred from the network kernel buffer to the file system page cache through a kernel-level zero-copy interface (such as sendfile), and the file system page cache is used to cache the data to be written to disk in kernel mode.

[0141] This embodiment relates to the system's transaction integrity control and breakpoint resume support.

[0142] Overall verification: The transaction control module 1202 is specifically used to: obtain the overall checksum of the data stream that has been written before updating the status flag of the transaction header to "completed"; write the overall checksum into the checksum reservation area in the transaction header; after the overall checksum is written, update the status flag of the transaction header to "completed" and persist the updated transaction header to disk.

[0143] Fragmentation Verification and Resumption: The transaction header also includes a fragmentation verification area. The transaction control module 1202 is further configured to: during the process of receiving the data stream to be written and writing it to the storage resource, when the written data meets the preset data size, update the fragmentation checksum in the fragmentation verification area. The fragmentation reclamation daemon 1205 is further configured to: when the storage resource is determined to be an abnormal fragment, read the last updated valid fragmentation checksum in the fragmentation verification area; determine the breakpoint offset based on the valid fragmentation checksum, and the breakpoint offset is used for subsequent breakpoint resumption based on the breakpoint offset.

[0144] This embodiment relates to read-side security control of the system. The streaming data writing system 12 is also used to process read requests, specifically including: The index management module 1204 is also used to retrieve the index unit in response to a read request from a received data object; The transaction control module 1202 is also used to read the status identifier in the transaction header of the corresponding storage resource when there is a corresponding index record in the index unit; The data writing module 1203 is further configured to read the data object in the storage resource when the status indicator is completed, and return the corresponding data in response to the read request of the data object; The transaction control module 1202 is further configured to intercept the read request of the data object when the status identifier is incomplete.

[0145] It should be noted that the streaming data writing system provided in the above embodiments is only an example of the division of the above functional modules. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the streaming data writing system and the streaming data writing method embodiments provided in the above embodiments belong to the same concept, and the specific implementation process is detailed in the streaming data writing method embodiments, which will not be repeated here.

[0146] This embodiment provides a method and system for writing streaming data in a large attachment mail server scenario. See also... Figure 12 The streaming data writing system 12 runs on a storage server and includes a resource management module 1201, a transaction control module 1202, a data writing module 1203, an index management module 1204, and a fragmentation recycling daemon process 1205.

[0147] In this specific application scenario, the above modules work together to achieve complete streaming storage and atomic control. The specific processing flow is as follows: I. Normal Write and Atomicity Guarantee Process In this scenario, when a client initiates an email upload, the system performs the following steps: Reception and Pre-check: Resource management module 1201 listens on a network port (e.g., SMTP port 25) and establishes a TCP connection. Resource management module 1201 receives the MAIL FROM command sent by the client and obtains metadata about the data object (e.g., sender, estimated size). It checks the currently available storage block space; if space is insufficient, it triggers expansion or rejects the request. Subsequently, resource management module 1201 allocates a physical storage block and records its file path and current write offset.

[0148] Write Transaction Header: Transaction control module 1202 writes a fixed-length "transaction header" (e.g., 512 bytes) at the beginning of the allocated physical storage block (Offset=0). The transaction header contains fields: Magic Number (e.g., 0xABCD), Status Bit (initialized to 0x01, representing "writing in progress"), Expected Size (total number of bytes obtained from the network stream), Timestamp (write start time), Checksum, and Reserved Area. Transaction control module 1202 calls the operating system's underlying fsync interface to ensure the transaction header has been physically written to disk. Key point: At this time, index management module 1204 does not create a record in the metadata index unit, so the upper layer cannot find the object when querying the index, ensuring the data is not visible.

[0149] Streaming data writing: The data writing module 1203 creates a mapping channel between the network socket handle and the storage medium file handle. The data writing module 1203 sets the memory buffer size to a fixed value (e.g., 64KB), which does not change with the total size of the data object. After the data stream enters the system, it is divided into multiple 64KB data packets. Each data packet is directly transferred from the network kernel buffer to the file system's page cache via the sendfile or splice system call, avoiding user-space memory copying. During the writing process, the data writing module 1203 continuously updates the current write offset of the file.

[0150] Write completion flag: When the data write module 1203 receives the end flag of the data stream (such as the end-of-stream marker for SMTP), it confirms that the data stream transmission is complete. The transaction control module 1202 calculates the CRC32 checksum of the entire data block and fills it into the "checksum reservation area" of the transaction header. The transaction control module 1202 updates the "status bit" in the transaction header from 0x01 to 0x00 (representing "complete") and calls fsync to ensure that the transaction header update is written to disk.

[0151] Update Index Unit: The index management module 1204 reads the "status bit" of the transaction header and confirms it is 0x00 (complete). The index management module 1204 extracts information such as "expected size," "timestamp," and "overall checksum" from the transaction header. The index management module 1204 inserts a new record into a separate "metadata index unit" (such as a database table or in-memory index structure), containing: data block path, object ID, checksum, and creation time. At this point, users or upper-layer applications can retrieve this object when querying the index.

[0152] Read Interception and Response: When an external client or upper-layer application initiates a data read request, the index management module 1204 first retrieves the metadata index unit. If no index record is found, it directly returns "data does not exist". If an index record is found, the transaction control module 1202 will synchronously read the transaction header status bit of the corresponding physical file. Only when the status bit is confirmed to be 0x00 (completed) will the data writing module 1203 read the data object in the storage resource and respond to the data read request by returning the corresponding data; if the transaction control module 1202 determines that the status bit is still 0x01 (writing), it will intercept the read request and return a "data locked / temporarily unavailable" flag to the client.

[0153] II. Abnormal Interruption and Fragment Reclamation Process If the write process is interrupted, the fragmentation recycling daemon 1205 will clean up the data through a background mechanism: Background scanning: The fragmentation recycling daemon 1205 responds to the arrival of the preset scan cycle (such as every 5 minutes) and starts a scan task, covering the "data block files" on all storage media.

[0154] Status bit reading and judgment: The fragmentation recycling daemon 1205 opens each data block file and reads the first 512 bytes (transaction header). It parses the "status bit" in the transaction header. If the status bit is 0x00 (complete), the file is skipped. If the status bit is 0x01 (writing), the time threshold check is performed. If the magic number check fails, it is determined to be an abnormal fragment.

[0155] Time threshold verification: The fragmentation reclamation daemon 1205 reads the "timestamp" in the transaction header and calculates the difference between the current system time and the timestamp. If the duration is less than the preset threshold (e.g., 300 seconds), it is determined that writing is in progress and no intervention is performed. If the duration is greater than or equal to the threshold, it is determined as "fragmentation with abnormal interruption".

[0156] Execution of fragmentation reclamation strategy: The fragmentation reclamation daemon 1205 reads the "expected size" from the transaction header and obtains the actual file size using fstat. If the actual data volume is less than the expected data volume, it confirms that the data object has been truncated. The fragmentation reclamation daemon 1205 executes the strategy according to the configuration: if it is forced reclamation, it directly deletes the file to release space; if it is archive retention, it moves the file to the recycle bin directory and sets the retention period.

[0157] Index cleanup: If the fragmentation recycling daemon 1205 performs a file deletion operation, it will instruct the index management module 1204 to check the "metadata index unit". If the index management module 1204 confirms that a record of the object exists in the index (partially created due to concurrent writes), it will delete the index record to ensure the consistency between the file and the index.

[0158] III. Streaming Optimization for Extremely Large Objects For emails with large attachments, the data writing module 1203 performs the following optimizations: Object size prediction: When the resource management module 1201 is in the early stage of establishing the connection, it parses the SIZE extension field in the protocol header and passes the expected data size to the data writing module 1203.

[0159] I / O path routing: Data writing module 1203 performs dynamic routing based on the expected data size. (1) Large object direct write channel: If SIZE > preset size threshold (e.g. 100MB), the data writing module 1203 does not allocate user space memory buffer, but directly calls the kernel-level Direct I / O interface, so that the data stream bypasses the operating system page cache and is written directly to disk. The memory usage only maintains a fixed Socket buffer size, fundamentally eliminating the risk of OOM.

[0160] (2) Ordinary object zero-copy channel: If SIZE≤ preset size threshold, the data writing module 1203 enables page cache writing mode, and directly transmits the data of the network kernel buffer to the file system PageCache through the kernel-level zero-copy interface, thereby accelerating I / O throughput by utilizing OS cache.

[0161] Fragmentation Verification and Resume from Breakpoint: To support resume from breakpoint, the transaction control module 1202 maintains a "fragmentation verification area" in the transaction header. During the write process, whenever the data meets the preset data size, such as 10MB, the transaction control module 1202 updates the fragmentation checksum. If the fragmentation reclamation daemon 1205 detects abnormal fragments, it can read the last valid fragmentation checksum to determine the breakpoint offset for the client to use for resuming the download.

[0162] As can be seen from the above comprehensive embodiments, this application effectively solves the problems of memory overflow of large attachments, atomicity of writing, and fragmentation recycling in email storage scenarios through a clear module division of labor and cooperation mechanism.

[0163] This embodiment provides an electronic device, which may specifically be a storage server, a cloud storage node, or a smart terminal with data writing capabilities. Figure 13 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application.

[0164] Typically, electronic device 13 includes one or more processors 1301 and one or more memories 1302.

[0165] The processor 1301 may include one or more processing cores, such as a multi-core ARM Cortex-A series processor or an Intel Xeon processor.

[0166] The memory 1302 may include one or more computer-readable storage media, which may be non-transitory. The memory 1302 may also include high-speed random access memory and non-volatile memory, such as one or more flash memory devices.

[0167] In some embodiments, the non-transitory computer-readable storage medium in memory 1302 is used to store at least one computer program, which is executed by processor 1301 to implement the streaming data writing method provided in the embodiments of the streaming data writing method described in this application.

[0168] Those skilled in the art will understand that Figure 13 The structure shown does not constitute a limitation on the electronic device 13, and may include more or fewer components than shown, or combine certain components, or use different component arrangements.

[0169] This embodiment provides a computer-readable storage medium storing computer program code. When the computer program code is run on a computer, the computer executes the streaming data writing method provided in the above embodiment.

[0170] This embodiment also provides a computer program product that, when run on a computer, causes the computer to perform the aforementioned related steps to implement the streaming data writing method provided in the above embodiment.

[0171] In this embodiment, the device, computer-readable storage medium, computer program product, or chip are all used to execute the streaming data writing method described above. Therefore, the beneficial effects they achieve can be referred to the beneficial effects of the streaming data writing method described above, and will not be repeated here. Through the description of the above embodiments, those skilled in the art will understand that, for the sake of convenience and brevity, only the division of the above functional modules is used as an example. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0172] In the embodiments provided in this application, it should be understood that the disclosed apparatus and the described streaming data writing method can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection of devices or units may be electrical, mechanical, or other forms.

[0173] The above content is only a specific implementation of this application, but the protection scope of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the protection scope of this application.

Claims

1. A method for writing streaming data, characterized in that, The method includes: In response to a received write command, storage resources are allocated and data attribute information is determined, wherein the data attribute information is used to represent the metadata of the data object to be written and the location information of the storage resources; The transaction header is persisted to a preset location of the storage resource, and the creation of the index of the storage resource in the index unit is temporarily suspended. The transaction header includes an initial incomplete status flag, the data attribute information, and a magic number used to identify the transaction data block. Receive the data stream to be written and write it to the storage resource; In response to the completion of the data stream writing, the status flag of the transaction header is updated to "completed", and the updated transaction header is persisted to complete the establishment of the index of the storage resource, so that the data object is visible to the outside world.

2. The method according to claim 1, characterized in that, The method further includes: In response to the arrival of a preset scan cycle, the transaction header of the storage resource is scanned to determine the status identifier of the transaction header; The magic number in the transaction header is validated to obtain the validation result; When the verification result fails, the storage resource is determined to be an abnormal fragment; When the verification result is passed and the status of the transaction header is incomplete, the time difference value is obtained. The time difference value is used to represent the difference between the current system time and the write start time recorded in the transaction header. If the time difference exceeds a preset time threshold, the storage resource is determined to be an abnormal fragment. When the storage resource is an abnormal fragment, the storage resource occupied by the abnormal fragment is released, and the index record corresponding to the abnormal fragment in the index unit is deleted.

3. The method according to claim 2, characterized in that, After determining that the storage resource is an abnormal fragment, and before releasing the storage resource occupied by the abnormal fragment, the method further includes: Read the expected data volume from the transaction header and obtain the actual data volume of the storage resource; If the actual data volume is less than the expected data volume, it is confirmed that the data object is truncated. Based on the existence of truncation in the data object, the storage resources occupied by the abnormal fragment are released, and the index record corresponding to the abnormal fragment in the index unit is deleted.

4. The method according to claim 2, characterized in that, Deleting the index record corresponding to the abnormal fragment in the index unit includes: Confirm that the index record containing the abnormal fragment exists in the index unit; If the index record of the abnormal fragment was partially created due to concurrent writes, delete the index record of the abnormal fragment.

5. The method according to claim 3, characterized in that, After confirming that the data object is truncated, the method further includes: The fragmentation recycling strategy is implemented by performing resource cleanup operations; The fragmentation recycling strategy includes a forced recycling strategy or an archive retention strategy; The forced recycling strategy includes: Delete the file corresponding to the abnormal fragment; The archiving retention strategy includes: Move the files corresponding to the abnormal fragments to the recycle bin directory and set a preset retention period.

6. The method according to claim 1, characterized in that, The write instruction also carries the expected data size of the data object; The process of receiving the data stream to be written and writing it to the storage resource includes: When the expected data size is greater than a preset size threshold, the received data stream is written to the storage resource through direct write mode. The direct write mode is used to enable the data stream to bypass the operating system's page cache and be written directly to disk without allocating a user-mode memory buffer. If the expected data size does not exceed the preset size threshold, the received data stream will be written to the storage resource through page cache write mode.

7. The method according to claim 2, characterized in that, The transaction header also includes a fragmentation verification area; The method further includes: During the process of receiving the data stream to be written and writing it to the storage resource, when the written data meets the preset data size, the fragment checksum in the fragment check area is updated. If the storage resource is determined to be an abnormal fragment, the last updated valid fragment checksum in the fragment check area is read. Based on the valid fragment checksum, the breakpoint offset is determined, and the breakpoint offset is used for subsequent breakpoint resumption based on the breakpoint offset.

8. The method according to claim 1, characterized in that, Before updating the status flag of the transaction header to "completed", the method further includes: Obtain the overall checksum of the data stream after writing is complete; Write the overall checksum into the checksum reservation area in the transaction header; The update of the transaction header status indicator to be completed includes: After the overall checksum is written, the status flag of the transaction header is updated to "completed", and the updated transaction header is persisted to disk.

9. The method according to claim 6, characterized in that, In the page cache write mode, receiving the data stream to be written and writing it to the storage resource includes: When the expected data size is not greater than a preset size threshold, the received data stream is segmented to obtain data fragments; The data fragments are transferred from the network kernel buffer to the file system page cache via the kernel-level zero-copy interface. The file system page cache is used to cache the data to be written to disk in kernel mode.

10. The method according to claim 8, characterized in that, The process of establishing the index for the storage resource includes: Extract the data attribute information and the overall checksum from the transaction header; Insert an index record into the metadata index unit. The index record contains the data block path, object identifier, overall checksum, and creation time.

11. The method according to claim 1, characterized in that, The method further includes: In response to a received data read request, the index unit is retrieved; If a corresponding index record exists in the index unit, read the status identifier in the transaction header of the corresponding storage resource; When the status is marked as completed, respond to the data read request and return the corresponding data object; When the status is marked as incomplete, the data read request is intercepted.

12. A streaming data writing system, characterized in that, include: The resource management module is used to allocate storage resources and determine data attribute information in response to the received write command. The data attribute information is used to represent the metadata of the data object to be written and the location information of the storage resource. The transaction control module is used to persist the transaction header to a preset location for storing the storage resource. The transaction header includes an initialized incomplete status flag, the data attribute information, and a magic number for identifying the transaction data block, and temporarily suspends the creation of the index of the storage resource in the index unit. The data writing module is used to receive the data stream to be written and write it to the storage resource; The index management module is used to update the status flag of the transaction header to "completed" in response to the completion of the data stream writing, and persist the updated transaction header to complete the establishment of the index of the storage resource so that the data object is visible to the outside world.

13. An electronic device, characterized in that, It includes a processor and a memory, the memory storing a computer program that, when executed by the processor, implements the method as described in any one of claims 1-11.

14. A computer-readable storage medium, characterized in that, It stores a computer program thereon, which, when executed by a processor, implements the method as described in any one of claims 1-11.

Images