Data recording method, recording device, electronic equipment and readable storage medium

Abstract

The application discloses a trigger type data recording method based on a ring buffer, a recording device, electronic equipment and a readable storage medium. The method comprises continuously writing original data into a preset ring buffer area of a first storage; in response to a trigger event, determining a trigger window corresponding to the trigger event, and locking original data in the trigger window; and writing the original data in the trigger window from the ring buffer area to a second storage. When there are multiple trigger events and multiple trigger windows corresponding to the trigger events overlap with each other, only one write operation is performed on the original data in the overlapping area. According to the data recording method and device, the service life consumption, thermal load and maintenance cost of the storage system can be reduced, and the system reliability can be improved.

Description

Technical Field

[0001] This application relates to the field of data processing, and in particular to a data recording method, recording device, electronic device, and readable storage medium. Background Technology

[0002] During product testing (e.g., DRAM, Dynamic Random Access Memory), data acquisition devices continuously output large amounts of raw read and write data. While continuously writing all this raw data into memory (e.g., SSD, Solid State Drive) would provide a complete test record, it would rapidly increase the total amount of data written to the memory. For high-speed sampling of large-scale data streams, prolonged continuous writing can significantly accelerate memory media wear, shorten memory lifespan, and increase the thermal load and maintenance costs of the storage system. Summary of the Invention

[0003] The purpose of this application is to address the deficiencies and shortcomings of the prior art by providing a data recording method, recording device, electronic device, and readable storage medium to reduce the lifespan consumption, thermal load, and maintenance costs of storage systems.

[0004] This application is achieved through the following technical solution:

[0005] In a first aspect, embodiments of this application provide a data recording method, which includes:

[0006] The raw data is continuously written to a preset circular buffer in the first memory. The circular buffer includes multiple buffer blocks. Each buffer block includes a header and a payload area. The header includes the buffer block status, global sequence number range, timestamp range, and persistent reference count. The persistent reference count is used to record the number of times the corresponding buffer block is referenced by the triggered event. The initial value of the persistent reference count is 0.

[0007] In response to a triggering event, a triggering window corresponding to the triggering event is determined, and a buffer block within the triggering window is locked. The triggering window includes a pre-triggering window that traces back from the time the triggering event occurred and a post-triggering window that extends backward.

[0008] The original data in the buffer block within the trigger window is written from the circular buffer to the second memory. After the original data in the buffer block is written to the second memory, the persistent reference count corresponding to the buffer block is incremented by 1, and an index entry is created for the trigger event in the global trigger index table. The index entry includes a list of logical blocks of the buffer block within the trigger window. The list of logical blocks includes physical block address pointers of each buffer block within the trigger window in the second memory, and a local sequence number of each buffer block in the current trigger event. The local sequence number is used to characterize the relative position of the corresponding buffer block in the timing of the current trigger event.

[0009] When there are multiple triggering events and their corresponding triggering windows overlap, only one write operation is performed on the original number in the buffer block within the overlapping area. The persistent reference count is increased for the buffer block within the overlapping area according to the number of corresponding triggering events. The buffer blocks within the overlapping area share the same physical storage address in the index entries corresponding to each triggering event in the global triggering index table.

[0010] In some embodiments, the buffer block status includes free, filling, filled, locked for writing, shared to disk, and reclaimable;

[0011] When the persistent reference count is greater than 0, the buffer block is in the state of being shared on disk, and the buffer block in the state of being shared on disk is prohibited from being physically overwritten.

[0012] In some embodiments, when the triggering event is successfully parsed or deleted by the user, the persistent reference count of all the corresponding buffer blocks is decremented by 1; when the persistent reference count of the buffer block is reduced to 0, the state of the buffer block changes from being shared to being reclaimable.

[0013] In some embodiments, when the buffer block is in a disk-shared state for a predetermined time, the persistent reference count of the buffer block is reduced to 0 and it is converted to a reclaimable state.

[0014] In some embodiments, when the number of buffer blocks in the disk-shared state exceeds a first threshold, the persistent reference count of the buffer blocks is reduced to 0 and they are converted to a reclaimable state in the order in which the buffer blocks entered the disk-shared state.

[0015] In some embodiments, the step of determining a trigger window corresponding to a trigger event and locking a buffer block within the trigger window in response to a trigger event includes:

[0016] The priority of the triggering event is determined based on the triggering reason, and the size of the triggering window corresponding to the triggering event is determined based on the priority of the triggering event; the priority includes a first priority, a second priority, and a third priority;

[0017] When the number of buffer blocks shared by the locked transcription and the already written disk exceeds the second threshold, the size of the trigger window corresponding to the trigger event of the second priority and the third priority is reduced;

[0018] When the number of buffer blocks that are in the locked transcription and those that have been written to disk exceeds a third threshold, a buffer pressure warning signal is issued.

[0019] In some embodiments, a data recovery step is also included:

[0020] When a data read request for a target trigger event is received, the global trigger index table is retrieved to obtain the corresponding list of logical blocks.

[0021] Based on the physical block address pointer in the logical block list, the data of the corresponding buffer block is read from the second memory;

[0022] Based on the local sequence number in the logical block list, the data of each buffer block read is reassembled and restored to recover the timing data corresponding to the target triggering event.

[0023] Secondly, embodiments of this application also provide a data recording device, which includes:

[0024] A first memory, wherein the first memory is provided with a circular buffer for storing raw data;

[0025] A trigger determination module is configured to respond to a trigger event, determine the trigger window corresponding to the trigger event, and lock the original data within the trigger window;

[0026] The second memory is used to store the original data within the trigger window.

[0027] Thirdly, embodiments of this application also provide an electronic device, including a memory and a processor, wherein the memory stores a computer program that can run on the processor, and the computer program, when executed by the processor, implements the above-described data recording method.

[0028] Fourthly, embodiments of this application also provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described data recording method.

[0029] The beneficial effects of this application include at least the following:

[0030] 1. In the data recording method of this application embodiment, the corresponding data is saved based on the triggering of the triggering event, which reduces the amount of data written, improves the life of the memory, and reduces the thermal load and maintenance cost of the storage system; the context data corresponding to the triggering event is completely saved through the pre-trigger window and the post-trigger window, which facilitates the analysis of the triggering event.

[0031] 2. In the data recording method of this application embodiment, the buffer block is protected by a persistent reference count set in the buffer block header and the associated disk-written shared state, which prevents the relevant buffer block from being overwritten before the trigger event is processed. This ensures that the original data can be retrieved from the buffer block again when the original data written into the second memory becomes unusable due to data errors or other accidents, effectively guaranteeing that the processing and analysis of the trigger event can proceed smoothly.

[0032] 3. In the data recording method of this application embodiment, by using preset time recycling rules and threshold recycling rules for the number of shared state buffer blocks already written to disk, it is possible to prevent data loss caused by the buffer blocks being protected by a large number of protected buffer blocks due to the failure to process triggered events such as failure of upper-layer analysis tools.

[0033] 4. In the data recording method of this application embodiment, by performing only one write operation on the original data in the buffer block in the overlapping area, the amount of data written is further reduced while ensuring complete recording of the data related to the triggering event, thereby reducing the lifespan consumption of the second memory.

[0034] 5. In the data recording method of this application embodiment, by establishing a local sequence number corresponding to each buffer block in the index entry corresponding to the triggering event, the correct data time series can be reconstructed according to the local sequence number during data recovery, ensuring the correctness of the playback order when multiple triggering events share data blocks. Attached Figure Description

[0035] Figure 1 This is a flowchart illustrating the data recording method according to an embodiment of this application;

[0036] Figure 2 This is a schematic diagram of a circular buffer according to an embodiment of this application;

[0037] Figure 3 This is a schematic diagram of the data writing process in the data recording method of this application embodiment;

[0038] Figure 4 This is a schematic diagram of the data recovery process in the data recording method of this application embodiment;

[0039] Figure 5 This is a schematic diagram of the structure of a data recording device according to an embodiment of this application;

[0040] Figure 6 This is a schematic diagram of the structure of an electronic device according to an embodiment of this application. Detailed Implementation

[0041] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. The components of the embodiments of this application described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely represents selected embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0042] In the description of this application, terms such as "comprising," "having," and "having" are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or apparatuses. Those skilled in the art will understand the specific meaning of the above terms in this application according to the specific circumstances. Furthermore, in the description of this application, unless otherwise stated, "a plurality of" means two or more.

[0043] Flowcharts are used in this application to illustrate the operations performed by the methods and apparatus according to embodiments of this application. It should be understood that the preceding or following operations are not necessarily performed in exact order. Instead, various steps can be processed in reverse order or simultaneously as needed. Furthermore, other operations can be added to these processes, or one or more steps can be removed from them.

[0044] When testing DRAM (Dynamic Random Access Memory), data generated during testing is typically sent to a host computer (e.g., a mainframe computer) via a data acquisition device. The host computer then analyzes the test data to obtain the test results. For example, during DRAM reliability analysis or system stress testing, it is necessary to reproduce, analyze, and trace production anomalies caused by errors, protocol anomalies, or human-induced events during the testing process. During high-speed monitoring of DRAM, the DRAM data acquisition device continuously outputs a large amount of raw read and write data, with the data volume reaching terabytes per minute (TB / minute). While continuously writing all the test data to an SSD array would provide a complete record, it would rapidly increase the total amount of SSD data written. Long-term continuous writing may significantly accelerate SSD media wear and shorten the memory's lifespan.

[0045] The key to data analysis is usually not to save all normal data, but to save sufficient context around errors, protocol anomalies, or human-triggered events. Data before the trigger point helps analyze the anomaly formation process; data after the trigger point helps determine whether the anomaly has spread, recovered, or recurred. Based on this, in one embodiment of this application provided by the inventors, raw data is written into a circularly overwriteable buffer in a first memory, and only when a specific trigger event occurs is the relevant raw data written to a second memory for data analysis and event processing. The method provided by this embodiment saves corresponding data based on the triggering of the event, reducing the amount of data written, thereby overcoming the problem of accelerated memory media wear caused by long-term continuous writing of large amounts of data in the prior art, improving memory lifespan, and reducing the thermal load and maintenance costs of the storage system.

[0046] Figure 1 This is a flowchart illustrating a data recording method 100 according to an embodiment of this application. The data recording method 100 includes steps S10 to S30. Of course, the method may also include more or fewer operations, and this application is not limited thereto.

[0047] Step S10: Continuously write the raw data into the preset circular buffer in the first memory.

[0048] The first memory uses devices with virtually unlimited write cycles or high durability, including but not limited to: DRAM (Dynamic Random Access Memory), FeRAM (Ferroelectric Random Access Memory), SRAM (Static Random-Access Memory), and MRAM (Magnetoresistive Random Access Memory). Based on the virtually unlimited write cycles or high durability of the first memory, a circular buffer is set up. Original data is continuously written to the circular buffer and cyclically overwritten. Even with long-term continuous writing of large amounts of data, the memory's lifespan will not be affected.

[0049] In some embodiments, the first memory is DRAM, which may be located on the motherboard of the host computer. The data acquisition device is connected to the host computer and writes raw data directly into the circular buffer in the DRAM via DMA (Direct Memory Access). Specifically, the raw data output by the data acquisition device enters the host server through the PCIe DMA receiving module. The host writes the physical address or IOMMU mapping address of the circular buffer into the DMA descriptor table. The PCIe DMA receiving module corresponding to the DRAM data acquisition device only needs to write according to the descriptor table in a loop, without requesting new host memory during the recording process. The PCIe DMA receiving module writes the data directly into the circular buffer in the host DRAM. Fixed pages, large pages, or pre-registered DMA memory are preferably used to avoid repeated memory allocation or repeated mapping during the recording process.

[0050] The write pointer, protect pointer, and release pointer constitute the core pointers of the circular buffer. The write pointer indicates the location of the next DMA write; the protect pointer indicates the earliest location that is still not overwriteable; and the release pointer indicates the location that can be released after the write is complete and the persistent reference count reaches zero. All three pointers are represented by the buffer block number and wraparound count to avoid ambiguity after wrapping.

[0051] In some embodiments, the circular buffer consists of multiple buffer blocks, each with a fixed or variable-length payload area and a header. The buffer block header records the buffer block status, global sequence number range, timestamp range, data semantic category, DDR address field, valid bitmap, checksum, and persistent reference count. The payload area stores the original read / write data, command address sequence, or other original records configured by the system. Buffer blocks are reused in circular order, and the oldest unprotected buffer block can be overwritten by new DMA data.

[0052] A circular buffer can be divided into multiple buffer blocks of a fixed size, or multiple pages can be dynamically combined to form a buffer block according to the data block size. Fixed buffer blocks facilitate state management and address calculation; dynamic buffer blocks are easier to adapt to different DRAM transaction lengths. The size of the buffer block can be set according to the data bandwidth and the trigger window. If the buffer block is too small, the number of state management and indexes increases; if the buffer block is too large, the trigger window boundaries are not fine enough, and more irrelevant data may be stored. Preferably, the buffer block size is set to an alignment size that is friendly to sequential writes to the second memory.

[0053] In some embodiments, the size of the circular buffer can be dynamically expanded as needed. Specifically, new buffer blocks can be dynamically added, or the storage space of existing buffer blocks can be dynamically expanded.

[0054] In some embodiments, the host DRAM (first memory) has a capacity of 512GB, and the circular buffer is divided into multiple 4MB buffer blocks.

[0055] In other embodiments, the host DRAM (first memory) has a capacity of 1TB, and the circular buffer dynamically allocates buffer blocks according to the DDR transaction length.

[0056] It should be noted that, regardless of the type of buffer block used, each buffer block is bound to the data stream sequence through a global sequence number and a timestamp.

[0057] The information recorded in the buffer block header includes: a global sequence number range to record the global sequence number interval covered by the data within the buffer block, facilitating sequential processing, deduplication, or merging of data streams; a timestamp range to indicate the minimum and maximum timestamps of the data in the buffer block, used for time range queries and data sorting; a data semantic category to indicate the data type stored in the buffer block (such as logs, indexes, materialized views, etc.), allowing upper-layer components to perform differentiated processing by category; a DDR address field to describe the starting address and length of the buffer block in DDR memory, allowing DMA or CPU to directly access the physical storage location; a valid bitmap to mark in bits whether each sub-unit (such as a record or sector) within the buffer block contains valid data, supporting fast leak detection and partial write-back; and a checksum to store the verification results (such as CRC and parity checks) of the header and data, used to verify the integrity of the buffer block during transmission or storage.

[0058] Buffer block states are used to manage the lifecycle and allocation / reclaim of buffer blocks, including free, filling, filled, locked and being written to disk, shared, and reclaimable. Specifically, when the persistent reference count is greater than 0, the buffer block is in the shared-to-disk state, and buffer blocks in this state cannot be physically overwritten.

[0059] The persistent reference count is initially 0. After the original data in the buffer block is written to the second memory, the persistent reference count corresponding to that buffer block is incremented by 1. For buffer blocks in overlapping regions, the persistent reference count is incremented based on the number of triggering events corresponding to that buffer block. For example, when a buffer block is first marked as "needs to be saved" by the triggering window of a certain triggering event and is actually written to the second memory, its persistent reference count is incremented by 1. If the triggering window of another subsequent triggering event also contains the buffer block, the data of the buffer block will not be written to the second memory again, but only its persistent reference count will be incremented by 1.

[0060] It should be noted that persistent reference counting refers to a memory management strategy that combines the "persistence" feature with traditional reference counting techniques. Traditional reference counting is a basic automatic memory management technique that maintains a counter for each object to track how many other objects or variables are "referring" to it. When the counter drops to 0, it indicates that the object is no longer in use, and the memory it occupies is immediately reclaimed. In contrast, the persistent reference counting in this application is a proactive protection mechanism. It ensures that the buffer block remains in a protected state (shared on disk) until a "reference release instruction" is received, preventing it from being physically overwritten and ensuring that the original data in the buffer block is not mistakenly cleaned up by the system.

[0061] For example, during the process of writing the original data in the buffer block to the second memory, the second memory will continuously read (i.e., reference) the buffer block. In traditional reference counting, once all the original data in the buffer block has been written to the second memory, the second memory will no longer perform read operations (i.e., no longer reference the buffer block), and the buffer block will become reclaimable (i.e., it can be overwritten). However, in this application, through persistent reference counting, even if other modules (including but not limited to the second memory) no longer reference the buffer block, as long as its persistent reference count is not cleared, the buffer block will remain in a protected state until the persistent reference count is cleared by a reference release instruction, at which point the buffer block will become reclaimable.

[0062] In some embodiments, after a triggering event is processed, a reference release command is issued, decrementing the persistent reference count of all corresponding buffer blocks by 1. When the persistent reference count of a buffer block reaches 0, the buffer block's state changes from shared to reclaimable. Processing a triggering event means that all original data corresponding to that triggering event is successfully parsed or the triggering event is confirmed for deletion by the user. Specifically, the system provides a reference release interface for upper-layer analysis tools or lifecycle management modules to call. When a triggering event referencing a buffer block is processed, the upper-layer analysis tool or lifecycle management module issues a reference release command, and the system decrements the persistent reference count of that buffer block by 1.

[0063] By using persistent reference counting to ensure that buffer blocks are not automatically reclaimed by the system before being explicitly released, it is guaranteed that the original data in the buffer block can be successfully parsed after being written to the secondary memory. For example, during the process of writing the original data to the secondary memory, there may be actual errors or other unexpected events such as software contention for resources that prevent the triggered events from being analyzed or processed. Therefore, the buffer block is not released immediately after being written to the secondary memory, but only after all the triggered events have been successfully parsed or the user has actively released it. The persistent reference count is then set to 0, and the buffer block is released.

[0064] In some embodiments, to prevent the buffer block from being protected for an extended period due to unprocessed trigger events, thus leading to insufficient capacity of the circular buffer, a reference release command is issued when the buffer block has been in a shared state on disk for a predetermined time. This reduces the persistent reference count of the buffer block to 0 and places it in a reclaimable state. The predetermined time can be set according to the maximum duration of trigger event processing, and is not limited to setting the predetermined time to be greater than the maximum duration of trigger event processing.

[0065] In some embodiments, when the number of buffer blocks in the disk-shared state exceeds a first threshold, the persistent reference counts of the buffer blocks are sequentially reduced to 0 and they are converted to a reclaimable state according to the order in which they entered the disk-shared state (i.e., the buffer blocks that entered the disk-shared state first are released first), until the number of buffer blocks in the disk-shared state falls below the first threshold. The first threshold may be, for example, 50%. When the number of buffer blocks in the disk-shared state exceeds 50% of the total number, the persistent reference counts of the buffer blocks are sequentially reduced to 0 and they are converted to a reclaimable state.

[0066] In step S10, the buffer block is protected by a persistent reference count set in the buffer block header and its associated disk-written shared state. This prevents the relevant buffer block from being overwritten before the trigger event is processed, ensuring that the original data can be retrieved from the buffer block again if data errors or other unforeseen circumstances render the original data written to the second memory unusable. This effectively guarantees the smooth processing and analysis of the trigger event. Furthermore, preset time-based recycling rules and disk-written shared state buffer block quantity threshold recycling rules prevent data loss caused by a large number of buffer blocks being protected and unable to write new data if the trigger event is not processed in time due to reasons such as upper-layer analysis tool failure.

[0067] Step S20: In response to the trigger event, determine the trigger window corresponding to the trigger event and lock the buffer block within the trigger window.

[0068] Triggering events can be triggered by hardware flags, software rules, and / or external ports. Hardware flags include data inconsistency, CRC errors, abnormal bus status, sampling loss, and front-side FIFO overflow. Software rules include abnormal command sequences, access to specific address ranges, bank conflict modes, and abnormal temperature or voltage-related risk scores. External port triggers can originate from GPIO, test programs, or user interfaces.

[0069] In some embodiments, in order to ensure that the trigger window can accurately save the relevant original data, the timestamp of the trigger event will be accurately located according to different trigger sources.

[0070] For external port triggering, a hardware timestamp is first generated by the FPGA board connected to the external port before the external trigger signal enters the host. Then, the host software maps the timestamp to the circular buffer block. In this way, even if there is a millisecond delay in the host software response, the original data before the external event occurs can be accurately saved.

[0071] For software triggering, when the user interface or test program sends a trigger command, the host records the command reception time and locates backward from the most recent DMA completion sequence number. If the software trigger needs to precisely correspond to a specific step in the program under test, the test program can simultaneously write an event flag so that the trigger event index entry includes the user-defined step name.

[0072] Please see Figure 2 In some embodiments, the trigger window includes a pre-trigger window that traces back from the time the trigger event occurred, and a post-trigger window that extends backward from the time the trigger event occurred. The original data in the pre-trigger window can be used to analyze the anomaly formation process; the original data in the post-trigger window can be used to determine whether the anomaly has spread, recovered, or recurred. By combining the pre-trigger window and the post-trigger window to form a complete trigger window, sufficient context data can be saved when the trigger event occurs, facilitating the processing and analysis of the trigger event.

[0073] In some embodiments, the trigger window size can be configured according to time, data volume, or number of transactions. A transaction refers to a complete read or write operation, consisting of a sequence of commands such as ACT (Activate), READ / WRITE, and PRECHARGE. The trigger window size can be configured, for example, according to time (5 seconds, 10 seconds, or 20 seconds before and after the trigger point); according to data volume (several GB of raw data before and after the trigger point); or according to the number of transactions (several cycles of ACT, READ, WRITE, and PRECHARGE commands).

[0074] In some embodiments, the size of the post-trigger window can be related to the type of triggering event. For example, for transient data inconsistency, the post-trigger window can be relatively short to confirm whether the error is repeated; for power disturbances, the post-trigger window can be longer to observe the recovery process; for user-manual triggering, a fixed recording duration can be configured by the user.

[0075] Optionally, the trigger window can be configured according to time, with the pre-trigger window configured to be 10 seconds and the post-trigger window configured to be 20 seconds.

[0076] Optionally, the trigger window can be automatically expanded to the most recent refresh boundary or Rank switching boundary according to the command sequence.

[0077] When a trigger event occurs, the system locates the buffer block in the circular buffer based on the global sequence number corresponding to the trigger point, marks the pre-trigger window forward and the subsequent trigger window backward, and transitions these buffer blocks from the reclaimable state to the locked transcription state. Specifically, to facilitate the sequential writing of raw data to the second memory, when the trigger window boundary is inside the buffer block, it is written as a complete buffer block, but the valid analysis interval is recorded in the index. When the trigger window boundaries of multiple trigger events are all inside the same buffer block, the entire buffer block is considered an overlapping region.

[0078] In some embodiments, to prevent false triggering, step S20 further includes: when a triggering event occurs, temporarily protecting the buffer block in the triggering window; determining whether the triggering event is a valid triggering through a user confirmation window; if the triggering event is a valid triggering, locking the buffer block; if the triggering event is not a valid triggering (false triggering), restoring the buffer block to a recyclable state.

[0079] In some embodiments, to further ensure system stability and prevent data loss caused by buffer blocks being heavily protected and unable to be written to, step S20 further includes:

[0080] The priority of a triggering event is determined based on its triggering cause, and the size of the corresponding triggering window is determined based on the priority of the triggering event.

[0081] Priorities are categorized into first, second, and third priorities. For example, data inconsistency triggers fall under the first priority, requiring the complete saving of both the pre-trigger and post-trigger windows; command exceptions fall under the second priority, requiring the saving of the command context and related raw data; and periodic sampling falls under the third priority, where the frequency can be reduced or only a summary can be saved when the second memory is under strain or the ring buffer is under high pressure.

[0082] When the number of buffer blocks under lock-and-write and those already written to disk exceeds the second threshold, the size of the trigger window corresponding to the second and third priority trigger events is reduced. The second threshold can be, for example, 70%. When the number of protected buffer blocks exceeds the second threshold, periodic sampling will be paused, meaning that trigger windows will no longer be set for third priority trigger events; at the same time, the post-trigger window for second priority trigger events will be shortened.

[0083] When the number of buffer blocks under lock-and-write operations and those already written to disk exceeds a third threshold, a buffer pressure warning signal is issued. The third threshold could be, for example, 80%. When the number of protected buffer blocks exceeds this threshold, the system reports the buffer pressure to the user and may stop new non-critical records.

[0084] Step S30: Write the original data in the buffer block within the trigger window from the circular buffer to the second memory, and create an index entry for the trigger event in the global trigger index table; wherein, when there are multiple trigger events and their corresponding multiple trigger windows overlap, only one write operation is performed on the original data in the overlapping area. The persistent reference count of the buffer block in the overlapping area is increased according to the number of trigger events, and the buffer blocks in the overlapping area share the same physical storage address in the index entries corresponding to each trigger event in the global trigger index table.

[0085] The second storage device can be any type of storage device, such as DRAM, SSD (Solid State Drive), HDD (Hard Disk Drive), Network Attached Storage (NAS), Storage Area Network (SAN), etc., without limitation.

[0086] In some embodiments, the second memory employs a non-volatile memory device.

[0087] In some embodiments, the second memory is an SSD, and data in the buffer block can be written to the second memory in a block-sequential manner using user-space NVMe, direct I / O, asynchronous I / O, or other mechanisms that can provide block sequential writes.

[0088] In some embodiments, when writing the raw data within the trigger window from the buffer block to the second memory, an index entry corresponding to the trigger event is simultaneously created in the global trigger index table. Specifically, a global trigger index table can be maintained in a fixed area of ​​the second memory or in a separate index file, and each index entry in the global trigger index table corresponds to a trigger event.

[0089] The index entries contain the unique identifier of the trigger event (Trigger ID), the trigger reason, the trigger time, the global sequence number range of the pre-trigger window, the global sequence number range of the subsequent trigger window, the check value, and the playback order.

[0090] The index entry also contains a logical block list of buffer blocks within the trigger window; the logical block list includes pointers to the physical block addresses of each buffer block in the second memory, as well as the local sequence number of each buffer block in the current trigger event. The local sequence number is used to characterize the relative position of the corresponding buffer block in the timing of the current trigger event. The local sequence number is mapped based on the sequential arrangement of the global sequence numbers of each buffer block.

[0091] When multiple trigger events share the same buffer block, each entry in the global trigger index table points to the same location on the second memory via a physical block address pointer. Using local sequence numbers, data order disruption during playback of multiple trigger events due to sharing the same buffer block can be avoided. After reading the local trigger index table, the offline analysis software can independently reconstruct the correct data time series for each trigger event, without relying on the original physical storage order of the data on the second memory.

[0092] Please see Figure 3 Before writing the original data within the trigger window from the circular buffer to the second memory, an overlap detection step is also included:

[0093] Confirm the set of buffer blocks within the trigger window, iterate through each buffer block in the set, and query its current persistent reference count;

[0094] If the persistent reference count is 0, add the buffer block to the "to be written list";

[0095] If the persistent reference count is greater than 0, skip the data writing, only record the physical address pointer of the buffer block and its local sequence number in the event into the corresponding index entry of the global trigger index table, and increment the persistent reference count of the buffer block by 1.

[0096] After completing the overlap detection step, for buffer blocks in the write list, adjacent buffer blocks are merged into a large write request and written to the pre-allocated record file or raw block device in the second memory, and its status is updated to "shared on disk" and the persistent reference count is incremented by 1.

[0097] To avoid blocking the DMA receive path when writing data to the second memory, a read-only reference is used when reading the protected buffer block in the circular buffer during the write process, without moving the DMA write pointer.

[0098] In some embodiments, to support rapid retrieval, a multi-level directory can be created for triggering events, organized by time, triggering reason, DDR address domain, and priority. When a user queries an anomaly for a specific address range or a specific bank, it is not necessary to scan all original files; only the corresponding directory needs to be located and the relevant trigger window needs to be read.

[0099] In some embodiments, to ensure the integrity of the trigger window data, a small amount of header and tail data is read back immediately after the original data is written to the second memory for quick verification, or a full verification is performed during idle time. The verification result is written to the index entry of the trigger event. If the verification fails, the system can rewrite the buffer block to the spare area and update the physical block address pointer in the logical block list.

[0100] In step S30, by performing only one write operation on the original data of the overlapping area, the amount of data written is further reduced while ensuring complete recording of the data related to the triggering event, thus reducing the lifespan of the second memory. By establishing a local sequence number corresponding to each buffer block in the index entry corresponding to the triggering event, the correct data time series can be reconstructed based on the local sequence number during data recovery. Furthermore, when the triggering event shares a buffer block, it does not depend on the physical address order of the data, preventing data playback chaos and improving system stability.

[0101] Please see Figure 4 In some embodiments, the data recording method 100 further includes data recovery steps S41 to S44:

[0102] Step S41: Receive a data read request for the target trigger event.

[0103] Step S42: Retrieve the global trigger index table to obtain the corresponding list of logical blocks;

[0104] Step S43: Read the data of the corresponding buffer block from the second memory according to the physical block address pointer in the logical block list;

[0105] Step S44: Based on the local sequence number in the logical block list, reassemble and restore the data of each buffer block to recover the timing data corresponding to the target trigger event.

[0106] In some embodiments, different content is configured to be saved for trigger events of different priorities. First-priority trigger events (e.g., data inconsistency) save the original read / write data, related command addresses, and comparison results; second-priority trigger events (e.g., command exceptions) save the command addresses and a shorter window of original data; third-priority trigger events (e.g., periodic sampling) save only compressed summaries or short window samples.

[0107] In some embodiments, the second memory write queue can be sorted by priority. Furthermore, when the ring buffer pressure is too high, trigger windows for third-priority trigger events that have not yet been written can be discarded, but trigger windows for confirmed first-priority and second-priority trigger events must not be discarded.

[0108] In some embodiments, when a new triggering event overlaps with an existing triggering window or the distance is less than the merging threshold, the new triggering event and the existing triggering window are merged into an extended window, and multiple triggering reasons are retained in the global triggering index table; when there are multiple triggering events and their corresponding multiple triggering windows overlap, for triggering events with different priorities among the multiple triggering events, the saving strategy corresponding to the higher priority triggering event is adopted.

[0109] Specifically, for consecutively triggered events, the system employs an event merging strategy. When a new triggered event overlaps with an existing triggered window or the distance is less than the merging threshold, the triggered window of the new event is merged with the existing triggered window into a single extended window, and multiple trigger reasons are retained in the corresponding index entry of the global trigger index table. If the two triggered events have different priorities, the extended window adopts the preservation strategy corresponding to the higher-priority triggered event. The preservation strategy includes the triggered window size, the content to be preserved, and the write priority, ensuring that the data integrity of important triggered events is not diluted by lower-priority events. The merging threshold is dynamically set based on the type of triggered event and the system load. For example, for data inconsistency-related triggered events, the merging threshold can be set to a smaller value, such as 100ms, to accurately capture each abnormal event; for periodic sampling-related triggered events, the merging threshold can be set to a larger value, such as 1s, to reduce the number of unnecessary index entries.

[0110] By using an event merging strategy, multiple triggering events that are close in time are managed together, reducing the number of index entries in the global trigger index table and lowering index storage overhead. At the same time, by retaining multiple triggering reasons and adopting a higher priority saving strategy, it is ensured that no key abnormal information is missed in the analysis of the merged events.

[0111] In some embodiments, the DRAM data acquisition device is connected to the host and writes raw data directly into a circular buffer in DRAM via Direct Memory Access (DMA). The circular buffer is divided into multiple buffer blocks of a fixed size, with the buffer block size set to an alignment size that is friendly to SSD sequential writes; or the circular buffer dynamically allocates buffer blocks according to the DDR transaction length.

[0112] The first memory is a dynamic random access memory (DRAM), which is located on the motherboard. The DRAM data acquisition device is connected to the host and writes the raw data directly into the circular buffer in the DRAM through the PCIe DMA receiving module.

[0113] The host writes the physical address or IOMMU mapped address of the circular buffer into the DMA descriptor table. The PCIe DMA receive module writes cyclically according to the DMA descriptor table, without requesting new host memory during the recording process. Fixed pages, large pages, or pre-registered DMA memory are preferably used to avoid redundant memory allocation or mapping during recording. The circular buffer is divided into multiple buffer blocks of a fixed size, each with a fixed or variable-length payload area and a buffer block header. The size of the buffer block is set according to the data bandwidth and trigger window. Preferably, the buffer block size is set to an alignment size that is friendly to sequential writes to the second memory to achieve optimal performance during subsequent large-block sequential writes to the SSD. For example, setting the buffer block size to 4MB matches the 4MB aligned write unit of the SSD, reducing write fragmentation.

[0114] In other embodiments, the circular buffer dynamically allocates buffer blocks according to the DDR transaction length and dynamically combines multiple pages to form a buffer block, in order to adapt to different DRAM transaction lengths and improve storage efficiency.

[0115] Direct writing via PCIe DMA avoids CPU involvement in data transfer, reducing CPU load and improving data transfer efficiency. By setting the buffer block size to an alignment size that is friendly to sequential writing to the SSD, optimal performance for subsequent sequential writes to the SSD is ensured, reducing write amplification effects.

[0116] In some embodiments, when writing raw data to the second memory, the writing rhythm is adjusted according to the temperature of the second memory. When the temperature exceeds a first temperature threshold, the system reduces the writing rate or frequency and increases the writing batch; when it exceeds a second temperature threshold, the writing of raw data for third-priority trigger events is paused; when it exceeds a third temperature threshold, only the trigger window of the first-priority trigger events is saved and an alarm is generated.

[0117] In some embodiments, the second memory includes multiple different types of memory. That is, the second memory is composed of a combination of various storage devices, such as SSDs, HDDs, network-attached storage, and storage area networks. The system writes the raw data of triggering events of first and second priority to the local SSD array, and writes the raw data of triggering events of third priority to the HDD, network-attached storage, or storage area network. This allows high-speed local SSD resources to be concentrated on exception windows that actually require the raw data, and also reduces the cost of the second memory.

[0118] In some embodiments, the first memory includes multiple circular buffers. Specifically, it includes a first circular buffer, a second circular buffer, and a third circular buffer. The first circular buffer stores the most recent few seconds of raw data at full speed, the second circular buffer stores longer-term raw data that has been downsampled or compressed, and the third circular buffer stores a long-term statistical summary. When a trigger event occurs, data from one or more circular buffers is selected and written to the second memory according to the trigger priority.

[0119] In some embodiments, the first memory includes multiple circular buffers. Specifically, for multi-channel or multi-board systems, circular buffers are established according to different DRAM channels, different data semantic categories, or different FPGA input queues. For example, raw read data, raw write data, command address data, and error records can each have independent circular buffers. When a trigger event occurs, the trigger cause determines whether to link and protect multiple partitions within the same time window; when linking and protecting, the trigger window will include raw data from other related circular buffers within the same time period for subsequent analysis of cross-channel correlations.

[0120] like Figure 5 As shown, based on the data recording method 100 described in the foregoing embodiments of this application, this application also provides a data recording device 200, which includes:

[0121] The first memory 210 has a circular buffer for storing raw data.

[0122] Trigger judgment module 220 is configured to respond to a trigger event, determine the trigger window corresponding to the trigger event, and lock the original data in the trigger window;

[0123] The second memory 230 is used to store the original data in the trigger window.

[0124] like Figure 6 As shown, this application also proposes an electronic device 300, which includes a memory 310 and a processor 320. The processor 320 and the memory 310 are communicatively connected. The memory 310 stores a computer program that can run on the processor 320. When the computer program is executed by the processor 320, it implements the data recording method 100 of the foregoing embodiments of this application.

[0125] Processor 320 can be a Central Processing Unit (CPU). Processor 320 can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or combinations thereof. The general-purpose processor can be a microprocessor or any conventional processor.

[0126] The memory 310, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs, non-transitory computer-executable programs, and modules, such as the program instructions / modules corresponding to the data recording method 100 in the embodiments of this application. The processor 320 executes various functional applications and data processing of the server by running the non-transitory software programs, instructions, and modules stored in the memory 310, thereby implementing the data recording method 100 in the aforementioned embodiments.

[0127] The memory 310 may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created according to the use of the data recording method 100, etc. Furthermore, the memory 310 may include high-speed random access memory and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, the memory 310 may optionally include memory remotely located relative to the processor 320, and these remote memories may be connected to the processor 320 via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0128] This application also proposes a computer-readable storage medium storing a computer program that, when executed by processor 320, implements the data recording method 100 of the foregoing embodiments of this application. The computer-readable storage medium may be a magnetic disk, optical disk, read-only memory (ROM), random access memory (RAM), flash memory, hard disk drive (HDD), or solid-state drive (SSD), etc. The computer-readable storage medium may also include combinations of the above-mentioned types of memory.

[0129] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit it. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

Claims

1. A data recording method, characterized by, include: The raw data is continuously written to a preset circular buffer in the first memory. The circular buffer includes multiple buffer blocks. Each buffer block includes a header and a payload area. The header includes the buffer block status, global sequence number range, timestamp range, and persistent reference count. The persistent reference count is used to record the number of times the corresponding buffer block is referenced by the triggered event. The initial value of the persistent reference count is 0. In response to a triggering event, a triggering window corresponding to the triggering event is determined, and a buffer block within the triggering window is locked. The triggering window includes a pre-triggering window that traces back from the time the triggering event occurred and a post-triggering window that extends backward. The original data in the buffer block within the trigger window is written from the circular buffer to the second memory. After the original data in the buffer block is written to the second memory, the persistent reference count corresponding to the buffer block is incremented by 1, and an index entry is created for the trigger event in the global trigger index table. The index entry includes a list of logical blocks of the buffer block within the trigger window. The list of logical blocks includes physical block address pointers of each buffer block within the trigger window in the second memory, and a local sequence number of each buffer block in the current trigger event. The local sequence number is used to characterize the relative position of the corresponding buffer block in the timing of the current trigger event. When there are multiple triggering events and their corresponding triggering windows overlap, only one write operation is performed on the original number in the buffer block within the overlapping area. The persistent reference count is increased for the buffer block within the overlapping area according to the number of corresponding triggering events. The buffer blocks within the overlapping area share the same physical storage address in the index entries corresponding to each triggering event in the global triggering index table.

2. The data recording method according to claim 1, characterized by, The buffer block status includes free, filling, filled, locked and being copied, shared to disk, and reclaimable; Specifically, when the persistent reference count is greater than 0, the buffer block is in the state of being shared on disk, and the buffer block in the state of being shared on disk is prohibited from being physically overwritten.

3. The data recording method according to claim 2, characterized by, When the triggering event is successfully parsed or the user confirms the deletion, the persistent reference count of all the corresponding buffer blocks is decremented by 1; when the persistent reference count of the buffer block is reduced to 0, the state of the buffer block changes from being shared to being reclaimable.

4. The data recording method according to claim 2, characterized by, When the buffer block is in a shared state on disk for a predetermined time, the persistent reference count of the buffer block is reduced to 0 and it is converted to a reclaimable state.

5. The data recording method according to claim 2, wherein When the number of buffer blocks in the disk-shared state exceeds a first threshold, the persistent reference count of the buffer blocks is reduced to 0 and they are converted to a reclaimable state in the order in which the buffer blocks entered the disk-shared state.

6. The data recording method according to claim 2, wherein The step of responding to a trigger event, determining the trigger window corresponding to the trigger event, and locking the buffer block within the trigger window includes: The priority of the triggering event is determined based on the triggering reason, and the size of the triggering window corresponding to the triggering event is determined based on the priority of the triggering event; the priority includes a first priority, a second priority, and a third priority; When the number of buffer blocks shared by the locked transcription and the already written disk exceeds the second threshold, the size of the trigger window corresponding to the trigger event of the second priority and the third priority is reduced; When the number of buffer blocks that are in the locked transcription and those that have been written to disk exceeds a third threshold, a buffer pressure warning signal is issued.

7. The data recording method of claim 1, wherein, It also includes data recovery steps: Received a data read request for a target-triggered event; Retrieve the global trigger index table to obtain the corresponding list of logical blocks; Based on the physical block address pointer in the logical block list, the data of the corresponding buffer block is read from the second memory; Based on the local sequence number in the logical block list, the data of each buffer block read is reassembled and restored to recover the timing data corresponding to the target triggering event.

8. A data recording apparatus, characterized by include: A first memory, wherein the first memory is provided with a circular buffer for storing raw data; A trigger determination module is configured to respond to a trigger event, determine the trigger window corresponding to the trigger event, and lock the original data within the trigger window; The second memory is used to store the original data within the trigger window.

9. An electronic device, comprising a memory and a processor, wherein the memory stores a computer program executable on the processor, characterized in that, When the computer program is executed by the processor, it implements the data recording method as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the data recording method as described in any one of claims 1 to 7.

Images