Time series data processing method and system based on memory file mapping

Abstract

The invention discloses a time series data processing method and system based on memory file mapping, and the method comprises the steps: carrying out the partitioning of time series data, so as to obtain a plurality of data blocks; a memory is divided into a main area, a temporary area and a playing area, each area comprises a plurality of slots with a fixed number, and the number of the slots corresponding to each area is allocated according to different working modes; determining a current working mode; if the mode is a real-time monitoring mode, sequentially mapping data blocks traced back at the current moment according to slot positions of a main area, and mapping the latest data blocks according to a rotation strategy; when the data blocks needing to be accessed are cold data which are not mapped in the main area, mapping and accessing are carried out by adopting the temporary area; if the playing mode is the playback mode, mapping data needing to be played back according to the idle slot position of the playing area and the playback time point; when the data blocks needing to be played are data which are not mapped in the playing area and the main area, the temporary area is adopted for mapping and playing; therefore, system calling overhead is reduced, and memory capacity limitation is broken through.

Description

Technical Field

[0001] This invention relates to the field of data storage technology, and in particular to a time-series data processing method and a time-series data processing system based on memory file mapping. Background Technology

[0002] In the field of sensor data analysis software, especially in applications that require real-time processing, storage and historical playback of large-scale, continuously sampled sensor time-series data, sensor data is usually generated continuously at a fixed frequency (e.g., hundreds to tens of thousands of times per second), forming a large amount of time-series data stream, which needs to meet the following requirements: (1) Real-time requirement: New data needs to be written quickly and analyzed in real time. (2) Historical query requirement: Data of any historical time period can be accessed efficiently. Existing processing methods include: (1) Pure memory storage scheme: All sensor time-series data are stored in RAM and accessed directly through arrays. Disadvantage: The amount of data is limited by the physical memory capacity. When the data scale increases, it is easy to exhaust memory and cannot support large-scale data storage. (2) File storage scheme: Data is stored in disk files and read and written through file I / O operations. Disadvantage: Frequent disk I / O results in slow access speed and cannot meet the time-series data processing requirements with high real-time requirements. (3) Standard memory-mapped file technology: Using the memory-mapped file interface provided by the operating system, the file is directly mapped to the process address space. Drawbacks: Lacks intelligent memory management strategies; frequent mapping / unmapping leads to high performance overhead; cannot dynamically adjust memory residency strategies based on data access frequency; lacks thread safety. Summary of the Invention

[0003] This invention aims to at least partially solve one of the technical problems in the aforementioned technologies. To this end, one objective of this invention is to propose a time-series data processing method based on memory file mapping, which dynamically selects the mapping region of data blocks according to the working mode and data usage frequency, thereby reducing system call overhead and overcoming memory capacity limitations.

[0004] To achieve the above objectives, this invention proposes a time-series data processing method based on memory file mapping, comprising the following steps: dividing the data into blocks according to the sampling rate of the time-series data to obtain multiple data blocks of fixed duration; pre-allocating memory regions to obtain a main region, a temporary region, and a playback region, wherein each region includes a fixed number of slots, the size of each slot matching the size of the data block, and allocating the number of slots corresponding to each region according to different working modes; determining the current working mode according to user instructions or running status; if the current working mode is a real-time monitoring mode, then mapping the slots of the main region sequentially to the current time-based backtracking data. Data blocks are mapped according to a rotation strategy. When a data block that needs to be accessed is cold data that is not mapped in the main area, the temporary area is used for mapping and access. If the current working mode is playback mode, the data to be played back is mapped according to the free slots in the playback area and the playback time point, and the last data block is mapped to the main area. When a data block that needs to be played back is data that is not mapped in the playback area and the main area, the temporary area is used for mapping and playback. Thus, the mapping area of ​​data blocks is dynamically selected according to the working mode and data usage frequency, thereby reducing system call overhead and breaking through memory capacity limitations.

[0005] In addition, the time-series data processing method based on memory file mapping proposed in the above embodiments of the present invention may also have the following additional technical features: Optionally, mapping the latest data block according to the round-robin strategy includes: when the latest data block is generated, finding the slot in the main area with the earliest mapping time and a reference count of 0; releasing the mapping relationship between the slot and the old data block; and mapping the latest data block to the vacated slot.

[0006] Optionally, when the data block to be accessed is mapped in the main area, its corresponding reference count is changed from 0 to 1 to ensure access security.

[0007] Optionally, when the data block to be accessed is cold data that is not mapped in the main area, the temporary area is used for mapping and access, including: finding a free slot in the temporary area to map the data block to be accessed to the free slot in the temporary area, and changing its corresponding reference count from 0 to 1; after the access is completed, the reference count is changed from 1 to 0 to automatically unmap and release the slot; when there is only one free slot left in the temporary area, the data block to be accessed is mapped to the last slot, and the data is immediately copied to the application layer buffer, and the mapping is unmapped to keep at least one free slot.

[0008] Optionally, mapping the data to be played back based on the free slots in the playback area and the playback time point includes: calculating the range of data blocks to be loaded based on the playback time point selected by the user, so as to map the corresponding data blocks to the free slots in the playback area; when the user adjusts the playback time, demapping all slots in the playback area and remapping the data blocks corresponding to the new time point, so as to ensure that the playback area always caches the data of the currently visible window.

[0009] Optionally, if the current working mode is real-time monitoring mode, the slot of the playback area is set to 0.

[0010] Optionally, if the current working mode is playback mode, the slot of the main area is set to 1 to map the last data block.

[0011] To achieve the above objectives, a first aspect of the present invention proposes a time-series data processing system based on memory file mapping, comprising: a data segmentation module for segmenting data into blocks according to the sampling rate of the time-series data to obtain multiple data blocks of fixed duration; a memory partitioning module for pre-allocating memory regions to obtain a main region, a temporary region, and a playback region, wherein each region includes a fixed number of slots, the size of each slot is matched with the size of the data block, and the number of slots corresponding to each region is allocated according to different working modes; a determination module for determining the current working mode according to user instructions or running status; and a first mapping processing module for... If the current working mode is real-time monitoring mode, the data blocks to be backed up at the current time are mapped sequentially according to the slots in the main area, and the latest data block is mapped according to the rotation strategy; when the data block to be accessed is cold data that is not mapped in the main area, the temporary area is used for mapping and access; the second mapping processing module is used to map the data to be played back according to the free slots in the playback area and the playback time point if the current working mode is playback mode, and map the last data block to the main area; when the data block to be played is data that is not mapped in the playback area and the main area, the temporary area is used for mapping and playback. Attached Figure Description

[0012] Figure 1 This is a flowchart illustrating a time-series data processing method based on memory file mapping according to an embodiment of the present invention. Figure 2 This is a schematic diagram of a memory pool design according to an embodiment of the present invention; Figure 3 This is a schematic diagram illustrating the mapping between the main area and the temporary area in the real-time monitoring mode according to an embodiment of the present invention; Figure 4 A schematic diagram illustrating the mapping of newly added data blocks in the real-time monitoring mode according to an embodiment of the present invention; Figure 5This is a schematic diagram of the mapping of playback mode data blocks according to an embodiment of the present invention; Figure 6 This is a schematic diagram of the mapping of data blocks after the playback time changes in the playback mode according to an embodiment of the present invention; Figure 7 This is a flowchart illustrating the memory pool master area mapping and rotation strategy according to an embodiment of the present invention; Figure 8 This is a block diagram of a time-series data processing system based on memory file mapping according to an embodiment of the present invention. Detailed Implementation

[0013] Embodiments of the present invention are described in detail below, examples of which are illustrated in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain the present invention, and should not be construed as limiting the present invention.

[0014] To better understand the above technical solutions, exemplary embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present invention are shown in the drawings, it should be understood that the present invention can be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided to enable a more thorough understanding of the present invention and to fully convey the scope of the invention to those skilled in the art.

[0015] To better understand the above technical solutions, the following will provide a detailed explanation of the technical solutions in conjunction with the accompanying drawings and specific implementation methods.

[0016] Figure 1 This is a flowchart illustrating a time-series data processing method based on memory file mapping according to an embodiment of the present invention. The method includes the following steps: S101, the data is divided into blocks according to the sampling rate of the time series data to obtain multiple data blocks of fixed duration.

[0017] In other words, the data is divided into blocks based on the amount of data per second (sampling rate) of the time-series data, and mapping is performed on a block-by-block basis.

[0018] S102 pre-allocates memory areas to obtain a main area, a temporary area, and a playback area. Each area includes a fixed number of slots, and the size of each slot matches the size of the data block. The number of slots corresponding to each area is allocated according to different working modes.

[0019] In other words, pre-allocated memory regions are used to map data block files. Based on usage scenarios and frequency, the memory regions are divided into: main area, temporary area, and playback area. Each region has several slots, and each slot can map one data block file. A memory partitioning diagram is shown below. Figure 2 As shown. The number of slots in each region remains constant during program execution. The functions of each partition are as follows: Main area: Used to cache frequently accessed hot data; Temporary area: Cold data used for on-demand mapping; Playback area: Dedicated to mapping the data currently being played in playback mode.

[0020] S103 determines the current working mode based on user instructions or operating status.

[0021] In other words, the determination of the working mode is mainly based on the user's operating scenario and the system's operating status. For example, if 1. the system is receiving sensor data in real time, 2. the user selects the "real-time monitoring" or "real-time viewing" function, and 3. the data is continuously generated and needs to be processed and displayed in real time, then the working mode is determined to be the real-time monitoring mode; if 1. the user selects the "historical playback" or "data playback" function, 2. the system loads the stored historical data file, and 3. the user specifies a specific playback time point or time period, then the working mode is determined to be the playback mode.

[0022] S104, if the current working mode is real-time monitoring mode, then map the data blocks back to the current time in sequence according to the slots of the main area, and map the latest data blocks according to the rotation strategy.

[0023] It should be noted that when viewing real-time data, users are most concerned with the data within the latest time period, which is usually only a few seconds. Earlier data is generally only viewed temporarily. Therefore, the latest N data blocks are mapped to N slots in the main area.

[0024] As one example, mapping the latest data block according to the round-robin strategy includes: when the latest data block is generated, finding the slot in the main area with the earliest mapping time and a reference count of 0; removing the mapping relationship between the slot and the old data block; and mapping the latest data block to the vacated slot.

[0025] In other words, when a new data block is added, the oldest data block is unmapped, and the new block is mapped to the vacated slot. The logic for adding a new block is as follows: Figure 7 As shown; see the main area mapping diagram. Figure 3 and 4 As shown.

[0026] It should be noted that when data from a mapped data block is being used, the reference count of the data block is incremented; only data blocks with a reference count of 0 can be unmapped.

[0027] S105, when the data block to be accessed is cold data that is not mapped in the main area, a temporary area is used for mapping and access.

[0028] As an example, when the data block to be accessed is cold data that is not mapped in the main area, a temporary area is used for mapping and access, including: finding a free slot in the temporary area to map the data block to be accessed to the free slot in the temporary area, and changing its corresponding reference count from 0 to 1; after the access is completed, the reference count is changed from 1 to 0 to automatically unmap and release the slot; when there is only one free slot left in the temporary area, the data block to be accessed is mapped to the last slot, and the data is immediately copied to the application layer buffer, and the mapping is unmapped to keep at least one free slot.

[0029] In other words, when viewing real-time data, if the data block containing the data to be viewed is not mapped to the main area, a free slot in the temporary area is found, and the data block is mapped to the corresponding slot. Reference counting is used to mark slot usage; when the reference count is 0, the corresponding data block is unmapped, and the memory slot in the temporary area is released. When only one available slot remains in the temporary area, the data block is mapped, its contents are copied, and then the mapping is immediately removed. This ensures that there is always at least one available temporary area slot; see the temporary area mapping diagram below. Figure 3 And Figure 4.

[0030] Additionally, when a data block that needs to be accessed is mapped in the primary area, its corresponding reference count is changed from 0 to 1 to ensure access security.

[0031] It should be noted that if the current working mode is real-time monitoring mode, the slots in the playback area are set to 0.

[0032] S106, if the current working mode is playback mode, then map the data to be played back according to the free slots in the playback area and the playback time point, and map the last data block to the main area.

[0033] In other words, when viewing playback data, the data no longer increases over time; instead, a specific time segment is selected from the existing data, typically only a few seconds. Therefore, it's necessary to locate the data block containing the data for the current playback time and the preceding period, and map this data block to the playback area. When the playback time changes, the new data block is located and remapped to ensure that the data for the currently playing segment is always within the playback area. See the playback area mapping diagram below. Figure 5 and Figure 6 .

[0034] As one example, the data to be played back is mapped according to the free slots in the playback area and the playback time point, including: calculating the range of data blocks to be loaded according to the playback time point selected by the user, so as to map the corresponding data blocks to the free slots in the playback area; when the user adjusts the playback time, the mapping of all slots in the playback area is de-mapped, and the data blocks corresponding to the new time point are remapped to ensure that the playback area always caches the data of the currently visible window.

[0035] It should be noted that if the current working mode is playback mode, the slot of the main area is set to 1 to map the last data block.

[0036] In other words, when viewing playback data, the main area only retains one slot, which is always mapped to the last data block and used to read data and create data block files.

[0037] S107: When the data block to be played is not mapped in the playback area and the main area, a temporary area is used for mapping and playback.

[0038] In other words, when viewing playback data, the temporary area function is the same as in real-time monitoring mode; when the data being searched is neither in the playback area nor the main area, the temporary area is used.

[0039] Therefore, this application uses memory file mapping technology to divide data into blocks, loads the data blocks that are in use into memory, and saves other data blocks on disk; among them, the data block mapping strategy, that is, to determine which data blocks should be loaded into memory and when, and which data blocks should be kept on disk according to the usage scenario, can be further broken down into the following key points: (1) Data block mapping strategy: dynamically select the mapping area of ​​data blocks and the timing of data block mapping and unmapping according to the working mode and data usage frequency; (2) Memory partition management: management of the memory area used to map data block files; (3) Reference counting protection: use the reference counting mechanism to protect the memory blocks that are being accessed, prevent them from being accidentally overwritten, and ensure thread safety and data consistency. To better understand the above technical solutions, let's take a data acquisition and analysis software as an example: This software receives sensor data in real time and displays the data in various types of chart controls, or applies certain algorithms to the data and allows real-time viewing of the calculation results. It also supports saving data and analyzing it in playback mode.

[0040] The detailed plan includes: 1. Develop a cache container using the C++ programming language based on the Windows API memory-mapped file interface. The cache container can accept template parameters to support data of different data types.

[0041] 2. The cache container divides the data into data blocks of the same size, and each data block is stored for 2 seconds. The cache is managed in units of data blocks.

[0042] 3. Develop a memory pool feature. The memory pool is divided into three zones, each with several slots. Each slot corresponds to a pre-allocated memory block, the size of which is the same as the data block. The number of slots remains unchanged during runtime. Detailed memory pool partition size settings: For real-time data, the number of slots in the three areas of the memory pool is set as follows:

[0043] For data playback, the number of slots in the three areas of the memory pool is set as follows:

[0044] 4. Develop a cache container to manage data blocks and implement the logic for controlling the mapping and unmapping of data blocks.

[0045] 5. Main area data block rotation strategy when caching real-time data; In real-time monitoring mode, data is continuously generated over time. The container maintains a logical pointer to the latest data block. During initialization, the eight slots of the main area are mapped sequentially to the eight data blocks that are traced back from the current moment (i.e., the eight latest 2-second data segments).

[0046] Table 1: Example of main area mapping after real-time monitoring mode initialization (assuming the latest block is Block_100)

[0047] When new 2-second data arrives, a new data block Block_101 is generated, and the container performs a rotation: Find the slot in the main area that has the earliest mapping time and a reference count of 0 (in this example, Slot 1, mapped to Block_93).

[0048] Remove the mapping relationship between Slot 1 and Block_93.

[0049] Map Block_101 to Slot 1. This first-in-first-out (FIFO) round-robin ensures that the main area always caches the latest 8 data blocks.

[0050] Table 2: Main Area Mapping Status After Rotation

[0051] 6. Data block access protection mechanism; When a thread needs to read data from Block_100 (assuming it's in Slot 8 of the main area), the container changes its reference count from 0 to 1. At this point, even if Block_100 becomes the "oldest block" over time, it won't be rounded up because its reference count is not zero, thus ensuring safe access.

[0052] Table 3: Initial State of the Temporary Zone

[0053] Scenario A: Normal on-demand mapping. A user requests to view Block_50 (cold data). The container finds Temp Slot 1, maps Block_50 into it, and sets the reference count to 1. After use, the reference count returns to 0, and the slot becomes free again.

[0054] Scenario B: Emergency Handling. Assume that 7 slots in the temporary area are already occupied, leaving only Temp Slot 8 free.

[0055] Table 4: Status of the Temporary Area Before Emergency Treatment

[0056] At this point, the user requests Block_60. The container executes: Map Block_60 to Temp Slot 8.

[0057] Immediately copy all data from Block_60 to the application layer buffer.

[0058] Immediately unmap Temp Slot 8 from Block_60, restoring it to an idle state.

[0059] This strategy ensures a minimum of "at least one free slot" to prevent the system from freezing.

[0060] 7. When caching real-time data, the playback area is disabled. At this time, the number of slots in the playback area is configured to 0, and there is no mapping table.

[0061] 8. Dynamic mapping strategy for the playback area when caching playback data; Assume the total data consists of Blocks 1 to 200. The user sets the playback time to T=150 seconds (corresponding to Block 75), and the display window is 8 seconds long (requiring four blocks: Blocks 72, 73, 74, and 75).

[0062] Table 5: Dynamic Updates of Playback Area Mapping When Playback Time Changes

[0063] 9. When buffering and replaying data, the main area function is simplified. The main area is fixed to 1 slot and always maps to the last data block (Block_200).

[0064] Table 6: Fixed mapping of the main area when cached and replaying data

[0065] 10. Temporary area auxiliary access strategy when replaying cached data; When buffering playback data (currently playing Block_88-91), the user opens a separate analysis view and requests to view Block_120.

[0066] Checking, Block_120 is neither in the playback area nor in the main area (the main area is Block_200).

[0067] Find an empty slot in the temporary area (such as Temp Slot 1) and map Block_120 into it for analysis.

[0068] Table 7: Examples of Temporary Area Access During Cache Replay Data

[0069] After the analysis is complete, the reference count of Block_120 is reset to 0, and Temp Slot 1 is released, without affecting core replay performance.

[0070] Additionally, it should be noted that the specific implementation of memory mapping can be based on either the Windows API or the Linux API, depending on the target platform. For storage space-sensitive scenarios, transparent compression can be added: block data is compressed, and a decompression process is added during reading.

[0071] In summary, this application achieves near-memory access speeds by using a memory mapping strategy to keep frequently accessed data resident in memory; it reduces unnecessary file mapping calls and system call overhead by using reference counting and partition mapping strategies; the reference counting mechanism ensures thread-safe access, thereby achieving efficient concurrent access; and it overcomes memory capacity limitations by occupying only limited physical memory.

[0072] To implement the above embodiments, this invention also proposes a time-series data processing system based on memory file mapping, such as... Figure 8 As shown, the time-series data processing system based on memory file mapping includes: a data block module 10, a memory partitioning module 20, a determination module 30, a first mapping processing module 40, and a second mapping processing module 50.

[0073] The system includes the following modules: a data segmentation module 10, which segments the data according to the sampling rate of the time-series data to obtain multiple data blocks of fixed duration; a memory partitioning module 20, which pre-allocates memory regions to obtain a main region, a temporary region, and a playback region, where each region includes a fixed number of slots, and the size of each slot matches the size of the data block. The number of slots corresponding to each region is allocated according to different working modes; a determination module 30, which determines the current working mode according to user instructions or running status; a first mapping processing module 40, which, if the current working mode is real-time monitoring mode, sequentially maps the data blocks back to the current time according to the slots in the main region, and maps the latest data block according to the round-robin strategy; when the data block to be accessed is cold data not mapped in the main region, it is mapped and accessed using the temporary region; and a second mapping processing module 50, which, if the current working mode is playback mode, maps the data to be played back according to the free slots in the playback region and the playback time point, and maps the last data block to the main region; when the data block to be played is data not mapped in the playback region and the main region, it is mapped and played using the temporary region.

[0074] It should be noted that the above description and examples of the time-series data processing method based on memory file mapping are also applicable to the time-series data processing system based on memory file mapping in this embodiment, and will not be repeated here.

[0075] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0076] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0077] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0078] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0079] It should be noted that any reference signs placed between parentheses in the claims should not be construed as limiting the claims. The word "comprising" does not exclude the presence of components or steps not listed in the claims. The word "a" or "an" preceding a component does not exclude the presence of a plurality of such components. The invention can be implemented by means of hardware comprising several different components and by means of a suitably programmed computer. In a unit claim enumerating several means, several of these means may be embodied by the same item of hardware. The use of the words first, second, and third, etc., does not indicate any order. These words can be interpreted as names.

[0080] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the invention.

[0081] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

[0082] In the description of this invention, it should be understood that the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0083] In this invention, unless otherwise explicitly specified and limited, the terms "installation," "connection," "linking," and "fixing," etc., should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral part; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; they can refer to the internal communication of two components or the interaction between two components. Those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances.

[0084] In this invention, unless otherwise explicitly specified and limited, "above" or "below" the second feature can mean that the first feature is in direct contact with the second feature, or that the first feature is in indirect contact with the second feature through an intermediate medium. Furthermore, "above," "over," and "on top" of the second feature can mean that the first feature is directly above or diagonally above the second feature, or simply that the first feature is at a higher horizontal level than the second feature. "Below," "below," and "under" the second feature can mean that the first feature is directly below or diagonally below the second feature, or simply that the first feature is at a lower horizontal level than the second feature.

[0085] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms should not be construed as necessarily referring to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0086] Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention.

Claims

1. A time-series data processing method based on memory file mapping, characterized in that, Includes the following steps: The data is divided into blocks based on the sampling rate of the time series data to obtain multiple data blocks of fixed duration; Pre-allocate memory areas to obtain the main area, temporary area and playback area. Each area includes a fixed number of slots, and the size of each slot matches the size of the data block. The number of slots corresponding to each area is allocated according to different working modes. The current working mode is determined based on user instructions or operating status; If the current working mode is real-time monitoring mode, then the data blocks that are being traced back at the current moment are mapped sequentially according to the slots of the main area, and the latest data blocks are mapped according to the rotation strategy; When the data block to be accessed is cold data that is not mapped in the main area, the temporary area is used for mapping and access. If the current working mode is playback mode, the data to be played back is mapped according to the free slots in the playback area and the playback time point, and the last data block is mapped to the main area; When the data block to be played is not mapped in the playback area and the main area, the temporary area is used for mapping and playback.

2. The time-series data processing method based on memory file mapping as described in claim 1, characterized in that, Map the latest data block according to the rotation strategy, including: When the latest data block is generated, find the slot in the main area that has the earliest mapping time and a reference count of 0; Remove the mapping relationship between the slot and the old data block; Map the latest data block to the vacated slot.

3. The time-series data processing method based on memory file mapping as described in claim 2, characterized in that, When the data block that needs to be accessed is mapped in the main area, its corresponding reference count is changed from 0 to 1 to ensure access security.

4. The time-series data processing method based on memory file mapping as described in claim 2, characterized in that, When the data block to be accessed is cold data that is not mapped in the main area, the temporary area is used for mapping and access, including: Find a free slot in the temporary area, map the data block that needs to be accessed to the free slot in the temporary area, and change its corresponding reference count from 0 to 1; After access is complete, the reference count will be changed from 1 to 0 to automatically unmap and release the slot; When only one free slot remains in the temporary area, the data block that needs to be accessed is mapped to the last slot, and the data is immediately copied to the application layer buffer. The mapping is then removed to keep at least one free slot.

5. The time-series data processing method based on memory file mapping as described in claim 1, characterized in that, The data to be replayed is mapped according to the available slots in the playback area and the playback time points, including: Based on the playback time point selected by the user, calculate the range of data blocks that need to be loaded, and map the corresponding data blocks to the free slots in the playback area; When a user adjusts the playback time, the mapping of all slots in the playback area is demapped, and the data blocks corresponding to the new time point are remapped to ensure that the playback area always caches the data of the currently visible window.

6. The time-series data processing method based on memory file mapping as described in claim 1, characterized in that, If the current working mode is real-time monitoring mode, then the slot of the playback area is set to 0.

7. The time-series data processing method based on memory file mapping as described in claim 1, characterized in that, If the current working mode is playback mode, the slot of the main area is set to 1 to map the last data block.

8. A time-series data processing system based on memory file mapping, characterized in that, include: The data segmentation module is used to segment the time-series data into blocks based on the sampling rate of the time-series data to obtain multiple data blocks of fixed duration. The memory partitioning module is used to pre-allocate memory regions to obtain the main region, temporary region and playback region. Each region includes a fixed number of slots, and the size of each slot matches the size of the data block. The number of slots corresponding to each region is allocated according to different working modes. The determination module is used to determine the current working mode based on user instructions or operating status. The first mapping processing module is used to map the data blocks back to the current time sequentially according to the slots of the main area if the current working mode is real-time monitoring mode, and to map the latest data blocks according to the round-robin strategy; when the data block to be accessed is cold data that is not mapped in the main area, the temporary area is used for mapping and access. The second mapping processing module is used to map the data to be played back according to the free slots of the playback area and the playback time point if the current working mode is playback mode, and map the last data block to the main area; when the data block to be played is data that is not mapped in the playback area and the main area, the temporary area is used for mapping and playback.

Images