Data processing method, apparatus, device, medium, and program product

Abstract

Embodiments of the present specification provide a data processing method, device, equipment, medium and program product, wherein the data processing method comprises: receiving a data access request for a target object, and determining a corresponding data management unit according to a target object identifier in the data access request; obtaining a data index corresponding to the data management unit, the data index indicating an association relationship between an object identifier and sequence description information, the sequence description information indicating region positioning information of data regions where different data segments of a same data sequence are located; if target sequence description information corresponding to the target object identifier is queried in the data index, determining a to-be-accessed data region based on a request type of the data access request and the target sequence description information, and processing the data access request based on region positioning information corresponding to the to-be-accessed data region. In this way, the complete sequence is replaced by an index to reside in memory, and the corresponding storage region is matched according to the request type, thereby reducing memory occupation and access delay.

Description

Technical Field

[0001] This specification relates to the field of computer technology, and in particular to a data processing method, apparatus, device, medium, and program product. Background Technology

[0002] With the development of online businesses such as recommendation, advertising, search, and e-commerce, ranking models increasingly rely on long-term user behavior sequences to characterize user interests. User actions such as clicks, browsing, favorites, searches, conversions, and product interactions do not occur in isolation; they often span multiple business scenarios and accumulate continuously throughout the user's entire lifecycle. For long-sequence models, if only the most recent dozens or hundreds of behaviors can be accessed, a large amount of long-term interest, periodic interest, and cross-scenario interest signals will be lost, affecting the model's judgment of the relevance of candidate content or advertisements. Therefore, online ranking chains need to have the ability to access historical behavior sequences of tens of thousands or even hundreds of thousands of users.

[0003] Currently, to meet the low-latency access requirements of online sorting systems for user historical behavior sequences, each user's complete behavior sequence is typically kept directly in memory. However, keeping the complete user sequence directly in memory incurs extremely high resource costs. Therefore, there is an urgent need for a data processing method that can balance low-cost storage, low-latency querying, and real-time writing requirements. Summary of the Invention

[0004] In view of the above, embodiments of this specification provide a data processing method. One or more embodiments of this specification also relate to a data processing apparatus, a computing device, a computer-readable storage medium, and a computer program product, to address the technical deficiencies existing in the prior art.

[0005] According to a first aspect of the embodiments of this specification, a data processing method is provided, comprising: Receive data access requests for a target object and determine the corresponding data management unit based on the target object identifier in the data access request; Obtain the data index corresponding to the data management unit. The data index indicates the association between the object identifier and the sequence description information. The sequence description information indicates the regional location information of the data area where different data segments of the same data sequence are located. If the target sequence description information corresponding to the target object identifier is found in the data index, the data area to be accessed is determined based on the request type of the data access request and the target sequence description information, and the data access request is processed based on the area location information corresponding to the data area to be accessed.

[0006] According to a second aspect of the embodiments of this specification, a data processing apparatus is provided, comprising: The management unit determination module is configured to receive data access requests for a target object and determine the corresponding data management unit based on the target object identifier in the data access request. The index acquisition module is configured to acquire the data index corresponding to the data management unit. The data index indicates the association between the object identifier and the sequence description information. The sequence description information indicates the regional location information of the data area where different data segments of the same data sequence are located. The request processing module is configured to, if the target sequence description information corresponding to the target object identifier is found in the data index, determine the data area to be accessed based on the request type of the data access request and the target sequence description information, and process the data access request based on the area location information corresponding to the data area to be accessed.

[0007] According to a third aspect of the embodiments of this specification, a computing device is provided, comprising: Memory and processor; The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions, which, when executed by the processor, implement the steps of the above-described data processing method.

[0008] According to a fourth aspect of the embodiments of this specification, a computer-readable storage medium is provided that stores computer-executable instructions, which, when executed by a processor, implement the steps of the data processing method described above.

[0009] According to a fifth aspect of the embodiments of this specification, a computer program product is provided, including a computer program / instructions that, when executed by a processor, implement the steps of the above-described data processing method.

[0010] This specification provides a data processing method in one embodiment. By setting a data index to indicate the association between object identifiers and sequence description information, only lightweight index information and sequence description information need to be stored in memory, without storing the complete data sequence. This significantly reduces memory usage, enabling online service clusters to economically deploy ultra-long sequence access capabilities under massive object scales. Simultaneously, by determining the data region to be accessed based on the request type, read and write requests can be routed to different data regions for processing, achieving separation and isolation of read and write paths, avoiding mutual blocking of read and write operations, and improving the system's concurrent processing capabilities and response speed. Furthermore, by directly processing data access requests based on region positioning information, the system can accurately locate and access the target data segment without reading the complete data sequence, avoiding unnecessary disk I / O and memory copying, and significantly reducing the access latency of online requests. Therefore, this method can achieve low-latency access and real-time writing of ultra-long data sequences while controlling memory costs, providing stable and efficient data support for sequence models in online services. Attached Figure Description

[0011] Figure 1 This is a flowchart illustrating a data processing method provided in one embodiment of this specification; Figure 2 This is a schematic diagram of the structure of a time-series indexed storage system provided in one embodiment of this specification; Figure 3 This is a schematic diagram of a data processing procedure provided in one embodiment of this specification; Figure 4 This is a flowchart illustrating the processing procedure of a data processing method provided in one embodiment of this specification. Figure 5 This is a schematic diagram of the structure of a data processing apparatus provided in one embodiment of this specification; Figure 6 This is a structural block diagram of a computing device provided in one embodiment of this specification. Detailed Implementation

[0012] Many specific details are set forth in the following description to provide a full understanding of this specification. However, this specification can be implemented in many other ways than those described herein, and those skilled in the art can make similar extensions without departing from the spirit of this specification. Therefore, this specification is not limited to the specific implementations disclosed below.

[0013] The terminology used in one or more embodiments of this specification is for the purpose of describing particular embodiments only and is not intended to be limiting of the one or more embodiments of this specification. The singular forms “a” and “the” as used in one or more embodiments of this specification and the appended claims are also intended to include the plural forms, unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items.

[0014] It should be understood that although the terms first, second, etc., may be used to describe various information in one or more embodiments of this specification, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, first may also be referred to as second without departing from the scope of one or more embodiments of this specification, and similarly, second may also be referred to as first. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to a determination."

[0015] Furthermore, it should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in one or more embodiments of this specification are all information and data authorized by the user or fully authorized by all parties. Moreover, the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0016] This specification provides a data processing method, and also relates to a data processing apparatus, a computing device, a computer-readable storage medium, and a computer program product, which will be described in detail in the following embodiments.

[0017] See Figure 1 , Figure 1 A flowchart of a data processing method according to an embodiment of this specification is shown, specifically including the following steps 102-106.

[0018] Step 102: Receive a data access request for the target object and determine the corresponding data management unit based on the target object identifier in the data access request.

[0019] Among them, a data access request is a network request initiated by the ranking link in an online business system (such as a recommendation system, advertising system, or search system) to the storage system to obtain or update the behavioral sequence data of the target object.

[0020] In practical applications, when a recommendation system needs to rank candidate content based on relevance, the ranking model requires the target object's historical behavior sequence as input features; in this case, the data access request is a read request. When the target object generates new interaction behavior (such as clicking, browsing, favoriting, purchasing, etc.), the business system needs to persistently store this behavior record; in this case, the data access request is a write request. This data access request must at least include the target object identifier and the request type identifier (e.g., read or write).

[0021] Optionally, read requests may also carry time range conditions (such as querying the most recent N behaviors or querying behaviors within a specified time period), and write requests may also carry the behavior data content to be stored.

[0022] Among them, the target object identifier is the original identity information used to uniquely identify a target object at the business level.

[0023] Specifically, the target object can be an entity with behavioral sequence data, such as a user, device terminal, merchant account, or content entry using the online business system. Correspondingly, the target object identifier can be a user ID, device ID, MAC address, IP address, mobile phone number, email address, or a user code uniformly encoded by the business system (such as OpenID or UnionID), etc. This invention does not impose specific limitations on this. It is understood that the specific format of the target object identifier depends on the actual implementation of the business system.

[0024] Step 104: Obtain the data index corresponding to the data management unit. The data index indicates the association between the object identifier and the sequence description information. The sequence description information indicates the regional positioning information of the data area where different data segments of the same data sequence are located.

[0025] The data management unit is used to manage the mapping relationship between object identifiers and sequence description information within a preset range.

[0026] A data index is a memory-stored mapping structure used to indicate the association between object identifiers and sequence description information. The data index uses the object identifier (or the internal key value obtained after mapping the object identifier) ​​as the lookup key and the storage location or reference of the corresponding sequence description information as the lookup value, thereby enabling rapid location of the target object's sequence description information.

[0027] Data indexing can be implemented using hash tables, which have a lookup time complexity of O(1), meeting the high throughput and low latency requirements of online services. Specifically, the key of the hash table is the hash value of the target object identifier (i.e., the object routing key), and the value of the hash table is a pointer or offset to the corresponding sequence description information (i.e., the UserSequence object). Since the hash table stores the mapping relationship between object identifiers and sequence description information, rather than the complete data sequence of the target object itself, the memory space occupied by the data index is extremely small, independent of the length of the data sequence, and only related to the number of objects that need to be indexed. Even if the length of the user behavior sequence reaches tens of thousands or even hundreds of thousands, only the lightweight index entry and sequence description information corresponding to the user need to be stored in memory, while the complete data sequence body is still stored on disk, thus significantly reducing memory consumption.

[0028] Understandably, data indexing can be implemented in ways other than hash tables. Skip lists, Bloom filters combined with hash tables, or ordered arrays can also be used, as long as they can support quick lookup of the corresponding sequence description information based on the object identifier.

[0029] In one optional implementation, different data segments of the same data sequence include: historical stock sequence segments that have been persistently stored on disk, real-time incremental data segments stored in memory, and hotspot cache segments loaded from disk into memory. Correspondingly, the sequence description information is a unified metadata description object of the complete data sequence of a single target object within the system, used to describe the distribution of the data sequence in the aforementioned different storage areas.

[0030] For example, the sequence description information can be specifically implemented as a UserSequence meta-object. This meta-object itself does not directly store the complete data sequence entity, but rather stores several location information points to different storage areas. Specifically, the sequence description information includes the following four types of location information: (1) Disk region location information (disk offset) is used to point to the historical data sequence that has been persisted in the disk storage region. The disk storage region is the main carrier layer of the complete data sequence, which stores the complete data that has been merged in the background and can be persisted for a long time. Even if the cached data or incremental data in memory is lost due to process restart, the system can still recover the complete data sequence from the disk through the disk region location information, thus ensuring the persistence and recoverability of the data.

[0031] (2) Cache location information (cache offset) is used to point to the hot sequence segments stored in the memory cache area. When a data segment on the disk is frequently accessed, the system can load the segment into the memory cache area and record the corresponding cache location information in the sequence description information. When accessing the same segment later, the system can read directly from the memory cache, thereby reducing disk I / O and reducing access latency.

[0032] (3) The writable region location information (mutable_delta offset) is used to point to the real-time newly added data stored in the writable incremental region of memory. When the system receives a write request, the newly added data is first written to the writable incremental region of memory without modifying the existing data on the disk. Through this mechanism, the write operation only involves a small-scale memory append operation, avoiding frequent disk read and write operations and significantly reducing write latency.

[0033] (4) Frozen region location information (immutable_delta offset), used to point to the incremental data to be merged stored in the frozen incremental region in memory. When the amount of data in the writable incremental region in memory reaches a preset threshold or meets a preset time condition, the region is switched to read-only state and becomes a frozen incremental region, waiting for the background merging thread to merge it with the existing data on disk. The frozen incremental region remains read-only during the merging process, available for query requests and background merging, thereby ensuring that write operations are not blocked during the merging process.

[0034] Through the above structure, the sequence description information associates different segments of the same data sequence, which are scattered in the disk storage area, memory cache area, memory writable incremental area and memory frozen incremental area, into a complete unified view, thereby decoupling "object location" from "data storage".

[0035] Specifically, object location is quickly accomplished through data indexing (a mapping structure in memory), ensuring access speed; the complete data sequence is stored on disk, ensuring controllable storage costs; hotspot segmentation and real-time incremental data are stored in memory, ensuring real-time performance and low latency for online queries. Foreground queries, real-time writes, background merging, and anomaly recovery can all start from the same sequence description information, accessing different storage areas pointed to by disk area location information, cache area location information, writable area location information, and frozen area location information, thus forming a unified data access entry point.

[0036] It is understood that the above four types of positioning information are all specific implementations of regional positioning information. In the embodiments of this specification, regional positioning information refers to all information used to locate the position of data segments in the storage area. Its specific form may be offset, pointer, address, or other information that can uniquely determine the storage location. The embodiments of this specification do not specifically limit this.

[0037] In the embodiments described in this specification, the data processing method can be deployed and executed within the online index service node.

[0038] In one optional implementation, the system organizes data in a hierarchical manner: "node-shard-part-actual bucket-object sequence". Specifically, each service node is responsible for carrying one or more data shards; each data shard is further divided into multiple parts; each part is responsible for a continuous or logically continuous range of actual buckets and independently manages the mapping relationship between object identifiers and sequence description information of all target objects within that range. Through this hierarchical structure, the data sequence of massive objects can be split into multiple relatively independent management units, facilitating parallel access, background merging, data recovery, and elastic scaling.

[0039] The Part is a specific implementation of a data management unit. It is understood that those skilled in the art can also implement the division of data management units in other ways, such as dividing them directly according to the hash value range of object identifiers, or according to the business attributes of object identifiers (such as region, membership level, etc.), as long as it can achieve the division of massive amounts of data into multiple independently managed data subsets. This invention does not impose specific limitations on this. Each Part independently maintains its corresponding data index (i.e., memory mapping table). This data index is used to store the mapping relationship between object identifiers and sequence description information within the scope of that Part, thereby supporting the rapid location of the sequence description information of the target object within the scope of that Part.

[0040] In one optional embodiment of this specification, determining the corresponding data management unit based on the target object identifier in the data access request includes: determining the target actual bucket corresponding to the target object identifier based on the mapping relationship between the target object identifier and the actual bucket.

[0041] In another optional embodiment of the embodiments of this specification, determining the corresponding data management unit based on the target object identifier in the data access request includes: The target mapping function maps target object identifiers to target object routing keys. Specifically, the target mapping function maps the same object identifier to the same object routing key and different object identifiers to different object routing keys. Based on the target object routing key and the set number of virtual buckets, determine the target virtual bucket corresponding to the target object routing key. The number of virtual buckets is a fixed value used to isolate the mapping relationship from object identifier to virtual bucket and the mapping relationship from virtual bucket to actual bucket. Based on the target virtual bucket and the mapping relationship between the virtual bucket and the actual bucket, the target actual bucket corresponding to the target object routing key is determined, and the data management unit to which the target actual bucket belongs is determined as the data management unit corresponding to the target object identifier. Here, the actual bucket is a storage container used to store data.

[0042] A virtual bucket is a stable logical mapping layer between an object identifier and its actual physical storage location. The number of virtual buckets is set during system initialization and remains constant during system operation. Virtual buckets themselves do not store any data; they merely act as an intermediary bridge connecting the "object identifier" and the "actual physical storage location." By setting up the virtual bucket layer, the system decouples the "mapping from object identifier to virtual bucket" from the "mapping from virtual bucket to physical bucket"—the former remains fixed, while the latter can be dynamically adjusted, thus achieving a stable mapping from object identifier to physical storage location.

[0043] An actual bucket is the smallest physical unit for storing data. It stores the mapping between object identifiers and sequence descriptions within that bucket, as well as the corresponding data sequence. Each data management unit (Part) corresponds to one or more actual buckets; that is, an actual bucket is a finer-grained data bucket within a Part. The number of actual buckets can be dynamically adjusted according to the growth of business scale (e.g., increasing the number of actual buckets during expansion), but changes in actual buckets will not affect the mapping between object identifiers and virtual buckets; only the mapping between virtual buckets and actual buckets needs to be adjusted.

[0044] It's important to note that in online indexing services, the system typically uses multiple shards and Parts to jointly host a massive sequence of object data. If the mapping method from object identifiers to Parts is poorly designed, issues such as uneven data distribution, localized hotspots, and excessive load on individual Parts can easily arise. Specifically, if a direct hash modulo method is used to map objects to actual buckets, any change in the total number of actual buckets (e.g., when the system expands and the number of actual buckets needs to be increased) will alter the divisor of the modulo operation, changing the mapping results for almost all objects. This will require a large number of object data to be migrated and reconstructed between different Parts, resulting in extremely high data migration costs and system overhead. Therefore, a stable mapping mechanism is urgently needed to ensure that object identifiers are evenly distributed across different actual buckets and Parts, and to maintain the stability of the object-to-Part mapping as much as possible when the number of actual buckets is adjusted, avoiding large-scale data reconstruction.

[0045] To address the aforementioned issues, this specification provides a stable mapping mechanism based on virtual buckets. This mechanism decouples the mapping from object identifiers to virtual buckets from the mapping from virtual buckets to physical buckets by introducing a stable logical mapping layer—the virtual bucket—between object identifiers and their actual physical storage locations. The mapping from object identifiers to virtual buckets is calculated based on a fixed number of virtual buckets and remains unchanged during system operation; the mapping from virtual buckets to physical buckets can be dynamically adjusted with system size. Therefore, when system expansion or contraction causes a change in the number of physical buckets, only the mapping between virtual buckets and physical buckets needs to be adjusted, without rehashing all object identifiers. This ensures uniform data distribution while avoiding large-scale data migration and reconstruction across parts.

[0046] Specifically, in the embodiments of this specification, determining the corresponding data management unit based on the target object identifier in the data access request includes the following steps: Step 1: Map the object identifier to the object routing key.

[0047] The system uses a target mapping function (such as a hash function, like MurmurHash3 or xxHash, which are non-cryptographic hash algorithms) to map variable-length target object identifiers (e.g., user IDs in string form) to fixed-length object routing keys (i.e., pkey_hash). This mapping function is deterministic and uniformly distributed—the same object identifier maps to the same object routing key, and different object identifiers map to different object routing keys. Through this mapping, the system can unify object identifiers of arbitrary formats into fixed-length integer values ​​that are easy to calculate and look up within the system, avoiding the comparison and storage overhead caused by directly using variable-length strings for routing calculations.

[0048] Step 2: Determine the target virtual bucket.

[0049] The system determines the target virtual bucket corresponding to an object's routing key based on the object's routing key and the preset number of virtual buckets (virtual_bucket_num). Specifically, the system performs a modulo operation on the object's routing key. Since the number of virtual buckets is a fixed value (e.g., 4096 or 16384), and the object's routing key is generated by a deterministic mapping function, the same object identifier always maps to the same virtual bucket, and this mapping relationship never changes during system operation.

[0050] Step 3: Determine the target actual bucket.

[0051] The system determines the target physical bucket corresponding to the target virtual bucket based on the mapping relationship between virtual buckets and physical buckets. This mapping relationship is a dynamic table, which the system can flexibly adjust based on changes in the number of physical buckets. For example, when expanding capacity, the system adds new physical buckets, and can adjust the mapping of some virtual buckets from the old physical buckets to the new ones without changing the mapping from object identifiers to virtual buckets.

[0052] Step 4: Determine the data management unit.

[0053] The system determines the data management unit (i.e., Part) to which the target actual bucket belongs based on the bucket range it is located in. Each Part is responsible for a continuous or logically continuous range of actual buckets (for example, Part 0 is responsible for actual buckets 0~100, and Part 1 is responsible for actual buckets 101~200). Therefore, once the actual bucket is determined, its corresponding Part can be determined.

[0054] Through the above four steps, the system completes the entire routing and location process from "target object identifier" to "data management unit".

[0055] To better understand the beneficial effects of the virtual bucket mechanism in scaling scenarios, the following specific examples illustrate this: Assume the system is initially configured with 10 Parts, each managing 10 physical buckets, totaling 100 physical buckets; the number of virtual buckets is fixed at 1000, with each virtual bucket pointing to a physical bucket through a mapping relationship. When business growth necessitates scaling the system to 20 Parts, the total number of physical buckets increases from 100 to 200, and the scope of physical buckets managed by each Part is adjusted accordingly. In a solution without a virtual bucket layer, the change in the total number of physical buckets leads to a change in the modulo divisor, altering the mapping results for almost all objects, requiring a full data migration across Parts.

[0056] In this solution, the mapping relationship between object identifiers and virtual buckets (calculated based on a fixed number of virtual buckets of 1000) remains unchanged; the system only needs to adjust the mapping relationship between virtual buckets and actual buckets (for example, remapping some virtual buckets that originally pointed to the old actual buckets to the newly added actual buckets), and only performs local data migration within the actual bucket range corresponding to the virtual buckets whose mapping relationship has been adjusted, without the need for full data redistribution across all parts.

[0057] By applying the embodiments in this specification, a virtual bucket layer is set up to decouple the mapping from object identifiers to virtual buckets from the mapping from virtual buckets to physical buckets. This ensures that the logical ownership relationship of object identifiers (i.e., the mapping from object identifiers to virtual buckets) remains permanently stable, while the physical storage location (i.e., the mapping from virtual buckets to physical buckets) can be dynamically adjusted with the system scale. Thus, when the number of physical buckets or parts changes, only the mapping relationship needs to be adjusted without rehashing all object identifiers. This limits the data migration scope to the part corresponding to the virtual bucket whose mapping relationship has changed, avoiding the redistribution of all data across parts. This significantly reduces data migration costs and operational complexity during system expansion or contraction. Simultaneously, by uniformly mapping object identifiers to object routing keys using a target mapping function, and then determining virtual buckets based on these routing keys, the number of objects carried by each virtual bucket remains statistically balanced. This results in a more even load distribution across all physical buckets and parts, avoiding localized hotspots and single-point overload issues. Furthermore, the virtual bucket, acting as an indirect layer, makes the partitioning of parts and changes in the number of physical buckets transparent to upper-layer business applications. The business layer only needs to provide object identifiers for the system to automatically complete routing and location, simplifying the development and maintenance costs of upper-layer applications.

[0058] Step 106: If the target sequence description information corresponding to the target object identifier is found in the data index, the data area to be accessed is determined based on the request type of the data access request and the target sequence description information, and the data access request is processed based on the area location information corresponding to the data area to be accessed.

[0059] In one optional implementation of the embodiments of this specification, the data region to be accessed is determined based on the request type of the data access request and the target sequence description information, including: If the request type is a write request, the writable incremental data area in the target sequence description information is determined as the data area to be accessed. The writable incremental data area is used to store newly written data. Based on the location information of the area corresponding to the data area to be accessed, the data access request is processed, including: Based on the writable region location information corresponding to the writable incremental data region, the write data carried by the write request is written to the writable incremental data region.

[0060] For write requests, the system typically does not directly modify the existing persisted sequence on the disk. Instead, it locates the writable incremental data area in memory through the writable area location information in the sequence description information, writes the new data to that area, and then the background merging process merges the incremental data with the existing data on the disk and writes it to disk.

[0061] To better understand the role of writable incremental data regions, the following is a brief explanation of their flow throughout the data lifecycle: (1) Writing phase: New data is written to the writable incremental data area (memory).

[0062] (2) Freezing phase: When the writable incremental data area meets the merge triggering conditions (such as the data volume reaching the preset threshold or the timer period arriving), the area is switched to read-only state and becomes the frozen incremental data area (immutable_delta). At the same time, the system creates a new writable incremental data area to take over subsequent writing.

[0063] (3) Merging phase: The background merging thread reads the incremental data in the frozen incremental data area, merges it with the existing data in the disk, generates a new disk data block and writes it to the disk.

[0064] (4) Release phase: After the merge is completed, the system releases the memory space occupied by the frozen incremental data area.

[0065] Through the above process, the data in the writable incremental data area is eventually persisted to the disk, achieving eventual data consistency, while the write operation is never blocked throughout the process.

[0066] By applying the embodiments in this specification, a writable incremental data area is set as a dedicated memory area for real-time writing, and corresponding writable area location information is configured in the sequence description information. This allows write requests to directly append new data to the writable incremental data area in memory based on this location information, without modifying existing data on the disk. This completely frees write operations from disk I / O, reducing write latency from milliseconds to microseconds and significantly improving the system's write throughput. Simultaneously, since write operations only involve memory appending and are completely isolated from existing disk data, read requests can concurrently access existing disk data without interference from write operations. This achieves physical isolation of read and write paths, avoiding read-write conflicts and lock contention, and significantly improving the system's concurrent processing capabilities. Furthermore, write operations do not frequently trigger disk writes, effectively reducing SSD write wear and extending the lifespan of the storage medium. Data in the writable incremental data area is subsequently merged and written to disk in a unified manner through a background process, transforming scattered real-time writes into batch sequential disk writes, further optimizing disk I / O performance.

[0067] In one optional implementation of the embodiments of this specification, the data region to be accessed is determined based on the request type of the data access request and the target sequence description information, including: If the request type is a read request, then determine whether there is disk area location information corresponding to the disk storage area in the target sequence description information, wherein the disk data area is used to store the persisted historical storage data; If the disk region location information does not exist, the incremental data region in the target sequence description information will be used as the data region to be accessed. Based on the location information of the area corresponding to the data area to be accessed, the data access request is processed, including: If the disk region location information does not exist, the incremental data of the target object is read from the incremental data region based on the incremental region location information corresponding to the incremental data region, and the incremental data is used as the response result of the read request.

[0068] The incremental data area is a memory region within the data management unit (Part) used to store newly added data that has not yet been persisted to disk. Located in memory, the incremental data area absorbs newly written data in real time, allowing write operations to be completed without modifying existing data on disk.

[0069] In some embodiments, the incremental data region may logically contain data in two states: (1) Writable incremental data (located in the mutable_delta area): Currently receiving new data being written in real time, supporting append write operations.

[0070] (2) Frozen incremental data (located in the immutable_delta area): Incremental data that has been frozen in a read-only state and is waiting to be merged with the existing data on the disk. Only read operations are supported, and writing is not supported.

[0071] The data in the incremental data area, together with the existing data on disk, constitutes a complete data sequence for the same target object. The existing data on disk represents the object's long-term historical data (cold data), while the incremental data represents recently added data (hot data) that has not yet been written to disk. During reading, both must be merged to obtain a complete data sequence view containing the latest data.

[0072] In the embodiments of this specification, when the system receives a read request for a target object, the system determines the data area to be accessed based on the request type (i.e., read) of the read request and the target sequence description information queried through the data index.

[0073] Specifically, the system first determines whether the target sequence description information contains disk region location information corresponding to the disk storage area. Disk region location information (i.e., disk offset) is used to point to the historical data that has been persisted in the disk storage area. The existence of disk region location information means that the target object has historical data stored on the disk; the absence of disk region location information means that the target object does not have any persisted data on the disk.

[0074] When disk area location information is not available, the system identifies the incremental data area in the target sequence description information as the data area to be accessed, and reads the incremental data of the target object from the incremental data area based on the incremental area location information corresponding to the incremental data area, and uses the incremental data as the response result of the read request.

[0075] It should be understood that in system design, the absence of disk region location information means that the target object has no persisted historical data on the disk. This situation typically occurs in the following scenarios: (1) Newly created target object: The target object has just appeared in the system and no data has been persisted to the disk yet, but several real-time new data entries may have been generated and written to the incremental data area.

[0076] (2) The data of the target object has been completely cleaned or archived: The existing data of the target object has been cleaned, but there may still be incremental data in the incremental data area that has not been cleaned.

[0077] (3) Merging has not been triggered after the first write: The target object has been written, and the new data has been written to the incremental data area, but the background merging process has not been triggered yet, so there is no persistent data of the object on the disk.

[0078] In the above scenario, the incremental data stored in the incremental data area constitutes all the valid data for the target object, and there are no other data sources that need to be merged with the existing data on disk. Therefore, the system can obtain a complete data sequence view of the target object by simply reading the incremental data from the incremental data area and returning it as the response result of the read request.

[0079] Compared to reading existing disk data, reading data from the incremental data area has the following characteristics: the incremental data area is located in memory, the read operation is only a memory access, the latency is extremely low (microsecond level); and the incremental data usually only contains recently added behavior, the data volume is much smaller than the complete disk storage sequence, and the read efficiency is higher.

[0080] When disk region location information is not available, the system performs the following operations: Step 1: Determine the target incremental data area.

[0081] The system determines the storage location of the incremental data of the target object in memory based on the incremental region location information (i.e., writable region location information and / or frozen region location information) contained in the target sequence description information.

[0082] Step 2: Read incremental data.

[0083] Based on the incremental region location information, the system reads all incremental data of the target object from the incremental data region. In one implementation, if data exists in both the writable incremental data region and the frozen incremental data region, the system reads data from both regions separately and merges them into a unified incremental dataset.

[0084] Step 3: Return the read results.

[0085] The system returns the incremental data read from the incremental data area as the response to the read request. Since the target object does not have existing data on the disk, the incremental data is the complete data sequence of the object, and there is no need to merge it with the existing data on the disk.

[0086] Step 4 (optional): Initialize disk region location information.

[0087] In one implementation, when the amount of data in the incremental data region reaches a preset threshold, the background merging thread can write the incremental data to disk and update the disk region location information in the sequence description information. Subsequently, when the object is read again, the disk region location information will still be present, and the system will process it according to the presence of this information.

[0088] By applying the embodiments of this specification, when disk area location information is absent, the incremental data area is used as the data area to be accessed, and incremental data is directly read from the incremental data area in memory as the response result. This allows read requests for newly created or target objects that do not yet have disk storage to be completed in a pure memory access manner, avoiding the invalid I / O overhead caused by attempting to access the disk due to the absence of disk area location information. The read latency remains at the microsecond level. At the same time, since there is no need to perform the merging operation between disk storage data and incremental data when disk area location information is absent, unnecessary CPU computation overhead is reduced, further reducing read latency. In addition, this mechanism allows new objects to be read in real time after the first write without waiting for the background merging process to complete, achieving the effect of seeing the data immediately after writing, ensuring real-time data consistency, and meeting the stringent requirements of online business for data real-time performance.

[0089] In one optional embodiment of this specification, after determining whether disk region location information corresponding to a disk storage region exists in the target sequence description information, the method further includes: If disk area location information exists, the existing disk area and incremental data area in the target sequence description information will be used as the data area to be accessed. Based on the location information of the area corresponding to the data area to be accessed, the data access request is processed, including: If disk region location information exists, then based on the disk region location information corresponding to the disk storage region, the target storage data segment of the historical storage data of the target object is read from the disk storage region. Based on the incremental region positioning information corresponding to the incremental data region, read the incremental data of the target object from the incremental data region; The target existing data is segmented and merged with the incremental data to serve as the response result for the read request.

[0090] In the embodiments of this specification, when the system receives a read request for a target object and determines that there is disk area location information corresponding to the disk storage area in the target sequence description information, the system determines both the disk storage area and the incremental data area as the data area to be accessed, and reads data from the two areas respectively and then merges them to obtain the complete data sequence of the target object.

[0091] Specifically, the system performs the following operations: Step 1: Determine the data area to be accessed.

[0092] When the system determines that the disk area location information exists, it indicates that the target object already has historical data stored on the disk. Simultaneously, the target object may also have real-time newly added data in the incremental data area that has not yet been persisted. Therefore, the system identifies both the disk existing data area and the incremental data area as data areas to be accessed.

[0093] Step 2: Read historical data segments from the disk storage area.

[0094] The system reads the target data segment of the historical data of the target object from the disk storage area based on the disk storage area location information corresponding to the disk storage area.

[0095] It's important to note that what's read from the disk storage area is the target data segment, not the complete historical data sequence. In one implementation, the system first obtains the sequence organization information of the target object based on the disk area location information. This sequence organization information includes multiple data segment descriptions (i.e., MetaArray) arranged in chronological order. Then, based on the query conditions carried in the read request (such as the target time range or the most recent N records), the system determines the target data segment description information from among the multiple data segment descriptions. Finally, based on the data location information in the target data segment description information, the corresponding target data segment is read from the disk storage area. Because the historical data on the disk is organized in chronological order, the system can quickly locate the target segment using the segment description information without reading the complete disk sequence, thus significantly reducing unnecessary I / O.

[0096] Step 3: Read incremental data from the incremental data area.

[0097] The system reads incremental data of the target object from the incremental data region based on the incremental region positioning information corresponding to the incremental data region. Specifically, the incremental region positioning information includes writable region positioning information and / or frozen region positioning information. The system reads data from the writable incremental data region and / or frozen incremental data region respectively, and merges the reading results into the incremental data of the target object (see the aforementioned embodiment for details).

[0098] Step 4: Merge the data and return.

[0099] The system will segment the target existing data read from the disk storage area and merge it with the incremental data read from the incremental data area, and return the merged data as the response result of the read request.

[0100] It should be understood that, in the embodiments of this specification, the complete data sequence of the same target object may be distributed in different storage areas at different times. Persisted historical data is stored in the disk storage area, while real-time new data that has not yet been persisted is stored in the incremental data area (in memory). Together, they constitute the complete data sequence of the target object.

[0101] Therefore, when disk region location information exists, reading only the existing disk region will result in the loss of real-time new data that has not yet been written to disk, leading to incomplete read results; reading only the incremental data region will result in the loss of persistent historical data. Only by reading data from both regions simultaneously and merging them can a complete data sequence view containing the latest data be obtained.

[0102] Specifically, "segmenting the target existing data and merging it with the incremental data" includes: (1) Sort and merge the read target existing data segments and incremental data according to timestamps. Since both existing data segments and incremental data have timestamp information, the system can determine the order of the data based on the timestamps.

[0103] (2) In one implementation, the system traverses both the existing data segments and the incremental data simultaneously using an iterator, and outputs the merged data sequence in chronological order. This merging process is performed in memory and does not require writing to disk.

[0104] (3) The merged data sequence is the complete behavioral sequence view of the target object under the query conditions. This view contains all data from historical stock to the latest addition, and maintains the consistency of time order.

[0105] Based on the foregoing embodiments, the reading path of the embodiments in this specification can be summarized as follows: When a read request arrives, the system first locates the sequence description information of the target object using the data index. Then, the system determines whether the disk region location information exists. If it does not exist, it means that the target object has no historical data. The system only reads data from the incremental data area as the response result (see the aforementioned example of "disk area location information does not exist").

[0106] If it exists, it means that the target object has both historical existing data and possibly real-time new data. The system reads the corresponding target existing data segments from the disk existing data area and the incremental data from the incremental data area, and returns the merged data as the response result.

[0107] Through the complete reading path described above, the system can return a complete and correct data sequence under various data distribution conditions.

[0108] By applying the embodiments of this specification, when disk region positioning information exists, both the existing disk region and the incremental data region are identified as the data regions to be accessed. Data is read from the two regions based on the disk region positioning information and the incremental region positioning information respectively, and then merged. This ensures that the read result includes both the persisted historical data and the unpersisted real-time incremental data, guaranteeing the integrity and real-time consistency of the read data. Simultaneously, the target data segments, rather than the complete sequence, are read from the existing disk region. The system quickly locates the target segments using the segment description information in the sequence organization information, avoiding the I / O amplification problem caused by reading the entire disk sequence and significantly reducing read latency. Furthermore, the reading of the incremental data region is a pure memory operation, and the merging process is performed in memory. Disk I / O in the overall read path is limited to reading only the necessary target segments, significantly reducing the number of disk accesses and the amount of I / O data. In addition, the complete isolation of the read and write paths ensures that write operations do not affect the concurrent execution of read operations, effectively improving system throughput.

[0109] In one optional embodiment of the embodiments of this specification, the incremental data area includes a writable incremental data area and / or a frozen incremental data area. The writable incremental data area is used to store newly written data, and the frozen incremental data area is used to store read-only incremental data that is frozen during the data merging process. The incremental area positioning information includes writable area positioning information corresponding to the writable incremental data area and / or frozen area positioning information corresponding to the frozen incremental data area. Based on the incremental region positioning information corresponding to the incremental data region, incremental data of the target object is read from the incremental data region, including: Based on the writable region location information, read the new data corresponding to the target object from the writable incremental data region; Based on the location information of the frozen area, read-only incremental data corresponding to the target object is read from the frozen incremental data area; New data and / or read-only incremental data are identified as incremental data for the target object.

[0110] In this embodiment of the invention, the incremental data area includes a writable incremental data area and / or a frozen incremental data area, both of which are located in memory and are used to store newly added data that has not yet been persisted to disk, but they differ in function and lifecycle.

[0111] The writable incremental data area (i.e., the mutable_delta area) is a memory area in the memory storage resources of the data management unit (Part) used to receive real-time written data. This area supports high-concurrency append write operations, and all new data is written to this area first. The writable incremental data area has the following characteristics: (1) writability - supports real-time data append writing; (2) exclusivity - only one writable incremental data area exists in the system at a time to receive the current write; (3) transient - when the merge trigger condition is met, this area will be frozen and switched to read-only state.

[0112] The frozen incremental data region (i.e., the immutable_delta region) is a memory region formed when the writable incremental data region meets the merge trigger conditions (such as the data volume reaching a preset threshold or the timer period arriving), and the writable incremental data region switches to a read-only state. The frozen incremental data region has the following characteristics: (1) Read-only - after freezing, no new writes are accepted, and only read operations are supported; (2) Waiting for merging - the data in the frozen incremental data region waits for the background merging thread to merge it with the existing disk data and write it to disk; (3) Limited lifespan - after the merging is completed, the memory space occupied by the frozen incremental data region will be released.

[0113] The core difference lies in the fact that a writable incremental data area is the area "currently being written to," while a frozen incremental data area is the area "where writing has stopped and is awaiting merging." At any given time, for the same target object, there may be one writable incremental data area and multiple frozen incremental data areas (multiple snapshots formed by freezing at different points in time). Therefore, an incremental data area actually includes "writable incremental data areas and / or frozen incremental data areas," depending on which incremental data areas currently exist in the system—newly written data resides in writable incremental data areas, while data that has been frozen but not yet merged resides in frozen incremental data areas. Correspondingly, the incremental area location information also includes the writable area location information corresponding to the writable incremental data area, and / or the frozen area location information corresponding to the frozen incremental data area.

[0114] In the embodiments described in this specification, when the system needs to read incremental data of a target object from the incremental data region, the system performs the following operations based on the incremental region positioning information corresponding to the incremental data region: Step 1: Read the writable incremental data area.

[0115] The system reads the newly added data corresponding to the target object from the writable incremental data area based on the writable area location information (i.e., mutable_delta offset). The writable incremental data area stores the latest data that is currently being received for real-time writing; this data has not been frozen and has not been merged with the existing data on disk.

[0116] Step 2: Read the frozen incremental data area.

[0117] The system reads the read-only incremental data corresponding to the target object from the frozen incremental data region based on the frozen region location information (i.e., the immutable_delta offset). The frozen incremental data region stores incremental data that has been frozen and is waiting to be merged with the existing data on disk.

[0118] It should be noted that steps one and two above are executed independently. If there is no writable incremental data area in the system (e.g., there are no new write operations on the target object), step one will not be executed or the read result will be empty; if there is no frozen incremental data area in the system (e.g., no freezing operation has occurred), step two will not be executed or the read result will be empty. Therefore, the data read from the incremental data area is actually the union of "data in the writable incremental data area" and "data in the frozen incremental data area".

[0119] Step 3: Merge into incremental data.

[0120] The system merges newly added data read from the writable incremental data area and read-only incremental data read from the frozen incremental data area, and determines the merged result as the incremental data of the target object. This incremental data is then merged with the existing disk data to form a complete data sequence view of the target object.

[0121] It should be noted that, from a system design perspective, dividing the incremental data area into a writable incremental data area and a frozen incremental data area is to achieve the following objectives: (1) Write without blocking: By setting a writable incremental data area, the write operation always has an available memory area to receive new data and will not be blocked by background merge operations.

[0122] (2) Merging does not affect reading: The background merge operation reads the data in the frozen incremental data area, while the foreground read operation reads the data in both the writable incremental data area and the frozen incremental data area simultaneously, without interfering with each other. Even if the merge thread is processing the frozen incremental data area, new writes can still continue to be written to the writable incremental data area, achieving complete parallelism between writing and merging.

[0123] (3) No data loss: Before the writable incremental data area is frozen to read-only, its data is always in an accessible state; after freezing, the data is still retained in memory until the merging is completed. Therefore, at any time, all data that has been written but not yet persisted exists in the writable incremental data area or the frozen incremental data area and can be read.

[0124] (4) Ensure the integrity of the read view: When reading, data is read from both the writable incremental data area and the frozen incremental data area at the same time, ensuring that the read incremental data includes "all data that has been written but not yet persisted" and that no written data is missed.

[0125] To better understand the relationship between the two regions, the conversion process is briefly described below: (1) Initial state: The system creates a writable incremental data area, and all newly added data is written to this area.

[0126] (2) Triggering freeze: When the amount of data in the writable incremental data area reaches the preset threshold or the timer period is reached, the system switches the area to read-only state, making it a frozen incremental data area.

[0127] (3) Create a new region: The system creates a new writable incremental data region, and subsequent write operations are transferred to the new region.

[0128] (4) Background merge: The background merge thread reads the data in the frozen incremental data area, merges it with the existing data on the disk, generates a new disk data block and writes it to the disk.

[0129] (5) Release area: After the merge is completed, the system releases the memory space occupied by the frozen incremental data area.

[0130] (6) Loop: The new writable incremental data area accumulates data again, waiting for the next round of freezing and merging.

[0131] Through the above transformation, the data in the writable incremental data area is finally persisted to the disk, while freezing the incremental data area as an intermediate state in the transformation process ensures the parallel execution of the three operations of writing, reading and merging.

[0132] By applying the embodiments of this specification, the incremental data area is divided into a writable incremental data area and a frozen incremental data area, and the corresponding writable area location information and frozen area location information are configured in the sequence description information respectively. This allows read operations to read data from both areas separately and merge them into complete incremental data, ensuring that all written but not yet persisted data can be read, thus guaranteeing data integrity and consistency. Simultaneously, since the background merge operation only operates on the frozen incremental data area, while the write operation always operates on the writable incremental data area, and the read operation reads both areas simultaneously, the three operations do not block each other, achieving true parallel processing of write, read, and merge, significantly improving the system's throughput and concurrency capabilities. Furthermore, the data in the frozen incremental data area remains in memory until the merge is complete, so even if an exception occurs during the merge process, the data will not be lost, and the merge can be re-executed after a failure, enhancing the system's fault tolerance and data reliability.

[0133] In one optional embodiment of this specification, a target storage data segment is read from the disk storage area based on the disk area location information corresponding to the disk storage area, including: Based on disk area location information, the corresponding sequence organization information is obtained, which includes multiple data segment description information arranged in chronological order; Based on the query conditions carried in the read request, the target data segment description information is determined from multiple data segment description information, and the storage location of the target data segment is determined from the data location information in the target data segment description information. If the storage location is a cached data area, the target stored data segment is read from the cached data area. The cached data area is used to store hot data that has been loaded into memory from the disk stored area. If the storage location is a disk data area, then read the target storage data segment from the disk data area.

[0134] In the embodiments of this specification, sequence organization information (i.e., TsSequence) is a logical object used to describe the organizational structure information of the historical stored data of a single target object on the disk. Sequence organization information does not directly store the data sequence itself, but rather achieves a unified description and rapid location of the stored data on the disk by recording the data segmentation distribution of the target object on the disk.

[0135] Specifically, the sequence organization information includes multiple data segment descriptions (i.e., MetaEntries) arranged in chronological order. Each data segment description corresponds to a data segment (i.e., SequenceArray) and records information such as the segment's start timestamp (begin_timestamp), data length (length), and storage location (offset) on disk or in memory. Because the data segment descriptions are arranged in chronological order, the system can quickly locate the target segment using time conditions without traversing the entire data sequence.

[0136] Each data segment description information should include at least the following fields: (1) Start timestamp (begin_timestamp): The timestamp corresponding to the first data in this data segment, used to identify the position of this segment in the complete time series. Since the segments are arranged in chronological order, the start timestamps of adjacent segments are monotonically increasing.

[0137] (2) Data length: The number of data entries or the amount of data contained in this data segment, used to determine the range of data to be read.

[0138] (3) Data location information: The actual storage location of the segment in the storage medium, including disk offset (d_offset, i.e., the storage location in the disk data area) or memory offset (m_offset, i.e., the storage location in the cache data area). With this location information, the system can directly read the data entity of the segment.

[0139] In this embodiment of the specification, the target data segment for reading historical data of a target object from the disk storage area based on the disk storage area location information corresponding to the disk storage area specifically includes the following steps: Step 1: Obtain sequence organization information.

[0140] The system obtains the sequence organization information (TsSequence) corresponding to the target object based on the disk region location information (i.e., disk offset). This sequence organization information is persistently stored on the disk, loaded from the disk into memory when the target object is accessed for the first time, and cached in the sequence description information.

[0141] Step 2: Locate the segmented description information of the target data.

[0142] Based on the query conditions carried in the read request, the system determines the target data segment description information from multiple data segment description information in the sequence organization information. The query conditions may include the target time range (e.g., querying data after a certain year, month, and day), the reading direction (e.g., forward or reverse order), and the number of reads (e.g., the most recent N records).

[0143] Specifically, since multiple data segment descriptions are arranged in ascending order of their starting timestamps, the system can use a binary search algorithm to quickly locate the target segment that matches the query conditions within the data segment descriptions. For example, when the query condition is "query data after July 1, 2025", the system uses binary search to locate the data segment description with the starting timestamp closest to July 1, 2025 as the target.

[0144] Step 3: Determine the storage location of the target data segments.

[0145] The system determines the actual storage location of the target data segment based on the data location information in the target data segment description information.

[0146] Specifically, the data location information can include the following two types: Cache offset (m_offset): When valid, it indicates that the data segment has been loaded from the disk into the memory cache data area, and the target data segment is stored in the cache data area.

[0147] Disk offset (d_offset): When valid, it indicates that the data segment is stored in the disk data area.

[0148] The system determines the actual storage location of the target data segment based on which offset is valid in the data positioning information.

[0149] Step 4: Read data from the target storage location.

[0150] Based on the results determined in step three, the system executes the following branch operations: If the target data segment is stored in the cached data area (i.e., m_offset is valid), the system reads the target data segment directly from the cached data area. The cached data area is used to store hot data that has been loaded into memory from the disk storage area. Reading from the cache does not require disk I / O, resulting in extremely low access latency.

[0151] If the target data segment is stored in the disk data area (i.e., d_offset is valid), the system reads the target data segment from the disk data area according to the disk offset. After reading, in one implementation, the system can also asynchronously load the segment into the cache data area and update the cache offset in the data location information so that it can be read directly from the cache in subsequent accesses.

[0152] Step 5: Return the read results.

[0153] The system will segment the target existing data it reads into the disk existing data area, and then merge it with the incremental data for subsequent processing.

[0154] To more clearly illustrate the working principle of the embodiments in this specification, the following specific example describes the collaborative working process of sequence organization information and caching mechanism: Suppose a target object is stored on disk in 10 data segments (segment 0 to segment 9), each segment containing 100 behavioral records. The sequence organization information of this object records the description information of the 10 segments, each segment description information including the start timestamp, length, and disk offset.

[0155] Scenario 1: First Access. When the system receives its first read request for this object (querying the last 200 actions), it uses a binary search to locate the target segments as segments 8 and 9 in the sequence organization information. At this point, these two segments have not yet been loaded into the cached data area, so only the disk offset is valid in the data location information. The system reads the data for segments 8 and 9 from the disk data area based on the disk offset. After reading, the system asynchronously loads these two segments into the cached data area and updates the data location information, increasing the cache offset.

[0156] Scenario 2: Re-access. When the system receives the same read request for the same object again, the system still locates the target segments as segments 8 and 9. In this case, the cache offset in the data location information is valid, and the system directly reads these two segments from the cached data area without needing to access the disk again.

[0157] Through the above mechanism, hot data segments are cached after being accessed for the first time, and subsequent accesses can be read directly from memory, which greatly reduces the number of disk I / O operations and access latency.

[0158] By applying the embodiments in this specification, the system obtains the sequence organization information (including multiple data segment descriptions arranged in chronological order) corresponding to the target object based on disk region location information. This allows the system to quickly locate the target data segment description information in the sequence organization information using only query conditions without reading the complete data sequence. Based on the data location information, the system determines the storage location of the target data segment, thus achieving precise on-demand reading of disk data. Each read accesses only the target segment matching the query conditions, avoiding the huge I / O overhead and memory copying costs associated with reading the entire disk sequence, significantly reducing read latency. Simultaneously, by determining whether the target data segment is already stored in the cached data area based on the data location information, frequently accessed hot segments are prioritized for reading from the memory cache, further reducing disk I / O operations and improving cache hit rate. This achieves optimal read / write performance with limited memory resources. Furthermore, cache loading of data segments does not affect the integrity and persistence of the original data on the disk. Cache eviction only requires clearing the copy in memory, ensuring that disk data is always recoverable and guaranteeing system reliability and data security.

[0159] In one optional implementation of the embodiments of this specification, the query conditions include a target start timestamp; based on the query conditions carried in the read request, the target data segment description information is determined from multiple data segment description information, including: The first and last data segment descriptions in the multiple data segment descriptions are used as the start and end positions of the current search range, respectively. Get the intermediate data segment description information located at the middle position between the start position and the end position in the current search range, and get the intermediate start timestamp in the intermediate data segment description information; Match the intermediate start timestamp and the target start timestamp, and determine the target data segment description information from multiple data segment description information based on the matching results.

[0160] The query conditions in the embodiments of this specification include the target start timestamp (i.e., the user-specified query start time point, such as "querying behavioral data after 00:00:00 on July 1, 2025"). Since the multiple data segment description information in the sequence organization information is arranged in ascending order of the start timestamp, the system can, based on this ordered arrangement characteristic, locate the target data segment description information that matches the query conditions among multiple data segment description information by comparing the target start timestamp with the start timestamp in the data segment description information.

[0161] Specifically, the system uses the first and last data segment descriptions from multiple data segment descriptions as the start and end positions of the current search range, respectively; it obtains the intermediate data segment description at the middle position between the start and end positions in the current search range, and obtains the intermediate start timestamp in the intermediate data segment description; it matches the intermediate start timestamp with the target start timestamp, and determines the next search direction or the target data segment description based on the matching result, until the target data segment description matching the query conditions is located.

[0162] By applying the embodiments of this specification, multiple data segment descriptions are arranged in order of their start timestamps, and a sequential search is performed among the multiple data segment descriptions based on the target start timestamp. This allows the system to quickly locate the target data segment description that matches the query conditions without traversing all data segment descriptions. Compared to the linear search method of sequential traversal, this significantly reduces CPU search overhead and query latency in large-scale data segmentation scenarios. Furthermore, the search process only involves comparison operations of the start timestamps, without needing to read the actual content of the data segment or load data entities into memory, resulting in minimal memory overhead and preventing the normal use of system memory resources from being affected by the search operation. In addition, this search method does not depend on specific timestamp formats and matching logic, and can adapt to various time expression methods and query conditions, exhibiting good versatility and scalability.

[0163] In one optional implementation of the embodiments of this specification, the intermediate start timestamp and the target start timestamp are matched, and the target data segment description information is determined from multiple data segment description information based on the matching result, including: If the target start timestamp matches the intermediate start timestamp, then the intermediate data segment description information is determined as the target data segment description information; If the target start timestamp is earlier than the intermediate start timestamp, the end position is updated to the position before the intermediate position, and the process returns to the step of obtaining the intermediate data segment description information at the intermediate position between the start and end positions in the current search range, until the target data segment description information is determined. If the target start timestamp is later than the intermediate start timestamp, the start position is updated to the position after the intermediate position, and the process returns to the step of obtaining the intermediate data segment description information at the intermediate position between the start and end positions in the current search range, until the target data segment description information is determined.

[0164] Since the multiple data segment descriptions in the sequence organization information are arranged in ascending order of the start timestamp, the system can use a binary search to quickly locate the target data segment description that matches the target start timestamp among the multiple data segment descriptions.

[0165] The core idea of ​​binary search is that since the data segment description information is already arranged in chronological order, the system does not need to traverse all segments from the beginning. Instead, it locates the target segment with logarithmic time complexity by continuously narrowing the search range, thereby greatly improving the positioning efficiency.

[0166] Specifically, determining the target data segment description information from multiple data segment description information based on the query conditions carried in the read request includes the following steps: Step 1: Initialize the search range.

[0167] The system uses the first and last data segment descriptions from multiple data segment descriptions as the start and end positions of the current search range, respectively. That is, initially, the current search range covers all data segment descriptions.

[0168] Step 2: Obtain the segment description information of the middle position.

[0169] The system obtains the intermediate data segment description information located at the middle position between the start and end positions in the current search range, and obtains the intermediate start timestamp in the intermediate data segment description information.

[0170] In one implementation, the middle position is calculated as: mid = (left + right) / / 2, where left is the starting index of the current search range and right is the ending index of the current search range.

[0171] Step 3: Match the intermediate start timestamp with the target start timestamp.

[0172] The system matches the intermediate start timestamp with the target start timestamp (i.e., compares the size relationship between the two), and determines the next search direction or the target data segmentation description information based on the matching result.

[0173] The matching results include the following three cases: If the intermediate start timestamp matches the target start timestamp (i.e., they are equal or meet the preset matching conditions), then the intermediate data segment description information is determined as the target data segment description information.

[0174] If the intermediate start timestamp is earlier than the target start timestamp (i.e., the data segment in the middle position is earlier than the target time), it means that the target segment is located in the latter half of the current search range. The system will take the next position after the middle position as the new starting position and narrow the search range to the latter half.

[0175] If the intermediate start timestamp is later than the target start timestamp (i.e., the data segment in the middle position is later than the target time), it means that the target segment is in the first half of the current search range. The system will take the position before the intermediate position as the new end position and narrow the search range to the first half.

[0176] Step 4: Repeat the process until the target is determined.

[0177] The system repeats steps two and three above, continuously narrowing the search range until the target data segment description information is located or it is determined that no data segment description information that meets the conditions exists.

[0178] To facilitate understanding, the following specific examples illustrate the matching process described above: Assume the sequence organization information contains descriptions of 8 data segments, with their start timestamps as follows: [2025-01-01, 2025-02-01, 2025-03-01, 2025-04-01, 2025-05-01, 2025-06-01, 2025-07-01, 2025-08-01].

[0179] The target start timestamp in the user's query criteria is 2025-05-15.

[0180] The system performs a binary search as follows: (1) The initial search range is all 8 segments (position 0 to position 7), with the middle position being position 3 and the middle starting timestamp being 2025-04-01. This timestamp is earlier than the target starting timestamp 2025-05-15, indicating that the target segment is located in the latter half. The system will then narrow the search range to positions 4 to 7.

[0181] (2) The current search range is from position 4 to position 7, with position 5 as the middle position and the middle starting timestamp being 2025-06-01. This timestamp is later than the target starting timestamp 2025-05-15, indicating that the target segment is located in the first half of the current range. The system will narrow the search range to position 4 to position 7.

[0182] (3) The current search range is from position 4 to position 4. The data segment description information corresponding to this position is the target data segment description information. Its starting timestamp is 2025-05-01, which is the first data segment later than the target starting timestamp 2025-05-15.

[0183] By applying the embodiments of this specification, multiple data segment descriptions are arranged in order of their starting timestamps, and a binary search is used to locate the target data segment description based on its starting timestamp within these multiple data segment descriptions. This allows the system to quickly locate the target data segment description matching the query conditions in logarithmic time complexity (O(log N)) without traversing all data segment descriptions. Compared to the linear search method of sequential traversal, this method improves search efficiency by several orders of magnitude in large-scale data segment scenarios (such as when a single user has tens of thousands of data segments), significantly reducing CPU search overhead and query latency. Furthermore, the binary search process only involves comparing the starting timestamps, without needing to read the actual content of the data segment or load data entities into memory. The memory overhead of the search process is minimal and will not affect the normal use of system memory resources. In addition, this search method does not depend on specific timestamp formats and matching logic, and can adapt to various time expression methods and query conditions, exhibiting good versatility and scalability.

[0184] In one optional embodiment of the embodiments of this specification, the above data processing method further includes: If the target sequence description information corresponding to the target object identifier is not found in the data index, and the data access request is a read request, then it is determined that the data sequence corresponding to the target object identifier does not exist, and the read result is returned as empty.

[0185] In the embodiments of this specification, when the system does not find the target sequence description information corresponding to the target object identifier in the data index (i.e., the memory mapping table), it indicates that the target object does not have a corresponding metadata record in the memory mapping table of the current data management unit. At this time, the system executes different processing logic according to the request type of the data access request.

[0186] Specifically, if the data access request is a read request, the system determines that the data sequence corresponding to the target object identifier does not exist and directly returns an empty read result.

[0187] For example, when a target object (such as a newly registered user, a newly listed product, or a newly connected device) has just appeared in the system, and no data has been persisted to disk or any real-time new data has been generated, the target object does not have corresponding sequence description information in the data index. At this time, if the system receives a read request for the target object, since the object has never generated any data, its data sequence does not exist, and the system directly returns an empty result.

[0188] By applying the embodiments in this specification, when the target sequence description information is not found in the data index and the request type is a read request, the system directly determines that the data sequence of the target object does not exist and returns an empty result. This allows the system to quickly respond to read requests for new or invalid objects without accessing the disk or creating additional metadata, avoiding the additional I / O overhead and memory usage caused by checking the disk or creating an empty record for each read request. At the same time, by distinguishing the different processing logic for read and write requests—creating a new record when a write request misses and returning an empty result when a read request misses—on-demand allocation of memory resources is achieved, preventing a large number of objects that are only read and never written from creating invalid entries in the memory map table, further ensuring the lightweight nature of the memory map table and reducing memory costs. In addition, the fast failure mechanism for read requests avoids invalid requests occupying subsequent data processing link resources, improving the overall throughput and resource utilization efficiency of the system.

[0189] In one optional embodiment of the embodiments of this specification, the above data processing method further includes: If the target sequence description information corresponding to the target object identifier is not found in the data index, and the data access request is a write request, then the target sequence description information corresponding to the target object identifier is created. The writable region location information corresponding to the writable incremental data region is recorded in the target sequence description information, and the association relationship between the target object identifier and the target sequence description information is stored in the data index. Based on the writable region location information corresponding to the writable incremental data region, the write data carried by the write request is written to the writable incremental data region.

[0190] In the embodiments of this specification, when the system does not find the target sequence description information corresponding to the target object identifier in the data index (i.e., the memory mapping table), it indicates that the target object does not have a corresponding metadata record in the memory mapping table of the current data management unit. At this time, the system executes different processing logic according to the request type of the data access request.

[0191] Specifically, if the data access request is a write request, the system performs the following operations: First, it creates target sequence description information (i.e., UserSequence meta-object) corresponding to the target object identifier, and records the writable region location information (i.e., mutable_delta offset) corresponding to the writable incremental data region in the target sequence description information; then, it stores the association between the target object identifier and the target sequence description information in the data index; finally, based on the writable region location information corresponding to the writable incremental data region, it writes the write data carried by the write request into the writable incremental data region.

[0192] Through the above operations, the system completes the entire process of creating metadata for the new target object, establishing an index, and writing the first data.

[0193] For example, when a target object (such as a newly registered user, a newly listed product, or a newly connected device) generates behavioral data in the system for the first time, a corresponding sequence description information for that target object has not yet been established in the data index. At this time, if the system receives a write request for that target object (such as recording a click, an interaction record, or a status change), the system needs to create sequence description information for that object to record its data storage location in the system, and store the association between the object's identifier and the sequence description information in the data index, so that subsequent read requests can quickly locate the object's sequence description information through the data index.

[0194] It should be understood that, unlike read requests, write requests carry the data to be stored, and the system needs to allocate storage resources for this data. If the system does not create sequence description information, the written data will not be accessible to subsequent read requests. Therefore, for write requests, if sequence description information is not found in the data index, the system needs to actively create sequence description information and establish an index association.

[0195] Through the above processing, the first write request for a new target object can complete the entire process of "creating metadata → building an index → ​​writing incremental data" in the same operation. After the write operation is completed, the sequence description information of the object already exists in the data index, and the incremental data has been stored in the writable incremental data area. Therefore, a read request issued immediately after the write request is completed can locate the sequence description information of the object through the data index and read the data just written from the writable incremental data area, achieving the data consistency effect of "visible immediately after writing".

[0196] The details regarding the writable incremental data region in this embodiment can be found in the detailed descriptions of the preceding embodiments. For ease of understanding, a brief explanation is provided below: The writable incremental data area is a dedicated storage area within the data management unit's memory storage resources for absorbing real-time write data. The sequence description information for each target object records writable area location information, indicating the target object's storage location within the writable incremental data area. When the system performs a write operation, it appends the new data to the corresponding storage location based on this location information.

[0197] By applying the embodiments of this specification, when the target sequence description information is not found in the data index and the request type is a write request, the sequence description information corresponding to the target object is automatically created and an index association is established. At the same time, the written data is stored in the writable incremental data area. This allows the first data write of a new target object to complete the entire process of metadata creation, index establishment, and data storage in a single operation, without the need for additional metadata pre-creation steps, thus simplifying the data processing flow. In addition, after the write request is completed, the sequence description information of the target object already exists in the data index and the incremental data has been stored in memory. Subsequent read requests can immediately access the data that has just been written, achieving data consistency that is visible upon writing and meeting the stringent requirements of online business for real-time data visibility. Furthermore, the differentiated handling of write and read requests when the data index is not hit—creating it during writing and not creating it during reading—avoids a large amount of invalid metadata generated by read requests occupying memory resources, ensuring the lightweight nature of the memory mapping table and the efficient utilization of memory resources.

[0198] See Figure 2 , Figure 2 A schematic diagram of the structure of a time-series indexed storage system provided according to an embodiment of this specification is shown.

[0199] The time-series index storage system in the embodiments of this specification can be divided into a request access layer, a routing mapping layer, an index service layer, a sequence metadata layer, and a physical storage layer.

[0200] like Figure 2 As shown, in the request access layer: the system receives online requests (data access requests) from online services (such as recommendations, advertisements, search, etc.). The request carries the object identifier (user_key) of the target object, which is used to uniquely identify the target object to which the data to be accessed belongs.

[0201] At the routing mapping layer, the system performs a hash calculation on the target object identifier to obtain a fixed-length primary key hash value (pkey_hash). Subsequently, the system determines the corresponding Virtual Bucket based on this hash value. Virtual buckets are stable logical buckets; their number is set during system initialization and remains constant during operation. The system further determines the actual Bucket corresponding to the object's hash value based on the mapping relationship between virtual buckets and actual buckets. By introducing the virtual bucket layer, the system decouples the logical ownership of object identifiers from their physical storage location, eliminating the need to re-hash all object identifiers when the number of actual buckets changes.

[0202] At the indexing service layer: Indexing service nodes organize data according to a "Shard-Part" hierarchy. Each service node is responsible for carrying one or more data shards, and each data shard is further divided into multiple Parts. Each Part is responsible for a contiguous or logically contiguous range of actual buckets. Each Part maintains an independent in-memory hash table (data index). The key of this hash table is the primary key hash value (pkey hash) of the target object, and the value is the offset or pointer of the sequence description information (UserSequence) corresponding to the target object. The in-memory hash table only stores the mapping relationship between object identifiers and sequence description information, and does not store the complete data sequence, thus its memory consumption is extremely small.

[0203] At the sequence metadata layer: UserSequence is a unified description object of a single target object's data sequence within the system. It does not directly store the complete data sequence, but rather stores four types of offset information pointing to different storage areas: Disk offset (disk offset / disk region location information): points to the persistent historical sequence of data in the disk storage layer (disk data sequence, complete historical behavior).

[0204] Cache offset: Points to the hot sequence segment in the memory cache layer that has been loaded from disk (memory read cache, hot sequence segment).

[0205] mutable_delta offset: Points to the currently receiving new data being written in real time in the mutable delta region of memory (mutable delta, real-time addition behavior).

[0206] immutable_delta offset (frozen increment offset): points to the read-only increment data that is frozen during the background merge process in the memory frozen increment area (memory frozen increment, background merge read).

[0207] Through the four types of offsets mentioned above, UserSequence associates different segments of the same data sequence that are scattered across different storage areas into a unified logical view.

[0208] In the physical storage layer, there are two main components: disk storage and memory storage. Disk stock sequence: Stores complete historical behavior sequences and is the main carrier layer of data sequences. The historical stock data of each target object is organized in the form of multiple SequenceArray data blocks on disk.

[0209] Memory read cache: Stores copies of hot sequence segments loaded from disk to accelerate reading of frequently accessed segments.

[0210] In-memory writable incremental: Stores real-time newly added data, used to store the write data corresponding to write requests, ensuring low latency of write operations.

[0211] Memory-frozen incremental: Stores background merged data (i.e., read-only incremental data frozen during the background merge process) to ensure that writes are not blocked during the merge.

[0212] At the sequence data organization layer: the existing data on disk is logically organized in the form of MetaArray and SequenceArray: MetaArray (Segment Meta-information): Records the metadata of multiple SequenceArray data segments of the target object on disk. Each MetaEntry corresponds to a SequenceArray segment, recording the segment's start timestamp (begin_ts), data length (length), and storage offset (offset). MetaArrays are arranged in chronological order, supporting quick location of the target segment based on time conditions.

[0213] SequenceArray (data segment): Stores the actual content of the data sequence. Each SequenceArray is a block of data arranged in chronological order.

[0214] Through the above hierarchical structure, the embodiments in this specification adopt a layered storage architecture that coordinates memory, disk, and incremental storage. Different storage layers undertake different responsibilities, and a unified data access entry point is formed through UserSequence and MetaArray. The core idea of ​​this architecture is to place frequently accessed data that requires fast location or real-time updates in memory, store the largest complete historical sequences on disk, cache hot historical segments in memory as needed, and save new behaviors that have not yet been written to disk in the incremental area, thereby achieving a balance between cost, latency, and real-time performance.

[0215] Specifically, the memory layer primarily stores lightweight indexing and location information and runtime metadata, including the memory hash table within the Part (used to quickly locate the corresponding UserSequence from the object identifier), the UserSequence meta-object (used to uniformly describe the distribution of the same data sequence across different storage layers), the MetaArray (used to describe the time range, length, and storage location of data segments), and some cached SequenceArrays and incremental sequences (mutable_delta and immutable_delta). Since the amount of the above information is much smaller than the complete historical sequence, storing it in memory significantly improves location speed while avoiding the high costs associated with the complete sequence residing in memory. The disk layer serves as the main carrier layer for the complete data sequence, storing the persisted complete historical data. The caching layer saves copies of hot segments loaded from disk on demand to accelerate repeated access. The incremental layer stores newly added data that has not yet been merged into disk, ensuring write performance and data real-time performance. Through the collaboration of these four layers, the system achieves a significant reduction in memory costs while ensuring low-latency access, providing an economical and feasible solution for storing and accessing ultra-long sequences in scenarios with massive amounts of objects.

[0216] See Figure 3 , Figure 3 A schematic diagram of a data processing procedure provided according to an embodiment of this specification is shown.

[0217] like Figure 3 As shown in the embodiments of this specification, the complete path of the data location process from requesting access to data return is as follows: When an online read request or a real-time write request arrives, the system first extracts the user primary key (user_key), also called the object identifier of the target object in this embodiment, which uniquely identifies the target object to which the data to be accessed belongs. Subsequently, the system performs a hash calculation on the user primary key to obtain a fixed-length hash value (pkey_hash), and calculates the Virtual_bucket based on Virtual_bucket_num, i.e., calculating the virtual bucket corresponding to the hash value based on the preset number of virtual buckets. Then, based on the mapping relationship between virtual buckets and actual buckets, the system determines the actual bucket number corresponding to the virtual bucket. Finally, based on the bucket range of the actual bucket number, the system determines the target Part to which the actual bucket number belongs and enters the target Part. Through this routing mechanism, the system distributes the data of massive objects into different Parts, achieving ordered data organization and load balancing.

[0218] Upon entering the target Part, the system queries the Part's internal in-memory hash table to find the entry with the primary key hash value (pkey_hash) as the key. This hash table stores the mapping relationship between primary key hash values ​​and UserSequences (sequence description information), used to quickly locate the sequence description information of the target object. The system executes different processing logic depending on whether the corresponding UserSequence is found in the in-memory hash table and the type of the current request.

[0219] When a request is a read request and no corresponding UserSequence is found in the in-memory hash table, it means that the target object does not exist in the current system, and the system directly returns an empty result / miss. When a request is a write request and no corresponding UserSequence is found in the in-memory hash table, it means that the target object is being written to for the first time. The system creates a new UserSequence and writes it to the in-memory hash table to establish an index association, then locates the writable incremental data area and writes the real-time increment carried by the write request to that area. When a request is a write request and a corresponding UserSequence is found in the in-memory hash table, the system directly locates the writable incremental data area recorded in that UserSequence and writes the real-time increment carried by the write request to that area, without modifying the existing data on disk.

[0220] When a read request is made and a corresponding UserSequence is found in the memory hash table, the system further reads the disk offset information in the UserSequence to check if the target object has a persistent data sequence on disk. If the target object's data sequence does not exist on disk, the system only checks if readable data exists in the memory cache and incremental data area. If it exists, it reads the data; otherwise, it returns an empty result. If the target object's data sequence exists on disk, the system obtains the TsSequence through the disk offset information and reads the MetaArray segment metadata. This segment metadata records the metadata of multiple data segments of the target object on disk. Each segment corresponds to a segment metadata entry, recording the segment's start timestamp, data length, and storage offset. Based on the query conditions carried in the read request (such as the target's start timestamp, read order, etc.), the system quickly locates the target MetaArray in the segment metadata using a binary search.

[0221] After locating the target MetaArray, the system determines the actual storage location of the target data segment based on the data location information within the target MetaArray: if m_offset (cache offset) is valid, it means that the data segment has been loaded from disk into the memory cache, and the system directly reads the memory cache segment from the memory cache area; if d_offset (disk offset) is valid, it means that the data segment has not yet been cached, and the system reads the disk segment from the disk data area based on the disk offset. After reading, the system merges the candidate behavior segment read from disk (or cache) with the delta data (incremental data) read from the incremental data area (writable incremental and frozen incremental) in chronological order to form a complete data sequence as the query result of the read request.

[0222] The above data location process achieves accurate location and access to the target data segment without reading the complete data sequence through virtual bucket routing, fast location using memory hash table, segmented addressing based on segmented metadata, and cache-first reading strategy. This significantly reduces I / O overhead and memory usage while ensuring low-latency access.

[0223] The following is in conjunction with the appendix Figure 4 Taking the application of the data processing method provided in this specification in processing extremely long user behavior sequences as an example, the data processing method will be further explained. Figure 4 A flowchart illustrating the processing procedure of a data processing method according to an embodiment of this specification is shown, specifically including the following steps.

[0224] Step 402: Receive a data access request for the target object and determine the corresponding data management unit based on the target object identifier in the data access request.

[0225] Step 404: Map the target object identifier to the target object routing key using the target mapping function. The target mapping function is used to map the same object identifier to the same object routing key and different object identifiers to different object routing keys.

[0226] Step 406: Determine the target virtual bucket corresponding to the target object routing key based on the target object routing key and the set number of virtual buckets. The number of virtual buckets is a fixed value used to isolate the mapping relationship from object identifier to virtual bucket and the mapping relationship from virtual bucket to actual bucket.

[0227] Step 408: Based on the target virtual bucket and the mapping relationship between the virtual bucket and the actual bucket, determine the target actual bucket corresponding to the target object routing key, and determine the data management unit to which the target actual bucket belongs as the data management unit corresponding to the target object identifier. Here, the actual bucket is a storage container used to store data.

[0228] Step 410: Obtain the data index corresponding to the data management unit. The data index indicates the association between the object identifier and the sequence description information. The sequence description information is used to describe the storage area where different data segments of the same data sequence are located. The storage area is at least one of the disk storage area, cached data area, writable incremental data area and frozen incremental data area. The sequence description information contains the area location information of each data segment in its storage area.

[0229] It should be noted that different data segments of the same data sequence can be stored in one or more of the four types of areas mentioned above, depending on their state; not all sequences will be divided into all four areas. A specific data segment will only reside in one of the storage areas at any given time. For example, persisted historical data segments are located in the disk storage area, newly added data segments are located in the writable incremental data area, frozen data segments awaiting merging are located in the frozen incremental data area, and hot data segments are located in the cached data area. Different data segments of the same target object may reside in different storage areas, but it is not necessarily true that data exists in all four areas.

[0230] Step 412: Determine whether the target sequence description information corresponding to the target object identifier is found in the data index.

[0231] If no target sequence description information corresponding to the target object identifier is found, and the data access request type is a read request, then proceed to step 414; if no target sequence description information corresponding to the target object identifier is found, and the data access request type is a write request, then proceed to step 416.

[0232] Step 414: Determine that the data sequence corresponding to the target object identifier does not exist, and return an empty read result.

[0233] Step 416: Create target sequence description information corresponding to the target object identifier, record the writable region location information corresponding to the writable incremental data region in the target sequence description information, and store the association relationship between the target object identifier and the target sequence description information in the data index; based on the writable region location information corresponding to the writable incremental data region, write the write data carried by the write request into the writable incremental data region.

[0234] If the target sequence description information corresponding to the target object identifier is found, and the data access request type is a write request, then proceed to step 418; if the target sequence description information corresponding to the target object identifier is found, and the data access request type is a read request, then proceed to steps 420 to 424.

[0235] Step 418: Determine that the data area to be accessed is a writable incremental data area, where the writable incremental data area is used to store newly written data; based on the writable area location information corresponding to the writable incremental data area, write the write data carried by the write request into the writable incremental data area.

[0236] Step 420: Determine whether there is disk area location information corresponding to the disk storage area in the target sequence description information.

[0237] If the disk region location information does not exist, proceed to step 422; if the disk region location information exists, proceed to steps 424 to 430.

[0238] Step 422: Designate the writable incremental data area and the frozen incremental data area as the data areas to be accessed; based on the writable area location information, read the new data corresponding to the target object from the writable incremental data area; based on the frozen area location information, read the read-only incremental data corresponding to the target object from the frozen incremental data area; determine the new data and the read-only incremental data as the incremental data of the target object, and use the incremental data as the response result of the read request.

[0239] Step 424: Select the disk storage area, writable incremental data area, and frozen incremental data area from the target sequence description information as the data area to be accessed.

[0240] Step 426: Based on the writable region location information, read the new data corresponding to the target object from the writable incremental data region; based on the frozen region location information, read the read-only incremental data corresponding to the target object from the frozen incremental data region; determine the new data and read-only incremental data as the incremental data of the target object; segment the target existing data and merge it with the incremental data as the response result of the read request.

[0241] Step 428: Based on the disk area location information, obtain the corresponding sequence organization information, wherein the sequence organization information includes multiple data segment description information arranged in chronological order; based on the query conditions carried by the read request, determine the target data segment description information from the multiple data segment description information, and determine the storage location of the target data segment based on the data location information in the target data segment description information; if the storage location is a cache data area, read the target existing data segment from the cache data area; if the storage location is a disk data area, read the target existing data segment from the disk data area.

[0242] Step 430: Segment the target existing data and merge it with the incremental data as the response result of the read request.

[0243] Corresponding to the above method embodiments, this specification also provides data processing apparatus embodiments. Figure 5A schematic diagram of the structure of a data processing apparatus according to one embodiment of this specification is shown. Figure 5 As shown, the device includes: The management unit determination module 502 is configured to receive data access requests for a target object and determine the corresponding data management unit based on the target object identifier in the data access request. The index acquisition module 504 is configured to acquire the data index corresponding to the data management unit. The data index indicates the association between the object identifier and the sequence description information. The sequence description information indicates the regional positioning information of the data area where different data segments of the same data sequence are located. The request processing module 506 is configured to, if the target sequence description information corresponding to the target object identifier is found in the data index, determine the data area to be accessed based on the request type of the data access request and the target sequence description information, and process the data access request based on the area location information corresponding to the data area to be accessed.

[0244] Optionally, the management unit determining module 502 is further configured as follows: The target mapping function maps target object identifiers to target object routing keys. Specifically, the target mapping function maps the same object identifier to the same object routing key and different object identifiers to different object routing keys. Based on the target object routing key and the set number of virtual buckets, determine the target virtual bucket corresponding to the target object routing key. The number of virtual buckets is a fixed value used to isolate the mapping relationship from object identifier to virtual bucket and the mapping relationship from virtual bucket to actual bucket. Based on the target virtual bucket and the mapping relationship between the virtual bucket and the actual bucket, the target actual bucket corresponding to the target object routing key is determined, and the data management unit to which the target actual bucket belongs is determined as the data management unit corresponding to the target object identifier. Here, the actual bucket is a storage container used to store data.

[0245] Optionally, the request processing module 506 is further configured as follows: If the request type is a write request, the writable incremental data area in the target sequence description information is determined as the data area to be accessed. The writable incremental data area is used to store newly written data. Based on the writable region location information corresponding to the writable incremental data region, the write data carried by the write request is written to the writable incremental data region.

[0246] Optionally, the request processing module 506 is further configured as follows: If the request type is a read request, then determine whether there is disk area location information corresponding to the disk storage area in the target sequence description information, wherein the disk data area is used to store the persisted historical storage data; If the disk region location information does not exist, the incremental data region in the target sequence description information will be used as the data region to be accessed. If the disk region location information does not exist, the incremental data of the target object is read from the incremental data region based on the incremental region location information corresponding to the incremental data region, and the incremental data is used as the response result of the read request.

[0247] Optionally, the request processing module 506 is further configured as follows: If disk area location information exists, the existing disk area and incremental data area in the target sequence description information will be used as the data area to be accessed. If disk region location information exists, then based on the disk region location information corresponding to the disk storage region, the target storage data segment of the historical storage data of the target object is read from the disk storage region. Based on the incremental region positioning information corresponding to the incremental data region, read the incremental data of the target object from the incremental data region; The target existing data is segmented and merged with the incremental data to serve as the response result for the read request.

[0248] Optionally, the incremental data area includes a writable incremental data area and / or a frozen incremental data area. The writable incremental data area is used to store newly written data, and the frozen incremental data area is used to store read-only incremental data that is frozen during the data merging process. The incremental area location information includes the writable area location information corresponding to the writable incremental data area and / or the frozen area location information corresponding to the frozen incremental data area. Request processing module 506 is further configured as follows: Based on the writable region location information, read the new data corresponding to the target object from the writable incremental data region; Based on the location information of the frozen area, read-only incremental data corresponding to the target object is read from the frozen incremental data area; New data and / or read-only incremental data are identified as incremental data for the target object.

[0249] Optionally, the request processing module 506 is further configured as follows: Based on disk area location information, the corresponding sequence organization information is obtained, which includes multiple data segment description information arranged in chronological order; Based on the query conditions carried in the read request, the target data segment description information is determined from multiple data segment description information, and the storage location of the target data segment is determined from the data location information in the target data segment description information. If the storage location is a cached data area, the target stored data segment is read from the cached data area. The cached data area is used to store hot data that has been loaded into memory from the disk stored area. If the storage location is a disk data area, then read the target storage data segment from the disk data area.

[0250] Optionally, the query criteria include the target start timestamp; the request processing module 506 is further configured to: The first and last data segment descriptions in the multiple data segment descriptions are used as the start and end positions of the current search range, respectively. Get the intermediate data segment description information located at the middle position between the start position and the end position in the current search range, and get the intermediate start timestamp in the intermediate data segment description information; Match the intermediate start timestamp and the target start timestamp, and determine the target data segment description information from multiple data segment description information based on the matching results.

[0251] Optionally, the request processing module 506 is further configured as follows: If the target start timestamp matches the intermediate start timestamp, then the intermediate data segment description information is determined as the target data segment description information; If the target start timestamp is earlier than the intermediate start timestamp, the end position is updated to the position before the intermediate position, and the process returns to the step of obtaining the intermediate data segment description information at the intermediate position between the start and end positions in the current search range, until the target data segment description information is determined. If the target start timestamp is later than the intermediate start timestamp, the start position is updated to the position after the intermediate position, and the process returns to the step of obtaining the intermediate data segment description information at the intermediate position between the start and end positions in the current search range, until the target data segment description information is determined.

[0252] Optionally, the request processing module 506 is also configured to: If the target sequence description information corresponding to the target object identifier is not found in the data index, and the data access request is a read request, then it is determined that the data sequence corresponding to the target object identifier does not exist, and the read result is returned as empty.

[0253] Optionally, the request processing module 506 is also configured to: If the target sequence description information corresponding to the target object identifier is not found in the data index, and the data access request is a write request, then the target sequence description information corresponding to the target object identifier is created. The writable region location information corresponding to the writable incremental data region is recorded in the target sequence description information, and the association relationship between the target object identifier and the target sequence description information is stored in the data index. Based on the writable region location information corresponding to the writable incremental data region, the write data carried by the write request is written to the writable incremental data region.

[0254] The above is an illustrative scheme of a data processing apparatus according to this embodiment. It should be noted that the technical solution of this data processing apparatus and the technical solution of the data processing method described above belong to the same concept. For details not described in detail in the technical solution of the data processing apparatus, please refer to the description of the technical solution of the data processing method described above.

[0255] Figure 6 A structural block diagram of a computing device 600 according to one embodiment of this specification is shown. The components of the computing device 600 include, but are not limited to, a memory 610 and a processor 620. The processor 620 is connected to the memory 610 via a bus 630, and a database 650 is used to store data.

[0256] The computing device 600 also includes an access device 640, which enables the computing device 600 to communicate via one or more networks 660. Examples of these networks include Public Switched Telephone Network (PSTN), Local Area Network (LAN), Wide Area Network (WAN), Personal Area Network (PAN), or combinations of communication networks such as the Internet. The access device 640 may include one or more of any type of wired or wireless network interface (e.g., a network interface card (NIC)), such as an IEEE 802.11 Wireless Local Area Network (WLAN) wireless interface, a Wi-MAX (Worldwide Interoperability for Microwave Access) interface, an Ethernet interface, a Universal Serial Bus (USB) interface, a cellular network interface, a Bluetooth interface, or a Near Field Communication (NFC) interface.

[0257] In one embodiment of this specification, the above-described components of the computing device 600 and Figure 6 Other components, not shown, can also be connected to each other, for example, via a bus. It should be understood that... Figure 6 The block diagram of the computing device shown is for illustrative purposes only and is not intended to limit the scope of this specification. Those skilled in the art can add or replace other components as needed.

[0258] The computing device 600 can be any type of stationary or mobile computing device, including mobile computers or mobile computing devices (e.g., tablet computers, personal digital assistants, laptop computers, notebook computers, netbooks, etc.), mobile phones (e.g., smartphones), wearable computing devices (e.g., smartwatches, smart glasses, etc.) or other types of mobile devices, or stationary computing devices such as desktop computers or personal computers (PCs). The computing device 600 can also be a mobile or stationary server.

[0259] The processor 620 is configured to execute the following computer-executable instructions, which, when executed by the processor, implement the steps of the above-described data processing method.

[0260] The above is an illustrative scheme of a computing device according to this embodiment. It should be noted that the technical solution of this computing device and the technical solution of the data processing method described above belong to the same concept. For details not described in detail in the technical solution of the computing device, please refer to the description of the technical solution of the data processing method described above.

[0261] An embodiment of this specification also provides a computer-readable storage medium storing computer-executable instructions that, when executed by a processor, implement the steps of the above-described data processing method.

[0262] The above is an illustrative scheme of a computer-readable storage medium according to this embodiment. It should be noted that the technical solution of this storage medium and the technical solution of the data processing method described above belong to the same concept. For details not described in detail in the technical solution of the storage medium, please refer to the description of the technical solution of the data processing method described above.

[0263] An embodiment of this specification also provides a computer program, wherein when the computer program is executed in a computer, it causes the computer to perform the steps of the above-described data processing method.

[0264] The above is an illustrative example of a computer program according to this embodiment. It should be noted that the technical solution of this computer program and the technical solution of the data processing method described above belong to the same concept. Details not described in detail in the technical solution of the computer program can be found in the description of the technical solution of the data processing method described above.

[0265] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0266] The computer instructions include computer program code, which may be in the form of source code, object code, executable file, or certain intermediate forms. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording media, USB flash drive, portable hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium may be appropriately added or removed according to the requirements of patent practice. For example, in some regions, according to patent practice, computer-readable media may not include electrical carrier signals and telecommunication signals.

[0267] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that the embodiments in this specification are not limited to the described order of actions, because according to the embodiments in this specification, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in this specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to the embodiments in this specification.

[0268] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0269] The preferred embodiments disclosed above are merely illustrative of this specification. The optional embodiments do not exhaustively describe all details, nor do they limit the invention to the specific implementations described. Clearly, many modifications and variations can be made based on the embodiments described herein. These embodiments are selected and specifically described in this specification to better explain the principles and practical applications of the embodiments, thereby enabling those skilled in the art to better understand and utilize this specification. This specification is limited only by the claims and their full scope and equivalents.

Claims

1. A data processing method, characterized in that, include: Receive a data access request for a target object, and determine the corresponding data management unit based on the target object identifier in the data access request; Obtain the data index corresponding to the data management unit. The data index indicates the association between the object identifier and the sequence description information. The sequence description information indicates the regional positioning information of the data region where different data segments of the same data sequence are located. If the target sequence description information corresponding to the target object identifier is found in the data index, the data region to be accessed is determined based on the request type of the data access request and the target sequence description information, and the data access request is processed based on the region positioning information corresponding to the data region to be accessed.

2. The method according to claim 1, characterized in that, The step of determining the corresponding data management unit based on the target object identifier in the data access request includes: The target object identifier is mapped to a target object routing key through a target mapping function, wherein the target mapping function is used to map the same object identifier to the same object routing key and different object identifiers to different object routing keys; Based on the target object routing key and the set number of virtual buckets, the target virtual bucket corresponding to the target object routing key is determined, wherein the number of virtual buckets is a fixed value used to isolate the mapping relationship from object identifier to virtual bucket and the mapping relationship from virtual bucket to actual bucket; Based on the target virtual bucket and the established mapping relationship between virtual buckets and actual buckets, the target actual bucket corresponding to the target object routing key is determined, and the data management unit to which the target actual bucket belongs is determined as the data management unit corresponding to the target object identifier, wherein the actual bucket is a storage container used to store data.

3. The method according to claim 1, characterized in that, The step of determining the data region to be accessed based on the request type of the data access request and the target sequence description information includes: If the request type is a write request, then the writable incremental data area in the target sequence description information is determined as the data area to be accessed, wherein the writable incremental data area is used to store newly written data. The process of processing the data access request based on the regional location information corresponding to the data region to be accessed includes: Based on the writable region location information corresponding to the writable incremental data region, the write data carried by the write request is written into the writable incremental data region.

4. The method according to claim 1, characterized in that, The step of determining the data region to be accessed based on the request type of the data access request and the target sequence description information includes: If the request type is a read request, then determine whether there is disk area location information corresponding to the disk storage area in the target sequence description information, wherein the disk data area is used to store persistent historical storage data; If the disk area location information does not exist, the incremental data area in the target sequence description information will be used as the data area to be accessed. The process of processing the data access request based on the regional location information corresponding to the data region to be accessed includes: If the disk area location information does not exist, then based on the incremental area location information corresponding to the incremental data area, the incremental data of the target object is read from the incremental data area, and the incremental data is used as the response result of the read request.

5. The method according to claim 4, characterized in that, After determining whether the target sequence description information contains disk area location information corresponding to the disk storage area, the method further includes: If the disk area location information exists, then the disk stock area and incremental data area in the target sequence description information are taken as the data area to be accessed; The process of processing the data access request based on the regional location information corresponding to the data region to be accessed includes: If the disk area location information exists, then based on the disk area location information corresponding to the disk storage area, the target storage data segment of the historical storage data of the target object is read from the disk storage area; Based on the incremental region positioning information corresponding to the incremental data region, the incremental data of the target object is read from the incremental data region; The target existing data is segmented and merged with the incremental data to serve as the response result of the read request.

6. The method according to claim 4, characterized in that, The incremental data area includes a writable incremental data area and / or a frozen incremental data area. The writable incremental data area is used to store newly written data, and the frozen incremental data area is used to store read-only incremental data that is frozen during the data merging process. The incremental area positioning information includes writable area positioning information corresponding to the writable incremental data area and / or frozen area positioning information corresponding to the frozen incremental data area. The step of reading incremental data of the target object from the incremental data region based on the incremental region positioning information corresponding to the incremental data region includes: Based on the writable region positioning information, new data corresponding to the target object is read from the writable incremental data region; Based on the location information of the frozen area, read-only incremental data corresponding to the target object is read from the frozen incremental data area; The newly added data and / or the read-only incremental data are determined as the incremental data of the target object.

7. The method according to claim 5, characterized in that, The target storage data segment, which reads the historical storage data of the target object from the disk storage area based on the disk area location information corresponding to the disk storage area, includes: Based on the disk area location information, the corresponding sequence organization information is obtained, wherein the sequence organization information includes multiple data segment description information arranged in chronological order; Based on the query conditions carried in the read request, the target data segment description information is determined from the multiple data segment description information, and the storage location of the target data segment is determined from the data location information in the target data segment description information. If the storage location is a cache data area, then the target stored data segment is read from the cache data area, wherein the cache data area is used to store hot data that has been loaded from the disk storage area into memory; If the storage location is a disk data area, then the target stored data segment is read from the disk data area.

8. The method according to claim 7, characterized in that, The query criteria include the target start timestamp; The step of determining the target data segment description information from the plurality of data segment description information based on the query conditions carried in the read request includes: The first and last data segment description information in the plurality of data segment description information are respectively used as the start and end positions of the current search range; Obtain the intermediate data segment description information located at the middle position between the start position and the end position in the current search range, and obtain the intermediate start timestamp in the intermediate data segment description information; The intermediate start timestamp and the target start timestamp are matched, and the target data segment description information is determined from the multiple data segment description information based on the matching result.

9. The method according to claim 8, characterized in that, The process of matching the intermediate start timestamp and the target start timestamp, and determining the target data segment description information from the multiple data segment description information based on the matching result, includes: If the target start timestamp matches the intermediate start timestamp, then the intermediate data segmentation description information is determined as the target data segmentation description information; If the target start timestamp is earlier than the intermediate start timestamp, then the end position is updated to the position before the intermediate position, and the process returns to the step of obtaining the intermediate data segment description information located at the intermediate position between the start position and the end position in the current search range, until the target data segment description information is determined. If the target start timestamp is later than the intermediate start timestamp, then the start position is updated to the position after the intermediate position, and the process returns to the step of obtaining the intermediate data segment description information located at the intermediate position between the start position and the end position in the current search range, until the target data segment description information is determined.

10. The method according to claim 1, characterized in that, The method further includes: If no target sequence description information corresponding to the target object identifier is found in the data index, and the data access request is a read request, then it is determined that the data sequence corresponding to the target object identifier does not exist, and the read result is returned as empty.

11. The method according to claim 1, characterized in that, The method further includes: If no target sequence description information corresponding to the target object identifier is found in the data index, and the data access request is a write request, then target sequence description information corresponding to the target object identifier is created, the writable region location information corresponding to the writable incremental data region is recorded in the target sequence description information, and the association relationship between the target object identifier and the target sequence description information is stored in the data index; Based on the writable region location information corresponding to the writable incremental data region, the write data carried by the write request is written into the writable incremental data region.

12. A data processing apparatus, characterized in that, include: The management unit determination module is configured to receive a data access request for a target object and determine the corresponding data management unit based on the target object identifier in the data access request. The index acquisition module is configured to acquire the data index corresponding to the data management unit. The data index indicates the association between the object identifier and the sequence description information. The sequence description information indicates the regional positioning information of the data region where different data segments of the same data sequence are located. The request processing module is configured to, if the target sequence description information corresponding to the target object identifier is found in the data index, determine the data region to be accessed based on the request type of the data access request and the target sequence description information, and process the data access request based on the region positioning information corresponding to the data region to be accessed.

13. A computing device, characterized in that, include: Memory and processor; The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions, which, when executed by the processor, implement the steps of the data processing method according to any one of claims 1-11.

14. A computer-readable storage medium, characterized in that, It stores computer-executable instructions that, when executed by a processor, implement the steps of the data processing method according to any one of claims 1-11.

15. A computer program product, characterized in that, Includes a computer program / instructions that, when executed by a processor, implement the steps of the data processing method according to any one of claims 1-11.

Images