Label data processing method based on multi-partition architecture, terminal equipment, storage medium and multi-partition architecture

Abstract

The invention discloses a label data processing method based on a multi-partition architecture, terminal equipment, a storage medium and the multi-partition architecture, and belongs to the field of data processing.The method comprises the steps that a to-be-executed service operation and a target identifier are obtained through analysis; according to the target identifier, determining a first target integer ID and a target first version number corresponding to the label in the target partition; and sending a service operation request to a TsdServer layer to enable the TsdServer layer to locate to a corresponding target storage unit according to the first target integer ID, calling a target second version number corresponding to the first target integer ID in the target storage unit, and after it is determined that the target first version number is consistent with the target second version number through verification, sending a service operation request to the TsdServer layer, so as to enable the TsdServer layer to locate to the corresponding target storage unit according to the first target integer ID. And executing service operation on the label of the label configuration area or the service data block of the data area in the target storage unit. By implementing the method and the device, the problems of low label change efficiency and over-high data processing cost caused by strong binding of labels and data in the prior art can be solved.

Description

Technical Field

[0001] This invention relates to the field of data processing, and in particular to a tag data processing method, terminal device, storage medium, and multi-partition architecture based on a multi-partition architecture. Background Technology

[0002] With the rapid development of fields such as the Internet of Things and industrial monitoring, the scale of business data is growing exponentially, with the daily data volume of a single system reaching tens of TB. As the core carrier of business data management, the architecture design and tag data management capabilities of the business database directly determine the scalability, performance and operation and maintenance costs of the system.

[0003] Currently, mainstream business databases have significant limitations: business databases with integrated tag and data design (such as InfluxDB 2.x and Prometheus) have strong binding between tags and data within partitions. Tag changes require modification of all associated data, resulting in tag update time increasing linearly with data volume. 100GB of data can take more than 1 hour. Moreover, during migration or backup, because tags and data are not separated, partition size expands by 20%-30%, significantly increasing bandwidth and storage costs. Summary of the Invention

[0004] This invention provides a tag data processing method, terminal device, storage medium, and multi-partition architecture based on a multi-partition architecture. The method can solve the problems of low tag change efficiency and high data processing cost caused by the strong binding of tags and data in the prior art.

[0005] An embodiment of the present invention provides a tag data processing method based on a multi-partition architecture. The multi-partition architecture includes an SDK layer and a TsdServer layer. The SDK layer caches the integer ID and first version number corresponding to the tags in each partition. The TsdServer layer includes a storage unit corresponding to each partition. The storage unit includes a tag configuration area and a data area. The tag configuration area stores tags, the integer ID corresponding to the tags, and the second version number. The data area includes several business data blocks, and the business data blocks are associated with tags through integer IDs. The tag data processing method, applicable to the SDK layer, includes: In response to the business operation command sent by the user, the business operation to be executed and the target identifier are parsed. Based on the target identifier, determine the first target integer ID and the corresponding first target version number of the tag within the target partition; A service operation request is sent to the TsdServer layer. The service operation request includes the first target integer ID and the first target version number, so that the TsdServer layer can locate the corresponding target storage unit based on the first target integer ID and call the target second version number in the target storage unit that corresponds to the first target integer ID. After verifying that the target first version number and the target second version number are consistent, the service operation is executed on the tag or data block of the tag configuration area or data area in the target storage unit. The business processing results returned by the TsdServer layer are fed back to the user.

[0006] Furthermore, the SDK layer includes a TagCache module; The initialization process of the TagCache module includes: For each partition in the TsdServer layer, a tag retrieval request for the current partition is sent to the TsdServer layer, so that after receiving the tag retrieval request, the TsdServer layer returns all the tags of the current partition, the integer ID corresponding to each tag, and the second version number; Cache all tags in the current partition and the integer IDs corresponding to each tag; The second version number corresponding to each tag of the current partition is used as the first version number corresponding to each tag of the current partition in the TagCache module, and then cached.

[0007] Furthermore, the tag data processing method based on a multi-partition architecture also includes: After verifying that the target first version number and the target second version number are inconsistent, the error code returned by the TsdServer layer is parsed to obtain an integer ID for the table tag version mismatch, which is used as the second target integer ID; Send a refresh request and the second target integer ID to the TsdServer layer, so that the TsdServer layer returns the tag and second version number stored in the corresponding tag configuration area based on the second target integer ID; Based on the returned tag and second version number, the corresponding tag and first version number in the TagCache module are overwritten.

[0008] Furthermore, the business operation instruction includes a business data write instruction; the business operation includes business data writing; and the business operation request includes a business data write request. The execution of the business operation on the tag configuration area or data area of ​​the business data block within the target storage unit includes: The process involves iterating through all the business data to be written in the business data write request. During the iteration, each business data point is written into the business data block in the target storage unit that is associated with the first target integer ID, based on the first target integer ID associated with each business data to be written.

[0009] Furthermore, the service operation instruction includes a label change instruction; the service operation includes a label change; and the service operation request includes a label change request. The execution of the business operation on the tag configuration area or data area of ​​the business data block within the target storage unit includes: Based on the first target integer ID carried in the tag change request, the corresponding tag is queried in the tag configuration area of ​​the target storage unit, and the queried tag is modified. Update the second version number of the modified tag to obtain the updated second version number.

[0010] Furthermore, the service operation instruction includes a partition migration instruction; the service operation includes partition migration; and the service operation request includes a partition migration request. The execution of the business operation on the tag configuration area or data area of ​​the business data block within the target storage unit includes: The target partition is used as the source partition for the data to be migrated, and the target storage unit is used as the source storage unit. Create a receive partition for receiving migration data; wherein the receive partition has the same identifier, time window, and storage path format as the source partition; Generate a source storage unit corresponding to the receiving partition; The source storage unit corresponding to the source partition is transferred to the initial storage unit corresponding to the receiving partition. After the transfer is completed, the storage unit of the migrated receiving partition is obtained. Delete the source storage unit corresponding to the source partition.

[0011] Furthermore, the service operation instruction includes a partition backup instruction; the service operation includes a partition backup; and the service operation request includes a partition backup request. The execution of the business operation on the tag configuration area or data area of ​​the business data block within the target storage unit includes: Copy the target storage unit, and package the copied storage unit to obtain a backup file; Upload the backup file to the preset target address.

[0012] An embodiment of the present invention also provides a multi-partition architecture, including: an SDK layer and a TsdServer layer; the SDK layer caches the integer ID and first version number corresponding to the tags in each partition; the TsdServer layer includes storage units corresponding to each partition, the storage unit includes a tag configuration area and a data area, the tag configuration area stores tags, the integer ID corresponding to the tags and the second version number, and the data area includes a plurality of business data blocks, the business data blocks being associated with tags through integer IDs; The SDK layer is used to receive business operation instructions sent by the user; in response to the business operation instructions sent by the user, it parses the business operation to be executed and the target identifier; based on the target identifier, it determines the first target integer ID and the corresponding first target version number of the tag in the target partition; and sends a business operation request to the TsdServer layer, the business operation request including the first target integer ID and the first target version number. The TsdServer layer is used to locate the corresponding target storage unit based on the first target integer ID, and call the target second version number corresponding to the first target integer ID in the target storage unit. After verifying that the target first version number and the target second version number are consistent, the layer executes the business operation on the tag in the tag configuration area or the business data block in the data area of ​​the target storage unit.

[0013] This application also provides a terminal device, including: One or more processors; A memory, coupled to the processor, for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the tag data processing method based on a multi-partition architecture as described in the above embodiments of the invention.

[0014] This application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the tag data processing method based on a multi-partition architecture as described in the above embodiments of the invention.

[0015] The following benefits can be obtained by implementing the present invention: This invention provides a tag data processing method, terminal device, storage medium, and multi-partition architecture based on a multi-partition architecture. The method caches the integer ID and first version number corresponding to the tag in the SDK layer, and combines it with the second version number stored in the tag configuration area of ​​TsdServer to achieve lightweight version verification based on integer comparison. It does not require distributed transactions or full data traversal, which significantly reduces the overhead of consistency maintenance. When the tag changes, only the tag and second version number in the tag configuration area need to be updated, without modifying the business data blocks in the data area. This effectively solves the problem that tag changes in integrated design require full data modification. Attached Figure Description

[0016] To more clearly illustrate the technical solution of this application, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0017] Figure 1 This is a flowchart illustrating a tag data processing method based on a multi-partition architecture provided in a certain embodiment of this application; Figure 2 This is a schematic diagram of the structure of a multi-partition architecture for tag data processing based on a multi-partition architecture provided in a certain embodiment of this application; Figure 3 This is a schematic diagram of the structure of a terminal device provided in a certain embodiment of this application. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings of the embodiments. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0019] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains; the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the application; the terms “comprising” and “having”, and any variations thereof, in the specification, claims, and foregoing description of the drawings are intended to cover non-exclusive inclusion.

[0020] In the description of the embodiments of this application, technical terms such as "first" and "second" are used only to distinguish different objects and should not be construed as indicating or implying relative importance or implicitly specifying the number, specific order, or primary and secondary relationship of the indicated technical features. In the description of the embodiments of this application, "multiple" means two or more, unless otherwise explicitly defined.

[0021] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0022] In the description of the embodiments in this application, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, the character " / " in this document generally indicates that the preceding and following related objects have an "or" relationship.

[0023] In the description of the embodiments of this application, the term "multiple" refers to two or more (including two), similarly, "multiple sets" refers to two or more (including two sets), and "multiple pieces" refers to two or more (including two pieces).

[0024] In the description of the embodiments of this application, unless otherwise expressly specified and limited, technical terms such as "installation," "connection," "joining," and "fixing" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral part; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; they can refer to the internal communication of two components or the interaction between two components. For those skilled in the art, the specific meaning of the above terms in the embodiments of this application can be understood according to the specific circumstances.

[0025] See Figure 1To address the problems of low efficiency in tag changes and high data processing costs caused by the strong binding of tags and data in existing technologies, an embodiment of the present invention provides a tag data processing method based on a multi-partition architecture. The multi-partition architecture includes an SDK layer and a TsdServer layer. The SDK layer caches the integer ID and first version number corresponding to the tags in each partition. The TsdServer layer includes storage units corresponding to each partition. The storage unit includes a tag configuration area and a data area. The tag configuration area stores tags, the integer ID corresponding to the tags, and the second version number. The data area includes several business data blocks, and the business data blocks are associated with tags through integer IDs. As an illustration, the TsdServer layer is divided into several independent partitions according to different business dimensions (such as device type, business line) or different time windows (such as by day, by hour). Each partition serves as the smallest unit for related business operations such as data migration and data backup, ensuring the independence and simplicity of partition business operations. Within each partition, tag configuration information (including tags and the corresponding second version number) and business data blocks are stored in the same storage unit, without the need for separate storage across nodes. Specifically, the SDK layer has a built-in lock-free mapping caching mechanism that maps target identifiers to corresponding integer IDs. It automatically converts the target identifiers entered by the user into integer IDs (reducing network transmission and server matching overhead) and caches the current version number of the tag as the first version number. The TsdServer layer maintains the tag version number. When it receives a business operation request sent by the SDK layer, it only needs to compare whether the first version number and the second version number are consistent. No distributed transaction is required. If the version expires, an error code is returned, which triggers the SDK layer to refresh the cache and retry, thus ensuring tag data consistency in a lightweight manner. Specifically, the SDK layer is deployed on the user end (such as application server, collection terminal), providing APIs for tag creation, deletion, modification and query, and data read and write. The core components are the TagCache module and the request adaptation module. The TagCache module is used to implement the mapping between target identifiers and integer IDs (4 bytes), as well as version caching on the user end. The request adaptation module is used to convert the target identifier into an integer ID and generate a business operation request carrying the corresponding first version number. The integer ID replaces the string target identifier for transmission, reducing bandwidth consumption. The TsdServer layer is deployed on the server side and adopts a multi-partition architecture. Its core components are a multi-partition management module (used for partition creation, migration and backup), storage units and a version verification module (comparing tag versions to ensure consistency).

[0026] The following is an illustrative representation of the data structure of the multi-partition architecture: constexpr uint32_t kTagNameLength = 256; constexpr uint32_t kUnitLength= 16; / / 1. Tag configuration structure (including first version number, used for consistency verification) struct TagConfig final{uint32_t tag_id; / / Integer ID, unique identifier, assigned within the TsdServer partition, cannot be modified after creation char tag_name[kTagNameLength]; / / Tag name (unique within the partition) uint32_t version; / / Version number (set to 1 upon creation, increments by 1 with each update, used for consistency verification) / / Extended fields (such as tag type, unit, etc., updated with version, does not affect historical data) TagType tag_type; / / Tag type (digital / analog) char unit[kUnitLength]; / / Data unit (such as ℃, kPa)}; / / 2. Business data structure (including integer ID and version number, associated with tags within the partition) struct TimeSeriesData {uint32_t tag_id; / / Integer ID (converted from tag_name in the SDK layer) int64_t timestamp; / / Timestamp (milliseconds) double value; / / Business data value uint32_t tag_version; / / Tag version number when writing (used for TsdServer verification)}; / / 3. Partition metadata structure (identifies partition boundaries and status, ensuring overall operation for migration / backup) struct PartitionMeta { uint64_t partition_id; / / Unique partition ID (format: business identifier + time window, such as "device_grp1_20240901") int64_t start_time; / / Partition start timestamp (such as 2024-09-01 00:00:00) int64_t end_time; / / Partition end timestamp (such as 2024-09-01 23:59:59) char storage_block_path

[256] ; / / Partition storage unit path bool is_migrating; / / Partition migration status (true = in migration, write prohibited) bool is_backing_up; / / Partition backup status (true = backup in progress, write disabled)}; / / 4.SDK-side tag cache structure (lock-free design, reducing thread contention) struct TagCache {unordered_map<string, uint32_t> name_to_id; / / Tag_name→tag_id mapping (lock-free hash table) unordered_map<uint32_t, TagConfig> id_to_config; / / Tag_id → TagConfig mapping (including first version number) atomic. <bool>need_refresh; / / Cache refresh flag (atomic variable, avoids lock overhead) uint64_t partition_id; / / Bound partition ID (ensures a one-to-one correspondence between the cache and the partition)}; Specifically, the core of the above data structure is the strong association between tags and data within a partition: TagConfig contains a version number for consistency verification; TimeSeriesData associates tags within a partition through tag_id (i.e., integer ID) and carries tag_version (i.e., the first version number) to ensure tag state matching during writing; PartitionMeta defines partition boundaries and states to ensure that complete storage units can be located during data migration and backup; TagCache implements SDK-level tag mapping and version caching to reduce network transmission and server overhead; tag_id (i.e., integer ID) is an integer (4 bytes), replacing the string tag_name (tag name, i.e., target identifier) ​​for transmission, reducing bandwidth consumption; version is only associated with TagConfig, so TimeSeriesData does not need to be modified when tags change, significantly reducing change costs.

[0027] The tag data processing method, applicable to the SDK layer, includes: S1. In response to the business operation command sent by the user, parse the business operation to be executed and the target identifier; In a preferred embodiment, the SDK layer includes a TagCache module; The initialization process of the TagCache module includes: For each partition in the TsdServer layer, a tag retrieval request for the current partition is sent to the TsdServer layer, so that after receiving the tag retrieval request, the TsdServer layer returns all the tags of the current partition, the integer ID corresponding to each tag, and the second version number; Cache all tags in the current partition and the integer IDs corresponding to each tag; The second version number corresponding to each tag of the current partition is used as the first version number corresponding to each tag of the current partition in the TagCache module, and cached. Specifically, when the SDK layer starts, it needs to specify the target partition_id and send a tag retrieval request for the current partition to the TsdServer layer. After the TsdServer layer returns all TagConfigs in the partition, the SDK layer writes them into the TagCache module, constructs a mapping between name_to_id (e.g., "temp_sensor1" → 1001) and id_to_config (e.g., 1001 → {tag_id:1001, version:2, ...}), and sets need_refresh to false to ensure that subsequent requests can directly use the cache.

[0028] In a preferred embodiment, the tag data processing method based on a multi-partition architecture further includes: After verifying that the target first version number and the target second version number are inconsistent, the error code returned by the TsdServer layer is parsed to obtain an integer ID for the table tag version mismatch, which is used as the second target integer ID; Send a refresh request and the second target integer ID to the TsdServer layer, so that the TsdServer layer returns the tag and second version number stored in the corresponding tag configuration area based on the second target integer ID; Based on the returned tag and second version number, the corresponding tag and first version number in the TagCache module are overwritten; Specifically, when the SDK layer receives the "tag version expired" error code (ERROR_TAG_VERSION_OUTDATED) returned by the TsdServer layer, it immediately sets need_refresh to true, triggers an asynchronous refresh thread, and retrieves the latest TagConfig (i.e., the returned tag and second version number) for the partition from the TsdServer layer. After overwriting the local TagCache module, need_refresh is set to false. The entire refresh process takes ≤10ms, does not block user requests, and avoids affecting business continuity.

[0029] S2. Based on the target identifier, determine the first target integer ID and the corresponding first target version number corresponding to the tag within the target partition; Specifically, the following is the request construction process for mapping tag_name to tag_id: / / SDK-side tag name conversion and request building function int SDKTagConverter::convertAndBuildRequest(const string&tag_name,int64_t timestamp,double value,uint64_t partition_id,WriteRequest&req) { / / 1. Check if the cache is bound to the target partition (avoid cross-partition cache mixing) if (tag_cache.partition_id != partition_id) { / / Reinitialize the cache when switching partitions if (initTagCache(partition_id) != 0) {return ERROR_PARTITION_CACHE_INIT_FAILED;}} / / 2. Check if the cache needs to be refreshed (prefer the latest cache) if (tag_cache.need_refresh.load(memory_order_acquire)) {if (refreshTagCache(partition_id) != 0){return ERROR_CACHE_REFRESH_FAILED;}} / / 3. tag_name→tag_id conversion (lock-free hash table lookup, time ≤ 100ns) auto name_it = tag_cache.name_to_id.find(tag_name); if (name_it == tag_cache.name_to_id.end()) { / / Cache miss: query the tag's TagConfig config in a single query to TsdServer; if (getTagConfigFromServer(partition_id, tag_name, config) != 0) {return ERROR_TAG_NOT_EXIST;} / / Update cache tag_cache.name_to_id[tag_name] = config.tag_id; tag_cache.id_to_config[config.tag_id] = config; name_it = tag_cache.name_to_id.find(tag_name);} / / 4. Get the current first version number of the tag uint32_t tag_id = name_it->second; uint32_t tag_version = tag_cache.id_to_config[tag_id].version; / / 5. Construct the write request (carrying tag_id and version number, without tag_name string) TimeSeriesData data; data.tag_id = tag_id; data.timestamp = timestamp; data.value = value; data.tag_version = tag_version; req.partition_id = partition_id; req.data_list.push_back(data); return 0;}. Specifically, the above process converts the user-input tag_name (i.e., the target identifier) ​​into an integer ID and constructs the entire process of the business operation request. Key steps include: Check if the cache is bound to the target partition to avoid cross-partition cache mixing (e.g., "tag_name of partition A" mistakenly mapped to "tag_id of partition B"); if the partition is switched, automatically reinitialize the cache; use the atomic variable need_refresh to determine if the cache has expired, and trigger a refresh if it has expired to ensure that the latest tag configuration is used; use a lock-free hash table to achieve fast lookup of tag_name (i.e., target identifier) ​​to integer ID (time ≤ 100ns); if the cache misses, query the TsdServer layer once and update the cache to avoid repeated queries in subsequent requests; business operation requests only carry tag_id (i.e., the first target integer ID, 4 bytes) and tag_version (i.e., the first target version number, 4 bytes), without the tag_name string (usually 20-50 bytes), reducing network transmission volume by 80%-90%.

[0030] S3. Send a service operation request to the TsdServer layer. The service operation request includes the first target integer ID and the target first version number, so that the TsdServer layer can locate the corresponding target storage unit according to the first target integer ID, and call the target second version number in the target storage unit corresponding to the first target integer ID. After verifying that the target first version number and the target second version number are consistent, the service operation is performed on the tag or data block of the tag configuration area or data area in the target storage unit. As an illustration, the TsdServer layer adopts an overall multi-partition architecture, where tags and data within each partition are stored in the same storage unit (such as the file pointed to by storage_block_path). Core functions include "integrated storage of tags and data within a partition", "lightweight version verification", and "overall partition migration / backup". The tag configuration area stores all TagConfigs within the partition, using a B-tree index (key is tag_id), supporting version queries with O(logN) complexity. The B-tree index is loaded into memory, ensuring that tag version verification requires no disk I / O. The data area stores business data in blocks according to tag_id. Each business data block corresponding to tag_id is arranged in order by timestamp and directly associated with the tag_id in the tag configuration area. For example, the data block with tag_id=1001 directly corresponds to the TagConfig with tag_id=1001 in the tag configuration area. Therefore, when reading data, the TsdServer layer can quickly locate the tag configuration and corresponding data from the same storage unit without cross-file or cross-node queries. When migrating or backing up data, the storage unit can be directly copied without separating tags and data, and the operation complexity is consistent with the traditional integrated design.

[0031] In a preferred embodiment, the service operation instruction includes a service data write instruction; the service operation includes service data writing; and the service operation request includes a service data write request. The execution of the business operation on the tag configuration area or data area of ​​the business data block within the target storage unit includes: Traverse all the business data to be written in the business data write request. During the traversal, according to the first target integer ID associated with each business data to be written, write each business data point into the business data block in the data area of ​​the target storage unit associated with the first target integer ID. Specifically, the process of writing the scene is as follows: / / Data write and consistency check within the TsdServer layer partition int Partition::handleWriteRequest(const WriteRequest&req) { / / 1. Check partition status (writes are prohibited during migration / backup to avoid data inconsistency) if (meta.is_migrating || meta.is_backing_up) {return ERROR_PARTITION_BUSY;} / / 2. Add a shared lock within the partition (allows concurrent writes by multiple threads, prohibits label changes) shared_lock<shared_mutex> lock(partition_rw_lock); / / 3. Traverse data points and batch verify tag versions (one lock operation within the same partition, reducing overhead) for (const auto&data : req.data_list) { / / 3.1 Query tags from the tag configuration area (in-memory B-tree index, time ≤ 500ns) auto tag_it = tag_configs.find(data.tag_id); if (tag_it == tag_configs.end()) {return ERROR_TAG_NOT_EXIST_IN_PARTITION;} / / 3.2 Version verification: only compare integer version numbers (no string matching, no distributed coordination) if(tag_it->second.version != data.tag_version) {return ERROR_TAG_VERSION_OUTDATED; / / Return error code, trigger SDK refresh}} / / 4. Write to the data area (within the same storage unit, sequential writing, time ≤ 1ms / 1000 data points) for (const auto&data : req.data_list) {data_blocks[data.tag_id].push_back(data);} / / 5. Asynchronous flush to disk (ensures data persistence without blocking write requests) asyncFlushToStorageBlock(); return 0;} Specifically, after completing the target partition status check (if the partition is in migration or backup, writing is prohibited and a partition busy error code is returned), and applying a shared lock to the target partition (allowing multi-threaded concurrent writing to improve throughput, while prohibiting label changes to avoid conflicts between writing and label updates), all the business data to be written in the business data write request are then traversed. During the traversal, for each business data to be written associated with the first target integer ID (i.e., the unique identifier corresponding to the label), the business data block bound to the integer ID in the target storage unit data area is quickly located through the memory index structure. Then, each business data point is written to the corresponding data block in the traversal order, realizing the precise association storage of business data and labels (the business data block in the data area is configured with label information in the label configuration area corresponding to the first target integer ID).

[0032] In a preferred embodiment, the service operation instruction includes a label change instruction; the service operation includes a label change; and the service operation request includes a label change request. The execution of the business operation on the tag configuration area or data area of ​​the business data block within the target storage unit includes: Based on the first target integer ID carried in the tag change request, the corresponding tag is queried in the tag configuration area of ​​the target storage unit, and the queried tag is modified. Update the second version number of the modified label to obtain the updated second version number; Specifically, the process of changing labels is as follows: / / TsdServer layer tag change handling int Partition::handleTagUpdate(constTagUpdateRequest&req) { / / 1. Add an exclusive lock within the partition (disallow writes, allow reads) unique_lock<shared_mutex> lock(partition_rw_lock); / / 2. Check if the tag exists in the current partition auto tag_it = tag_configs.find(req.tag_id); if (tag_it == tag_configs.end()) {return ERROR_TAG_NOT_EXIST_IN_PARTITION;} / / 3. Check the requested version number (to avoid concurrent change conflicts) if(tag_it->second.version != req.current_version) {return ERROR_TAG_VERSION_CONFLICT;} / / 4. Update the tag configuration (only modify the tag configuration area, do not touch the data area) TagConfig new_config = req.new_config; new_config.version = tag_it->second.version + 1; / / Increment the version number tag_configs[req.tag_id] = new_config; / / 5. Synchronize and flush the tag configuration area to disk (to ensure the version number is persistent) flushTagConfigToStorageBlock(); return 0;} Specifically, after applying an exclusive lock to the target partition (disabling data write operations to avoid version confusion caused by tag changes and concurrent writes, while allowing read operations without affecting query services), the corresponding tag is queried in the tag configuration area of ​​the target storage unit based on the first target integer ID carried in the tag change request; if the tag is not found, a tag not found error code is returned; if the tag is found, it is necessary to first verify whether the current first version number carried in the request is consistent with the second version number stored in the tag configuration area (to avoid concurrency conflicts caused by multiple threads modifying at the same time), and after the verification is successful, the configuration information (such as extended fields) of the tag is modified; After modification, the second version number of the tag is incremented according to the rule of "original version number + 1" to obtain the updated second version number, and then synchronously updated to the tag configuration area. Finally, the latest content of the tag configuration area (including the updated second version number) is persisted to the storage medium through synchronous disk flushing operation to ensure that the version change will not be lost due to node failure, and the tag change operation is finally completed.

[0033] In a preferred embodiment, the service operation instruction includes a partition migration instruction; the service operation includes partition migration; and the service operation request includes a partition migration request. The execution of the business operation on the tag configuration area or data area of ​​the business data block within the target storage unit includes: The target partition is used as the source partition for the data to be migrated, and the target storage unit is used as the source storage unit. Create a receive partition for receiving migration data; wherein the receive partition has the same identifier, time window, and storage path format as the source partition; Generate a source storage unit corresponding to the receiving partition; The source storage unit corresponding to the source partition is transferred to the initial storage unit corresponding to the receiving partition. After the transfer is completed, the storage unit of the migrated receiving partition is obtained. Delete the source storage unit corresponding to the source partition; Specifically, during the partition migration process, the target partition is first used as the source partition for the data to be migrated, and its corresponding storage unit is used as the source storage unit. The source partition is marked as "migrating" (meta.is_migrating=true) and set to "write prohibited, read allowed" status to avoid inconsistencies between the source and target data due to data changes during the migration. Subsequently, a receiving partition with metadata (including identifier, time window, and storage path format) completely identical to that of the source partition is created on the target TsdServer, and an initial storage unit containing an empty label configuration area and a data area is generated. Next, the source TsdServer transmits the complete source storage unit (including the label configuration area and data area) to the initial storage unit of the receiving partition in a binary file copy manner. The transmission process does not require splitting files or handling the association between labels and data. After the transmission is completed, the target TsdServer verifies the integrity by comparing the file hash value. If the verification is successful, the global routing table is updated (pointing the partition identifier to the target TsdServer node), and the migrated receiving partition storage unit is obtained. Finally, the source TsdServer retains the source storage unit for 72 hours for fault rollback, then removes the "migrating" mark and deletes the source storage unit, completing the entire migration process. Because this process does not require additional handling of the association between tags and data, the migration time is reduced by more than 48% compared to the separate design.

[0034] In a preferred embodiment, the service operation instruction includes a partition backup instruction; the service operation includes a partition backup; and the service operation request includes a partition backup request. The execution of the business operation on the tag configuration area or data area of ​​the business data block within the target storage unit includes: Copy the target storage unit, and package the copied storage unit to obtain a backup file; Upload the backup file to the preset target address; Specifically, in the partition backup process, the target partition is first marked as "backing up" (meta.is_backing_up=true) and set to "write prohibited, read allowed" status to ensure the integrity of the backup data. Then, the target storage unit (including the tag configuration area and data area) corresponding to this partition is directly copied. The copied storage unit is packaged into a backup file in the format "partition_id_YYYYMMDD_HHMMSS.bak" (e.g., "device_grp1_20240901_143000.bak"). The packaging process does not require separating the tag configuration area and the data area, nor does it require additional compression. Next, the backup file is uploaded to a preset target address (e.g., local NAS, object storage S3, etc.). The backup log only records the partition identifier (partition_id), backup time, and file path; there is no need to separately record the association between the tag and the data. Finally, the "backing up" status of the target partition is removed. The tag (meta.is_backing_up=false) restores its write permissions; the entire process does not require additional tag and data association verification steps, the backup time is reduced by more than 60% compared to the separate design, and the recovery can be done by simply uncompressing the backup file without rebuilding the tag and data association.

[0035] S4. Feedback the business processing results returned by the TsdServer layer to the user; Specifically, in this invention, tag changes only require updating the second version number of the tag configuration area within the partition (memory operation, time ≤1ms), without modifying the historical business data blocks in the data area. Actual testing shows a 30% increase in throughput. This invention integrates the tag configuration area and data area within the partition, directly packaging the complete storage block file (including the tag_id (integer ID) of the tag configuration area, tag information, and business data blocks of the data area) during migration / backup, eliminating the need for file splitting or verifying relationships. Secondly, this invention caches the tag_id (integer ID) and first version number corresponding to the tag at the SDK layer, and verifies the consistency between the first and second version numbers in memory at the TsdServer layer, eliminating the need for distributed transactions. Furthermore, this invention converts the tag identifier into a 4-byte tag_id (integer ID) at the SDK layer, reducing tag transmission volume by 80%-90%, and the TsdServer layer only requires integer version comparison, reducing CPU overhead by 60%. Actual testing shows that under the same request volume, network bandwidth usage is reduced by 55% compared to existing technologies.

[0036] See Figure 2 This invention provides a multi-partition architecture, comprising: an SDK layer and a TsdServer layer; the SDK layer caches integer IDs and first version numbers corresponding to tags in each partition; the TsdServer layer includes storage units corresponding to each partition, each storage unit including a tag configuration area and a data area, the tag configuration area storing tags, integer IDs corresponding to the tags, and second version numbers, the data area including several business data blocks, the business data blocks being associated with tags through integer IDs; The SDK layer is used to receive business operation instructions sent by the user; in response to the business operation instructions sent by the user, it parses the business operation to be executed and the target identifier; based on the target identifier, it determines the first target integer ID and the corresponding first target version number of the tag in the target partition; and sends a business operation request to the TsdServer layer, the business operation request including the first target integer ID and the first target version number. The TsdServer layer is used to locate the corresponding target storage unit based on the first target integer ID, and call the target second version number corresponding to the first target integer ID in the target storage unit. After verifying that the target first version number and the target second version number are consistent, the layer executes the business operation on the tag in the tag configuration area or the business data block in the data area of ​​the target storage unit.

[0037] See Figure 3 One embodiment of this application also provides a terminal device, including: One or more processors; A memory, coupled to the processor, for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the tag data processing method based on the multi-partition architecture as described above.

[0038] The processor controls the overall operation of the terminal device to complete all or part of the steps of the tag data processing method based on the multi-partition architecture described above. The memory stores various types of data to support the operation of the terminal device. This data may include, for example, instructions for any application or method operating on the terminal device, as well as application-related data. The memory can be implemented using any type of volatile or non-volatile storage device or a combination thereof, such as Static Random Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read-Only Memory (EPROM), Programmable Read-Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0039] In an exemplary embodiment, the terminal device may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the tag data processing method based on a multi-partition architecture as described in any of the foregoing embodiments, and to achieve the same technical effects as the methods described above.

[0040] In another exemplary embodiment, a computer-readable storage medium including a computer program is also provided. When executed by a processor, the computer program implements the steps of the tag data processing method based on a multi-partition architecture as described in any of the foregoing embodiments. For example, the computer-readable storage medium may be the aforementioned memory including the computer program, which may be executed by a processor of a terminal device to complete the tag data processing method based on a multi-partition architecture as described in any of the foregoing embodiments and achieve the same technical effects as the aforementioned method.

[0041] The above description represents the preferred embodiments of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of the present invention, and these improvements and modifications are also considered to be within the scope of protection of the present invention.< / bool>

Claims

1. A tag data processing method based on a multi-partition architecture, characterized in that, The multi-partition architecture includes an SDK layer and a TsdServer layer. The SDK layer caches the integer ID and first version number corresponding to the tags in each partition. The TsdServer layer includes storage units corresponding to each partition. Each storage unit includes a tag configuration area and a data area. The tag configuration area stores tags, the integer ID corresponding to the tags, and the second version number. The data area includes several business data blocks, and the business data blocks are associated with tags through integer IDs. The tag data processing method, applicable to the SDK layer, is characterized by comprising: In response to the business operation command sent by the user, the business operation to be executed and the target identifier are parsed. Based on the target identifier, determine the first target integer ID and the corresponding first target version number of the tag within the target partition; A service operation request is sent to the TsdServer layer. The service operation request includes the first target integer ID and the first target version number, so that the TsdServer layer can locate the corresponding target storage unit based on the first target integer ID and call the target second version number in the target storage unit that corresponds to the first target integer ID. After verifying that the target first version number and the target second version number are consistent, the service operation is executed on the tag or data block of the tag configuration area or data area in the target storage unit. The business processing results returned by the TsdServer layer are fed back to the user.

2. The tag data processing method based on a multi-partition architecture as described in claim 1, characterized in that, The SDK layer includes a TagCache module; The initialization process of the TagCache module includes: For each partition in the TsdServer layer, a tag retrieval request for the current partition is sent to the TsdServer layer, so that after receiving the tag retrieval request, the TsdServer layer returns all the tags of the current partition, the integer ID corresponding to each tag, and the second version number; Cache all tags in the current partition and the integer IDs corresponding to each tag; The second version number corresponding to each tag of the current partition is used as the first version number corresponding to each tag of the current partition in the TagCache module, and then cached.

3. The tag data processing method based on a multi-partition architecture as described in claim 2, characterized in that, Also includes: After verifying that the target first version number and the target second version number are inconsistent, the error code returned by the TsdServer layer is parsed to obtain an integer ID for the table tag version mismatch, which is used as the second target integer ID; Send a refresh request and the second target integer ID to the TsdServer layer, so that the TsdServer layer returns the tag and second version number stored in the corresponding tag configuration area based on the second target integer ID; Based on the returned tag and second version number, the corresponding tag and first version number in the TagCache module are overwritten.

4. The tag data processing method based on a multi-partition architecture as described in claim 1, characterized in that, The business operation instruction includes a business data write instruction; the business operation includes business data writing; the business operation request includes a business data write request. The execution of the business operation on the tag configuration area or data area of ​​the business data block within the target storage unit includes: The process involves iterating through all the business data to be written in the business data write request. During the iteration, each business data point is written into the business data block in the target storage unit that is associated with the first target integer ID, based on the first target integer ID associated with each business data to be written.

5. The tag data processing method based on a multi-partition architecture as described in claim 1, characterized in that, The business operation instruction includes a label change instruction; the business operation includes a label change; the business operation request includes a label change request. The execution of the business operation on the tag configuration area or data area of ​​the business data block within the target storage unit includes: Based on the first target integer ID carried in the tag change request, the corresponding tag is queried in the tag configuration area of ​​the target storage unit, and the queried tag is modified. Update the second version number of the modified tag to obtain the updated second version number.

6. The tag data processing method based on a multi-partition architecture as described in claim 1, characterized in that, The business operation instructions include partition migration instructions; the business operation includes partition migration. The business operation request includes a partition migration request; The execution of the business operation on the tag configuration area or data area of ​​the business data block within the target storage unit includes: The target partition is used as the source partition for the data to be migrated, and the target storage unit is used as the source storage unit. Create a receive partition for receiving migration data; wherein the receive partition has the same identifier, time window, and storage path format as the source partition; Generate a source storage unit corresponding to the receiving partition; The source storage unit corresponding to the source partition is transferred to the initial storage unit corresponding to the receiving partition. After the transfer is completed, the storage unit of the migrated receiving partition is obtained. Delete the source storage unit corresponding to the source partition.

7. The tag data processing method based on a multi-partition architecture as described in claim 1, characterized in that, The business operation instruction includes a partition backup instruction; the business operation includes a partition backup; the business operation request includes a partition backup request. The execution of the business operation on the tag configuration area or data area of ​​the business data block within the target storage unit includes: Copy the target storage unit, and package the copied storage unit to obtain a backup file; Upload the backup file to the preset target address.

8. A multi-partition architecture, characterized in that, include: SDK layer and TsdServer layer; The SDK layer caches the integer ID and first version number corresponding to the tags in each partition; the TsdServer layer includes storage units corresponding to each partition, the storage unit includes a tag configuration area and a data area, the tag configuration area stores tags, the integer ID corresponding to the tags and the second version number, and the data area includes several business data blocks, and the business data blocks are associated with tags through integer IDs; The SDK layer is used to receive business operation instructions sent by the user; In response to the business operation command sent by the user, the business operation to be executed and the target identifier are parsed. Based on the target identifier, determine the first target integer ID and the corresponding first target version number corresponding to the tag within the target partition; send a service operation request to the TsdServer layer, the service operation request including the first target integer ID and the first target version number; The TsdServer layer is used to locate the corresponding target storage unit based on the first target integer ID, and call the target second version number corresponding to the first target integer ID in the target storage unit. After verifying that the target first version number and the target second version number are consistent, the layer executes the business operation on the tag in the tag configuration area or the business data block in the data area of ​​the target storage unit.

9. A terminal device, characterized in that, include: One or more processors; A memory, coupled to the processor, for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the tag data processing method based on a multi-partition architecture as described in any one of claims 1-7.

10. A storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the tag data processing method based on a multi-partition architecture as described in any one of claims 1-7.

Images