Method and apparatus for controlling label data flow based on write filtering and frequency routing

Abstract

Embodiments of the present disclosure disclose a label data flow control method and device based on write filtering and frequency routing. A specific implementation of the method includes: performing change amount filtering processing on an obtained to-be-stored label data set to obtain a filtered label data set; performing time limit anchoring write processing on the filtered label data set to obtain a retained label data set; performing rolling window aggregation processing on the retained label data set to obtain a write frequency set; performing bidirectional amplitude limiting routing processing on a storage attribution information set to obtain a target storage attribution information set; performing migration processing on the retained label data set to obtain a migrated label data set; performing per-dimension pruning processing on the migrated label data set to obtain a to-be-output label data set; and performing label data flow control on a data storage server set. The implementation can reduce waste of storage resources and computing resources of the data storage server set.

Description

Technical Field

[0001] The embodiments disclosed herein relate to the field of computer technology, and more specifically to a method and apparatus for controlling tag data flow based on write filtering and frequency routing. Background Technology

[0002] The tag data flow control method based on write filtering and frequency routing refers to a method for controlling the writing, storage location, and output range of tag data in a data storage server cluster. Write filtering refers to comparing the tag data to be written with the tag data already stored in the data storage server cluster, and determining whether to execute the write operation based on whether the difference between the two reaches a preset range. Frequency routing refers to determining the storage location of the tag data based on the number of times the tag data is actually written within a preset time period.

[0003] In recent years, the number of active users of information service platforms has been large and has been growing (for example, the average daily active users of securities service applications in April 2026 were approximately 72.17 million, and the monthly active users increased from approximately 167 million in April 2025 to approximately 180 million in April 2026). Information service platforms maintain multiple tags for each user, and the number of tag data write requests arriving at the data storage server set per unit time increases accordingly, and is significantly higher during peak user operation periods (e.g., the opening and closing periods of trading days) than during off-peak periods.

[0004] The typical approach to controlling the flow of tag data is as follows: First, storage locations are divided according to preset tag categories, and the quantity and duration of historical tag data retained in each storage location are configured according to these preset categories. Then, each collection of tag data is treated as a write request and written to the storage location corresponding to the tag's category. Finally, in response to a received retrieval request, tag data with a timestamp (i.e., the time recorded when the tag data was written) later than the last retrieval time indicated by the retrieval request is read from the corresponding storage location and output.

[0005] However, in practice, it has been found that when using the above method to control the flow of tag data, the following technical problem often occurs: Since the tag data generated in each acquisition is written without distinguishing whether the tag value has changed, tag data with unchanged values ​​is repeatedly written and occupies storage space. Furthermore, the timestamps of these unchanged tag data are refreshed with each write. When responding to an acquisition request, this changed tag data is identified and read and output. To address this, the conventional solution is to compare the tag data to be written with the already stored tag data before writing, skipping write requests where the difference does not reach a preset threshold. However, conventional... The current solution still divides storage locations according to the preset categories of tags. The quantity and duration of the historical tag data retained are configured according to the above preset categories, which is not compatible with the frequency of actual tag data writing. The number of historical tag data retained in the data storage server set increases due to tag data with frequently changing values, which increases the storage space occupied by the data storage server set. At the same time, tag data with different frequency of value changes are still stored in the same storage location. Tag data with long-term unchanged values ​​are read together when reading the above storage location, which increases the amount of data read in a single time, leading to an increase in the load on the data storage server set and wasting the storage and computing resources of the data storage server set.

[0006] The information disclosed in this background section is only intended to enhance the understanding of the background of the present disclosure concept, and therefore may contain information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0007] The summary portion of this disclosure is intended to provide a brief overview of the concepts, which will be described in detail in the detailed description portion. This summary portion is not intended to identify key or essential features of the claimed technical solutions, nor is it intended to limit the scope of the claimed technical solutions.

[0008] Some embodiments of this disclosure propose a tag data flow control method and apparatus based on write filtering and frequency routing to solve one or more of the technical problems mentioned in the background section above.

[0009] In a first aspect, some embodiments of this disclosure provide a tag data flow control method based on write filtering and frequency routing, comprising: in response to receiving a tag dataset to be stored, obtaining a stored tag dataset and a storage ownership information set corresponding to the tag dataset to be stored from a data storage server set; performing change filtering processing on the tag dataset to be stored according to the stored tag dataset to obtain a filtered tag dataset; performing time-limited anchored write processing on the filtered tag dataset according to the storage ownership information set to obtain a retained tag dataset; performing rolling window aggregation processing on the retained tag dataset to obtain a write frequency set; performing bidirectional amplitude-limited routing processing on the storage ownership information set according to the write frequency set to obtain a target storage ownership information set; performing migration processing on the retained tag dataset according to the target storage ownership information set to obtain a migrated tag dataset; in response to receiving a tag acquisition request, performing dimension-by-dimensional pruning processing on the migrated tag dataset according to the target storage ownership information set and the synchronization timestamp included in the tag acquisition request to obtain a tag dataset to be output; and performing tag data flow control on the data storage server set according to the migrated tag dataset and the tag dataset to be output.

[0010] Secondly, some embodiments of this disclosure provide a tag data flow control device based on write filtering and frequency routing, comprising: an acquisition unit configured to, in response to receiving a tag dataset to be stored, acquire from a data storage server set an existing tag dataset and a storage ownership information set corresponding to the tag dataset to be stored; a filtering unit configured to perform change filtering processing on the tag dataset to be stored based on the existing tag dataset to obtain a filtered tag dataset; a writing unit configured to perform time-limited anchored writing processing on the filtered tag dataset based on the storage ownership information set to obtain a retained tag dataset; and an aggregation unit configured to perform a scrolling window aggregation of the retained tag dataset. The system is configured to: obtain a write frequency set; a routing unit, configured to perform bidirectional bandwidth-limited routing processing on the storage ownership information set based on the write frequency set to obtain a target storage ownership information set; a migration unit, configured to perform migration processing on the retained tag dataset based on the target storage ownership information set to obtain a migrated tag dataset; and a control unit, configured to, in response to receiving a tag acquisition request, perform dimension-by-dimensional pruning processing on the migrated tag dataset based on the target storage ownership information set and the synchronization timestamp included in the tag acquisition request to obtain a tag dataset to be output, and perform tag data flow control on the data storage server set based on the migrated tag dataset and the tag dataset to be output.

[0011] Thirdly, some embodiments of this disclosure provide an electronic device, including: one or more processors; and a storage device having one or more programs stored thereon, such that when the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any implementation of the first aspect.

[0012] Fourthly, some embodiments of this disclosure provide a computer-readable medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the method as described in any implementation of the first aspect.

[0013] The above embodiments of this disclosure have the following beneficial effects: The tag data flow control method based on write filtering and frequency routing in some embodiments of this disclosure can reduce the storage space occupation of the data storage server set and reduce the load of the data storage server set. Specifically, the reason for the waste of storage and computing resources of the data storage server set is that: the tag data generated by each collection is written without distinguishing whether the tag value has changed. Tag data with unchanged values ​​is repeatedly written and the data timestamp is refreshed, which increases the number of retained historical tag data, and is read and output together when responding to the acquisition request; at the same time, the storage location is divided according to the preset category of the tag, and the number and duration of the retained historical tag data are not commensurate with the frequency of actual tag writing. Tag data with different value change frequencies are stored in the same storage location, which increases the amount of data read in a single time. Based on this, the tag data flow control method based on write filtering and frequency routing in some embodiments of this disclosure can first, in response to receiving the tag dataset to be stored, obtain the stored tag dataset and storage ownership information set corresponding to the tag dataset to be stored from the data storage server set. Here, the existing tag dataset provides a comparison benchmark for subsequent change filtering, and the storage ownership information set provides a storage location for subsequent time-limited anchored write processing. Next, based on the existing tag dataset, change filtering is performed on the tag dataset to be stored, resulting in a filtered tag dataset. Here, the number of write requests and the amount of retained historical tag data are reduced accordingly. Then, based on the storage ownership information set, time-limited anchored write processing is performed on the filtered tag dataset, resulting in a retained tag dataset. Here, expired tag data automatically becomes invalid, and the data storage server set does not require further cleanup. Next, rolling window aggregation processing is performed on the retained tag dataset to obtain a write frequency set. Here, the write frequency is updated according to the recent write activity of the tags and is not smoothed by distant historical records. Next, based on the write frequency set, bidirectional bandwidth-limited routing processing is performed on the storage ownership information set to obtain the target storage ownership information set. Here, redundant historical versions are reduced, and upper and lower limits ensure that the duration is not too short, preventing duplicate writes. Next, based on the aforementioned target storage ownership information set, the retained tag dataset is migrated to obtain a migrated tag dataset. Here, the amount of historical tag data retained in each storage location matches the configured storage duration of that storage location. Finally, in response to a received tag retrieval request, based on the aforementioned target storage ownership information set and the synchronization timestamp included in the tag retrieval request, the migrated tag dataset is pruned dimension-by-dimensional to obtain a tag dataset to be output. Furthermore, based on the migrated tag dataset and the tag dataset to be output, tag data flow control is performed on the aforementioned data storage server set.Here, the scanning range, storage space usage, and memory cache usage of the data storage server set are reduced accordingly. Therefore, this tag data flow control method based on write filtering and frequency routing can reduce the storage space usage of the data storage server set and lower its load. Attached Figure Description

[0014] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and elements are not necessarily drawn to scale.

[0015] Figure 1 This is a flowchart of some embodiments of the tag data flow control method based on write filtering and frequency routing according to the present disclosure; Figure 2 This is a schematic diagram of the structure of some embodiments of the tag data flow control device based on write filtering and frequency routing according to the present disclosure; Figure 3 This is a schematic diagram of the structure of an electronic device suitable for implementing some embodiments of the present disclosure. Detailed Implementation

[0016] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0017] It should also be noted that, for ease of description, only the parts relevant to the invention are shown in the accompanying drawings. Unless otherwise specified, the embodiments and features described in this disclosure can be combined with each other.

[0018] It should be noted that the concepts of "first" and "second" mentioned in this disclosure are used only to distinguish different devices, modules or units, and are not used to limit the order of functions performed by these devices, modules or units or their interdependencies.

[0019] It should be noted that the terms "a" and "a plurality of" used in this disclosure are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0020] The names of messages or information exchanged between multiple devices in the embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.

[0021] This disclosure will now be described in detail with reference to the accompanying drawings and embodiments.

[0022] Figure 1 A flowchart 100 is shown, illustrating some embodiments of a tag data flow control method based on write filtering and frequency routing according to this disclosure. This tag data flow control method based on write filtering and frequency routing includes the following steps: Step 101: In response to receiving the tag dataset to be stored, retrieve the existing tag dataset and storage ownership information set corresponding to the tag dataset to be stored from the data storage server set.

[0023] In some embodiments, the execution entity (e.g., an electronic device) of the tag data flow control method based on write filtering and frequency routing described above can, in response to receiving a tag dataset to be stored, obtain the existing tag dataset and storage ownership information set corresponding to the tag dataset to be stored from the data storage server set. The data storage server in the data storage server set can be a storage server that stores multiple versions of the same tag data in an append-only manner and supports configuring the number and duration of versions retained according to storage location. For example, the data storage server set can be a server cluster deployed with Apache HBase. The storage location can be a logical partition in the data storage server set used to store tag data of different types. For example, the storage location can be a column family in Apache HBase. The tag data to be stored in the tag dataset to be stored can be key-value data describing a characteristic of an object (e.g., a user of an information service platform) to be written to the data storage server set. The tag data to be stored can include, but is not limited to, at least one of the following: primary key identifier, tag identifier, data type identifier, and tag value. The primary key identifier can be an identifier used to uniquely determine the object to which the tag data belongs in the data storage server set. For example, the primary key identifier can be a row key in Apache HBase. The aforementioned tag identifiers can be identifiers used to distinguish different tags under the same object. For example, the aforementioned tag identifiers can be name strings for tags such as "belonging organization identifier," "common item category identifier," "number of logins in the last seven days," and "number of visits in the last hour." The aforementioned data type identifiers can be identifiers representing the data type of the tag value. For example, the aforementioned data type identifiers can be identifiers representing numeric types or identifiers representing category types. The aforementioned tag values ​​can be the values ​​obtained by the tags in this data collection. For example, numeric tag values ​​can be the integer values ​​corresponding to "number of logins in the last seven days," and category tag values ​​can be the category name string corresponding to "common item category identifier." The aforementioned stored tag data in the aforementioned stored tag dataset can be tag data already stored in the data storage server set, whose primary key identifier and tag identifier are the same as the aforementioned tag data to be stored. The aforementioned stored tag values ​​can be the values ​​stored by the tags corresponding to the aforementioned stored tag data during the most recent write operation. The aforementioned storage ownership information in the aforementioned storage ownership information set can be information representing the storage location of the tag data to be stored in the data storage server set. The aforementioned storage attribution information may include, but is not limited to, at least one of the following: storage location identifier and storage validity period. The storage location identifier may be an identifier used to uniquely identify a storage location within a data storage server set. For example, the storage location identifier may be an Apache HBase column family name. The storage validity period may be the duration for which the tagged data is retained in the storage location corresponding to the aforementioned storage location identifier.For example, the storage validity period mentioned above can be the Time To Live (TTL) configured by column family in Apache HBase. The storage ownership information is determined by the tag identifier, and each tag data corresponding to the same tag identifier has the same storage ownership information.

[0024] In some optional implementations of certain embodiments, obtaining the existing tag dataset and storage attribution information set corresponding to the tag dataset to be stored from the data storage server set may include the following steps: The first step involves querying the data storage server set based on the primary key identifier set included in the aforementioned tag dataset to obtain the historical tag dataset and attribution metadata. The historical tag data in the historical tag dataset can be various versions of tag data already stored in the data storage server set, corresponding to the same primary key identifier and the same tag identifier. The attribution metadata can be data recording the storage attribution information currently corresponding to each tag identifier. For example, the attribution metadata can be an attribution metadata table stored independently of the tag data in the data storage server set. Each row of the attribution metadata table records a tag identifier, along with the corresponding storage location identifier, storage validity period, migration timestamp, and number of consecutive low-frequency windows. In practice, the execution entity can first determine each primary key identifier in the aforementioned primary key identifier set as the row key for a tag data read request, obtaining a tag data read request set. Then, using the `readAllVersions` method of the Apache HBase `Get` class, the number of returned versions for each tag data read request in the aforementioned tag data read request set is determined to be all versions, obtaining a full-version read request set. Next, the `get` method of the Apache HBase `Table` interface (with a list of `Get` objects as input) is used to submit the entire set of read requests to the aforementioned data storage server set, obtaining the historical tag dataset. Finally, the `get` method is used to retrieve the rows corresponding to the tag identifier sets included in the aforementioned tag dataset to be stored from the aforementioned attribution metadata table, obtaining the attribution metadata.

[0025] The second step involves performing time-series filtering on the aforementioned historical tag dataset to obtain the existing tag dataset. In practice, the executing entity can, for each historical tag data in the aforementioned historical tag dataset, determine the tag value of the latest version of the data timestamp in the aforementioned historical tag data as the corresponding existing tag value, thus obtaining the existing tag dataset.

[0026] The third step involves matching the attribution metadata with the tag identifier set included in the aforementioned tag dataset to obtain an existing attribution information set and a missing tag identifier set. The existing attribution information in the existing attribution information set can be storage attribution information already recorded in the attribution metadata. The missing tag identifiers in the missing tag identifier set can be tag identifiers in the attribution metadata for which corresponding storage attribution information has not yet been recorded. For example, a missing tag identifier could be the tag identifier corresponding to a tag written for the first time. In practice, the executing entity can first determine each tag identifier in the attribution metadata as the recorded tag identifier set. Then, it can determine the storage attribution information corresponding to each tag identifier in the tag identifier set that is included in the recorded tag identifier set as the existing attribution information set. Finally, it can determine the missing tag identifier set for each tag identifier in the tag identifier set that is not included in the recorded tag identifier set.

[0027] The fourth step involves filling the missing tag identifier set with the preset default attribution information to obtain the filled attribution information set. The preset default attribution information can be pre-defined storage attribution information used by tags for which storage attribution information has not yet been recorded. For example, the preset default attribution information could be storage attribution information with a storage location identifier of "tag_mid" and a storage validity period of 604800 seconds, where "tag_mid" can be the name of the column family with the middle storage validity period. The filled attribution information in the filled attribution information set can be the storage attribution information corresponding to the missing tag identifier, with the same value as the preset default attribution information. In practice, the executing entity can determine the preset default attribution information as the filled attribution information corresponding to each missing tag identifier in the missing tag identifier set, thus obtaining the filled attribution information set.

[0028] The fifth step is to determine the existing attribution information set and the filled attribution information set as the storage attribution information set.

[0029] Step 102: Based on the existing label dataset, perform change filtering on the label dataset to be stored to obtain the filtered label dataset.

[0030] In some embodiments, the execution entity can perform change amount filtering on the tag dataset to be stored based on the existing tag dataset to obtain a filtered tag dataset. The filtered tag data in the filtered tag dataset can be the tag data to be stored in the tag dataset where the tag change amount relative to the corresponding existing tag data reaches a preset change amount threshold. The tag change amount can characterize the degree of difference between the tag value of the tag data to be stored and the corresponding existing tag value. The method for determining the tag change amount corresponds to the data type identifier included in the tag data to be stored. The preset change amount threshold can be a pre-set lower limit value for tag change amount used to distinguish whether the tag value has changed. For example, the preset change amount threshold can include a lower limit value corresponding to an identifier representing a numerical type and a lower limit value corresponding to an identifier representing a categorical type. The lower limit value corresponding to an identifier representing a categorical type can be 1, meaning that the write operation is only performed when the tag value is different from the existing tag value.

[0031] In some optional implementations of certain embodiments, the process of performing change filtering on the tag dataset to be stored based on the existing tag dataset to obtain a filtered tag dataset may include the following steps: The first step involves dividing the dataset into numeric and categorical label datasets based on the data type identifiers. The numeric label data in the numeric label dataset consists of labels whose values ​​can be compared by their differences. The categorical label data in the categorical label dataset consists of labels whose values ​​can only be compared by whether they are the same. In practice, the executing entity can first identify the numeric label data as the numeric label dataset, and then identify the categorical label data as the categorical label dataset.

[0032] The second step involves performing a difference comparison on the numerical label dataset based on the existing label dataset to obtain a first label change quantity set. The first label change quantity in this set characterizes the deviation of the label value of the numerical label data from its corresponding existing label value. In practice, the executing entity can determine the first label change quantity for each numerical label data in the numerical label dataset by taking the absolute value of the difference between the label value of the numerical label data and the corresponding existing label value in the existing label dataset. Alternatively, for numerical label data with a large value range, the executing entity can also determine the first label change quantity by taking the ratio of the absolute value of the difference to the corresponding existing label value.

[0033] The third step involves performing a consistency comparison on the existing label dataset to obtain a second label change quantity set. The second label change quantity in this set indicates whether the label value of the category label data is the same as the corresponding existing label value. For example, the second label change quantity can be a value of 0 when the label value is the same as the existing label value, and a value of 1 when they are different. In practice, the executing entity can determine the second label change quantity for each category label data in the existing label dataset as follows: if the label value of the category label data is equal to the string of the corresponding existing label value in the existing label dataset, 0 is defined as the second label change quantity for that category label data; if the two label values ​​are not equal, 1 is defined as the second label change quantity for that category label data, thus obtaining the second label change quantity set. In addition, for category-type label data with a hierarchical structure, the aforementioned execution entity can also determine 1 as the second label change amount corresponding to the aforementioned category-type label data when the strings of the first-level categories are not equal, and determine 0 as the second label change amount corresponding to the aforementioned category-type label data when only the strings of the second-level categories are not equal.

[0034] The fourth step is to determine the first set of label change quantities and the second set of label change quantities as the label change quantity set.

[0035] Fifth, based on the aforementioned set of label change amounts, the aforementioned dataset of labels to be stored is subjected to threshold filtering to obtain a filtered label dataset. In practice, the executing entity can determine each label data in the aforementioned dataset of labels to be stored whose corresponding label change amount reaches the aforementioned preset change amount threshold as the filtered label dataset. Additionally, for label data to be stored that does not have corresponding existing label data in the aforementioned existing label dataset, the executing entity can directly determine the label data to be stored as the filtered label data in the filtered label dataset.

[0036] Step 103: Based on the storage attribution information set, perform time-limited anchoring writing processing on the filtered tag dataset to obtain the retained tag dataset.

[0037] In some embodiments, the execution entity can perform time-limited anchored write processing on the filtered tag dataset based on the storage ownership information set to obtain a retention tag dataset. The retention tag data in the retention tag dataset can be a retention record formed after the filtered tag data is written to the data storage server set. The retention tag data can include, but is not limited to, at least one of the following: filtered tag data, storage location identifier, data timestamp, cell validity duration, write status identifier, and migration identifier. The cell validity duration can be the duration for which the retention tag data is retained from the data timestamp, submitted along with this write request, and the cell validity duration does not exceed the storage validity duration corresponding to the storage location identifier. For example, the cell validity duration can be the cell-level liveness time submitted by the setTTL method of a change request in Apache HBase, in milliseconds. The write status identifier can be an identifier indicating whether the write request has been successfully written to disk in the data storage server set. The migration identifier can be an identifier indicating whether the retention tag data was generated by migration processing. For example, the migration identifier of the retention tag data generated by the above time-limited anchoring write process can be 0, and the migration identifier of the retention tag data generated by the migration process can be 1.

[0038] In the process of adopting technical solutions to address the technical problems mentioned above, for the application scenario: controlling the flow of tag data stored in storage locations with short effective storage durations and with intermittent periods where the values ​​remain unchanged for several consecutive days (for example, during several consecutive statutory holidays, user operations on the information service platform are significantly reduced, and the values ​​of tag data that change with user operations remain unchanged for several consecutive days), the following technical problem often arises: Since the effective storage duration is uniformly configured according to the storage location, and the storage location is determined by the number of writes within the scrolling window, the number of writes for each tag during the aforementioned intermittent periods... The data is all zero, the intervals between value changes are indistinguishable, and changing the storage location only affects newly written tag data; existing tag data remains valid for its original storage duration. This storage duration is shorter than the extended value change interval caused by the aforementioned discontinuity period. Tag data is cleared before the next value change and rewritten with the same value during the first collection after the discontinuity period ends. This causes a surge in write requests after the discontinuity period ends, and the rewritten tag data is repeatedly scanned when responding to retrieval requests due to refreshed data timestamps, increasing the load on the data storage server set. Considering the following requirements for this application scenario: adapting to long intervals between value changes, adapting to a fixed cell validity period with varying value change intervals, and cell validity period constrained by upper and lower limits, we have decided to adopt the following solution: In some optional implementations of certain embodiments, the process of performing time-limited anchoring write processing on the filtered tag dataset based on the aforementioned storage attribution information set to obtain the retained tag dataset may include the following steps: The first step is to retrieve the historical tag data sequence set and migration timestamp set from the aforementioned data storage server set. The historical tag data sequence set can be various versions of retained tag data corresponding to the same primary key identifier and the same tag identifier, arranged chronologically by the data timestamps. The migration timestamps in the migration timestamp set can be the moment when the tag's storage location identifier was last changed. For example, the migration timestamp can be recorded in a row corresponding to the tag identifier in the aforementioned attribution metadata table. In practice, the execution entity can first determine each primary key identifier in the primary key identifier set included in the filtered tag dataset as the row key of a sequence read request, obtaining a sequence read request set. Then, using the `readAllVersions` method of the Apache HBase `Get` class and the `get` method of the `Table` interface, the sequence read request set is submitted to the aforementioned data storage server set to obtain the retained tag data for each version. Subsequently, for each filtered tag data in the above filtered tag dataset, the retained tag data of each version of the retained tag data, whose primary key identifier and tag identifier are the same as those of the above filtered tag data, are arranged from earliest to latest according to the above data timestamps to obtain the corresponding historical tag data sequence, and then the historical tag data sequence set is obtained.

[0039] The second step involves performing timestamp matching and filtering on the historical tag data sequence set based on the aforementioned migration timestamp set, resulting in a filtered data sequence set. The filtered data sequences in this set can be the retained tag data from the historical tag data sequences whose timestamps are later than the migration timestamp corresponding to the same tag identifier. In practice, the executing entity can identify the retained tag data from each historical tag data sequence in the historical tag data sequence set whose timestamps are later than the migration timestamp corresponding to the historical tag data sequence as the filtered data sequence corresponding to that historical tag data sequence. Furthermore, for historical tag data sequences that do not have a corresponding migration timestamp in the aforementioned migration timestamp set, the executing entity can directly identify the historical tag data sequence as the filtered data sequence corresponding to that historical tag data sequence.

[0040] The third step involves performing a truncation difference processing on the filtered data sequence set to obtain a set of effective interval sequences and a set of truncation counts. The effective interval sequence in the effective interval sequence set can be a sequence composed of the differences between the data timestamps of two adjacent versions of the retained tag data in the filtered data sequence. These differences do not include those that span multiple collection periods due to unchanged values. The truncation count in the set of truncation counts can be the number of differences in the filtered data sequence that span multiple collection periods due to unchanged values. In practice, the execution entity can perform the following difference steps for each filtered data sequence in the filtered data sequence set: First, determine the differences between the data timestamps of two adjacent versions of the retained tag data in the filtered data sequence as the initial interval sequence. Second, determine each difference in the initial interval sequence that is greater than a preset truncation multiple of the preset collection interval as a truncation interval sequence. The preset collection interval can be the time interval between the successive generation of tag data for the same tag, in milliseconds (e.g., 5000). The aforementioned preset truncation factor can be a pre-defined multiple used to distinguish whether the interval spans multiple acquisition cycles (e.g., 2). The third step involves determining each difference in the initial interval sequence, excluding the truncated interval sequence, as the valid interval sequence corresponding to the filtered data sequence. The fourth step involves determining the number of differences in the truncated interval sequence as the truncation number corresponding to the filtered data sequence.

[0041] The fourth step involves alternating recursive processing of the aforementioned effective interval sequence set to obtain an interval benchmark value set and an interval fluctuation value set. The interval benchmark values ​​in the interval benchmark value set characterize the central tendency of each effective interval of the tag. The interval fluctuation values ​​in the interval fluctuation value set characterize the degree of deviation of each effective interval of the tag relative to the interval benchmark values. In practice, the executing entity can, for each effective interval sequence in the aforementioned effective interval sequence set, firstly, determine the first effective interval in the aforementioned effective interval sequence as the current interval benchmark value. Then, determine half of the first effective interval as the current interval fluctuation value. Afterwards, in order of data timestamp from earliest to latest, perform the following update steps for each effective interval in the aforementioned effective interval sequence following the first effective interval: First, determine the absolute value of the difference between the aforementioned effective interval and the aforementioned current interval benchmark value as the interval deviation value. Second, use an exponentially weighted moving average algorithm, with a preset fluctuation coefficient (e.g., 0.25), to smooth the aforementioned current interval fluctuation value and the aforementioned interval deviation value, obtaining the updated interval fluctuation value. The third step involves using the aforementioned exponentially weighted moving average algorithm to smooth the current interval benchmark value and the aforementioned effective interval by a preset smoothing coefficient (e.g., 0.125), resulting in an updated interval benchmark value. The fourth step involves determining the updated interval fluctuation value and the updated interval benchmark value as the current interval fluctuation value and the current interval benchmark value, respectively. Finally, the current interval benchmark value and the current interval fluctuation value are determined as the interval benchmark value and interval fluctuation value corresponding to the aforementioned effective interval sequence.

[0042] Fifth, based on the aforementioned interval fluctuation value set, perform forward bias synthesis processing on the aforementioned interval benchmark value set to obtain an initial effective duration set. The initial effective duration in the aforementioned initial effective duration set can be the duration required to ensure that the retained tag data persists until the next write of the same tag. In practice, the aforementioned execution entity can perform the following steps for each interval benchmark value in the aforementioned interval benchmark value set: First, expand the interval fluctuation value corresponding to the aforementioned interval benchmark value by a preset bias multiple to obtain an interval margin value. The aforementioned preset bias multiple can be a pre-set multiple used to determine the interval margin value (e.g., 4). Then, the sum of the aforementioned interval benchmark value and the aforementioned interval margin value is determined as the initial effective duration corresponding to the aforementioned interval benchmark value. Additionally, when the aforementioned interval margin value is less than a preset duration granularity (i.e., a pre-set lower limit allowed by the interval margin value, e.g., 1000 milliseconds), the aforementioned execution entity can also determine the sum of the aforementioned interval benchmark value and the aforementioned preset duration granularity as the initial effective duration corresponding to the aforementioned interval benchmark value. In addition, for valid interval sequences that are empty in the above valid interval sequence set, the execution entity can determine the preset initial valid duration (i.e., the duration set in advance for tags with fewer than two historical records, for example, 3,600,000 milliseconds) as the initial valid duration corresponding to the above valid interval sequence.

[0043] Step 6: Based on the aforementioned set of truncation counts, perform a doubling correction process on the aforementioned initial effective duration set to obtain a corrected effective duration set. The corrected effective duration in the aforementioned corrected effective duration set can be the duration obtained by multiplying the initial effective duration by the number of truncation counts. In practice, the aforementioned execution entity can use an exponential backoff algorithm for each initial effective duration in the aforementioned initial effective duration set, doubling the initial effective duration successively by the corresponding number of truncation counts to obtain the corresponding corrected effective duration. Alternatively, when the corresponding number of truncation counts exceeds a preset upper limit (e.g., 6), the aforementioned execution entity can also double the initial effective duration successively by the preset upper limit to obtain the corresponding corrected effective duration.

[0044] Step 7: Based on the preset lower bound set and the preset upper bound set of valid duration, perform interval truncation on the modified valid duration set to obtain the target valid duration set. The preset lower bound set of the preset valid duration can be a pre-defined lower limit value allowed for cell valid duration. The preset upper bound set of the preset upper bound set of the preset valid duration can be a pre-defined upper limit value allowed for cell valid duration. For example, the preset upper bound set of the preset valid duration can be the duration obtained by converting the storage valid duration configured for the storage location identifier corresponding to the same label identifier to milliseconds, because the cell-level lifespan cannot exceed the column-family-level lifespan. The target valid duration in the target valid duration set can be the modified valid duration falling between the preset lower bound and the preset upper bound set of the preset valid duration. The initial valid duration, the modified valid duration, the target valid duration, and the preset lower and upper bound sets of the preset valid duration are all in milliseconds. In practice, the aforementioned implementing entity can, for each modified effective duration in the aforementioned set of modified effective durations, firstly, if the modified effective duration is less than the corresponding preset lower bound of effective duration, determine the preset lower bound of effective duration as the corresponding target effective duration. Then, if the modified effective duration is greater than the corresponding preset upper bound of effective duration, determine the preset upper bound of effective duration as the corresponding target effective duration. Finally, if the modified effective duration is neither less than the corresponding preset lower bound of effective duration nor greater than the corresponding preset upper bound of effective duration, determine the modified effective duration as the corresponding target effective duration.

[0045] Step 8: Based on the aforementioned target valid duration set and storage ownership information set, perform duration-attached write processing on the filtered tag dataset to obtain the retention tag dataset. In practice, the execution entity can first, for each filtered tag data in the filtered tag dataset, determine the storage location identifier, including the corresponding storage ownership information, as the column family name of the write request for the filtered tag data. Then, using the `setTTL` method of Apache HBase's change request, determine the corresponding target valid duration as the cell valid duration of the write request. Next, write the value representing non-migration into the migration identifier column of the write request. Then, using the `batch` method of Apache HBase's Table interface, submit each write request to the aforementioned data storage server set to obtain a result array. Finally, determine whether the element in the result array corresponding to each write request is an exception object, and use this value as the corresponding write status identifier to obtain the retention tag dataset.

[0046] The above technical solution and its related content, as an inventive point of this disclosure, combined with step "107" below, solves the second technical problem: "The storage validity period is uniformly configured according to the storage location, which is not compatible with the value change interval of the tag itself." Factors that increase the load on the data storage server set are often as follows: the storage location is determined by the number of writes within the scrolling window, while the write count for each tag whose value has not changed for several consecutive days is zero, and the value change interval between them is indistinguishable; furthermore, changing the storage location only affects newly written tag data, and the previously retained tag data remains invalid according to the original storage validity period; the above storage validity period is shorter than the value change interval lengthened by the interruption period, and the tag data is cleared before the next value change, and is rewritten with the same value during the first collection after the interruption period ends. If the above factors are solved, the number of write requests can be reduced, and the load on the data storage server set can be lowered. To achieve this effect, this disclosure first obtains a historical tag data sequence set and a migration timestamp set from the above data storage server set. Next, based on the aforementioned migration timestamp set, the historical tag data sequence set is subjected to timestamp matching and filtering to obtain a filtered data sequence set. Here, intervals formed by resetting data timestamps when changing storage locations are excluded, and the calculated intervals only reflect the value changes of the tags themselves. Next, the filtered data sequence set is subjected to same-value truncation and differencing to obtain a valid interval sequence set and a truncation count set. Here, intervals formed by being cleaned and rewritten due to insufficient duration are no longer included in the valid intervals, and the number of such breakpoints is retained as evidence that the duration is too short. Next, the valid interval sequence set is subjected to alternating recursive processing to obtain an interval baseline value set and an interval fluctuation value set. Next, based on the interval fluctuation value set, the interval baseline value set is subjected to positive bias synthesis processing to obtain an initial valid duration set. Here, the recursion only needs to retain the baseline value and the fluctuation value, without needing to save all historical intervals; the bias ensures that the duration covers the upper edge of the interval fluctuation, preventing premature cleanup of tag data when the value change interval accidentally lengthens. Next, based on the aforementioned set of truncation counts, the initial valid duration set is doubled and corrected to obtain the corrected valid duration set. Then, based on the preset lower and upper bounds of the valid duration, the corrected valid duration set is truncated to obtain the target valid duration set. Here, tags that were previously cleared are given longer durations, preventing repeated writes; the lower bound ensures tags remain readable when responding to retrieval requests, and the upper bound prevents storage space usage from increasing. Finally, based on the target valid duration set and the storage ownership information set, the filtered tag dataset is appended with additional durations to obtain the retained tag dataset. Here, the number of write requests generated after the interruption period ends decreases, and the number of repeatedly scanned tag data also decreases.Therefore, the above technical solution and its related content, combined with step "107" below, can reduce the number of write requests and reduce the load on the data storage server set.

[0047] Step 104: Perform rolling window aggregation processing on the retention tag dataset to obtain the write frequency set.

[0048] In some embodiments, the execution entity may perform rolling window aggregation processing on the aforementioned retained tag dataset to obtain a write frequency set. The write frequency in this set represents the number of times a tag is written within a time interval ending at the current time. The write count only includes writes where the value changes and the data is successfully written to disk.

[0049] In some optional implementations of certain embodiments, the above-mentioned rolling window aggregation process on the retention tag dataset to obtain the write frequency set may include the following steps: The first step is to perform write status filtering on the aforementioned retention tag dataset to obtain the written tag dataset. The written tag data in the written tag dataset can be retention tag data that has been persisted to disk in the data storage server. In practice, the executing entity can identify each retention tag data in the aforementioned retention tag dataset whose write status identifier indicates that it has been persisted to disk as the written tag dataset.

[0050] The second step involves performing migration flag filtering on the aforementioned written label dataset to obtain a non-migrated label dataset. The non-migrated label data in this dataset can be any written label data whose label values ​​have changed. For example, written label data whose label values ​​have not changed but have been rewritten due to a change in storage location is not considered non-migrated label data. In practice, the executing entity can identify each written label data in the aforementioned written label dataset with a migration flag value of 0 as a non-migrated label dataset.

[0051] The third step involves dividing the non-migrated label dataset into windows based on a preset scrolling window duration, resulting in a label dataset within the window. The preset scrolling window duration can be a pre-defined time interval (e.g., 300,000 milliseconds) that traces backward from the current time. The label data within the window in the label dataset can be non-migrated label data whose timestamps fall within the time interval corresponding to the preset scrolling window duration. In practice, the executing entity can first obtain the current time value. Then, the difference between the current time value and the preset scrolling window duration is determined as the window start time value. Afterward, each non-migrated label data in the non-migrated label dataset whose timestamp is neither earlier than the window start time value nor later than the current time value is identified as the label dataset within the window.

[0052] The fourth step involves grouping and aggregating the aforementioned in-window label dataset to obtain a write frequency set. In practice, the execution entity can first, for each label identifier in the label identifier set included in the aforementioned in-window label dataset, identify the in-window label data with the same label identifier as the corresponding in-window label data group, thus obtaining an in-window label data group set. Then, for each in-window label data group in the aforementioned in-window label data group set, determine the number of in-window label data in the in-window label data group as the write frequency of the label identifier corresponding to the in-window label data group, thus obtaining a write frequency set.

[0053] Step 105: Based on the write frequency set, perform bidirectional bandwidth-limited routing processing on the storage ownership information set to obtain the target storage ownership information set.

[0054] In some embodiments, the execution entity may perform bidirectional bandwidth-limited routing processing on the storage ownership information set based on the write frequency set to obtain a target storage ownership information set. The target storage ownership information in the target storage ownership information set may be information characterizing the storage location of the tag data to be stored, determined by its write frequency.

[0055] In the process of adopting technical solutions to address the technical problems mentioned above, for the application scenario: scenarios where the retention requirements and writing frequency of tag data are inconsistent (for example, the "Affiliation Organization Identifier" tag remains unchanged for several years, but its writing frequency suddenly increases within a scrolling window due to a concentrated writing; the "Common Item Category Identifier" tag is rarely written, but its tag data is not suitable for long-term retention), the following technical problem three often arises: Since the storage location is determined only by the writing frequency, and the writing frequency is measured by the scrolling window and changes with short-term writing fluctuations, even if a tag that needs to be readable for a long time is pointed to the storage location with the shortest duration during a concentrated writing, its tag data becomes invalid when it is still needed to be read, and is rewritten with the same value in the next collection; while tags with infrequent writing are pointed to the storage location with the longest duration, and their tag data is retained even after it is no longer needed, which increases the number of write requests and the amount of historical tag data retained, resulting in an increase in the storage space occupied by the data storage server set and an increase in load. To address the following requirements for this application scenario: adapting to discrepancies between retention needs and write frequency, adapting to a storage duration no shorter than the tag's readability requirement, and adapting to a storage duration no longer than the tag's retention period, we have decided to adopt the following solution: In some optional implementations of certain embodiments, the process of performing bidirectional bandwidth-limited routing on the storage ownership information set based on the write frequency set to obtain the target storage ownership information set may include the following steps: The first step is to perform interval mapping processing on the aforementioned write frequency set to obtain a candidate storage allocation information set. The candidate storage allocation information in this set can be information representing the storage location corresponding to a preset frequency interval into which the write frequency falls. The preset frequency interval can be a pre-defined range of write frequencies (e.g., 0 to 2 times, 3 to 10 times, or more than 11 times). In practice, the executing entity can, for each write frequency in the aforementioned write frequency set, determine the storage allocation information corresponding to the preset frequency interval into which the write frequency falls in the preset frequency allocation lookup table as the candidate storage allocation information for that write frequency, thus obtaining the candidate storage allocation information set. The aforementioned preset frequency attribution table can be a pre-set table that records the correspondence between each preset frequency interval and each storage attribution information. In the above correspondence, the higher the write frequency represented by the preset frequency interval, the shorter the corresponding storage validity period (for example, 0 to 2 times corresponds to a storage location with a storage validity period of 2,592,000 seconds, 3 to 10 times corresponds to a storage location with a storage validity period of 604,800 seconds, and more than 11 times corresponds to a storage location with a storage validity period of 86,400 seconds).

[0056] The second step involves applying a lower bound to the candidate storage attribution information set based on a preset lower bound set of storage duration, resulting in a lower bounded attribution information set. The preset lower bound of storage duration in the preset lower bound set can be a pre-defined lower limit value allowed for the effective storage duration of a tag. For example, for tags whose write frequency surges due to short-term concentrated writes but still require long-term readability (e.g., "attribution organization identifier"), the preset lower bound of storage duration could be 2,592,000 seconds. The lower bounded attribution information in the lower bounded attribution information set can be storage attribution information with an effective storage duration not less than the preset lower bound of storage duration. In practice, the executing entity can, for each candidate storage attribution information in the candidate storage attribution information set, first determine the preset lower bound of storage duration corresponding to the tag identifier in the preset lower bound set of storage duration as the current storage duration lower bound. Then, when the effective storage duration included in the aforementioned candidate storage attribution information is less than the lower bound of the aforementioned current storage duration, the storage attribution information corresponding to the storage location whose effective storage duration is not less than the lower bound of the aforementioned current storage duration and is closest to the lower bound of the aforementioned current storage duration is determined as the lower bounded attribution information corresponding to the aforementioned candidate storage attribution information. Subsequently, when the effective storage duration included in the aforementioned candidate storage attribution information is not less than the lower bound of the aforementioned current storage duration, the aforementioned candidate storage attribution information is determined as the lower bounded attribution information corresponding to the aforementioned candidate storage attribution information, thus obtaining the lower bounded attribution information set.

[0057] The third step involves applying upper limit scaling to the aforementioned lower limit-following attribution information set based on the preset upper limit set of storage duration, resulting in the upper limit-following attribution information set. The preset upper limit of storage duration in the preset upper limit set can be a pre-defined upper limit value allowed for the effective storage duration of a tag. For example, for tags that should not be stored for more than seven days according to a preset retention period, the preset upper limit of storage duration could be 604,800 seconds. The upper limit-following attribution information in the upper limit-following attribution information set can be storage attribution information whose effective storage duration is not greater than the preset upper limit of storage duration. In practice, the executing entity can, for each lower limit-following attribution information in the aforementioned lower limit-following attribution information set, first determine the preset upper limit of storage duration corresponding to the tag identifier in the aforementioned preset upper limit set of storage duration as the current upper limit of storage duration. Then, when the effective storage duration included in the lower limit post-attribution information is greater than the upper limit of the current storage duration, the storage attribution information corresponding to the storage location whose effective storage duration is not greater than the upper limit of the current storage duration and is closest to the upper limit of the current storage duration is determined as the upper limit post-attribution information corresponding to the lower limit post-attribution information. Afterwards, when the effective storage duration included in the lower limit post-attribution information is not greater than the upper limit of the current storage duration, the lower limit post-attribution information is determined as the upper limit post-attribution information corresponding to the lower limit post-attribution information, thus obtaining the upper limit post-attribution information set.

[0058] The fourth step involves replacing the aforementioned storage ownership information set based on the upper limit width subsequent ownership information set to obtain the target storage ownership information set. In practice, the executing entity can, for each storage ownership information set, firstly, identify the upper limit width subsequent ownership information in the upper limit width subsequent ownership information set that corresponds to the tag identifier of the aforementioned storage ownership information as the target storage ownership information corresponding to the aforementioned storage ownership information. Then, for storage ownership information for which there is no corresponding upper limit width subsequent ownership information in the upper limit width subsequent ownership information set, that storage ownership information is identified as the target storage ownership information corresponding to the aforementioned storage ownership information. Finally, the target storage ownership information set is obtained.

[0059] The above-described technical solution and its related content, as an inventive point of this disclosure, combined with step "Step 107" below, solves technical problem three: "The storage location is determined solely by the write frequency, which is incompatible with the retention requirements of the tags themselves." Factors leading to increased storage space usage and load on the data storage server set are often as follows: write frequency is measured by a rolling window and varies with short-term write fluctuations. Even if a tag requiring long-term readability is pointed to the storage location with the shortest duration during a concentrated write, its tag data becomes invalid before it is even read, and is rewritten with the same value in the next collection; sparsely written tags are pointed to the storage location with the longest duration, and their tag data is retained even after it is no longer needed. Solving these factors can reduce the amount of retained historical tag data, reduce the storage space usage of the data storage server set, and lower the load. To achieve this effect, this disclosure first performs interval mapping processing on the above-described write frequency set to obtain a candidate storage attribution information set. Next, based on the preset lower bound set of storage duration, the candidate storage ownership information set is subjected to lower bound amplitude processing to obtain the lower bound amplitude ownership information set. Here, short-term write fluctuations no longer unilaterally determine the storage duration; tags requiring long-term readability will not expire within their readability period, and the same value does not need to be rewritten. Next, based on the preset upper bound set of storage duration, the lower bound amplitude ownership information set is subjected to upper bound amplitude processing to obtain the upper bound amplitude ownership information set. Here, tags with sparse writes are no longer retained long-term due to their position, and historical versions exceeding their retention period decrease accordingly. Finally, based on the upper bound amplitude ownership information set, the storage ownership information set is replaced to obtain the target storage ownership information set. Here, the storage location after both constraints takes effect immediately, without waiting for the next round of write frequency updates. Therefore, the above technical solution and its related content, combined with step "Step 107" below, can reduce the amount of retained historical tag data, reduce the storage space occupied by the data storage server set, and reduce the load on the data storage server set.

[0060] Step 106: Based on the target storage attribution information set, perform migration processing on the retained tag dataset to obtain the migrated tag dataset.

[0061] In some embodiments, the executing entity may perform migration processing on the retained tag dataset based on the target storage attribution information set to obtain a migrated tag dataset. The migrated tag data in the migrated tag dataset may be tag data retained in the data storage server set whose storage location matches the storage location pointed to by the attribution correction information. The attribution correction information may be storage attribution information consistent with the actual storage location of the tag data.

[0062] In some optional implementations of certain embodiments, the process of migrating the retained tag dataset based on the target storage attribution information set to obtain the migrated tag dataset may include the following steps: The first step involves dividing the aforementioned target storage ownership information set and the aforementioned storage ownership information set into migration direction sets, resulting in an upgrade ownership information set and a downgrade ownership information set. Specifically, the upgrade ownership information in the upgrade ownership information set can be target storage ownership information whose storage validity period is shorter than the storage validity period corresponding to the same tag identifier in the aforementioned storage ownership information set. Similarly, the downgrade ownership information in the downgrade ownership information set can be target storage ownership information whose storage validity period is longer than the storage validity period corresponding to the same tag identifier in the aforementioned storage ownership information set. In practice, the executing entity can perform the following division steps for each target storage ownership information in the aforementioned target storage ownership information set: First, identify the storage ownership information in the aforementioned storage ownership information set that shares the same tag identifier as the target storage ownership information as the original ownership information. Second, in response to the determination that the storage validity period included in the target storage ownership information is shorter than the storage validity period included in the original ownership information, identify the target storage ownership information as an upgrade ownership information. The third step is to determine that the storage validity period included in the target storage attribution information is greater than the storage validity period included in the previous attribution information, and to classify the target storage attribution information as a downgraded attribution information.

[0063] The second step involves performing a single-window comparison of the aforementioned upgrade attribution information set and the aforementioned write frequency set based on a preset upgrade frequency threshold, to obtain an upgrade confirmation attribution information set. The preset upgrade frequency threshold can be a pre-defined lower limit for write frequency used to confirm changes to storage locations with shorter effective storage durations. This preset upgrade frequency threshold is not greater than the minimum boundary value between two adjacent intervals in each preset frequency interval (e.g., 3 times). The upgrade confirmation attribution information in the aforementioned upgrade confirmation attribution information set can be upgrade attribution information whose write frequency reaches the aforementioned preset upgrade frequency threshold. In practice, the executing entity can perform the following comparison steps for each upgrade attribution information in the aforementioned upgrade attribution information set: First, determine the write frequency in the aforementioned write frequency set that belongs to the same tag as the aforementioned upgrade attribution information as the current write frequency. Second, in response to determining that the current write frequency is not less than the aforementioned preset upgrade frequency threshold, determine the aforementioned upgrade attribution information as an upgrade confirmation attribution information.

[0064] The third step involves performing cross-window continuous verification on the aforementioned downgrade attribution information set and the aforementioned write frequency set based on a preset downgrade frequency threshold and a preset number of consecutive confirmation windows, to obtain a downgrade confirmation attribution information set. The preset downgrade frequency threshold can be a pre-set upper limit on the write frequency for confirming changes to storage locations with longer effective storage durations (e.g., 2 times). The preset number of consecutive confirmation windows can be a pre-set number of consecutive low-frequency windows (e.g., 3). The number of consecutive low-frequency windows represents the number of rolling windows where the write frequency of a tag does not exceed the preset downgrade frequency threshold consecutively. The downgrade confirmation attribution information in the aforementioned downgrade confirmation attribution information set can be downgrade attribution information where the number of consecutive low-frequency windows reaches the preset number of consecutive confirmation windows. In practice, the executing entity can perform the following verification steps for each downgrade attribution information in the aforementioned downgrade attribution information set: First, determine the write frequency in the aforementioned write frequency set that belongs to the same tag identifier as the aforementioned downgrade attribution information as the current write frequency. The second step is to determine the current low-frequency window count as the number of consecutive low-frequency windows in the aforementioned attribution metadata table that belong to the same tag identifier as the aforementioned downgrade attribution information. The third step is to determine the updated low-frequency window count by summing the current low-frequency window count with 1, in response to the determination that the current write frequency is not greater than the aforementioned preset downgrade frequency threshold. The fourth step is to determine the updated low-frequency window count as 0, in response to the determination that the current write frequency is greater than the aforementioned preset downgrade frequency threshold. The fifth step is to write the updated low-frequency window count into a row in the aforementioned attribution metadata table that belongs to the same tag identifier as the aforementioned downgrade attribution information. The sixth step is to determine the downgrade attribution information as a single downgrade confirmation attribution information, in response to the determination that the updated low-frequency window count is not less than the aforementioned preset consecutive confirmation window count.

[0065] The fourth step involves performing affiliation change filtering on the aforementioned upgrade confirmation affiliation information set and downgrade confirmation affiliation information set, and then processing the affiliation change of the aforementioned retained tag dataset and target storage affiliation information set to obtain the tag dataset to be migrated and the affiliation correction information set. The tag data to be migrated in the aforementioned tag dataset can be the retained tag data corresponding to tag identifiers whose storage location has been confirmed to have changed. The affiliation correction information in the aforementioned affiliation correction information set can be storage affiliation information consistent with the actual storage location of the tag data. This affiliation correction information can include a storage location identifier and storage validity period; when the tag's storage location changes, it can also include a migration timestamp. The migration timestamp can be the time when the tag's storage location identifier was last changed. In practice, the aforementioned executing entity can perform the following filtering steps for each target storage ownership information in the aforementioned target storage ownership information set: First, in response to determining that the aforementioned target storage ownership information exists in either the aforementioned upgrade confirmation ownership information set or the aforementioned downgrade confirmation ownership information set, each retained tag data in the aforementioned retained tag data set that belongs to the same tag identifier as the aforementioned target storage ownership information is identified as tag data to be migrated; the aforementioned target storage ownership information is identified as ownership correction information; and the obtained current time value is identified as the migration timestamp included in the aforementioned ownership correction information. Second, in response to determining that the aforementioned target storage ownership information does not exist in either the aforementioned upgrade confirmation ownership information set or the aforementioned downgrade confirmation ownership information set, the storage ownership information in the aforementioned storage ownership information set that belongs to the same tag identifier as the aforementioned target storage ownership information is identified as ownership correction information.

[0066] Step 5: Based on the obtained current time value, reset the time information of the aforementioned label dataset to be migrated to obtain the reset label dataset. The current time value can be the time value at the time of the migration process, using the same timing base as the data timestamp. The reset label data in the reset label dataset can be the label data to be migrated whose data timestamp has been updated to the current time value, and whose expiration time remains the same as before the update. The expiration time can be the time pointed to by the sum of the data timestamp and the cell's valid duration. In practice, the executing entity can first obtain the current time value. Then, for each label data to be migrated in the aforementioned label dataset, perform the following reset steps: Step 1: Determine the original expiration time by the sum of the data timestamp and the cell's valid duration of the aforementioned label data. Step 2: Replace the data timestamp in the aforementioned label data to be migrated with the aforementioned current time value. Step 3: Replace the cell's valid duration in the aforementioned label data to be migrated with the difference between the original expiration time and the aforementioned current time value to obtain the reset label data. In addition, if the difference between the original failure time and the current time value is not greater than 0, the execution entity may also remove the label data to be migrated from the label dataset to be migrated and stop writing to the label data to be migrated.

[0067] Step 6: Based on the aforementioned storage ownership information set, perform version masking on the aforementioned tag dataset to be migrated to obtain the original ownership deletion mark set. The original ownership deletion marks in the original ownership deletion mark set can be marks used to ensure that each version of the tag data to be migrated in its storage location before the change of storage location is no longer read. For example, the original ownership deletion marks can be deletion marks pointing to each version of a column generated using the addColumns method of the Apache HBase Delete class. In practice, the execution entity can perform the following masking steps for each tag data to be migrated in the aforementioned tag dataset: First, determine the storage location identifiers included in the storage ownership information set that belong to the same tag identifier as the tag data to be migrated as the pre-change storage location identifiers. Second, determine a combination of the pre-change storage location identifiers, the primary key identifier of the tag data to be migrated, and the tag identifier as an original ownership deletion mark.

[0068] Step 7 involves merging and writing the reset label dataset, the original attribution deletion mark set, and the attribution correction information set to obtain the migrated label dataset. In practice, the execution entity can first set the migration flag of each reset label data in the reset label dataset to 1. Then, using the batch method of the Apache HBase Table interface, the reset label dataset and the original attribution deletion mark set are submitted to the data storage server set in the same batch. Next, using the batch method, the attribution correction information set is submitted to the attribution metadata table. Then, each attribution correction information in the attribution correction information set replaces the target storage attribution information in the target storage attribution information set that belongs to the same label identifier. Finally, the label data remaining in the data storage server set after the attribution correction information set is submitted is determined as the migrated label dataset.

[0069] Step 107: In response to receiving a tag acquisition request, the migrated tag dataset is pruned dimension by dimension according to the target storage ownership information set and the synchronization timestamp included in the tag acquisition request to obtain the tag dataset to be output. The tag data flow control is performed on the data storage server set according to the migrated tag dataset and the tag dataset to be output.

[0070] In some embodiments, the execution entity may, in response to receiving a tag acquisition request, perform dimension-by-dimensional pruning on the migrated tag dataset according to the target storage ownership information set and the synchronization timestamp included in the tag acquisition request to obtain a tag dataset to be output, and perform tag data flow control on the data storage server set according to the migrated tag dataset and the tag dataset to be output. The tag acquisition request may be a request to obtain changes in the tag data of an object since the last acquisition. The tag acquisition request may include a synchronization timestamp. The synchronization timestamp may be the time of the last acquisition of tag data. The tag data to be output in the tag dataset to be output may be tag data whose values ​​have changed since the last acquisition. The tag data to be output may include tag identifiers and tag values. The tag data flow control includes at least one of the following: limiting the scanning range of the data storage server set to the storage location of the tag dataset to be output; triggering the data storage server set to reclaim storage space for tag data that has exceeded the cell's valid duration; and configuring tag data in the migrated tag dataset that has not been output to not be written to the memory cache of the data storage server set.

[0071] In the process of adopting technical solutions to solve the technical problems mentioned above, for the application scenario: the scenario of controlling the flow of tag data that is statistically taken according to a fixed backtracking interval and the backtracking interval is shorter than the interval between two adjacent acquisitions (for example, the value of the "number of visits in the last hour" tag increases due to the user's access to the information service platform's page, and falls back to the original value after one hour of the access, while two adjacent tag acquisition requests are initiated at the same time every day), the following technical problem four often occurs: Since whether the tag value has changed is determined only by whether the data timestamp is later than the synchronization timestamp, while the write filter is based on the change in the tag value relative to the previous write, the two are based on different benchmarks. The tag undergoes multiple increases and decreases between two adjacent acquisitions. Each change that reaches the preset change threshold is written and the data timestamp is refreshed. Even if the change in the tag value relative to the previous acquisition does not reach the preset change threshold, it is still judged as a change and output, which increases the number of tag data output when responding to a single tag acquisition request, resulting in an increase in the network transmission volume of the data storage server set. To meet the following requirements of this application scenario: storage locations where no writes have occurred can be completely ignored; change detection and write filtering use the same benchmark; and the adaptation judgment result does not change with the boundary position of the tolerance interval, we have decided to adopt the following solution: In some optional implementations of certain embodiments, the above-mentioned step-by-step pruning of the migrated tag dataset based on the target storage attribution information set and the synchronization timestamp included in the tag acquisition request to obtain the tag dataset to be output, and the tag data flow control of the data storage server set based on the migrated tag dataset and the tag dataset to be output, may include the following steps: The first step is to perform time-series merging on the migrated tag dataset based on the aforementioned target storage ownership information set, resulting in a set of last-time ownership timestamps. The last-time ownership timestamp in this set can be the last data timestamp among the data timestamps of all tag data stored in a storage location by the object pointed to by the tag retrieval request, arranged from earliest to latest. In practice, the executing entity can first determine the storage location identifiers included in the target storage ownership information set as a set of storage location identifiers to be merged. Then, for each storage location identifier in the set of storage location identifiers to be merged, the following merging steps are performed: First, merge the data timestamps of all migrated tag data in the migrated tag dataset whose primary key identifier is the same as the object pointed to by the tag retrieval request and whose storage location identifier is the same as the storage location identifier to be merged, arranged from earliest to latest, into a single timestamp sequence. Second, determine the last data timestamp in the timestamp sequence as the last-time ownership timestamp corresponding to the storage location identifier to be merged.

[0072] The second step involves performing a batch filtering process on the target storage ownership information set based on the aforementioned synchronization timestamp and the aforementioned set of last ownership timestamps, resulting in a filtered ownership information set. The filtered ownership information in this set can be target storage ownership information whose last ownership timestamp is later than the aforementioned synchronization timestamp. In practice, the executing entity can perform the following filtering steps for each target storage ownership information in the target storage ownership information set: First, determine the last ownership timestamp in the aforementioned set of last ownership timestamps that corresponds to the storage location identifier included in the target storage ownership information as the current last timestamp. Second, in response to determining that the current last timestamp is later than the aforementioned synchronization timestamp, determine the target storage ownership information as a filtered ownership information. Furthermore, for target storage ownership information for which there is no corresponding last ownership timestamp in the aforementioned set of last ownership timestamps, the executing entity may choose not to determine the target storage ownership information as filtered ownership information.

[0073] The third step involves incrementally extracting data from the migrated label dataset based on the filtered attribution information set and the synchronization timestamp, resulting in a candidate label dataset. Candidate label data in this dataset can be migrated label data located in the storage location indicated by the filtered attribution information and whose timestamp is later than the synchronization timestamp. In practice, the execution entity can first use the `addFamily` method of the Apache HBase `Scan` class to identify the storage location identifiers included in the filtered attribution information set as the column families targeted by the scan request. Then, it can use the `setTimeRange` method of the `Scan` class to determine the lower bound of the time interval targeted by the scan request by summing the synchronization timestamp with 1, and the upper bound of the maximum possible time value. Next, it can use the `setCacheBlocks` method of the `Scan` class to configure the scan request to not write the data blocks traversed by the scan request to the memory cache of the data storage server set. Finally, the scan request is submitted to the data storage server set to obtain the candidate label dataset.

[0074] The fourth step involves performing multi-phase tolerance mapping on the candidate label dataset based on a preset change threshold and a preset phase offset value, resulting in a tolerance identifier set. The tolerance identifier set includes a tolerance identifier that represents the number of the tolerance interval into which a label value falls. The tolerance interval can be a range of values ​​with a width equal to the lower limit of the numerical change amount. This lower limit of the numerical change amount can be the lower limit value corresponding to the identifier representing the numerical value within the preset change threshold. For example, when the lower limit of the numerical change amount is 10, each tolerance interval can be a left-closed, right-open range such as 0 to 10, 10 to 20, and 20 to 30. The tolerance interval into which the label value 23 falls is 20 to 30, and the tolerance identifier can be 2. The preset phase offset value can be a pre-defined offset between two sets of tolerance intervals. For example, when the preset phase offset value is 5, another set of tolerance intervals can be left-closed and right-open value intervals such as 5 to 15, 15 to 25, and 25 to 35. The tolerance interval that the label value 23 falls into in the other set of tolerance intervals is 15 to 25, and the tolerance identifier can be 1. The tolerance identifier set includes the tolerance identifier groups that are the respective tolerance identifiers of the same label value in the two sets of tolerance intervals. For example, the tolerance identifier group corresponding to the label value 23 can be 2 and 1. In practice, the execution entity can first determine the tolerance intervals with a width equal to the lower limit of the numerical change amount and arranged sequentially from 0 in the direction of increasing value as the first phase tolerance interval set. Then, the tolerance intervals with a width equal to the lower limit of the numerical change amount and arranged sequentially from the preset phase offset value in both the direction of increasing value and the direction of decreasing value are determined as the second phase tolerance interval set. Subsequently, for each candidate label data in the aforementioned candidate label dataset, the following mapping steps are performed: First, the index of the tolerance interval into which the label value of the candidate label data falls within the first phase tolerance interval set is determined as the first tolerance identifier. Second, the index of the tolerance interval into which the label value of the candidate label data falls within the second phase tolerance interval set is determined as the second tolerance identifier. Third, the first tolerance identifier and the second tolerance identifier are determined as the tolerance identifier group corresponding to the candidate label data. Additionally, for candidate label data with categorical label values, the executing entity can determine the label value of the candidate label data as the first tolerance identifier and the second tolerance identifier corresponding to the candidate label data, respectively.

[0075] The fifth step involves performing phase-by-phase hashing on the aforementioned tolerance identifier set to obtain a set of check values. The check values ​​in this set can be fixed-length values ​​corresponding to combinations of tag identifiers and tolerance identifiers. The check values ​​are the same when the combinations are the same, and generally different when the combinations are different. For example, the check values ​​can be integers ranging from 0 to 4294967295. The check value set can also include the check values ​​of two tolerance identifiers within the same tolerance identifier set. For example, if the tolerance identifier set corresponding to the tag value 23 is 2 and 1, the check value set could be the check value corresponding to tolerance identifier 2 and the check value corresponding to tolerance identifier 1. In practice, the execution entity can perform the following hashing steps for each tolerance identifier set: First, concatenate the tag identifier of the candidate tag data corresponding to the tolerance identifier set with the first tolerance identifier in the tolerance identifier set to obtain the first string to be hashed. The second step involves hashing the first string to be hashed using the MurmurHash3 algorithm to obtain the first verification value. The third step concatenates the tag identifier of the candidate tag data corresponding to the tolerance identifier group with the second tolerance identifier in the tolerance identifier group to obtain the second string to be hashed. The fourth step uses the MurmurHash3 algorithm to hash the second string to be hashed to obtain the second verification value. The fifth step combines the first and second verification values ​​to determine the verification value group corresponding to the tolerance identifier group.

[0076] Step 6: Based on the reference check value set included in the tag acquisition request, perform multi-phase comparison processing on the check value set to obtain a set of difference tag identifiers. The tag acquisition request may also include a set of reference check values. The reference check values ​​included in the reference check value set can be the check values ​​corresponding to a tolerance identifier from the previous tag data acquisition. The difference tag identifiers in the difference tag identifier set can be tag identifiers corresponding to tags whose check values ​​are different from the two reference check values. For example, if a tag's tag value was 21 in the previous tag data acquisition, and the tolerance identifier set was 2 and 1, and the tag value changes to 23, the tolerance identifier set remains 2 and 1, but the two check values ​​are the same as the two reference check values, so the tag identifier does not belong to the difference tag identifier category. If the tag value changes to 37, the tolerance identifier set is 3 and 3, and the two check values ​​are different from the two reference check values, so the tag identifier belongs to the difference tag identifier category. In practice, the aforementioned executing entity can perform the following comparison steps for each check value group in the aforementioned check value set: First, determine the reference check value group in the aforementioned reference check value set that corresponds to the tag identifier of the candidate tag data corresponding to the aforementioned check value group as the current reference check value group. Second, in response to determining that the aforementioned current reference check value group does not exist, determine the tag identifier of the aforementioned candidate tag data as a difference tag identifier. Third, in response to determining that the aforementioned current reference check value group exists and that the first check value in the aforementioned check value group is different from the first reference check value in the aforementioned current reference check value group, determine the tag identifier of the aforementioned candidate tag data as an identifier in the first phase difference identifier set. Fourth, in response to determining that the aforementioned current reference check value group exists and that the second check value in the aforementioned check value group is different from the second reference check value in the aforementioned current reference check value group, determine the tag identifier of the aforementioned candidate tag data as an identifier in the second phase difference identifier set. Fifth step: In response to determining that the tag identifier of the above candidate tag data belongs to both the first phase difference identifier set and the second phase difference identifier set, the tag identifier of the above candidate tag data is determined as a difference tag identifier.

[0077] Step 7: Based on the aforementioned set of difference label identifiers, the candidate label dataset is filtered to obtain the label dataset to be output. In practice, the executing entity can determine the candidate label data whose label identifiers belong to the aforementioned set of difference label identifiers as the label dataset to be output.

[0078] Step 8: Based on the migrated label dataset and the label dataset to be output, control the label data flow of the data storage server set. In practice, the execution entity can first determine the scanning range of the data storage server set for each storage location of the label dataset to be output. Then, determine the storage locations of label data in the migrated label dataset that have exceeded the cell validity period as the set of storage locations to be reclaimed. Next, use the majorCompact method of the Apache HBase Admin interface to trigger the data storage server set to reclaim storage space for the set of storage locations to be reclaimed. Finally, determine the label data in the migrated label dataset that has not been output as label data that will not be written to the memory cache of the data storage server set.

[0079] The above-mentioned technical solution and related content, as an inventive point of this disclosure, solves the fourth technical problem: "The benchmark for output is different from the benchmark for write filtering, and tag data whose value change relative to the previous acquisition has not reached the preset change threshold is still output." Factors leading to increased network transmission volume in the data storage server set are often as follows: whether the tag value has changed is determined solely by whether the data timestamp is later than the synchronization timestamp, while write filtering is based on the change in the tag value relative to the previous write; when the backtracking interval of the value is shorter than the interval between two adjacent acquisitions, the tag experiences multiple increases and decreases between two acquisitions, and each change where the change reaches the preset change threshold is written and the data timestamp is refreshed; even if the change in the tag value relative to the previous acquisition has not reached the preset change threshold, it is still judged as a change and output. Solving these factors can reduce the amount of output tag data and reduce the network transmission volume of the data storage server set. To achieve this effect, this disclosure first performs time-series merging processing on the migrated tag dataset according to the target storage ownership information set to obtain the last ownership timestamp set. Next, based on the aforementioned synchronization timestamp and the aforementioned last time stamp set, the aforementioned target storage ownership information set is subjected to batch filtering to obtain a filtered ownership information set. Then, based on the filtered ownership information set and the aforementioned synchronization timestamp, the aforementioned migrated tag dataset is subjected to incremental extraction to obtain a candidate tag dataset. Here, storage locations that have not been written to since the last acquisition are completely skipped, eliminating the need to read tag data from each location individually, thus reducing the amount of data read. Next, based on a preset change threshold and a preset phase offset value, the aforementioned candidate tag dataset undergoes multi-phase tolerance mapping to obtain a tolerance identifier set. Then, the aforementioned tolerance identifier set undergoes phase-by-phase hashing to obtain a checksum set. Here, value fluctuations not exceeding the threshold are grouped into the same tolerance range and no longer produce differences; furthermore, the reference information used for comparison has a fixed length and can be carried with each request, eliminating the need for the data storage server set to save the previously output tag values ​​for each requester. Next, based on the set of reference check values ​​included in the aforementioned tag acquisition request, multi-phase comparison processing is performed on the check value set to obtain a set of difference tag identifiers. Then, based on the aforementioned set of difference tag identifiers, the aforementioned candidate tag dataset is filtered to obtain the tag dataset to be output. Here, a change is only determined when both sets of phases are different, eliminating misjudgments near interval boundaries in single interval divisions. Finally, based on the aforementioned migrated tag dataset and the aforementioned tag dataset to be output, tag data flow control is implemented on the aforementioned data storage server set. Here, the amount of tag data output by the data storage server set and the network transmission volume are reduced accordingly. Therefore, the above technical solution and its related content can reduce the amount of output tag data and reduce the network transmission volume of the data storage server set.

[0080] The above embodiments of this disclosure have the following beneficial effects: The tag data flow control method based on write filtering and frequency routing in some embodiments of this disclosure can reduce the storage space occupation of the data storage server set and reduce the load of the data storage server set. Specifically, the reason for the waste of storage and computing resources of the data storage server set is that: the tag data generated by each collection is written without distinguishing whether the tag value has changed. Tag data with unchanged values ​​is repeatedly written and the data timestamp is refreshed, which increases the number of retained historical tag data, and is read and output together when responding to the acquisition request; at the same time, the storage location is divided according to the preset category of the tag, and the number and duration of the retained historical tag data are not commensurate with the frequency of actual tag writing. Tag data with different value change frequencies are stored in the same storage location, which increases the amount of data read in a single time. Based on this, the tag data flow control method based on write filtering and frequency routing in some embodiments of this disclosure can first, in response to receiving the tag dataset to be stored, obtain the stored tag dataset and storage ownership information set corresponding to the tag dataset to be stored from the data storage server set. Here, the existing tag dataset provides a comparison benchmark for subsequent change filtering, and the storage ownership information set provides a storage location for subsequent time-limited anchored write processing. Next, based on the existing tag dataset, change filtering is performed on the tag dataset to be stored, resulting in a filtered tag dataset. Here, the number of write requests and the amount of retained historical tag data are reduced accordingly. Then, based on the storage ownership information set, time-limited anchored write processing is performed on the filtered tag dataset, resulting in a retained tag dataset. Here, expired tag data automatically becomes invalid, and the data storage server set does not require further cleanup. Next, rolling window aggregation processing is performed on the retained tag dataset to obtain a write frequency set. Here, the write frequency is updated according to the recent write activity of the tags and is not smoothed by distant historical records. Next, based on the write frequency set, bidirectional bandwidth-limited routing processing is performed on the storage ownership information set to obtain the target storage ownership information set. Here, redundant historical versions are reduced, and upper and lower limits ensure that the duration is not too short, preventing duplicate writes. Next, based on the aforementioned target storage ownership information set, the retained tag dataset is migrated to obtain a migrated tag dataset. Here, the amount of historical tag data retained in each storage location matches the configured storage duration of that storage location. Finally, in response to a received tag retrieval request, based on the aforementioned target storage ownership information set and the synchronization timestamp included in the tag retrieval request, the migrated tag dataset is pruned dimension-by-dimensional to obtain a tag dataset to be output. Furthermore, based on the migrated tag dataset and the tag dataset to be output, tag data flow control is performed on the aforementioned data storage server set.Here, the scanning range, storage space usage, and memory cache usage of the data storage server set are reduced accordingly. Therefore, this tag data flow control method based on write filtering and frequency routing can reduce the storage space usage of the data storage server set and lower its load.

[0081] Further reference Figure 2 As an implementation of the methods shown in the above figures, this disclosure provides some embodiments of a tag data flow control device based on write filtering and frequency routing. These device embodiments are similar to... Figure 1 Corresponding to the method embodiments shown, this tag data flow control device based on write filtering and frequency routing can be specifically applied to various electronic devices.

[0082] like Figure 2 As shown, a tag data flow control device 200 based on write filtering and frequency routing includes: an acquisition unit 201, a filtering unit 202, a writing unit 203, an aggregation unit 204, a routing unit 205, a migration unit 206, and a control unit 207. The acquisition unit 201 is configured to: in response to receiving a tag dataset to be stored, acquire the existing tag dataset and storage ownership information set corresponding to the tag dataset to be stored from a data storage server set. The filtering unit 202 is configured to: perform change filtering processing on the tag dataset to be stored based on the existing tag dataset to obtain a filtered tag dataset. The writing unit 203 is configured to: perform time-limited anchored write processing on the filtered tag dataset based on the storage ownership information set to obtain a retained tag dataset. The aggregation unit 204 is configured to: perform rolling window aggregation processing on the retained tag dataset to obtain a write frequency set. The routing unit 205 is configured to: perform bidirectional amplitude-limited routing processing on the storage ownership information set based on the write frequency set to obtain a target storage ownership information set. Migration unit 206 is configured to: perform migration processing on the retained tag dataset according to the target storage ownership information set to obtain a migrated tag dataset. Control unit 207 is configured to: in response to receiving a tag acquisition request, perform dimension-by-dimensional pruning processing on the migrated tag dataset according to the target storage ownership information set and the synchronization timestamp included in the tag acquisition request to obtain a tag dataset to be output; and perform tag data flow control on the data storage server set according to the migrated tag dataset and the tag dataset to be output.

[0083] It is understandable that the units and references described in the tag data flow control device 200 based on write filtering and frequency routing are... Figure 1The steps in the described method correspond to each other. Therefore, the operations, features, and beneficial effects described above for the method also apply to the tag data flow control device 200 based on write filtering and frequency routing and the units contained therein, and will not be repeated here.

[0084] The following is for reference. Figure 3 It shows a schematic diagram of the structure of an electronic device (e.g., an electronic device) 300 suitable for implementing some embodiments of the present disclosure. Figure 3 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments of this disclosure.

[0085] like Figure 3 As shown, the electronic device 300 may include a processing unit (e.g., a central processing unit, a graphics processing unit, etc.) 301, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 302 or a program loaded from a storage device 308 into a random access memory (RAM) 303. The RAM 303 also stores various programs and data required for the operation of the electronic device 300. The processing unit 301, ROM 302, and RAM 303 are interconnected via a bus 304. An input / output (I / O) interface 305 is also connected to the bus 304.

[0086] Typically, the following devices can be connected to I / O interface 305: input devices 306 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 307 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 308 including, for example, magnetic tapes, hard disks, etc.; and communication devices 309. Communication device 309 allows electronic device 300 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 3 An electronic device 300 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively. Figure 3 Each box shown can represent a device or multiple devices as needed.

[0087] In particular, according to some embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, some embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication device 309, or installed from storage device 308, or installed from ROM 302. When the computer program is executed by processing device 301, it performs the functions defined in the methods of some embodiments of this disclosure.

[0088] It should be noted that, in some embodiments of this disclosure, the computer-readable medium described above may be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In some embodiments of this disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In some embodiments of this disclosure, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0089] In some implementations, clients and servers can communicate using any currently known or future-developed network protocol such as HTTP (Hypertext Transfer Protocol) and can interconnect with digital data communication (e.g., communication networks) of any form or medium. Examples of communication networks include local area networks (“LANs”), wide area networks (“WANs”), the Internet (e.g., the Internet of Things), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any currently known or future-developed networks.

[0090] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device. The aforementioned computer-readable medium carries one or more programs that, when executed by the electronic device, cause the electronic device to: in response to receiving a tag dataset to be stored, retrieve from a data storage server set the corresponding stored tag dataset and storage ownership information set; perform change filtering on the tag dataset to be stored based on the stored tag dataset to obtain a filtered tag dataset; perform time-limited anchoring write processing on the filtered tag dataset based on the storage ownership information set to obtain a retained tag dataset; and perform rolling window aggregation processing on the retained tag dataset. A write frequency set is obtained; based on the write frequency set, bidirectional bandwidth-limited routing is performed on the storage ownership information set to obtain a target storage ownership information set; based on the target storage ownership information set, the retained tag dataset is migrated to obtain a migrated tag dataset; in response to receiving a tag acquisition request, based on the target storage ownership information set and the synchronization timestamp included in the tag acquisition request, the migrated tag dataset is pruned dimension by dimension to obtain a tag dataset to be output; and based on the migrated tag dataset and the tag dataset to be output, tag data flow control is performed on the data storage server set.

[0091] Computer program code for performing operations of some embodiments of this disclosure can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0092] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0093] The units described in some embodiments of this disclosure can be implemented in software or in hardware. The described units can also be located in a processor.

[0094] The functions described above in this document can be performed at least in part by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), system-on-a-chip (SoCs), complex programmable logic devices (CPLDs), and so on.

[0095] The above description is merely a selection of preferred embodiments of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of the invention involved in the embodiments of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described inventive concept. For example, technical solutions formed by substituting the above-described features with (but not limited to) technical features with similar functions disclosed in the embodiments of this disclosure.

Claims

1. A tag data flow control method based on write filtering and frequency routing, comprising: In response to receiving a tag dataset to be stored, the system retrieves the existing tag dataset and storage attribution information set corresponding to the tag dataset to be stored from the data storage server set. Based on the existing tag dataset, the tag dataset to be stored is subjected to change filtering to obtain the filtered tag dataset; Based on the storage attribution information set, the filtered tag dataset is subjected to time-limited anchoring writing processing to obtain the retained tag dataset; The retention tag dataset is subjected to a rolling window aggregation process to obtain a write frequency set; Based on the write frequency set, the storage ownership information set is subjected to bidirectional bandwidth limiting routing processing to obtain the target storage ownership information set; Based on the target storage attribution information set, the retained tag dataset is migrated to obtain the migrated tag dataset; In response to receiving a tag acquisition request, the migrated tag dataset is pruned dimension by dimension according to the target storage ownership information set and the synchronization timestamp included in the tag acquisition request to obtain a tag dataset to be output. The tag data flow control is then performed on the data storage server set according to the migrated tag dataset and the tag dataset to be output.

2. The method according to claim 1, wherein, The step of obtaining the existing tag dataset and storage attribution information set corresponding to the tag dataset to be stored from the data storage server set includes: Based on the primary key identifier set included in the tag dataset to be stored, query the data storage server set to obtain the historical tag dataset and attribution metadata; The historical tag dataset is subjected to time-series filtering to obtain the existing tag dataset; Based on the tag identifier set included in the tag dataset to be stored, the attribution metadata is matched to obtain the existing attribution information set and the missing tag identifier set; The missing tag identifier set is filled in according to the preset default attribution information to obtain the filled attribution information set; The existing attribution information set and the filled attribution information set are determined as the storage attribution information set.

3. The method according to claim 1, wherein, The step of performing change filtering on the tag dataset to be stored based on the existing tag dataset to obtain the filtered tag dataset includes: Based on the data type identifier set included in the label dataset to be stored, the label dataset to be stored is divided into numerical label datasets and categorical label datasets. Based on the existing label dataset, the numerical label dataset is subjected to difference comparison processing to obtain the first label change set; Based on the existing label dataset, a consistency comparison process is performed on the categorical label dataset to obtain the second label change set; The first set of label change values ​​and the second set of label change values ​​are determined as the label change value set; Based on the set of label changes, the dataset of labels to be stored is subjected to threshold filtering to obtain a filtered dataset of labels.

4. The method according to claim 1, wherein, The step of performing rolling window aggregation on the retention tag dataset to obtain the write frequency set includes: The written status filtering process is performed on the retained tag dataset to obtain the written tag dataset; The migration identifier filtering process is performed on the written label dataset to obtain a non-migration label dataset; Based on the preset scrolling window duration, the non-migrating label dataset is divided into windows to obtain the label dataset within the window; The label dataset within the window is grouped and aggregated to obtain the write frequency set.

5. The method according to claim 1, wherein, The storage ownership information set includes: storage location identifier and storage validity period.

6. The method according to claim 1, wherein, The step of migrating the retained tag dataset according to the target storage attribution information set to obtain the migrated tag dataset includes: The target storage ownership information set and the storage ownership information set are divided by migration direction to obtain the upgrade ownership information set and the downgrade ownership information set. Based on a preset upgrade frequency threshold, a single-window comparison process is performed on the upgrade attribution information set and the write frequency set to obtain the upgrade confirmation attribution information set. Based on the preset downgrade frequency threshold and the preset number of consecutive confirmation windows, cross-window continuous verification processing is performed on the downgrade attribution information set and the write frequency set to obtain the downgrade confirmation attribution information set. Based on the upgrade confirmation attribution information set and the downgrade confirmation attribution information set, the attribution change filtering process is performed on the retained tag dataset and the target storage attribution information set to obtain the tag dataset to be migrated and the attribution correction information set. Based on the obtained current time value, the time information of the label dataset to be migrated is reset to obtain the reset label dataset; Based on the storage attribution information set, version masking is performed on the tag dataset to be migrated to obtain the original attribution deletion tag set; The reset label dataset, the original attribution deletion tag set, and the attribution correction information set are merged and written to obtain the migrated label dataset.

7. A tag data flow control device based on write filtering and frequency routing, comprising: The acquisition unit is configured to, in response to receiving a tag dataset to be stored, acquire from the data storage server set the existing tag dataset and storage attribution information set corresponding to the tag dataset to be stored. The filtering unit is configured to perform change filtering on the tag dataset to be stored based on the existing tag dataset to obtain a filtered tag dataset. The writing unit is configured to perform time-limited anchoring writing processing on the filtered tag dataset according to the storage attribution information set to obtain the retained tag dataset. The aggregation unit is configured to perform a rolling window aggregation process on the retention tag dataset to obtain a write frequency set; The routing unit is configured to perform bidirectional bandwidth-limited routing processing on the storage ownership information set according to the write frequency set to obtain the target storage ownership information set; The migration unit is configured to perform migration processing on the retained tag dataset according to the target storage attribution information set to obtain the migrated tag dataset; The control unit is configured to, in response to receiving a tag acquisition request, perform dimension-by-dimensional pruning on the migrated tag dataset according to the target storage attribution information set and the synchronization timestamp included in the tag acquisition request to obtain a tag dataset to be output, and perform tag data flow control on the data storage server set according to the migrated tag dataset and the tag dataset to be output.

8. An electronic device, comprising: One or more processors; Storage device, on which one or more programs are stored, When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1-6.

9. A computer-readable medium having a computer program stored thereon, wherein, When the computer program is executed by a processor, it implements the method as described in any one of claims 1-6.

Images