Multi-scene application data acquisition method based on atomization design

Abstract

The invention discloses a multi-scene application data acquisition method based on atomization design. The method comprises the following steps: analyzing reported data into an atomization data structure; performing semantic recognition on the data labels to generate a label semantic mapping result; constructing a layered cache structure; generating a compressed semantic mapping rule by applying a multi-level semantic difference coding algorithm; identifying an optimal query path from the tag affinity matrix; and executing historical data cleaning based on the data value. Intelligent mapping of data labels is achieved through semantic recognition and version control, the storage efficiency is improved through predictive lazy loading and compression coding, the query path is optimized based on the affinity matrix, and the technical problem of multi-scene application data collection is effectively solved.

Description

Technical Field

[0001] The present invention belongs to the fields of energy management and Internet of Things, and in particular, to a multi-scenario application data acquisition method based on atomic design. Background Art

[0002] With the rapid development of Internet of Things technology and intelligent devices, data acquisition plays a key role in multiple fields such as energy management, environmental monitoring, and industrial control. Devices in different scenarios generate a large amount of heterogeneous data, which have different formats, semantics, and values. How to efficiently collect, parse, and manage these multi-scenario application data has become an important research topic. An effective data acquisition method can promote the interconnection and interoperability of data, improve the ability to mine data value, and is of great significance for realizing intelligent decision-making and refined management.

[0003] Current data acquisition methods are mostly developed based on specific scenarios, and mainly use predefined templates and fixed mapping rules to process data. Traditional methods usually pre-design data structure templates and map the collected raw data into the predefined structure. Some improved solutions introduce metadata management and describe the data structure through configuration files, improving the flexibility of the system. Other methods use a middleware layer to achieve unified conversion of different data source interfaces. These technologies have achieved certain effects in specific field applications and basically meet the data acquisition requirements in a single scenario.

[0004] However, the existing technologies still have obvious deficiencies in dealing with multi-scenario heterogeneous data. First, the predefined template is difficult to adapt to the dynamically changing data semantics. When new data tags are added or the data meaning is adjusted, it is necessary to re-design the template and mapping rules, lacking flexibility. Second, the traditional full-volume caching strategy is inefficient in a large-scale data environment and cannot be optimized for loading according to the data usage characteristics, resulting in waste of system resources and performance bottlenecks. In addition, the existing data query paths usually adopt fixed optimization strategies, lacking the ability to learn query relevance and unable to perform adaptive optimization according to the actual usage patterns, restricting the data query efficiency. Summary of the Invention

[0005] The object of the invention is to provide a multi-scenario application data acquisition method based on atomic design, in order to solve at least one technical problem existing in the prior art.

[0006] Technical solution: A multi-scenario application data acquisition method based on atomic design includes:

[0007] Receiving the reported data, parsing to form an atomic data structure including device identifier, freeze time, data tag, and data value; performing semantic recognition on the data tags therein, and outputting the tag semantic mapping result;

[0008] Based on the tag semantic mapping results, a tag mapping rule set is generated through a hierarchical caching structure; it is encoded and compressed to form a compressed semantic mapping rule;

[0009] Receive a data query request and a compressed semantic mapping rule, identify the query path from a pre-configured tag affinity matrix, execute the query and return the query result; accordingly, monitor the system storage space and perform historical data cleaning based on data value evaluation.

[0010] Advantageous effects: The present invention can automatically adapt to newly added data tags or adjusted data meanings, improving semantic adaptability and semantic correct mapping rate; at the same time, it also optimizes resource utilization, improving memory usage efficiency and query efficiency. Description of the Drawings

[0011] Figure 1 It is a step flow chart of a multi-scenario application data collection method based on atomization design provided by an embodiment of the present application.

[0012] Figure 2 It is a step flow chart of parsing to form an atomized data structure provided by an embodiment of the present application.

[0013] Figure 3 It is a step flow chart of outputting tag semantic mapping results provided by an embodiment of the present application.

[0014] Figure 4 It is a step flow chart of generating a tag mapping rule set provided by an embodiment of the present application. Detailed Embodiments

[0015] In order to enable those skilled in the art to better understand the solution of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0016] It should be particularly noted that, for the sake of clearly showing the step flow of the present application, serial numbers are marked for each step in the specification. These serial numbers are only for the convenience of description and do not limit the execution order of the steps. In actual operation, according to the technical requirements of the specific implementation scenario, each step can be executed in an order different from that shown in the specification, and in some cases, parallel processing between steps can also be achieved.

[0017] As Figure 1 shown, the multi-scenario application data collection method based on atomization design includes the following steps:

[0018] S1. Receive the reported data and parse it to form an atomized data structure containing device identification, freezing time, data label, and data value.

[0019] Specifically, the reported data is the data actively sent by a device or application to the system. For example, a smart meter reports electricity consumption. Obtain the content of the reported data, determine whether it is compressed, and if so, decompress the message and identify the reporting protocol. Parse it to form an atomized data structure. Among them, the device identification shows which specific device uploaded the data, such as the ID of a certain smart home device; the freezing time is the time point of the data, such as the data recorded by a temperature sensor at a certain time; the data label is the content represented by the data, such as "temperature", "humidity", "electricity", etc.; the data value is the specific numerical value, such as "temperature = 22.5°C".

[0020] S2. Perform semantic recognition on the data labels in the atomized data structure and output the label semantic mapping result.

[0021] Specifically, read the data labels in the atomized data structure, analyze the label semantics through a semantic recognition engine, generate label semantic feature vectors, match them with the existing labels in the label dictionary library, determine the label version information, and output the label semantic mapping result. Among them, semantic recognition is to understand the true meaning of the data label, rather than just treating it as a string of characters. The label semantic mapping result is to associate the label with its true meaning. For example: "TEMP" → "temperature"; "BATTERY" → "battery power"; "HUMID" → "humidity"; the final mapping result can be used to help the system understand the data and ensure that it can correctly use this information, such as for data analysis, report generation, intelligent monitoring, etc.

[0022] S3. Based on the label semantic mapping result, generate a label mapping rule set through a hierarchical caching structure.

[0023] Specifically, receive the label semantic mapping result, construct a hierarchical caching structure based on the semantic association predictive lazy loading strategy, store the frequently used label mapping rules in the first-level semantic cache, store the associated label mapping rules in the second-level semantic cache, and store the complete mapping history in the persistent storage. The hierarchical caching structure can be understood as a multi-level storage system. For example: the fast access layer (such as memory cache): stores the most frequently used data and has a fast reading speed; the secondary storage layer (such as a local database): stores the data that needs to be used but is not particularly frequently used; the long-term storage layer (such as cloud storage): stores a larger scale of data, with a slightly slower access speed but a larger storage capacity; this can improve the query efficiency and reduce the system burden. The label mapping rule set refers to formulating standardized mapping rules for different types of labels to ensure consistency.

[0024] S4. Encode and compress the label mapping rule set to form a compressed semantic mapping rule;

[0025] Specifically, obtain the label mapping rule set, apply a multi-level semantic difference encoding algorithm for compression, separate the core semantics from the extended semantics, generate a compressed semantic mapping rule, and store it in the storage system to reduce the storage space requirement and improve the access efficiency. The label mapping rule set has already defined the correspondence between data labels and their true meanings. However, if all mapping rules are directly stored, it may occupy a relatively large amount of storage space and affect the query efficiency. Therefore, the purpose of encoding and compression is to reduce storage occupancy while maintaining the integrity and availability of the data.

[0026] S5. Receive a data query request and the compressed semantic mapping rule, identify the query path from the pre-configured label affinity matrix, execute the query, and return the query result;

[0027] Specifically, the affinity matrix can be the association relationship between labels, helping the system find the most suitable query path. For example, "TEMP" (temperature) and "HUMID" (humidity) may belong to the environmental parameter category; "BATTERY" (battery power) and "VOLTAGE" (voltage) may belong to the power management category; in this way, the system can quickly find the most relevant labels during querying without having to perform a large number of unnecessary calculations. Identify the optimal query path from the label affinity matrix, dynamically adjust the query strategy according to the current system load status, and execute the query operation.

[0028] S6. According to the query result, monitor the system storage space and perform historical data cleaning based on data value evaluation.

[0029] Specifically, regularly detect the available storage space of the system. When the space is insufficient, perform historical data cleaning based on the data time characteristics and usage value evaluation to ensure the continuous and stable operation of the system. Not all data is equally important. Some data may be critical business data, while some data may be historical data with a low usage frequency. The role of data value evaluation is to judge which data is worth retaining for a long time and which data can be cleaned. The evaluation criteria may include: access frequency: whether it has been queried recently; business importance: whether it has an impact on the core business; storage cost: whether it occupies a large amount of storage resources.

[0030] In this embodiment, through semantic recognition, the system can automatically understand the true meaning of data tags, reduce manual intervention, and improve the intelligent level of data processing; encoding compression reduces storage occupancy, makes data storage more compact, and at the same time maintains the efficiency of queries; data value evaluation ensures that the system only retains high-value data, automatically clears low-value historical data, and optimizes the storage space; through the label mapping rule set and the affinity matrix, the system can quickly identify the query path, reduce unnecessary calculations, and improve the query speed. This embodiment can improve the quality of data collection and analysis.

[0031] As Figure 2 shown, according to one aspect of the present application, the steps of parsing to form an atomized data structure include:

[0032] S11. Receive the MQTT message reported by the device, and parse the message topic to obtain the original data packet;

[0033] S12. Check the message header feature identifier of the original data packet to determine whether it is compressed. If it is compressed, call the decompression algorithm to obtain the decompressed packet;

[0034] S13. Identify the protocol identification field in the decompressed packet to determine the protocol specification type;

[0035] S14. According to the protocol specification type, call the corresponding parsing logic to extract and organize the device identifier, freeze time, data tag, and data value to form a complete atomized data structure.

[0036] Specifically, receive the MQTT message reported by the device, parse the message topic, judge the data source and type, obtain the message payload content, and get the original data packet. Check whether the original data packet is compressed, which is determined by checking the message header feature identifier. If the compression identifier is found, call the decompression algorithm to decompress the data to obtain the decompressed packet. Identify the protocol specification type of the decompressed packet, which is judged by the protocol identification field in the packet, and call the corresponding parsing logic according to different protocol specifications to extract the data content in the packet. Parse the packet data, extract four key elements: device identifier, freeze time, data tag, and data value, and organize them into an atomized data structure as the basic unit for subsequent processing.

[0037] As Figure 3 shown, according to one aspect of the present application, the steps of outputting the label semantic mapping result include:

[0038] S21. Extract the data tag from the atomized data structure, perform processing such as removing special characters, unifying case, and word segmentation to obtain the standardized tag text;

[0039] S22. Input the standardized tag text into the semantic recognition engine to extract multi-dimensional semantic features and generate a label semantic feature vector;

[0040] S23. Calculate the similarity between the label semantic feature vector and the existing labels in the label dictionary library to determine the most matching label and its version information;

[0041] S24. Based on the most matching label and its version information, obtain the context information of the device type, spatial location, and data characteristics;

[0042] S25. Combine the context information with the most matching label and its version information to generate the label semantic mapping result.

[0043] Specifically, extract data labels from the atomized data structure, preprocess the label text, including operations such as removing special characters, unifying case, and word segmentation, to obtain a standardized label text. Input the standardized label text into the semantic recognition engine, extract the semantic features of the label through the semantic analysis algorithm, and generate a label semantic feature vector that describes the label semantics. The semantic feature vector adopts a multi-dimensional vector form, including multiple feature components such as the function dimension, unit dimension, and precision dimension of the label. Design and apply a semantic version control architecture, assign a three-level version number (major version.minor version.revision number) to each label, where the major version represents a major semantic change, the minor version represents a semantic adjustment, and the revision number represents a compatibility adjustment. Calculate the similarity between the label semantic feature vector and the feature vectors of the existing labels in the label dictionary library to determine the most matching label and its version information. Build a context-aware label system, associate the label with its usage environment, including dimensions such as device type context, spatial location context, and data characteristics context, to form a complete label semantic expression. Combine the label, version information, and context information to generate the label semantic mapping result.

[0044] According to one aspect of the present application, the steps of generating a label semantic feature vector include:

[0045] Decompose the standardized label text into a set of morphemes to obtain a label morpheme set; map the morphemes in the label morpheme set to the domain ontology library, obtain the domain semantic weights of each morpheme, and generate an initial morpheme semantic mapping set;

[0046] Perform context-aware word sense disambiguation on the initial morpheme semantic mapping set, identify and process polysemous words, and generate a disambiguated morpheme semantic mapping set; based on the disambiguated morpheme semantic mapping set, construct a semantic space representation including function, object, location, unit, and precision dimensions, and calculate the initial semantic vector;

[0047] Normalize the initial semantic vector so that the values of each dimension are mapped to the [0, 1] interval to form a label semantic feature vector.

[0048] Specifically, receive the standardized label text, and use the N-gram tokenization technique to decompose the text into sets of unigrams, bigrams, and trigrams, obtaining the label token set. For example, "indoor temperature sensor" is decomposed into tokens such as {"indoor", "temperature", "sensor", "indoor temperature", "temperature sensor"}. Map each token in the label token set to a predefined domain ontology library to obtain the domain semantic weight of each token. The ontology library contains the concept hierarchy and relationship definitions in the energy monitoring field. For example, "temperature" belongs to the "physical quantity" category, and "indoor" belongs to the "spatial location" category. Generate a token semantic mapping set, where each element contains the token, its corresponding domain category, and weight. Based on the token semantic mapping set, construct a multi-dimensional semantic space representation of the label. Design a nine-dimensional semantic vector structure, including: Function dimension (F1 - F3): Describes the main function categories of the label, such as "measurement", "control", "alarm"; Object dimension (O1 - O2): Describes the physical objects associated with the label, such as "temperature", "voltage", "power"; Location dimension (L1 - L2): Describes the spatial location information of the label, such as "indoor", "power distribution room"; Unit dimension (U1): Describes the measurement unit of the label, such as "degree Celsius", "volt", "watt"; Precision dimension (P1): Describes the precision requirement of the label, such as "high precision", "low precision".

[0049] Construct and implement a context-aware word sense disambiguation algorithm to solve the problem of polysemous words in the label text. This algorithm is processed through the following steps: Identify tokens with multiple possible semantic meanings from the token semantic mapping set; Analyze the context relationship of the token in the label, such as adjacent tokens, modification relationships; Calculate the probability distribution of different semantic interpretations in the current context; Select the semantic interpretation with the highest probability as the final mapping result.

[0050] Based on the processed token semantic mapping set, calculate the specific values of the nine-dimensional semantic vector. The calculation formula is: V[i] = Σ(W_j * R_ji) / Σ(W_j); where V[i] is the value of the i-th dimension of the semantic vector; W_j is the weight of token j (reflecting the importance of the token in the label); R_ji is the contribution degree of token j to dimension i; Σ represents the sum over all relevant tokens. Normalize the calculated semantic vector so that the value of each dimension is mapped to the interval [0, 1] to obtain a standardized semantic feature vector, which serves as the semantic representation of the label, that is, the label semantic feature vector.

[0051] According to one aspect of the present application, the steps of determining the most matching label and its version information and generating the label semantic mapping result include:

[0052] Calculate the cosine similarity between the semantic feature vector of the label and each label vector in the label dictionary library, and generate a sorted similarity list; according to the sorted similarity list, judge whether a new version of the label needs to be created through a set similarity threshold; according to the judgment result, determine the most matching label and its version information;

[0053] Calculate the dimension-level difference between the target vector and the most matching label, mark the significantly changed dimensions, and generate a dimension difference report; based on the dimension difference report, execute the version evolution rule to determine the upgrade method of the major version number, minor version number, or revision number, generate a label version identifier and associate it with the semantic feature vector of the label to form a label semantic mapping result. The format of the label version identifier is "label name@major version.minor version.revision number".

[0054] Specifically, receive the standardized semantic feature vector and retrieve similar labels in the label dictionary library. Use the cosine similarity algorithm to calculate the similarity between the target vector and each label vector in the library. The formula is: sim(A, B) = (A·B) / (||A||·||B||); where A is the target vector and B is the label vector in the library, and the result value ranges from [-1, 1]. The larger the value, the higher the similarity. Generate a sorted similarity list and sort the candidate labels in descending order of similarity. Based on the sorted similarity list, construct a semantic fault detection algorithm to judge whether a new version needs to be created. The core logic of this algorithm is: if the highest similarity exceeds the high threshold (default 0.95), it is judged as a perfect match of the same label and no version update is required; if the highest similarity is between the medium threshold and the high threshold (default 0.85 - 0.95), it is judged as a semantic adjustment of the same label and the minor version number needs to be updated; if the highest similarity is between the low threshold and the medium threshold (default 0.70 - 0.85), it is judged as a major change of the same label and the major version number needs to be updated; if the highest similarity is lower than the low threshold (default 0.70), it is judged as a new label and a new label record needs to be created.

[0055] Implement a vector difference analyzer to precisely locate the specific dimensions of label semantic changes. Calculate the dimension-level differences between the target vector and the most matching label vector: Diff[i] = |A[i] - B[i]|; where Diff[i] represents the difference value of the i-th dimension, and the dimensions with difference values greater than the preset threshold are marked as significantly changed dimensions. Generate a dimension difference report, including the difference values of each dimension and the list of significantly changed dimensions. Based on the dimension difference report, execute the version evolution rule engine to determine the version upgrade method. The core rules include: if there are significant changes in the function dimension (F1 - F3), upgrade the major version number; if there are significant changes in the object dimension (O1 - O2) or the location dimension (L1 - L2), upgrade the minor version number; if there are significant changes only in the unit dimension (U1) or the precision dimension (P1), upgrade the revision number; the version number format is "X.Y.Z", where X is the major version number, Y is the minor version number, and Z is the revision number. Create a version change record, which details the reasons for version changes, the changed dimensions, and the semantic vector values before and after the changes, as the historical record of label version evolution. The version change record adopts an incremental storage method, only recording the changed parts to reduce storage overhead. Generate a label version identifier in the format of "label name@major version.minor version.revision number", such as "indoor temperature@2.3.1", as the unique identifier of the label. Associate the label version identifier with the label semantic feature vector to form the label semantic mapping result.

[0056] According to one aspect of the present application, the steps of generating a label mapping rule set through a hierarchical cache structure include:

[0057] S31. Receive the label semantic mapping result, record it in the label mapping history library of the system, and update the usage frequency and context information statistics of the label;

[0058] S32. Construct and implement a semantic association predictive lazy loading strategy, which is different from traditional lazy loading and has the following characteristics: build a semantic association network based on the historical query pattern and the current context to predict the labels that may be needed in the short term; design a semantic distance attenuation function D(t, c) = e -α·R(t)·S(t，c) , where R(t) is the access frequency of label t, S(t, c) is the semantic similarity between label t and the current context c, and α is a system adjustable parameter; dynamically determine the preloading range according to the semantic distance, and preload the label into the cache when D(t, c) is less than the preset threshold;

[0059] S33. Implement a dynamic switching mechanism that senses the network state. By monitoring the system load and network state, adaptively adjust the preloading strategy: when the system load is light, expand the preloading range and cache more semantically related labels in advance; when the system load is heavy, narrow the preloading range and only cache the most core labels; when the network state is unstable, preferentially cache the core labels locally to reduce network dependence;

[0060] S34. Construct a three - layer cache structure and store the label mapping rules according to the importance and usage frequency respectively: Primary semantic cache: Store the core label mapping rules with high usage frequency, using in - memory storage to support millisecond - level access; Secondary semantic cache: Store the label mapping rules related to the semantics of the labels in the primary cache, using local fast storage to support sub - second - level access; Persistent storage: Store the complete label mapping history and rules, using a distributed database for storage to support second - level access; Finally, obtain the label mapping rule set.

[0061] As Figure 4 shown, according to one aspect of the present application, the steps of generating the label mapping rule set include:

[0062] Receive the label semantic mapping result and the current context information, extract the query records from the query history log library, and construct a query sequence set; Analyze the temporal query pattern between labels based on the query sequence set, calculate the temporal correlation strength of label pairs, and generate a label temporal correlation matrix;

[0063] Based on the label temporal correlation matrix, calculate the semantic similarity between the label and the current device, user, time, and operation context, and generate a label context relevance table; Combine the label context relevance table with the label temporal correlation matrix to calculate the semantic distance attenuation value of the label;

[0064] Implement a three - level pre - loading strategy according to the semantic distance attenuation value, load labels with different priorities into the primary, secondary, and tertiary caches respectively, and generate the label mapping rule set.

[0065] Specifically, receive the label semantic mapping result and the current system context information, extract the most recent N (default value 100) query records from the query history log library, and construct a query sequence set. The query sequence contains information such as query time, query label, and query context. Based on the query sequence set, implement a label temporal correlation analyzer to mine the temporal query pattern between labels. Use the sliding time window method (the window size is configurable, default is 30 minutes), count the query sequences of labels within the window, and calculate the temporal correlation strength of label pairs (ti, tj): TS(ti, tj)=Count(ti→tj) / Count(ti); where Count(ti→tj) represents the number of times label tj is queried within a certain time (default 5 minutes) after label ti is queried, and Count(ti) represents the total number of queries of label ti. Generate a label temporal correlation matrix to record the temporal correlation strength of all label pairs.

[0066] Build and implement a context-sensitive label relevance calculator to calculate the semantic similarity between a label and the current context. The context information includes: device context: the type of the currently operated device, device ID, device location, etc.; user context: user role, user's historical operation preferences, etc.; time context: the current time period, whether it is a holiday, etc.; operation context: the type of the currently ongoing business operation, etc. The calculation formula is: S(t, c) = w1·S_dev(t, c_dev) + w2·S_user(t, c_user) + w3·S_time(t, c_time) + w4·S_op(t, c_op); where S_dev, S_user, S_time, and S_op respectively represent the similarity between label t and device context c_dev, user context c_user, time context c_time, and operation context c_op, and w1~w4 are weight coefficients, and w1 + w2 + w3 + w4 = 1. Generate a label context relevance table.

[0067] Combine the label temporal correlation matrix and the label context relevance table to implement a predictive loading decision engine. For each candidate label t, calculate its semantic distance decay value: D(t, c) = e -α·R(t)·S(t，c) ; where R(t) is the normalized access frequency of label t (value range [0, 1]), S(t, c) is the semantic similarity between label t and the current context c (value range [0, 1]), and α is a system adjustable parameter (default value is 2.0). The smaller the distance value D(t, c), the more important the label is and the more it should be pre-loaded. Based on the semantic distance value, implement a three-level pre-loading strategy: core pre-loading: labels with a distance value D(t, c) < 0.3 are immediately loaded into the first-level cache; associated pre-loading: labels with a distance value 0.3 ≤ D(t, c) < 0.6 are asynchronously loaded into the second-level cache; background pre-loading: labels with a distance value 0.6 ≤ D(t, c) < 0.8 are loaded into the third-level cache when the system is idle. Build a pre-loading effect feedback mechanism to record the actual usage of pre-loaded labels, including metrics such as hit rate, loading time, and cache residence time. Based on the feedback data, dynamically adjust the α parameter and each threshold to optimize the pre-loading performance. Generate a pre-loading label set and a pre-loading performance report.

[0068] According to one aspect of the present application, the implementation steps of the three-level pre-loading strategy include:

[0069] Real-time collect the CPU usage rate, memory usage rate, network latency, network packet loss rate, and I / O waiting time of the system to generate a system state vector; based on the system state vector, classify the system load status into light, medium, heavy, or critical load levels, and output a load level identifier;

[0070] Obtain network latency and packet loss rate, evaluate the network status and classify it into stable, fluctuating or unstable states, and output the network status rating; combine the network status rating with the load level identifier to dynamically adjust the preloading range, priority and execution timing, and generate a dynamic policy configuration; based on the dynamic policy configuration, analyze the system status duration and change trend to achieve a smooth transition of policy switching and avoid system jitter caused by frequent switching.

[0071] Specifically, construct and implement a multi-dimensional system status monitor to collect key system performance indicators in real time, including: CPU usage rate: sample once every 5 seconds and calculate the average usage rate; memory usage rate: sample once every 5 seconds and calculate the percentage of available memory; network latency: perform a ping test once every 10 seconds and calculate the average latency time; network packet loss rate: count the communication packet loss rate once a minute; I / O waiting time: sample once every 10 seconds and calculate the average I / O waiting time. Generate a system status vector containing the latest values and trend values of each indicator. Based on the system status vector, implement a load level classifier to classify the system load status into four levels: Light load (Level 1): All indicators are within the normal range and the system runs smoothly; Medium load (Level 2): Some indicators are close to the warning line, but the system runs stably; Heavy load (Level 3): Multiple indicators exceed the warning line and the system performance begins to decline; Critical load (Level 4): Key indicators exceed the critical value and the system performance seriously declines. The classification uses a fuzzy logic algorithm to map each indicator to a membership function in the [0, 1] interval, and then obtains the overall load level through weighted calculation. Output the load level identifier.

[0072] Construct a network status evaluator to evaluate the current network reliability based on network latency and packet loss rate. The network status is divided into three levels: Stable state: Low and stable latency, packet loss rate close to 0; Fluctuating state: Obvious latency fluctuations or occasional packet loss; Unstable state: High and unstable latency, or high packet loss rate. The evaluation uses a time series analysis method, which not only considers the current network metric values, but also considers the recent change trend to improve the accuracy and forward-looking of the evaluation, and outputs the network status rating. Implement an adaptive policy adjuster to dynamically adjust the preloading policy according to the load level identifier and network status rating. The adjustment content includes: Preloading range: Adjust the semantic distance threshold, the higher the load, the smaller the threshold, and reduce the number of preloads; Preloading priority: Adjust the priority policy between local cache and remote acquisition according to the network status; Preloading timing: Adjust the preloading execution timing according to the system load to avoid peak loads. The specific adjustment rules are as follows: Light load + Stable network: Preload in the maximum range, and expand the distance threshold by 50%; Medium load + Stable network: Preload in the standard range, and use the default distance threshold; Heavy load + Stable network: Reduce the preloading range, and shrink the distance threshold by 30%; Any load + Unstable network: Only preload core tags and give priority to using local cache.

[0073] Build a smooth transition mechanism to avoid system jitter caused by frequent policy switching. Introduce state duration and change trend analysis. Only when the system state stabilizes at a new level for a certain period (default 30 seconds) or shows a clear change trend, trigger policy switching. At the same time, policy parameters are adjusted progressively rather than jumpily. Implement a policy execution feedback loop to record the system performance changes before and after policy adjustment, including indicators such as response time, resource utilization, and preloading hit rate. Optimize and adjust the policy based on the feedback data to improve system adaptability. Output dynamic policy configurations to guide preloading behavior.

[0074] According to one aspect of the present application, the steps of performing encoding compression to form compressed semantic mapping rules include:

[0075] S41. Obtain a set of label mapping rules, cluster and group them according to semantic similarity to form label groups with similar semantics, and identify the common semantic features of each group as the basic semantic template of the group;

[0076] S42. Design and implement a multi-level semantic difference encoding algorithm with the following characteristics: decompose the complete semantics of a label into two parts: core semantics and extended semantics; store the core semantics in the basic semantic template of the group, which is shared by all group members; store the extended semantics in a differential encoding manner with respect to the basic semantic template, and only record the differential part;

[0077] S43. Construct a semantic compression encoding function C(t, b) = {M, ΔS}, where t is the target label, b is the basic semantic template, M is the template identifier, and ΔS is the semantic difference set between the label and the template. Specific encoding steps: calculate the complete semantic feature vector St of the label t; find the most matching basic semantic template b and obtain its semantic feature vector Sb; calculate the difference vector ΔS = St – Sb; sparsify the difference vector, only retaining the significant difference components; combine the template identifier M and the difference vector ΔS to form a compressed encoding;

[0078] S44. Implement a frequency-aware adaptive compression strategy to dynamically adjust the compression intensity according to the usage frequency of labels: high-frequency labels use a lower compression ratio to retain more semantic details and improve decoding efficiency; low-frequency labels use a higher compression ratio to reduce storage space occupancy; regularly detect changes in label usage frequency and dynamically adjust compression parameters;

[0079] S45. Store the compressed compressed semantic mapping rules in a storage system, and at the same time maintain a compressed rule index for quick positioning and decompression.

[0080] Specifically, the implementation of the usage frequency-aware adaptive compression strategy is as follows: Extract the access records of each tag from the system usage logs, count the access frequencies within the past N days (default 30 days), and generate a tag usage frequency table. The frequency calculation uses a time decay model, where recent accesses have a higher weight than earlier ones: F(t) = Σ(Count(t, i) * e -λ*(N-i) ), where * represents the multiplication operation, Count(t, i) represents the number of accesses to tag t on the i-th day, λ is the time decay factor (default 0.1), and N is the number of days for statistics. Based on the tag usage frequency table, design a frequency stratification mechanism to divide the tags into four usage frequency levels: Very High Frequency (VHF) level: Tags with usage frequencies in the top 5%; High Frequency (HF) level: Tags with usage frequencies in the range of 5% - 20%; Medium Frequency (MF) level: Tags with usage frequencies in the range of 20% - 60%; Low Frequency (LF) level: Tags with usage frequencies in the bottom 40%.

[0081] Design differentiated compression strategies for each frequency level and construct a hierarchical compression configuration table: Very High Frequency level: The lowest compression ratio (about 1.5:1), retain all semantic dimensions, and the difference significance threshold θVHF = 0.05; High Frequency level: A low compression ratio (about 3:1), retain most semantic dimensions, and the difference significance threshold θHF = 0.1; Medium Frequency level: A medium compression ratio (about 6:1), retain important semantic dimensions, and the difference significance threshold θMF = 0.15; Low Frequency level: A high compression ratio (about 10:1), only retain the core semantic dimensions, and the difference significance threshold θLF = 0.2. Implement a compression efficiency monitor to continuously evaluate the effectiveness of the compression strategies at each level, including: Average compression ratio: The ratio of the data size before and after compression; Decoding time: The time required to restore the complete semantic information from the compressed data; Query impact: The degree of impact of compression on query performance; Generate a compression performance report as the basis for policy adjustment.

[0082] An adaptive threshold adjustment algorithm is built to dynamically adjust compression thresholds based on system storage status and query performance. When system storage pressure increases, the compression ratio of all tiers is increased; when query latency increases, the compression ratio of high-frequency tiers is reduced. The threshold adjustment formula is: θ'_i = θ_i * (1 + α * (S_current / S_target - 1) - β * (T_current / T_target - 1)), where θ'_i is the adjusted threshold for tier i; θ_i is the baseline threshold for tier i; S_current is the current storage utilization; S_target is the target storage utilization; T_current is the current average query response time; T_target is the target query response time; α is the storage pressure weight (default 0.5); and β is the query performance weight (default 0.3). A frequency change monitor is implemented to regularly analyze tag usage frequency trends (by default, weekly) and identify tags whose frequency tiers have changed. For tags with changed tiers, recompression is triggered, using the new tier's compression strategy. The monitoring algorithm uses an exponential moving average to mitigate the impact of temporary fluctuations and focus on long-term trends. An incremental update mechanism is implemented to incrementally update the storage for tags that require recompression, avoiding full rebuilds. During the update process, the original compressed data is retained until the newly compressed data is verified, ensuring system stability. A compression policy configuration and label compression allocation table are generated to guide subsequent compression operations.

[0083] According to one aspect of the present application, the step of generating compression semantic mapping rules includes:

[0084] Cluster the labels in the label mapping rule set according to semantic similarity to form a label group set;

[0085] Based on the tag group set, the core semantic template of each group is calculated, the common features of the groups are identified, and the core semantic template set of the groups is generated; the semantic feature vector of the tag is compared with the core semantic template of the group to which it belongs, the difference information is extracted, and the original difference code is generated;

[0086] Analyze the information redundancy in the original differential coding, remove the redundant parts with entropy values below the threshold, and obtain the optimized differential coding; select the optimal storage format according to different types of data, convert the optimized differential coding into an efficient storage structure, and generate the final compression semantic mapping rules.

[0087] Specifically, the algorithm receives a set of label mapping rules, extracts the standardized semantic feature vector for each label, and constructs a label set T = {t1, t2, ..., tn} and its corresponding semantic vector set V = {v1, v2, ..., vn}. An adaptive semantic clustering algorithm is constructed and implemented to group semantically similar labels. This algorithm improves upon the traditional K-means clustering algorithm and has the following features: dynamic cluster number determination: instead of presetting the number of clusters K, the optimal number of clusters is automatically found using the Silhouette Coefficient; semantic distance weighting: different semantic dimensions are weighted differently in distance calculations, with functional dimensions being given higher weight than other dimensions; and density awareness: the density distribution of labels in the semantic space is considered to prevent outliers from affecting the clustering effect. The clustering process is as follows: the initial K value is set to 2, and the maximum K value is set to sqrt(n) (n is the total number of labels); weighted K-means clustering is performed for each K value, and the silhouette coefficient S(K) is calculated; the K value that maximizes S(K) is selected as the final number of clusters; and the final clustering is performed to obtain the label population set G = {g1, g2, ..., gK}.

[0088] For each label cluster gi, its core semantic template is calculated. The core semantics are not simply vector averages, but are generated through the following steps: The distribution statistics (mean μj, standard deviation σj, skewness Sj) of each semantic dimension within the cluster are calculated; the cluster's characteristic dimensions are identified: dimensions with a smaller coefficient of variation (CV = σj / μj) are considered common to the cluster; the core semantic value for characteristic dimensions is the median (to resist outlier interference); the core semantic value for non-characteristic dimensions is set to null (indicating that the dimension is not considered a core feature); and a set of cluster core semantic templates M = {m1, m2, ..., mK} is generated, with each template containing dimension values and dimension importance weights. A multi-level semantic difference encoder is implemented to compare the complete semantic vector of a label (the label semantic feature vector) with the core semantic template of the cluster to extract differential information. The coding adopts a three-level structure: the first-level coding: records the ethnic group ID to which the label belongs, and the ethnic group core semantic template can be queried through the ID; the second-level coding: records the slight deviation of the label in the core feature dimension, expressed as a relative value; the third-level coding: records the complete value of the label in the non-core dimension, only when the dimension has significant meaning to the label.

[0089] Construct a semantic information redundancy detection algorithm to analyze the encoded differential information and remove redundant parts. This algorithm is based on the information entropy theory, calculates the entropy value of each part of the differential information, and determines it as redundant information when the entropy value is lower than the threshold, which can be safely removed. This ensures maximizing the compression ratio while retaining the key semantic differences. Implement an encoding format optimizer to select the optimal storage format according to different types of data: Sparse matrix format: suitable for differential vectors with most dimensional values being zero or empty; Bitmap index format: suitable for discrete dimensions with limited value ranges; Incremental encoding format: suitable for continuous dimensions close to the template value; Generate compressed semantic encoding, including ethnic group ID and optimized differential encoding, for efficient storage of label semantic information.

[0090] According to one aspect of the present application, the steps of extracting differential information and generating the original differential encoding include:

[0091] Receive the label semantic feature vector and the corresponding ethnic group core semantic template, and prepare for differential encoding;

[0092] Adopt corresponding calculation methods for different types of dimensions to generate the original differential vector;

[0093] Calculate the significance evaluation value of each dimension difference, set the significance threshold, and screen the significant difference dimensions;

[0094] Dynamically adjust the encoding precision for continuous numerical values according to the dimension characteristics and importance;

[0095] Convert the screened differential vector into a sparse matrix format and optimize the index using variable-length encoding;

[0096] Combine the template identifier, version identifier, and differential set to generate a complete compressed semantic encoding in binary format.

[0097] Specifically, receive the label semantic feature vector St and its corresponding ethnic group core semantic template Sb, and prepare for differential coding processing. Construct and implement a semantic vector difference analyzer to calculate the difference between the label vector and the template vector. Different from simple vector subtraction, this analyzer considers the characteristics of semantic dimensions: for continuous dimensions (such as precision values), calculate the relative difference: ΔS_cont[i] = (St[i] - Sb[i]) / Sb[i]; for discrete dimensions (such as function categories), calculate the similarity deviation: ΔS_disc[i] = 1 - sim(St[i], Sb[i]); for boolean dimensions, directly calculate the exclusive OR result: ΔS_bool[i] = St[i] Θ Sb[i]; where Θ represents the exclusive OR operation; generate the original difference vector ΔS_raw, retaining the difference values of all dimensions. Implement a significant difference filter to analyze the significance of each dimension difference in the original difference vector. The significance evaluation is based on two factors: the magnitude of the difference: the degree of difference from the template value; the semantic importance: the contribution of this dimension to the label semantics. The significance calculation formula: Sig(i) = |ΔS_raw[i]| * W[i]; where W[i] is the semantic importance weight of dimension i, which is predefined. Set a significance threshold θ (default 0.1), and when Sig(i) < θ, it is determined that the dimension difference is not significant and can be ignored. Generate the filtered difference vector ΔS_filtered, which only contains the values of significantly different dimensions.

[0098] Build an adaptive coding precision controller to dynamically adjust the coding precision according to the characteristics and importance of different dimensions. For continuous numerical values: Core semantic dimension: retain 4 significant digits; Secondary semantic dimension: retain 2 significant digits; Peripheral semantic dimension: retain 1 significant digit. Implement a sparse matrix compression encoder to convert the filtered difference vectors into an efficient storage format. For sparse vectors (most dimensions are 0 or filtered), store the non-zero elements in the form of "index-value" pairs: ΔS_sparse = {(i1, v1), (i2, v2), ..., (ik, vk)}; where i represents the dimension index and v represents the difference value. Different from traditional sparse matrix storage, by performing variable-length coding on the indexes based on semantic importance, shorter index codes are used for important dimensions to further improve the compression efficiency. Build the final compressed semantic code C(t, b), which consists of three parts: Template identifier M: pointing to the basic semantic template to which the label belongs; Version identifier V: recording the coding format version for subsequent decoding; Difference set ΔS: storing the processed sparse difference vectors. The coding uses a binary format with the structure: [2-byte template ID][1-byte version number][1-byte number of difference items n][n (dimension index, difference value) pairs], where the dimension index uses variable-length coding and the difference value uses different numbers of bytes according to the data type and precision requirements. Output the final compressed semantic code for efficient storage of the label semantic mapping rules.

[0099] According to one aspect of the present application, the steps of performing a query and returning a query result include:

[0100] S51. Receive a data query request initiated by the user, parse the query conditions, identify the data tags, time range, and filtering conditions involved, and form a structured query expression;

[0101] S52. Build and maintain a label affinity matrix to record the query association strength between labels. The element A[i, j] in the affinity matrix A represents a comprehensive measure of the frequency and correlation of co-occurrence of labels i and j in the query. Affinity calculation formula: A[i, j] = α·F(i, j) + β·S(i, j) + γ·T(i, j); where: F(i, j) is the frequency of co-occurrence of labels i and j in the same query; S(i, j) is the semantic similarity of labels i and j; T(i, j) is the time correlation of label j being queried after label i is queried; α, β, γ are weight coefficients satisfying α + β + γ = 1;

[0102] S53. Construct and implement a label affinity path optimization algorithm. Based on the label affinity matrix, construct a query execution path, convert the multi-label query into a directed graph, with the edge weights being the label affinity values, and improve the query efficiency by finding the optimal path. Path optimization steps: Construct a query graph G for the query label set; Calculate the total affinity scores of all possible query paths; Select the path with the highest affinity score as the query execution order;

[0103] S54. Implement a query path heat tracking mechanism to record the usage frequency and performance of each query path, providing real-time reference for path selection. Heat calculation formula: H(p) = λ·F(p) + (1 - λ)·P(p); where H(p) is the heat value of path p; F(p) is the usage frequency of path p; P(p) is the performance score of path p (a comprehensive evaluation of query efficiency and resource consumption); λ is a balance parameter that can be dynamically adjusted according to the system state;

[0104] S55. Dynamically select query strategies according to the current system load status. Prioritize paths with low resource consumption in high-load situations and select paths with high query efficiency in low-load situations to balance query performance and system stability;

[0105] S56. Execute the optimized query operation to obtain the query result, and at the same time update the affinity matrix of relevant labels and path heat information as the basis for subsequent query optimization.

[0106] According to one aspect of the present application, the configuration steps of the label affinity matrix include:

[0107] Extract query records from the query log library to construct a query history sequence including query time, query labels, and user identifiers; Based on the query history sequence, calculate the association strengths in three dimensions: co-occurrence frequency, semantic similarity, and temporal correlation between labels;

[0108] Based on the association strengths, optimize the weight coefficients in the three dimensions to minimize the query path prediction error; Apply the optimized weight coefficients to calculate the affinity values of label pairs and construct a label affinity matrix; Optimize the storage efficiency of the label affinity matrix using a sparse matrix storage structure to obtain an optimized label affinity matrix.

[0109] Specifically, query records of the most recent M days (default 90 days) are extracted from the query log library, including information such as query time, query tags, user identification, etc., to construct a query history sequence. The format of each record is {timestamp, user ID, [query tag set], query context}. A multi-dimensional association analyzer is constructed and implemented to calculate the association strength of tag pairs (i, j) from three different dimensions: Co-occurrence frequency dimension F(i, j): Calculate the normalized frequency of tags i and j co-occurring in the same query. Calculation steps: Count the number of times tags i and j co-occur in the query, denoted as Count(i, j); count the total number of times tag i appears, denoted as Count(i), and the total number of times tag j appears, denoted as Count(j); calculate the co-occurrence frequency F(i, j) = Count(i, j) / sqrt(Count(i) * Count(j)). Semantic similarity dimension S(i, j): Calculate the similarity degree of tags i and j in the semantic space. Calculation steps: Extract the semantic feature vectors Vi and Vj of tags i and j; apply the cosine similarity algorithm to calculate the vector similarity S(i, j) = cos(Vi, Vj). Temporal correlation dimension T(i, j): Analyze the temporal association pattern of tags i and j in the user's query behavior. Calculation steps: For each user, extract the number of times tag j is queried within a certain time window (default 30 minutes) after tag i in the query sequence; calculate the conditional probability P(j|i), which represents the probability of querying tag j after querying tag i; apply a time decay factor, where the closer the query, the higher the weight; perform normalization to obtain the temporal correlation T(i, j).

[0110] Implement a weight adaptive optimizer to dynamically adjust the weight coefficients α, β, γ of the three dimensions according to the system operation situation. The optimization goal is to minimize the query path prediction error, that is, to improve the prediction accuracy of the user's next query tag. The gradient descent method is used for weight optimization: α' = α - η * ΨE / Ψα; β' = β - η * ΨE / Ψβ; γ' = γ - η * ΨE / Ψγ; where E is the prediction error function, η is the learning rate (default 0.01), and Ψ is the partial derivative. At the same time, it is constrained that α + β + γ = 1, which is achieved through normalization. Construct a sparse matrix optimized storage structure to efficiently store a large-scale tag affinity matrix. Considering that the number of tags in the actual system can reach tens of thousands, and the affinity between most tag pairs is weak (close to 0), using a sparse matrix storage structure significantly reduces the storage space requirements. Store in CSR (Compressed Sparse Row) format: Only store non-zero elements with affinity values greater than the threshold (default 0.01); use three arrays of row pointers, column indices, and numerical values to represent the matrix; further optimize according to the symmetry characteristics of the affinity matrix, and only store the upper triangular part.

[0111] Implement an incremental update mechanism to avoid recalculating the entire affinity matrix each time. Design a sliding update algorithm based on a time window: process newly added query logs daily to calculate the incremental impact on affinity values; apply a time decay factor to gradually reduce the weight of historical data; perform a full reconstruction regularly (default once a month) to ensure the accuracy of the matrix. Build an affinity change monitoring mechanism to detect significant changes in label relationships and respond promptly to changes in user behavior patterns. When the affinity value change of a certain label pair exceeds a threshold (default 30%), trigger the recalculation of relevant paths to ensure that the query path remains optimal. Output the label affinity matrix and the affinity change report.

[0112] According to one aspect of the present application, the steps of identifying a query path include:

[0113] Parse the data query request, extract the involved data labels, and construct a query label set;

[0114] Combine the query label set with the label affinity matrix to construct a query label affinity subgraph;

[0115] Based on the query label affinity subgraph, calculate the total affinity score of the path and find the initial query path;

[0116] Based on the initial query path, process the logical dependency constraints in the query, convert the logical dependency constraints into mandatory path segments, adjust the initial query path, and obtain an adjusted query path; search for historical paths that match the adjusted query path in the pre-stored multi-level path cache to avoid repeated calculations and obtain candidate paths;

[0117] Evaluate the actual execution cost of the candidate paths, and comprehensively consider the total affinity score and the actual execution cost to generate the final optimized query path.

[0118] Specifically, upon receiving a data query request, parse the query conditions, extract the set of relevant data tags Q = {q1, q2,..., qn}, and construct a query tag set. Based on the tag affinity matrix A and the query tag set Q, construct a query tag affinity subgraph G=(V, E); where the vertex set V = Q, the edge set E contains all possible connections of tag pairs, and the weight of the edge e(qi, qj) is the affinity value A[i, j], generating a query affinity subgraph. Construct and implement a multi-objective path planning algorithm to find the optimal query execution path. Different from the traditional single-source shortest path algorithm, this embodiment needs to access all tags in the query tag set, which belongs to a variant of the Traveling Salesman Problem (TSP). Considering that the number of query tags is usually small (<10), an improved exact algorithm is adopted: when the number of tags ≤ 4, use the exhaustive method to find the global optimal path; when the number of tags > 4, use the improved genetic algorithm to solve the approximate optimal path. The path evaluation criterion is the total path affinity score: Score(P) = Σ A[i, j] for all (i, j) in path P.

[0119] Implement a path constraint processor to handle the logical dependency constraints in the query. In some queries, there are logical dependency relationships between tags and they must be accessed in a specific order. For example, "device basic information" must be queried before "device status record". The constraint processing steps are as follows: extract the tag access constraint conditions from the query semantics; convert the constraints into mandatory path segments in path planning; find the optimal path on the premise of meeting the constraint conditions. Construct a multi-level path caching mechanism to avoid repeated calculation of the optimal paths for similar queries. The cache structure includes: first-level cache: a complete match cache, which stores the recently calculated query paths (the tag sets are exactly the same); second-level cache: a partial match cache, which stores the query paths with inclusion relationships (the current query is a subset of the historical query); third-level cache: a pattern match cache, which stores the query paths with similar structures. When looking up the cache, first try a complete match, then try a partial match and make appropriate adjustments, and finally try a pattern match and reconstruct the path. Implement a path execution cost estimator to evaluate the actual execution cost of the candidate paths. A high path affinity score does not necessarily mean a low execution cost. Other factors need to be considered: data access cost: the difference in access speeds of different storage media; data volume factor: the impact of the size of the intermediate result set on subsequent processing; potential for parallel execution: the benefits of parallel execution for some tag queries. Considering both the affinity score and the execution cost, generate the final optimized query path as the guiding plan for query execution.

[0120] According to one aspect of the present application, the steps of dynamically adjusting the query strategy include:

[0121] Generate a unique identifier for each executed query path to form a path identifier set;

[0122] Record query path execution time, CPU usage, memory consumption, and I / O operation metrics, and calculate performance scores;

[0123] Count the query path usage frequency and apply a time decay model to give more weight to recent queries than historical queries.

[0124] Dynamically adjust frequency weight and performance weight according to system operation status to balance common use and resource consumption;

[0125] Calculate the path heat value and classify the query paths into four levels: popular, common, general, and unpopular; analyze the trend of path heat changes, predict changes in popular paths, optimize resource allocation in advance, and generate path heat reports.

[0126] Specifically, a query path identifier is designed and implemented to generate a unique identifier for each executed query path. The identifier format is a hash value of an ordered tag ID sequence, ensuring that identical query paths have the same identifier, facilitating statistics and analysis. A path identifier set is generated. A path performance monitor is established to record the execution performance metrics of each query path, including: total execution time (the total duration from query start to result return); CPU utilization (the average CPU utilization during execution); memory consumption (the maximum memory usage during execution); number of I / O operations (the number of disk reads and writes during execution); and intermediate result set size (the amount of temporary data generated during the query). These metrics are collected and normalized to convert them into a performance score P(p) in the interval [0, 1].

[0127] Implement a path usage frequency counter to record the usage frequency F(p) of each query path. Frequency statistics use a time decay model, with recent queries being weighted higher than historical queries: F(p) = Σ(Count(p, i) * e -Δ*(N-i) ), where Count(p, i) represents the number of times path p is used on day i, Δ is the time decay factor (default 0.05), and N is the number of days counted (default 30 days). An adaptive weight adjuster is constructed to dynamically adjust the frequency weight λ and performance weight (1-λ) based on the system's operating status. When system resources are sufficient, the frequency weight is increased to prioritize frequently used paths; when system resources are limited, the performance weight is increased to prioritize paths with low resource consumption. The weight adjustment formula is: λ = λ_base + η * (R_target - R_current) / R_target; where λ_base is the baseline weight (default 0.6), η is the adjustment coefficient (default 0.2), R_current is the current system resource utilization, and R_target is the target resource utilization.

[0128] Build and implement a path heat level classification mechanism. According to the heat value, query paths are divided into multiple levels: Hot paths (H≥0.8): with the highest priority and the most resources configured; Frequently used paths (0.5≤H<0.8): with relatively high priority and moderate resources configured; General paths (0.2≤H<0.5): with standard priority and basic resources configured; Cold paths (H<0.2): with the lowest priority and the least resources configured. For hot paths, the system also performs preheating processing, preloading relevant data into the cache to further improve query performance. Implement a heat change trend analyzer to detect the change trend of path heat, and promptly discover emerging hot paths and declining paths. Trend analysis uses the linear regression method to fit the heat values of the most recent N days and extracts the slope as the trend indicator. For paths with rapidly rising heat, optimize and allocate resources in advance to ensure a smooth transition of the system. Generate path heat reports and heat trend predictions to provide a decision-making basis for system resource scheduling.

[0129] According to one aspect of the present application, the steps of performing historical data cleaning include:

[0130] S61. Regularly detect the available storage space of the system, compare it with a preset threshold, and determine whether historical data cleaning is required;

[0131] S62. When the storage space is insufficient, calculate the data retention priority based on the data time characteristics and usage value. The priority calculation formula is: P(d) = w1·(1 / A(d)) + w2·U(d) + w3·I(d); where P(d) is the retention priority of data item d; A(d) is the age of data item d (the difference between the current time and the data freezing time); U(d) is the usage frequency of data item d; I(d) is the business importance score of data item d; w1, w2, and w3 are weight coefficients, satisfying w1 + w2 + w3 = 1;

[0132] S63. Delete historical data batch by batch in ascending order of data retention priority until the available space of the system reaches a safe level. Before data deletion, downsample and compress and archive data that may still be valuable to reduce storage space occupancy while retaining data value;

[0133] S64. Record the operation log of data cleaning, including information such as cleaning time, amount of cleaned data, and remaining space, to provide a reference for system operation and maintenance and optimization.

[0134] According to another aspect of the present application, a multi-scenario application data collection method based on atomized design includes: storage structure adjustment, a storage table with device identifier, freeze time, and data tag as primary keys and data value as a field; upstream application adjustment, reporting new regulations or data items, and the data storage application is responsible for data storage; the platform configures new data items to facilitate users to obtain and analyze data.

[0135] Specifically, design the storage table structure with device identifier, data freeze time, data tag, and data value. The data tag and data value are stored as strings, and only the data source and data user need to agree on the tag name and tag data type. Since converting from specific data tags to a vertical table increases the number of stored records and required space, the data table is divided into sub-tables. The sub-tables are divided according to the data freeze time and into 12 tables according to months. When receiving a message, identify the reporting topic and whether the message is compressed. If it is compressed, decompress the message. Identify the reported protocol. The old business logic is still decomposed according to the original logic (such as electrical data reporting, device reporting, analog quantity reporting, etc.). For atomic tag classes, according to the agreed structure, parse out the device identifier, freeze time, data tag, and data value, and confirm the name of the storage table according to the month to which the freeze time belongs, and store it in the database. For the client query logic, the client initiates a data query request, and the request parameters are the device identifier, data item, and query time interval. The backend confirms the name of the table to be queried according to the query time interval, grabs the data and aggregates it and returns it to the calling client. After the system runs for a long time, it may cause insufficient data storage space, and historical records need to be deleted to ensure that new reported data can be stored. The application regularly queries the available disk capacity of the current application. When the capacity is too small, perform the action of cleaning historical data. Query the oldest records by day and delete them. After completion, query the available space. If it is still insufficient, continue to delete historical records. If the space is sufficient, end the cleaning action.

[0136] This embodiment simplifies the storage structure, avoids modifying the data structure, and saves development time; for data items not within the agreement, if there is reported data, it can also be stored in time, which is convenient for later traceability and analysis of historical data; both data query and calculation are unified and simplified, and unified design can be carried out for data items; to adapt to the complexity of the energy network, the increase in protocols and monitoring indicators, and the resulting diversification of tags, realize the self-adaptation of data storage and transmission.

[0137] In a specific embodiment of the present application, a multi-scenario application data collection method based on atomized design is applicable to an intelligent power data management system. This system receives multi-type data reports from substations, distribution rooms, and electrical equipment and needs to be uniformly managed and efficiently queried. The specific steps include:

[0138] Step 1: Form an atomized data structure.

[0139] An intelligent power distribution device reports temperature sensor data through the MQTT protocol. The received MQTT message topic is "device / ABC123 / temperature", and the message payload is compressed JSON data: "eJyrVspOLVGyUjDUMwQCJR2FksSiVB2l6sTcgszEzDwdpcT0zBKlWgDlJQud".

[0140] 1.1 Receive the MQTT message reported by the device. Analyze the message topic to know that the data source is the temperature data of device ABC123, and obtain the message payload content "eJyrVspOLVGyUjDUMwQCJR2FksSiVB2l6sTcgszEzDwdpcT0zBKlWgDlJQud".

[0141] 1.2 By checking the header characteristics of the message payload, it is judged that the data is compressed. Call the Base64 decoding and GZIP decompression algorithms to obtain the decompressed JSON data: {"deviceId": "ABC123", "timestamp": "2025-04-27T10:30:15Z", "dataTag": "indoor_temperature_sensor", "value": 23.5, "unit": "celsius"}.

[0142] 1.3 Identify the temperature sensor data protocol in JSON format through the protocol identification field in the message.

[0143] 1.4 Analyze the JSON data and extract four key elements: device identifier "ABC123", freeze time "2025-04-27T10:30:15Z", data tag "indoor_temperature_sensor", and data value "23.5°C", and organize them into an atomized data structure.

[0144] Step Two: Semantic Recognition and Label Mapping.

[0145] 2.1 Extract the data tag "indoor_temperature_sensor" from the atomized data structure and perform preprocessing: uniformly convert it to lowercase and perform word segmentation to obtain the standardized tag text "indoor temperature sensor".

[0146] 2.2 Input the standardized tag text into the semantic recognition engine to generate a label semantic feature vector. The specific implementation is as follows:

[0147] 2.2.1. Use the N-gram tokenization technique to decompose "indoor temperature sensor" into a set of morphemes: {"indoor", "temperature", "sensor", "indoor temperature", "temperaturesensor"}.

[0148] 2.2.2. Map to the domain ontology library to obtain the domain semantic weights of each morpheme: "indoor": 0.8 (spatial location category); "temperature": 0.9 (physical quantity category); "sensor": 0.7 (device type category); "indoor temperature": 0.85 (compound concept); "temperature sensor": 0.75 (compound concept).

[0149] 2.2.3. Construct a nine-dimensional semantic vector structure: Function dimension (F1 - F3): [0.1, 0.8, 0.0] (indicating that the main function is "measurement"); Object dimension (O1 - O2): [0.9, 0.0] (indicating that the measurement object is "temperature"); Location dimension (L1 - L2): [0.8, 0.0] (indicating the location is "indoor"); Unit dimension (U1): [0.95] (indicating the unit is "Celsius"); Precision dimension (P1): [0.7] (indicating "standard precision").

[0150] 2.2.4. Apply the context-aware word sense disambiguation algorithm to the morpheme "temperature"; in the current context, "temperature" is adjacent to "sensor", and its semantics is determined to be "physical temperature" rather than "heating state" with a probability of 0.98.

[0151] 2.2.5. Semantic vector dimension value calculation formula: V[i] = Σ(W_j * R_ji) / Σ(W_j); where: V[i] is the i-th dimension value of the semantic vector; W_j is the weight of morpheme j; R_ji is the contribution degree of morpheme j to dimension i; Σ represents the summation of all relevant morphemes. Specifically, calculate the nine-dimensional semantic vector; taking the function dimension F1 as an example, the calculation formula is: V[F1] = (0.1×0.7 + 0.2×0.9 + 0.0×0.8 + 0.15×0.75 + 0.1×0.85) / (0.7 + 0.9 + 0.8 + 0.75 + 0.85) = 0.323 / 4 = 0.081; after calculation, the complete nine-dimensional vector [0.081, 0.783, 0.021, 0.886, 0.035, 0.762, 0.024, 0.933, 0.687] is obtained.

[0152] 2.2.6. Normalize the calculated semantic vector and map it to the interval [0, 1] to obtain the standardized semantic feature vector [0.1, 0.8, 0.0, 0.9, 0.0, 0.8, 0.0, 0.9, 0.7].

[0153] 2.3. Calculate the similarity between the label semantic feature vector and the existing labels in the label dictionary library to determine the most matching label and its version information.

[0154] 2.3.1. Cosine similarity calculation formula: sim(A, B) = (A·B) / (||A||·||B||); where A is the target vector; B is the label vector in the library; A·B is the vector inner product; ||A|| and ||B|| are the Euclidean norms of vectors A and B respectively. Calculate the cosine similarity between the target vector and each label vector in the dictionary library; the similarity with the label "indoor_temp_sensor@1.2.3" is 0.92, and the similarity with "room_temperature_sensor@2.0.1" is 0.88, generating a similarity sorted list.

[0155] 2.3.2. Judge the version update requirement based on the similarity sorted list; the highest similarity 0.92 is between the medium threshold 0.85 and the high threshold 0.95, and it is determined to be a semantic adjustment of the same label, and the minor version number needs to be updated.

[0156] 2.3.3. Calculate the dimension difference; the difference in the object dimension O1 is 0.05, the difference in the location dimension L1 is 0.02, the difference in the unit dimension U1 is 0.01, and the differences in the remaining dimensions are less than 0.01, generating a dimension difference report.

[0157] 2.3.4. Execute the version evolution rule; the difference in the object dimension O1 exceeds the significance threshold of 0.03. According to the rule, the minor version number should be upgraded, from 1.2.3 to 1.3.0.

[0158] 2.3.5. Create a version change record, record the reason for the change as "significant change in object dimension O1", the value before the change is 0.85, and the value after the change is 0.9.

[0159] 2.3.6. Generate the tag version identifier "indoor_temp_sensor@1.3.0", associate it with the tag semantic feature vector, and form the tag semantic mapping result.

[0160] 2.4. Combine the context information of the device type "intelligent power distribution equipment", the spatial location "power distribution room", and the data characteristic "real-time monitoring" to generate the complete tag semantic mapping result.

[0161] Step 3. Construct a hierarchical cache structure.

[0162] 3.1. Receive the tag semantic mapping result "indoor_temp_sensor@1.3.0", record it in the tag mapping history library, update the usage frequency to 1, and the context information to "real-time monitoring in the power distribution room".

[0163] 3.2. Construct a semantic association predictive lazy loading strategy.

[0164] 3.2.1. Extract the last 100 query records from the query history log library to construct a query sequence set, which includes 20 queries related to "indoor_temp_sensor". Common subsequent queries include "humidity_sensor" (8 times), "power_consumption" (6 times), etc.

[0165] 3.2.2. Analyze the tag temporal association, calculate the temporal association strength, and the calculation formula is: TS(ti,tj) = Count(ti→tj) / Count(ti); where Count(ti→tj) represents the number of times tag tj is queried within a certain time after tag ti is queried; Count(ti) represents the total number of queries for tag ti. Specifically, the association strength of "indoor_temp_sensor→humidity_sensor" is TS = 8 / 20 = 0.4, and the association strength of "indoor_temp_sensor→power_consumption" is TS = 6 / 20 = 0.3, generating a tag temporal association matrix.

[0166] 3.2.3. Calculate the semantic similarity between the tag and the current context; S(t, c) = w1·S_dev(t, c_dev) + w2·S_user(t, c_user) + w3·S_time(t, c_time) + w4·S_op(t, c_op); where S_dev, S_user, S_time, and S_op represent the similarity between the tag t and the device context, user context, time context, and operation context respectively; w1 to w4 are weight coefficients and satisfy w1 + w2 + w3 + w4 = 1. Taking "humidity_sensor" as an example, the similarity with the device context is 0.8, the similarity with the user context is 0.7, the similarity with the time context is 0.9, and the similarity with the operation context is 0.7. The total similarity S is calculated by weighted summation as S = 0.3×0.8 + 0.2×0.7 + 0.2×0.9 + 0.3×0.7 = 0.79.

[0167] 3.2.4. Calculate the semantic distance attenuation value; D(t, c) = e -α·R(t)·S(t,c) ; where R(t) is the normalized access frequency of the tag t; S(t, c) is the semantic similarity between the tag t and the current context c; α is a system adjustable parameter. Taking "humidity_sensor" as an example, its access frequency R = 0.4, the similarity with the current context S = 0.79, and α = 2.0. Calculate D(t, c) = e -2.0×0.4×0.79 = e -0.632 = 0.532.

[0168] 3.2.5. Implement a three - level pre - loading strategy according to the semantic distance; the distance value 0.532 of "humidity_sensor" is between 0.3 and 0.6, which is classified as associated pre - loading and asynchronously loaded into the secondary cache.

[0169] 3.2.6. Record the feedback of the pre - loading effect; the hit rate of "humidity_sensor" being actually queried within 10 minutes is 75%, the average loading time is 20ms, and the cache residence time is 15 minutes.

[0170] 3.3. Implement a dynamic switching mechanism for perceiving the network state.

[0171] 3.3.1. Collect system state metrics; the current CPU usage rate is 45%, the memory usage rate is 60%, the network latency is 30ms, the network packet loss rate is 0.1%, and the I / O waiting time is 5ms, generating a system state vector [45, 60, 30, 0.1, 5].

[0172] 3.3.2. Classification system load status: According to the system status vector, the CPU and memory utilization rates are at a medium level, and the network and I / O metrics are good. It is classified as medium load (Level 2), and the load level identifier "L2" is output.

[0173] 3.3.3. Evaluate network status: The network latency of 30 ms is low and stable, and the packet loss rate of 0.1% is close to 0. It is evaluated as a stable state, and the network status rating "S1" is output.

[0174] 3.3.4. According to the load level "L2" and network status "S1", adopt the standard range preloading strategy and keep the default distance threshold unchanged.

[0175] 3.3.5. Monitor the system status duration: The medium load status has lasted for 45 seconds, exceeding the threshold of 30 seconds, and confirm to execute the standard preloading strategy.

[0176] 3.3.6. Record the policy execution feedback: After the standard preloading strategy is executed, the query response time drops from 150 ms to 120 ms, the resource utilization rate increases by 2%, and the preloading hit rate is 68%.

[0177] 3.4. Build a three-layer cache structure: Store "indoor_temp_sensor@1.3.0" in the first-level semantic cache, store "humidity_sensor@2.1.0" and "power_consumption@1.5.2" in the second-level semantic cache, store the complete mapping history in the persistent storage, and generate a tag mapping rule set.

[0178] Step Four: Multi-level semantic difference encoding.

[0179] 4.1. Obtain the tag mapping rule set and perform clustering grouping according to semantic similarity: "indoor_temp_sensor", "room_temperature_sensor", and "ambient_temperature" are grouped into one cluster, and the common feature "temperature measurement" is identified to generate a basic semantic template.

[0180] 4.2. Design a multi-level semantic difference encoding algorithm.

[0181] 4.2.1. Extract the standardized semantic feature vector [0.1, 0.8, 0.0, 0.9, 0.0, 0.8, 0.0, 0.9, 0.7] of "indoor_temp_sensor@1.3.0" and construct a tag set.

[0182] 4.2.2. Apply the adaptive semantic clustering algorithm; by calculating the silhouette coefficient, determine the optimal number of clusters K = 3, perform weighted K-means clustering to obtain the set of label ethnic groups, and "indoor_temp_sensor" belongs to the "ambient temperature monitoring" ethnic group.

[0183] 4.2.3. Calculate the core semantic template of the "ambient temperature monitoring" ethnic group; the coefficient of variation CV of the function dimension F2 is 0.05, and the CV of the object dimension O1 is 0.03, both of which are less than the threshold 0.1, so they are determined as the common characteristics of the ethnic group. Take the median values as 0.8 and 0.9 respectively, and set the remaining dimensions as empty, generating the core semantic template [null, 0.8, null, 0.9, null, null, null, null, null].

[0184] 4.2.4. Implement multi-level semantic difference encoding; the first-level encoding records the ethnic group ID as 2, the second-level encoding records the deviation of the F2 dimension as 0.0, the deviation of the O1 dimension as 0.0, the third-level encoding records the value of the L1 dimension as 0.8, the value of the U1 dimension as 0.9, and the value of the P1 dimension as 0.7.

[0185] 4.2.5. Detect semantic information redundancy; the information entropy of the F2 and O1 dimensions is 0, which is lower than the threshold 0.1, so it is determined as redundant information and removed from the difference encoding.

[0186] 4.2.6. Optimize the encoding format; the L1 dimension adopts the bitmap index format, and the U1 and P1 dimensions adopt the incremental encoding format, generating the final compressed semantic encoding {2, [3:0.8, 7:0.9, 8:0.7]}.

[0187] 4.3. Construct a semantic compression encoding function.

[0188] 4.3.1. Receive the label semantic feature vector [0.1, 0.8, 0.0, 0.9, 0.0, 0.8, 0.0, 0.9, 0.7] and the corresponding ethnic group core semantic template [null, 0.8, null, 0.9, null, null, null, null, null], and prepare for difference encoding processing.

[0189] 4.3.2. Calculate the vector difference; for the F2 dimension, ΔS_cont[1]=(0.8 - 0.8) / 0.8 = 0; for the O1 dimension, ΔS_cont[3]=(0.9 - 0.9) / 0.9 = 0; for the L1 dimension, there is no template value, directly record 0.8; for the U1 dimension, there is no template value, directly record 0.9; for the P1 dimension, there is no template value, directly record 0.7; generate the original difference vector ΔS_raw = [0.1, 0, 0, 0, 0, 0.8, 0, 0.9, 0.7].

[0190] 4.3.3. Evaluate the significance of differences; significance calculation formula: Sig(i) = |ΔS_raw[i]| * W[i]; where |ΔS_raw[i]| is the absolute value of the raw difference in the i-th dimension; W[i] is the semantic importance weight of dimension i. For the F2 dimension, the significance Sig(1)=|0|×0.8 = 0, which is lower than the threshold of 0.1; for the O1 dimension, the significance Sig(3)=|0|×0.9 = 0, which is lower than the threshold of 0.1; for the L1 dimension, the significance Sig(5)=|0.8|×0.6 = 0.48, which is higher than the threshold of 0.1; for the U1 dimension, the significance Sig(7)=|0.9|×0.5 = 0.45, which is higher than the threshold of 0.1; for the P1 dimension, the significance Sig(8)=|0.7|×0.4 = 0.28, which is higher than the threshold of 0.1; generate the filtered difference vector ΔS_filtered=[null, null, null, null, null, 0.8, null, 0.9, 0.7].

[0191] 4.3.4. Adjust the coding precision; for the L1 dimension (core semantics), retain 4 significant digits: 0.8000; for the U1 dimension (secondary semantics), retain 2 significant digits: 0.90; for the P1 dimension (peripheral semantics), retain 1 significant digit: 0.7; obtain the difference vector with optimized precision [null, null, null, null, null, 0.8000, null, 0.90, 0.7].

[0192] 4.3.5. Convert the difference vector into a sparse matrix format; generate the form of "index - value" pairs: {(5, 0.8000), (7, 0.90), (8, 0.7)}, where the dimension index uses variable - length coding, and the index code of the important dimension L1 is shorter.

[0193] 4.3.6. Combine the final compressed coding; the template identifier is the ethnic group ID: 2; the version identifier is the coding format version: 1; the difference set is represented as a sparse vector: {(5, 0.8000), (7, 0.90), (8, 0.7)}; generate the binary format: [00000010][0000 0001][0000 0011][0000 0101][1000 0000][0000 0000][0000 0000][00000111][1001 0000][0000 1000][0111 0000], with a total length of 11 bytes.

[0194] 4.4. Implement an adaptive compression strategy with frequency awareness.

[0195] 4.4.1. Statistical label usage frequency; frequency calculation formula: F(t) = Σ(Count(t,i) * e -λ*(N-i) ); where Count(t,i) represents the number of accesses of label t on the i-th day; λ is the time decay factor; N is the number of days for statistics. Specifically, the access frequency of "indoor_temp_sensor" in the past 30 days is 65 times per day. Applying the time decay model, F(t)=∑(65×e -0.1×(30-i) ) = 512.3.

[0196] 4.4.2. Frequency stratification of labels; F(t) = 512.3 is in the top 15% of the system and is classified as the high-frequency layer (HF).

[0197] 4.4.3. Set compression strategy for the high-frequency layer; adopt a low compression ratio of 3:1, retain most semantic dimensions, and set the difference significance threshold θHF = 0.1.

[0198] 4.4.4. Monitor compression effect; the average compression ratio of the high-frequency layer is 2.8:1, the average decoding time is 0.5 ms, and the impact on query performance is an increase in response time by 1.2%.

[0199] 4.4.5. Dynamically adjust the compression threshold; threshold adjustment formula: θ'_i = θ_i * (1 + α * (S_current / S_target - 1) - β * (T_current / T_target - 1)); where θ_i is the baseline threshold of layer i; α is the storage pressure weight; β is the query performance weight. The current storage utilization rate S_current = 65%, the target is S_target = 70%; the current query response time T_current = 120 ms, the target is T_target = 100 ms; according to the formula θ'_HF = 0.1×(1 + 0.5×(65 / 70 - 1) - 0.3×(120 / 100 - 1)) = 0.1×(1 - 0.036 - 0.06) = 0.1×0.904 = 0.0904, the adjusted threshold of the high-frequency layer is 0.09.

[0200] 4.4.6. Monitor frequency changes; the usage frequency of "indoor_temp_sensor" shows an upward trend, with an average of 50 times per day at the beginning of the month and 80 times per day at the end of the month, but it is still within the high-frequency layer range and there is no need to adjust the compression strategy.

[0201] 4.4.7. Implement incremental update; recompress according to the adjusted threshold of 0.09, and replace the original storage with the new compressed data after verification to complete the policy update.

[0202] 4.5. Store the compressed semantic mapping rules in the storage system and maintain a compression rule index for quick location and decompression.

[0203] Step Five: Optimize the query of the label affinity path.

[0204] 5.1. Receive the data query request initiated by the user: "Query the change trend of indoor temperature and humidity in the distribution room ABC123 in the past 24 hours", parse out the query labels "indoor_temp_sensor" and "humidity_sensor", and the time range is "the past 24 hours" to form a structured query expression.

[0205] 5.2. Construct a label affinity matrix.

[0206] 5.2.1. Extract the query records in the recent 90 days from the query log library to construct a query history sequence containing 3,000 records.

[0207] 5.2.2. Calculate the label association strength; the number of times "indoor_temp_sensor" and "humidity_sensor" appear together in the query is 720, the total number of times "indoor_temp_sensor" appears is 1,200, and the total number of times "humidity_sensor" appears is 900. Calculate the co-occurrence frequency F = 720 / sqrt(1200 × 900) = 0.693; the semantic similarity S of the two labels is 0.75; the probability P of querying "humidity_sensor" within 5 minutes after querying "indoor_temp_sensor" is 0.65, and calculate the temporal correlation T = 0.65.

[0208] 5.2.3. Optimize the weight coefficients; the current weights for the system operation are α = 0.4, β = 0.3, γ = 0.3, and the prediction error is E = 0.15; apply gradient descent, ΨE / Ψα = 0.08, ΨE / Ψβ = -0.12, ΨE / Ψγ = 0.04, and the learning rate η = 0.01. Update to get α' = 0.4 - 0.01 × 0.08 = 0.399, β' = 0.3 - 0.01 × (-0.12) = 0.301, γ' = 0.3 - 0.01 × 0.04 = 0.2996. After normalization, α = 0.399, β = 0.301, γ = 0.3.

[0209] 5.2.4. Affinity calculation formula: A[i,j] = α·F(i,j) + β·S(i,j) + γ·T(i,j); where F(i,j) is the frequency of co-occurrence of tags i and j in the same query; S(i,j) is the semantic similarity between tags i and j; T(i,j) is the temporal correlation that tag j is queried after tag i is queried; α, β, and γ are weight coefficients, satisfying α + β + γ = 1. Specifically, calculate the affinity value A[i,j] = 0.399×0.693 + 0.301×0.75 + 0.3×0.65 = 0.697; according to the affinity value A[i,j] = 0.697 being greater than the threshold 0.01, store this value in the sparse matrix in CSR format.

[0210] 5.2.5. Perform incremental update; process 100 newly added query logs of the day, calculate the incremental impact on the affinity value as +0.002, and update it to A[i,j] = 0.697×0.99 + 0.002 = 0.692 after applying the time decay factor.

[0211] 5.2.6. Detect affinity change; the change compared to last month is -0.005, and the change rate is 0.7%, which is less than the threshold 30%, so there is no need to trigger recalculation of the path.

[0212] 5.3. Implement the label affinity path optimization algorithm.

[0213] 5.3.1. Parse the query request, extract the tag set Q = {"indoor_temp_sensor", "humidity_sensor"}, and construct the query tag set.

[0214] 5.3.2. Construct the query affinity subgraph G; the vertex set V = {"indoor_temp_sensor", "humidity_sensor"}, and the edge set E contains an edge e("indoor_temp_sensor", "humidity_sensor") with a weight of the affinity value 0.692.

[0215] 5.3.3. Apply the path planning algorithm; in this example, there are only two tags, and only the query order needs to be determined. Calculate the affinity scores of two possible paths: Score(indoor_temp→humidity) = 0.692, Score(humidity→indoor_temp) = 0.692, and the scores of the two paths are the same.

[0216] 5.3.4. Process query constraints; analyze query semantics, determine that "indoor temperature" is the main query object, "humidity" is the associated query object, there is a logical dependency from "temperature" to "humidity", and determine the query order as "indoor_temp_sensor" → "humidity_sensor".

[0217] 5.3.5. Check the path cache; find a path record that exactly matches in the first-level cache. The last query time was 10 minutes ago, and directly reuse the path.

[0218] 5.3.6. Evaluate the execution cost; the data of "indoor_temp_sensor" is stored in SSD, and the access cost is 2 points; the data of "humidity_sensor" is stored in HDD, and the access cost is 5 points; the estimated total execution cost of the path is 7 points, and it is confirmed as the optimal path.

[0219] 5.4. Implement a query path heat tracking mechanism.

[0220] 5.4.1. Generate an identifier for the query path; perform a hash calculation on the label ID sequence [5, 12] to obtain the path identifier "p78a2f9".

[0221] 5.4.2. Record the execution performance; the total execution time of the query path "p78a2f9" is 180 ms, the average CPU occupancy rate is 12%, the maximum memory usage is 80 MB, the number of I / O operations is 15 times, the size of the intermediate result set is 2 MB, and the normalized performance score P(p) = 0.85.

[0222] 5.4.3. Count the usage frequency; the query path "p78a2f9" has been used 150 times in the past 30 days. Apply the time decay model to calculate F(p) = ∑(Count(p, i) × e -0.05×(30-i) ) = 120.8, and after normalization, F(p) = 0.75.

[0223] 5.4.4. Adjust the weight parameter; the weight adjustment formula: λ = λ_base + η * (R_target - R_current) / R_target; where λ_base is the base weight; η is the adjustment coefficient; R_current is the current system resource utilization rate; R_target is the target resource utilization rate. Specifically, the current system resource utilization rate R_current = 55%, the target utilization rate R_target = 60%, the base weight λ_base = 0.6, and the adjustment coefficient η = 0.2. Calculate λ = 0.6 + 0.2 × (60 - 55) / 60 = 0.6 + 0.2 × 0.083 = 0.617.

[0224] 5.4.5. Calculate the path heat value; the heat calculation formula: H(p) = λ·F(p) + (1-λ)·P(p); where H(p) is the heat value of path p; F(p) is the usage frequency of path p; P(p) is the performance score of path p; λ is the balance parameter. Specifically, H(p)=0.617×0.75+(1-0.617)×0.85=0.617×0.75+0.383×0.85=0.463+0.326=0.789, which is classified as a commonly used path.

[0225] 5.4.6. Analyze the heat trend; the heat values in the past 7 days are [0.75, 0.76, 0.78, 0.77, 0.79, 0.78, 0.79] respectively, and the slope of the linear regression fit is +0.005, showing a stable upward trend. It is expected to enter the hot path level in the future.

[0226] 5.5. Dynamically select the query strategy according to the current system load status; the current system load is medium (L2), select the standard query strategy for commonly used paths, and allocate moderate resources.

[0227] 5.6. Execute the optimized query operation, query in the order of "indoor_temp_sensor" → "humidity_sensor", obtain the data of the past 24 hours, and return the result; at the same time, update the affinity matrix and path heat information.

[0228] Step Six. Storage management based on data value.

[0229] 6.1. The system periodically detects the storage space; the current available storage space is 150GB, the total capacity is 500GB, and the available rate is 30%, which is lower than the preset threshold of 35%. It is judged that historical data cleaning is required.

[0230] 6.2. Calculate the data retention priority; the priority calculation formula: P(d) = w1·(1 / A(d)) + w2·U(d)+ w3·I(d); where P(d) is the retention priority of data item d; A(d) is the age of data item d; U(d) is the usage frequency of data item d; I(d) is the business importance score of data item d; w1, w2, w3 are weight coefficients, satisfying w1+w2+w3=1. Taking the "indoor_temp_sensor" data as an example, the age A = 90 days, the usage frequency U = 0.8, the business importance I = 0.7, and the weight coefficients w1 = 0.3, w2 = 0.4, w3 = 0.3. Calculate P = 0.3×(1 / 90)+0.4×0.8+0.3×0.7=0.003+0.32+0.21=0.533.

[0231] 6.3. Perform data cleaning according to priority; batch delete data with a priority lower than 0.3, and a total of 120 GB of data is cleaned; downsample and compress data with a priority between 0.3 and 0.4 at a ratio of 5:1, and a total of 80 GB of data is processed, which occupies 16 GB after compression; after cleaning, the available space of the system increases to 254 GB, and the available rate is 50.8%, reaching a safe level.

[0232] 6.4. Record the cleaning operation log, including information such as the cleaning time "2025-04-27 15:30:00", the amount of cleaned data "200 GB", and the remaining space "254 GB", as a reference for system operation and maintenance.

[0233] Through actual application tests, this embodiment solves the key problems of traditional data collection methods: Improvement in semantic adaptability: Through semantic recognition and version control, the system can automatically adapt to newly added data tags or adjusted data meanings, achieving a semantic correct mapping rate of 99.2%, which is 13.6% higher than the 85.6% of the traditional template mapping method. Optimization of storage efficiency: The multi-level semantic difference coding algorithm reduces the data storage space by 68%. At the same time, through the frequency-aware adaptive compression strategy, the access performance of high-frequency data only drops by 2.5%, while the performance of traditional compression methods drops by 12.8% at the same compression ratio. Improvement in query efficiency: Based on the query path optimization of the label affinity matrix and combined with the heat tracking mechanism, the average query response time of the system is reduced from 350 ms to 120 ms, and the performance is improved by 65.7%; the response time of high-frequency query paths is further optimized to 80 ms, with a 77.1% improvement. Optimization of resource utilization: The semantic association predictive lazy loading strategy improves the memory usage efficiency by 45%, increases the cache hit rate from the original 52% to 85%, and increases the overall system throughput by 3.5 times.

[0234] The preferred embodiments of the present invention have been described in detail above. However, the present invention is not limited to the specific details in the above embodiments. Within the technical concept of the present invention, various equivalent transformations can be made to the technical solutions of the present invention, and these equivalent transformations all fall within the protection scope of the present invention.

Claims

1. A multi-scenario application data collection method based on atomized design, characterized in that It includes: Receiving reported data and parsing it to form an atomized data structure containing device identification, freeze time, data tags, and data values; Performing semantic recognition on the data tags therein and outputting a tag semantic mapping result; Based on the tag semantic mapping result, generating a tag mapping rule set through a hierarchical caching structure; encoding and compressing it to form a compressed semantic mapping rule; Receiving a data query request and a compressed semantic mapping rule, identifying a query path from a pre-configured tag affinity matrix, executing the query, and returning a query result; Monitoring the storage space of the system accordingly and performing historical data cleaning based on data value evaluation.

2. The method according to claim 1, wherein The steps of parsing to form an atomized data structure include: Receiving an MQTT message reported by a device, parsing the message topic to obtain an original data packet; checking the feature identifier of its packet header to determine whether it is compressed, and if so, calling a decompression algorithm to obtain the decompressed packet; Identifying the protocol identification field in the decompressed packet to determine the protocol specification type; accordingly, calling the corresponding parsing logic to extract and organize the device identification, freeze time, data tags, and data values to form a complete atomized data structure.

3. The method according to claim 1, characterized in that, The steps of outputting a tag semantic mapping result include: Extracting data tags from the atomized data structure, performing special character removal, case normalization, and word segmentation processing to obtain a standardized tag text; inputting it into a semantic recognition engine to extract multi-dimensional semantic features and generate a tag semantic feature vector; Calculating the similarity between the tag semantic feature vector and existing tags in the tag dictionary library to determine the most matching tag and its version information; accordingly, obtaining context information on device type, spatial location, and data characteristics; Combining the context information with the most matching tag and its version information to generate a tag semantic mapping result.

4. The method according to claim 3, characterized in that, The steps of determining the most matching tag and its version information and generating a tag semantic mapping result include: Calculating the cosine similarity between the tag semantic feature vector and each tag vector in the tag dictionary library to generate a similarity sorted list; accordingly, judging whether a new version of the tag needs to be created through a set similarity threshold; based on the judgment result, determining the most matching tag and its version information; Performing a dimension-level difference calculation on the target vector and the most matching tag, marking significantly changed dimensions, generating a dimension difference report; accordingly, executing a version evolution rule, determining an upgrade method, generating a tag version identifier and associating it with the tag semantic feature vector to form a tag semantic mapping result.

5. The method according to claim 1, wherein The steps of generating a tag mapping rule set include: Receiving a tag semantic mapping result and current context information, extracting query records from a query history log library, and constructing a query sequence set; accordingly, analyzing the temporal query pattern between tags, calculating the temporal correlation strength of tag pairs, and generating a tag temporal correlation matrix; Based on the tag temporal correlation matrix, calculating the semantic similarity between tags and the current device, user, time, and operation context to generate a tag context relevance table; combining it with the tag temporal correlation matrix to calculate the semantic distance attenuation value of tags; Implementing a three-level preloading strategy according to the semantic distance attenuation value, loading tags with different priorities into the first-level, second-level, and third-level caches respectively to generate a tag mapping rule set.

6. The method according to claim 3, wherein The steps of generating a tag semantic feature vector include: Decompose the standardized label text into a set of morphemes to obtain a label morpheme set; map the morphemes to the domain ontology library, obtain the domain semantic weight of each morpheme, and generate an initial morpheme semantic mapping set; The initial morpheme semantic mapping set is subjected to context-aware word sense disambiguation to identify and process polysemous words and generate a disambiguated morpheme semantic mapping set. Based on this, a semantic space representation containing function, object, location, unit, and precision dimensions is constructed, and the initial semantic vector is calculated and normalized to form a label semantic feature vector.

7. The method according to claim 5, wherein The implementation steps of the three-level preloading strategy include: Collect system CPU usage, memory usage, network latency, network packet loss rate, and I / O wait time in real time to generate a system state vector. Based on this, the system load state is classified into light, moderate, heavy, or critical load levels, and a load level identifier is output. Obtain network latency and packet loss rate, evaluate network status and classify it as stable, fluctuating, or unstable, and output a network status rating. Combine this with the load level identifier to dynamically adjust the preloading range, priority, and execution timing to generate dynamic policy configuration.

8. The method according to claim 3, characterized in that, The steps of generating compression semantic mapping rules include: The tags in the tag mapping rule set are clustered according to semantic similarity to form a tag group set; based on this, the core semantic template of each group is calculated; Compare the label semantic feature vector with the core semantic template of the group to which it belongs, extract the difference information, and generate the original difference code; analyze the information redundancy, remove the redundant parts with entropy values below the threshold, and obtain the optimized difference code; The optimal storage format is selected according to different types of data, the optimized difference encoding is converted into an efficient storage structure, and the final compression semantic mapping rules are generated.

9. The method according to claim 1, wherein The steps to configure the tag affinity matrix include: Extract query records from the query log library and construct a query history sequence containing query time, query tags, and user ID. Based on this, calculate the correlation strength between tags in three dimensions: co-occurrence frequency, semantic similarity, and temporal correlation. Based on the association strength, the weight coefficients of the three dimensions are optimized to minimize the query path prediction error; the affinity values of the label pairs are calculated using the optimized weight coefficients to construct the label affinity matrix; The sparse matrix storage structure is used to optimize the storage efficiency of the label affinity matrix, and the optimized label affinity matrix is obtained.

10. The method according to claim 1, wherein The steps to identify the query path include: Parse the data query request, extract the involved data labels, and construct the query label set; combine it with the label affinity matrix to construct the query label affinity subgraph; Based on the query label affinity subgraph, the total path affinity score is calculated to find the initial query path. Based on this, the logical dependency constraints in the query are processed and converted into mandatory path segments. The initial query path is adjusted to obtain the adjusted query path. Search the pre-stored multi-level path cache for historical paths that match the adjusted query path to obtain candidate paths; evaluate the actual execution costs of the candidate paths, comprehensively consider the total affinity score and actual execution cost, and generate the final optimized query path.

Images