A cloud platform-based Internet of Things data processing method and system

Abstract

The present invention relates to the field of information processing technology, specifically a cloud platform-based Internet of Things data processing method and system, comprising the following steps: based on the multimodal data uploaded by the Internet of Things devices, analyzing the device node data synchronization delay and timestamp matching degree, counting the node communication frequency and device association, screening the optimal communication node pair, and performing identity allocation, calculating the data packet transmission success rate, and forming a node identity allocation result. In the present invention, by analyzing the multimodal data uploaded by the Internet of Things devices, the node communication efficiency is optimized and the communication resources are accurately allocated, the transmission delay and failure rate are reduced, the data transmission efficiency is improved, the data access heat is identified, the storage resource utilization is improved and the equipment life is extended, the high-frequency data retrieval path is dynamically optimized, the retrieval speed and accuracy are improved, the response time is reduced, and the data processing efficiency and resource utilization are comprehensively improved from collection to storage, transmission, and retrieval.

Description

Technical Field

[0001] The present invention relates to the field of information processing technology, and in particular to a cloud platform-based Internet of Things data processing method and system. Background Art

[0002] The field of information processing technology covers the entire process of data collection, transmission, storage, analysis and utilization, and is widely used in many fields such as computer science, communication technology, artificial intelligence and cloud computing. In the field of information processing technology, the core content includes data acquisition and analysis, efficient data storage and management, data analysis and mining, and data visualization. This technical field focuses on the orderly organization and processing of various types of data through technical means to meet the needs of data utilization in different scenarios.

[0003] Among them, the IoT data processing method refers to a specific technical solution for standardized collection, storage, processing and transmission of massive data generated by IoT devices. The technical matters covered by this patent subject include data storage and computing resource management based on cloud computing environment, data interaction optimization between IoT devices, and processing and organization of large-scale heterogeneous data. This method provides a unified data processing architecture for IoT devices through a cloud platform, adopts distributed storage technology to achieve storage management of massive data, combines specific data cleaning and format conversion technologies to standardize the transmitted data, and uses a task allocation mechanism to complete data transmission between devices and the reasonable allocation of computing resources, thereby realizing the processing and scheduling of multi-source data in the IoT environment.

[0004] When processing the massive amounts of data generated by IoT devices, existing technologies have insufficient optimization of device node communication efficiency, and fail to effectively solve the problems of high communication delays and data transmission failure rates, resulting in transmission bottlenecks in multi-node communications in complex networks. In terms of data transmission, existing technologies have weak dynamic adjustment capabilities for bandwidth resource allocation, adopt fixed rules or simple traffic distribution modes, and fail to flexibly optimize according to the transmission needs of real-time data, resulting in waste of bandwidth resources or delays in transmission tasks. At the data storage level, there is a lack of effective means for dynamic identification of data access popularity and priority use of high-performance storage media, resulting in low reading efficiency of high-frequency access data, while low-frequency data occupies a large amount of storage space, causing waste of storage resources. Existing technologies do not adequately optimize the access path of high-frequency data, resulting in low retrieval efficiency, which can easily lead to extended response time or resource contention in high-concurrency access scenarios, limiting the processing capabilities of multi-source data and the efficiency of overall data utilization in the IoT environment, and failing to fully meet the dynamic needs in complex application scenarios. Summary of the Invention

[0005] The purpose of the present invention is to solve the shortcomings of the prior art and to propose a cloud platform-based Internet of Things data processing method and system.

[0006] In order to achieve the above objectives, the present invention adopts the following technical solution: a cloud platform-based Internet of Things data processing method, comprising the following steps:

[0007] S1: Based on the multimodal data uploaded by IoT devices, analyze the device node data synchronization delay and timestamp matching, count the node communication frequency and device association, select the optimal communication node pair, assign identifiers, calculate the data packet transmission success rate, and form the node identifier assignment result;

[0008] S2: Based on the node identifier allocation results, analyze the path communication delay, data transmission success rate, and loss rate, select the path data with the best transmission efficiency, generate a communication path efficiency ranking list, adjust the transmission order of the communication tasks within the group through bandwidth allocation, redistribute the data traffic, and generate a node task bandwidth allocation dataset;

[0009] S3: Based on the node task bandwidth allocation data set, count the access popularity of device data, identify the access frequency and time interval of the data, store high-popularity data in high-performance storage media, obtain high-popularity data storage distribution data, perform storage compression on low-popularity data, allocate storage space based on the compression rate and remaining capacity, and establish a data storage optimization result;

[0010] S4: Based on the data storage optimization results, extract high-frequency data access records, count the data sets with high access frequency, optimize the access order through multi-layer index node path distribution, adjust the index level order, optimize the dynamic retrieval of high-frequency data, and establish dynamic retrieval results for high-frequency IoT data.

[0011] As a further solution of the present invention, the step of obtaining the node identifier allocation result is specifically as follows:

[0012] S111: Based on the multimodal data uploaded by the IoT devices, extract the timestamps uploaded by the devices, calculate the time difference between the data uploaded by each device node, compare the time difference with the set synchronization threshold, select the node pairs whose time difference meets the threshold, and generate a time synchronization matching index;

[0013] S112: Based on the time synchronization matching index, select node pairs whose time synchronization matching meets the requirements, count the communication frequencies between the nodes, sort the communication frequencies of the node pairs, and generate an optimal selection list of node pairs;

[0014] S113: Based on the optimal selection list of node pairs, combined with the communication frequency and time synchronization matching degree between nodes, the formula is adopted:

[0015]

[0016] Calculate the transmission success rate and generate the node identification allocation result;

[0017] Among them, S represents the transmission success rate, α represents the weight factor of synchronization delay, β represents the weight factor of communication frequency, γ represents the correction coefficient of node association, D represents the synchronization delay value, F represents the communication frequency, and L represents the device association index.

[0018] As a further solution of the present invention, the step of obtaining the communication path efficiency ranking list is specifically as follows:

[0019] S211: calling the node pair identifier in the node identifier allocation result, using the cloud platform's data processing capabilities to retrieve the communication records associated with the node pair, extracting the communication timestamp data, identifying the communication delay between the nodes, and integrating the node pairs that meet the conditions by comparing with a preset delay threshold to obtain a list of preferred communication routes;

[0020] S212: Analyze the data transmission success rate and loss rate according to each path in the preferred communication route list, using the formula:

[0021]

[0022] Calculate the path transmission efficiency score and generate the path transmission efficiency analysis results;

[0023] Among them, E represents the path transmission efficiency score, T s Represents the number of successfully transmitted data packets, T t Represents the total number of data packets, L r represents the rate of lost packets, H represents the communication delay, and λ is the weight factor of the impact of delay on efficiency;

[0024] S213: Calling the path transmission efficiency analysis result, sorting the recorded transmission efficiency data, arranging the paths according to the order of data transmission efficiency, and generating a communication path efficiency sorting list.

[0025] As a further solution of the present invention, the steps of obtaining the node task bandwidth allocation data set are specifically as follows:

[0026] S221: Based on the communication path efficiency ranking list, calculate the efficiency value of each path, compare the relevance of task paths, record the correspondence between tasks and paths, extract path bandwidth and task bandwidth requirements, and obtain a communication task path priority allocation list;

[0027] S222: Based on the communication task path priority allocation list, adjust according to the difference between the task bandwidth requirement and the path bandwidth, allocate task traffic that exceeds the bandwidth capacity to a low-priority path, verify the path and task data, and obtain a task bandwidth allocation adjustment list;

[0028] S223: Based on the task bandwidth allocation adjustment list, sequentially adjust the transmission task of each communication node, allocate the adjusted data traffic according to the data traffic occupancy ratio in the bandwidth allocation list, summarize the task data and bandwidth allocation information of the communication nodes, and generate a node task bandwidth allocation data set.

[0029] As a further solution of the present invention, the step of obtaining the high-heat data storage distribution data is specifically as follows:

[0030] S311: Based on the node task bandwidth allocation data set, extract the access records associated with the device, count the data access frequency of each device, including the corresponding access log, calculate the timestamp difference of each log, analyze the frequency and time interval data, and generate an access popularity index for the device data;

[0031] S312: Based on the access popularity index of the device data, a popularity threshold is set and the data is classified. Data above the threshold is identified as high popularity data using the formula:

[0032]

[0033] Calculate the heat value of the data item and generate the classification results of high-heat data;

[0034] Where H represents the popularity value of the data item, Y represents the access frequency of the target data, ΔT represents the average access interval, M is the popularity weight factor, and R represents the bandwidth allocation ratio of the associated device;

[0035] S313: Based on the high-heat data classification result, migrate the data marked as high-heat to a high-performance storage medium, update the storage location information table, including the data item identifier and the new storage medium location, and obtain high-heat data storage distribution data.

[0036] As a further solution of the present invention, the step of obtaining the data storage optimization result is specifically as follows:

[0037] S321: Based on the high-heat data storage distribution data, extract the heat parameters of the storage units in the storage location, calculate the heat data in segments according to the storage capacity usage rate, mark the high-heat areas and extract the storage capacity distribution, integrate the distribution characteristics and capacity information of the storage units, and obtain the high-heat data storage distribution characteristics;

[0038] S322: Based on the high-heat data storage distribution characteristics, select storage locations corresponding to low-heat data, analyze the remaining capacity of each location and the storage demand of the low-heat data, match the storage demand with the remaining capacity, adjust the storage location of the low-heat data according to the compression potential, and obtain a low-heat data storage allocation data set;

[0039] S323: Based on the low-heat data storage allocation data set, a compression operation is performed on the low-heat data according to the compression potential parameter, the compressed data is reallocated to a matching storage location, the status and distribution parameters of the storage unit are updated, the overall storage data is counted, and a data storage optimization result is constructed.

[0040] As a further solution of the present invention, the steps for obtaining the dynamic search results of the high-frequency Internet of Things data are specifically as follows:

[0041] S411: Calling the high-frequency data identifiers and corresponding access records in the data storage optimization result, extracting the access timestamps, associating the timestamp data with the high-frequency identifiers, identifying access intervals based on the access times, and counting access frequencies to generate a high-access frequency data set;

[0042] S412: Based on the high-access frequency data set, the access sequence is adjusted through distribution optimization, using the formula:

[0043]

[0044] Optimize the access priority in the index path, dynamically adjust the index level order, and obtain the optimized path priority value;

[0045] Among them, P opt Indicates the optimized path priority value, Q i represents the access frequency of the i-th data, H i represents the number of nodes in the access path, W i is the weight factor set according to the device priority;

[0046] S413: Calling the optimized path priority value, associating the dynamic retrieval logic of high-popularity data with the index optimization order, and optimizing the access order of high-popularity data to generate dynamic retrieval results for high-frequency IoT data.

[0047] A cloud-based IoT data processing system, configured to execute the aforementioned cloud-based IoT data processing method, comprises:

[0048] The node ID allocation module compares the timestamp difference and delay parameters based on the multimodal data uploaded by IoT devices, analyzes the frequency of associated communications between device nodes, extracts high-frequency node pairs, calculates the transmission success rate based on the data packet transmission records, and establishes the node ID allocation results;

[0049] The communication path optimization module analyzes the communication delay and transmission success rate of the device node path based on the node identifier allocation result, selects the path according to the loss rate, extracts the optimal path data, rearranges the path bandwidth allocation, adjusts the task data flow transmission order, and generates the task bandwidth allocation result;

[0050] The data storage distribution module counts the device data access frequency and time interval based on the task bandwidth allocation result, filters the access frequency data and merges and allocates it to the high-speed storage medium, analyzes the compression rate of the low-frequency data and allocates the storage location, and generates the data storage optimization result;

[0051] Based on the data storage optimization results, the high-frequency data retrieval module extracts the high-frequency data storage path and original access records, analyzes the path access frequency and index level, optimizes the index order, adjusts the high-frequency data access path, and generates dynamic retrieval results for high-frequency Internet of Things data.

[0052] Compared with the prior art, the advantages and positive effects of the present invention are:

[0053] In the present invention, the efficiency of data processing and utilization is improved by conducting in-depth analysis of the multimodal data uploaded by IoT devices, and the communication efficiency between nodes is optimized by analyzing the data synchronization delay and timestamp matching degree of device nodes. In combination with the statistical communication frequency and device correlation, the optimal node combination is selected to achieve accurate allocation of communication resources, thereby reducing the delay and failure rate in data transmission. In the analysis of path communication delay, transmission success rate and loss rate, the priority optimization of transmission tasks and the reasonable allocation of traffic are achieved by generating a path efficiency ranking list and adjusting bandwidth allocation, thereby improving the overall data transmission efficiency. In terms of data storage optimization, Identify the access popularity and frequency of data, store high-popularity data in high-performance media, and compress low-popularity data to achieve efficient allocation and utilization of storage resources, thereby reducing the capacity pressure of storage devices and extending their service life. Dynamic retrieval optimization improves the speed and accuracy of high-frequency data retrieval and reduces the response time of data queries by optimizing and adjusting high-popularity data access records and index node paths. The overall processing logic is hierarchically optimized from data collection to storage, transmission, and retrieval, significantly improving the efficiency of data processing and resource utilization in the Internet of Things environment, and enhancing the processing capabilities of multi-source data and the accuracy of dynamic scheduling. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1It is a schematic diagram of the workflow of the present invention;

[0055] Figure 2 A flowchart of the node identifier allocation result in the present invention;

[0056] Figure 3 A flow chart showing a ranking list of communication path efficiencies in the present invention;

[0057] Figure 4 A flowchart of the node task bandwidth allocation data set in the present invention;

[0058] Figure 5 This is a flow chart of high-heat data storage and distribution data in the present invention;

[0059] Figure 6 This is a flow chart of the data storage optimization results in the present invention;

[0060] Figure 7 This is a flow chart of the dynamic retrieval results of high-frequency IoT data in the present invention. DETAILED DESCRIPTION

[0061] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0062] In the description of the present invention, it should be understood that the terms "length," "width," "up," "down," "front," "back," "left," "right," "vertical," "horizontal," "top," "bottom," "inside," "outside," and the like, indicating positions or relationships, are based on the positions or relationships shown in the accompanying drawings and are intended only to facilitate the description of the present invention and simplify the description. They do not indicate or imply that the devices or elements referred to must have a specific orientation, be constructed, or operate in a specific orientation. Therefore, they should not be construed as limiting the present invention. Furthermore, in the description of the present invention, "plurality" means two or more, unless otherwise expressly and specifically defined.

[0063] Example 1

[0064] See also Figure 1 The present invention provides a technical solution: a method for processing Internet of Things data based on a cloud platform, comprising the following steps:

[0065] S1: Based on the multimodal data uploaded by IoT devices, analyze the device node data synchronization delay and timestamp matching, count the node communication frequency and device association, select the optimal communication node pair, assign identifiers, calculate the data packet transmission success rate, and form the node identifier assignment result;

[0066] S2: Based on the node identification allocation results, analyze the path communication delay, data transmission success rate and loss rate, select the path data with the best transmission efficiency, generate a communication path efficiency ranking list, adjust the transmission order of the communication tasks within the group through bandwidth allocation, redistribute the data flow, and generate a node task bandwidth allocation dataset;

[0067] S3: Allocates data sets based on node task bandwidth, counts device data access popularity, identifies data access frequency and time intervals, stores high-popularity data in high-performance storage media, obtains high-popularity data storage distribution data, compresses low-popularity data, allocates storage space based on compression ratio and remaining capacity, and establishes data storage optimization results.

[0068] S4: Based on the data storage optimization results, extract the access records of high-frequency data, count the data sets with high access frequency, optimize the access order through multi-layer index node path distribution, adjust the index level order, optimize the dynamic retrieval of high-frequency data, and establish dynamic retrieval results for high-frequency IoT data.

[0069] The node identification allocation results include device node synchronization delay, timestamp matching, node communication frequency, and device correlation analysis results. The communication path efficiency ranking list includes path delay, data transmission success rate, data loss rate, and path efficiency ranking. The node task bandwidth allocation data set includes bandwidth allocation plan, transmission order adjustment, data traffic redistribution, and node task bandwidth configuration. High-heat data storage distribution data includes access popularity statistics, high-heat data storage location, and high-performance storage media configuration. Data storage optimization results include data compression rate, remaining storage space, low-heat data compression plan, and storage space allocation strategy. The dynamic retrieval results of high-frequency IoT data include access frequency, index path optimization, and retrieval order adjustment.

[0070] See also Figure 2 , the specific steps for obtaining the node identification assignment result are:

[0071] S111: Based on the multimodal data uploaded by the IoT devices, extract the timestamps uploaded by the devices, calculate the time difference between the data uploaded by each device node, compare the time difference with the set synchronization threshold, select the node pairs whose time difference meets the threshold, and generate a time synchronization matching index;

[0072] Call the timestamp record file uploaded by the device, classify each data point in the timestamp record according to the device node, calculate the data upload time difference between each two nodes one by one, establish a time difference matrix for node pairs based on the time difference data, compare the time difference values ​​in the matrix with the set synchronization threshold item by item, mark the node pairs whose time difference values ​​exceed the threshold range as out of sync, eliminate the out of sync node pairs and rebuild the time difference matrix, take the average of the time difference values ​​of the remaining node pairs to generate a new time difference matrix, calculate the time synchronization matching index for each pair of nodes based on the matrix, and the matching degree is obtained by subtracting the ratio of the time difference value to the synchronization threshold from 1. After the calculation is completed, the node pairs with synchronization matching degrees greater than the threshold setting ratio are screened out as synchronization nodes to generate the time synchronization matching degree index.

[0073] S112: Based on the time synchronization matching index, select node pairs that meet the time synchronization matching requirements, count the communication frequencies between the nodes, sort the communication frequencies of the node pairs, and generate an optimal selection list of node pairs;

[0074] The filtered node pairs are imported into the communication frequency table, and the communication log files of all nodes are classified in chronological order. The communication records between nodes in the log are counted as a communication frequency table. The communication frequency of each pair of nodes is calculated based on the frequency table. The communication frequency is obtained by dividing the number of communications by the communication cycle. The nodes are sorted from high to low according to the communication frequency. The top several pairs of node pairs with the highest communication frequency are extracted as the preferred node pair set. A communication frequency distribution diagram is generated based on the node frequency data in the preferred node pair set. The dense distribution range of the node pair communication frequency is determined, and the node pairs within the dense distribution range are preferentially marked to generate the optimal selection list of node pairs.

[0075] S113: Based on the optimal selection list of node pairs, combined with the communication frequency and time synchronization matching degree between nodes, the formula is used:

[0076]

[0077] Calculate the transmission success rate and generate the node identification allocation result;

[0078] Where S represents the transmission success rate, α represents the weight factor of synchronization delay, β represents the weight factor of communication frequency, γ represents the correction coefficient of node association, D represents the synchronization delay value, F represents the communication frequency, and L represents the device association index;

[0079] The benefit of this formula lies in that by combining the three parameters of synchronization delay D, communication frequency F, and device correlation index L, the weight factors α, β, and γ of delay, frequency, and correlation are reasonably set. By comprehensively adjusting the weights of the three parameters, the transmission success rate is dynamically optimized, improving the flexibility and accuracy of the model.

[0080] Take a node pair, assume synchronization delay D = 0.02 seconds, communication frequency F = 150 times / minute, device correlation index L = 0.8, and set weight factors α = 0.5, β = 0.3, and γ = 0.2 according to the actual monitoring environment;

[0081] Substitute into the formula to calculate:

[0082] Calculate the first part: exp(-0.5 0.02 - 0.3 150) = exp(-45.01) ≈ 2.55 × 10 -20 ;

[0083] therefore:

[0084] Calculate the second part:

[0085] Multiply the two results: S≈1·0.862=0.862;

[0086] The results show that the transmission success rate of the node pair is 86.2%, indicating that the node pair has high reliability in data transmission. Combining the transmission success rate data of the node pair can further optimize the node identification allocation result, and establish an efficient communication network by selecting the node pair with a higher transmission success rate.

[0087] See also Figure 3 ,The steps for obtaining the communication path efficiency sorted list are as follows:

[0088] S211: Calling the node pair identifier in the node identifier allocation result, using the cloud platform's data processing capabilities to retrieve the communication records associated with the node pair, extracting the communication timestamp data, identifying the communication delay between the nodes, and integrating the node pairs that meet the conditions by comparing with a preset delay threshold to obtain a list of preferred communication routes;

[0089] Communication data associated with node pairs are filtered from the cloud platform communication records. The start and end timestamps of the timestamp information record file are read step by step. The time difference calculation is performed for each node pair to obtain the communication delay value. All communication delay values ​​are accumulated one by one to form the total communication delay value of the path. The total delay result of each path is compared with the set delay threshold in turn, and the paths with total delay values ​​less than the set threshold are filtered out and added to the valid path set. For the paths that fail the screening, their corresponding node pairs are marked as ineligible and removed from the set. After completing the delay calculation and screening of all node pairs, a list of preferred communication routes is generated. At the same time, the filtered path identifiers and corresponding delay values ​​are recorded for subsequent use. The results of the valid path set support subsequent transmission efficiency analysis and path sorting.

[0090] S212: Analyze the data transmission success rate and loss rate for each path in the preferred communication route list using the formula:

[0091]

[0092] Calculate the path transmission efficiency score and generate the path transmission efficiency analysis results;

[0093] Among them, E represents the path transmission efficiency score, T s Represents the number of successfully transmitted data packets, T t Represents the total number of data packets, L r represents the rate of lost packets, H represents the communication delay, and λ is the weight factor of the impact of delay on efficiency;

[0094] The benefit of the formula is that by combining the transmission success rate T s / T t , loss rate L r , delay H and adjustment weight factor λ, realize the comprehensive evaluation of path transmission performance, which helps to optimize path selection to improve network communication efficiency and reliability;

[0095] Calculate a path in the valid path set and set the following parameters: the number of successfully transmitted data packets T s =9500, total number of data packets T t =10000, loss rate L r =0.05(by 1-T s / T t Calculated), path delay H = 0.02, delay weight factor λ = 0.1;

[0096] The first part of the calculation:

[0097]

[0098] 1-L r =1-0.05=0.95;

[0099]

[0100] The second part of the calculation:

[0101]

[0102]

[0103] Final transmission efficiency: E = 0.9025 + 5 = 5.9025

[0104] The results show that the transmission efficiency score of this path is 5.9025. The high transmission efficiency indicates that this path is suitable for priority consideration as a communication path. Combined with the efficiency scores of other paths, the path selection and sorting can be further optimized.

[0105] S213: Calling the path transmission efficiency analysis results, sorting the recorded transmission efficiency data, arranging the paths according to the order of data transmission efficiency, and generating a communication path efficiency sorting list;

[0106] By reading the transmission efficiency value of each path in turn, sorting them from high to low according to transmission efficiency, and storing the sorted path identifiers and corresponding efficiency values ​​in the communication path efficiency sorting table, the sorted data is combined and the sorting is further corrected by setting the path stability index. Paths with path stability lower than the set threshold are re-marked as secondary priority paths and adjusted in sorting. After the sorting adjustment is completed, the consistency between the transmission efficiency score and the stability index is reconfirmed. Through cross-validation of the efficiency value and stability score, paths with high transmission efficiency and good stability are screened out. Finally, a communication path efficiency sorting list is formed, which contains the path identifier, transmission efficiency value and stability score. The sorting results are combined to complete the path selection optimization, providing support for subsequent communication optimization decisions.

[0107] See also Figure 4 ,The steps for obtaining the node task bandwidth allocation dataset are as follows:

[0108] S221: Based on the communication path efficiency ranking list, calculate the efficiency value of each path, compare the relevance of the task paths, record the correspondence between the tasks and the paths, extract the path bandwidth and task bandwidth requirements, and obtain a communication task path priority allocation list;

[0109] The real-time throughput, latency, and packet loss rate of the path are used as key input parameters. The path efficiency value is calculated by comparing the communication indicators of each path. The efficiency value is obtained by comprehensive weighted calculation after real-time monitoring of the path throughput (measured in Mbps), latency (measured in ms), and packet loss rate (percentage). While calculating the path efficiency value, the correlation of task paths is compared. By retrieving the communication requirements of the task and the characteristics of the path resources, their matching degree is determined, and a corresponding relationship between the task and the path is established. The maximum bandwidth value of the path and the bandwidth requirement of the current communication task are further extracted, and the matching between the two is recorded. Paths that cannot meet the bandwidth requirement are marked as secondary paths. After obtaining the correlation of task paths, a communication task path priority allocation list is constructed based on the task priority and path efficiency value.

[0110] S222: Based on the communication task path priority allocation list, adjustments are made according to the difference between the task bandwidth requirement and the path bandwidth, and task traffic exceeding the bandwidth capacity is allocated to low-priority paths. The paths and task data are verified to obtain a task bandwidth allocation adjustment list;

[0111] First, the task's required bandwidth and the corresponding path's available bandwidth are extracted from the priority allocation results. The difference between the task's required bandwidth and the path's bandwidth is categorized into three scenarios: fully satisfied, partially satisfied, and unsatisfied. For tasks that exceed the path's bandwidth capacity, the excess bandwidth flow is extracted, and path resources that meet the bandwidth requirements are re-searched on low-priority paths, ensuring that path efficiency meets the minimum available standard. After bandwidth adjustment and allocation are completed, all task flows are rechecked to check whether the adjusted paths meet the task requirements and are consistent with the communication task's priority. If the adjusted paths still have remaining available bandwidth, this remaining resource is allocated to tasks with lower priorities but whose requirements have not yet been fully met. Finally, the adjusted task and path allocation is recorded to form a task bandwidth allocation adjustment list, noting the total flow and task occupancy ratio of each path.

[0112] S223: Based on the task bandwidth allocation adjustment list, sequentially adjust the transmission task of each communication node, allocate the adjusted data traffic according to the data traffic occupancy ratio in the bandwidth allocation list, summarize the task data and bandwidth allocation information of the communication nodes, and generate a node task bandwidth allocation data set;

[0113] First, the tasks of each communication node are classified and sorted, and the data traffic of high-priority tasks is allocated first. The node's bandwidth resources are redistributed according to the task data traffic occupancy ratio to ensure that the bandwidth allocation is consistent with the allocation data in the list. The communication data of each node is checked one by one, and the traffic is redistributed according to the occupancy ratio specified in the list to avoid resource waste or path overload. For nodes with multi-path task transmission, the task traffic distribution between paths must be balanced to ensure that the multi-path bandwidth allocation ratio is consistent with the task requirements. After the allocation is completed, the task data, path allocation information, bandwidth utilization, etc. of each node are summarized to generate a node task bandwidth allocation dataset. This dataset includes the task traffic details, path allocation results and bandwidth utilization of each communication node for subsequent optimization and monitoring.

[0114] See also Figure 5 ,The steps for obtaining the high heat data storage distribution data are as follows:

[0115] S311: Based on the node task bandwidth allocation data set, extract the access records associated with the device, count the data access frequency of each device, including the corresponding access log, calculate the timestamp difference of each log, analyze the frequency and time interval data, and generate the access popularity index of the device data;

[0116] Extract access records associated with each device ID from the dataset, count the access frequency and timestamp of each record, determine the data access frequency of each device by calculating the time interval of each access record, store the access frequency and time interval in a statistical table with the device ID as the key, and further use the data to analyze access heat. Read the device ID and timestamp information, store the difference between consecutive access timestamps as the access interval, accumulate and count the number of accesses for all records under the same device ID, and record the total access time. Calculate the preliminary heat value of each record through the operational relationship between the access frequency and time interval, merge the heat values ​​into a device access heat statistics table, smooth the extreme heat values ​​through normalization and generate an overall heat distribution, and finally generate an access heat index for the device data.

[0117] S312: Based on the access popularity index of the device data, a popularity threshold is set and the data is classified. Data above the threshold is identified as high popularity data using the formula:

[0118]

[0119] Calculate the heat value of the data item and generate the classification results of high-heat data;

[0120] Where H represents the popularity value of the data item, Y represents the access frequency of the target data, ΔT represents the average access interval, M is the popularity weight factor, and R represents the bandwidth allocation ratio of the associated device;

[0121] The formula is beneficial in that, by combining access frequency Y, time interval ΔT, popularity weight factor M, and bandwidth ratio R, it can evaluate device usage frequency and resource usage, providing a reliable basis for accurately identifying high-popularity data.

[0122] Detailed explanation of the formula and the process of formula calculation and derivation:

[0123] Take a device data item and set the parameters as follows: access frequency Y = 100, average access interval ΔT = 10, bandwidth allocation ratio R = 0.2, and heat weight factor M = 0.5. Substitute into the formula:

[0124] The first part of the calculation:

[0125] The second part of the calculation:

[0126]

[0127]

[0128] Final heat value: H = 10 + 0.224 = 10.224;

[0129] The result shows that the access popularity value of this device data is 10.224, which is higher than the set popularity threshold. This data is marked as high-popularity data and enters the next step of the storage optimization process.

[0130] S313: Based on the high-hot data classification result, migrate the data marked as high-hot to a high-performance storage medium, update the storage location information table, including the data item identifier and the new storage medium location, and obtain high-hot data storage distribution data;

[0131] Call the high-heat data marking results, filter data records above the threshold from the device access heat index, match the record identification information with the original storage location one by one, transfer the data content to the preset high-performance storage medium, update the new storage location to the storage location table, and record the identifier, original location and current storage medium type of the migrated data in the storage distribution table. Reallocate storage resources based on the bandwidth allocation ratio and heat index. The storage distribution table contains the identifier of each high-heat data and its corresponding latest storage medium, and finally obtains the high-heat data storage distribution data.

[0132] See also Figure 6 , the specific steps for obtaining data storage optimization results are:

[0133] S321: Based on the high-popularity data storage distribution data, extract the popularity parameters of the storage units in the storage location, calculate the popularity data in segments according to the storage capacity usage rate, mark the high-popularity areas and extract the storage capacity distribution, integrate the distribution characteristics and capacity information of the storage units, and obtain the high-popularity data storage distribution characteristics;

[0134] By real-time monitoring of the read and write frequency, data request times, and access duration of each storage unit, the access volume of each storage unit within a set period is recorded and its average value is calculated to obtain the heat parameters of the storage unit. According to the storage capacity utilization rate, the heat data of all storage units are segmented and statistically analyzed, and the heat data is divided into three intervals: high, medium, and low. Each interval is further refined according to the capacity utilization ratio of the storage unit. When marking high-heat areas, the storage units are classified according to the set heat threshold, and the storage units above the threshold are marked as high-heat areas. At the same time, the storage capacity distribution of the high-heat units is extracted, including the distribution of total capacity, used capacity, and remaining capacity. The distribution characteristics, capacity information, and heat data of each storage unit are integrated to generate high-heat data storage distribution characteristics results, forming a comprehensive view reflecting the resource utilization and data access characteristics of each storage unit, providing a basis for subsequent optimization of the storage layout.

[0135] S322: Based on the distribution characteristics of high-hot data storage, select storage locations corresponding to low-hot data, analyze the remaining capacity of each location and the storage demand of low-hot data, match the storage demand with the remaining capacity, adjust the storage location of the low-hot data according to the compression potential, and obtain a low-hot data storage allocation data set;

[0136] By extracting storage units below the set heat threshold from the storage distribution characteristics, the low-heat data corresponding to the storage units are screened out. When analyzing the remaining capacity of each low-heat storage location, its current remaining capacity parameters are extracted. Combined with the storage demand of the low-heat data, the remaining capacity is matched according to the data volume. For low-heat data that cannot be stored directly, it is distributed to multiple storage locations after sharding. To improve storage efficiency, combined with the analysis of compression potential, the compressibility ratio of the low-heat data is calculated based on data redundancy, structural characteristics and compression ratio parameters, and the storage location of the data is adjusted so that the storage units with higher remaining capacity are concentrated in the low-heat data storage dataset. After the matching is completed, the new storage location of each low-heat data and its corresponding capacity occupancy are recorded to generate a low-heat data storage allocation dataset.

[0137] S323: Based on the low-heat data storage allocation data set, a compression operation is performed on the low-heat data according to the compression potential parameter, the compressed data is reallocated to a matching storage location, the state and distribution parameters of the storage unit are updated, the overall storage data is counted, and a data storage optimization result is constructed;

[0138] The compression ratio is calculated for each low-heat data, and the compressible volume is estimated and compression is performed based on its redundancy and compression algorithm characteristics. The compressed data is reallocated to the corresponding storage unit according to the storage location parameters in the data allocation dataset. The status of all reallocated storage locations is updated, including the remaining capacity, used capacity and re-statistics of heat data. To ensure data consistency, it is verified whether the compressed data is completely allocated to the new storage location, and the compression ratio and storage location changes are recorded. When counting the overall storage data, the distribution parameters of each storage unit are summarized, including total capacity, usage rate and heat status. The data storage optimization results are generated based on the reallocated data. The optimization results can reflect the overall heat distribution changes of the storage unit and the compression and re-layout of low-heat data, laying the foundation for further improving the efficiency of the storage system.

[0139] See also Figure 7 ,The steps for obtaining dynamic retrieval results of high-frequency IoT data are as follows:

[0140] S411: Calling the high-frequency data identifiers and corresponding access records in the data storage optimization results, extracting the access timestamps, associating the timestamp data with the high-frequency identifiers, identifying access intervals based on access time, and counting access frequencies to generate a high-access frequency data set;

[0141] Filter high-frequency data access records and conduct detailed statistics on the access time of the records. The main purpose is to identify data sets with high access frequency, extract the access timestamp of each data from the optimization results, and calculate the time interval. The time interval is classified and stored according to the unique identifier of the data for the next step of processing. Perform frequency analysis on the classified time interval data, use statistical methods such as standard deviation and mean calculation to find data sets with access frequency higher than the average level, further mark and store the data sets, ensure that the marks correspond to the statistical results in the original data table, thereby generating a high-access frequency data set. The final data set will be used for subsequent indexing and storage optimization operations.

[0142] S412: Based on the high-access frequency data set, the access order is adjusted through distribution optimization using the formula:

[0143]

[0144] Optimize the access priority in the index path, dynamically adjust the index level order, and obtain the optimized path priority value;

[0145] Among them, P opt Indicates the optimized path priority value, Q i represents the access frequency of the i-th data, H i represents the number of nodes in the access path, W i is the weight factor set according to the device priority;

[0146] The benefit of the formula is that by integrating the access frequency F i , path length H i and data weight W i , which improves the efficiency of data access and the responsiveness of dynamic retrieval;

[0147] When determining the optimized access order, we first select representative data records based on the popular data set and calculate their path priorities. For example, we select three data records for calculation, and the record parameters are set as follows:

[0148] Record 1 has an access frequency of Q1 = 50, a path length of H1 = 4, and a data weight of W1 = 1.2;

[0149] Record 2 has an access frequency of Q2 = 40, a path length of H2 = 5, and a data weight of W2 = 1.0;

[0150] Record 3 has an access frequency of Q3 = 60, a path length of H3 = 3, and a data weight of W3 = 1.5;

[0151] The path priority of each record is determined by the following calculation:

[0152] Priority of record 1:

[0153] Priority of record 2:

[0154] Priority of record 3:

[0155] The sum of these calculation results gives the total path priority: P opt =15+8+30=53;

[0156] The results show that the optimized path priority value is 53, indicating that the optimized data index and access order can significantly improve the efficiency and response speed of data retrieval, further enhancing the performance of the data storage system and the reliability of data access. This optimization method is particularly critical in IoT data processing and can ensure efficient and accurate data processing and analysis.

[0157] S413: Calling the optimized path priority value, associating the dynamic retrieval logic of high-popularity data with the index optimization order, and optimizing the access order of high-popularity data to generate dynamic retrieval results for high-frequency IoT data;

[0158] By dynamically adjusting the index path and optimizing the data retrieval order, we first match the data access records with the new index path, prioritize those data with high access frequency, and apply dynamic retrieval strategies to the data to ensure that the required information can be quickly located during the data retrieval process to determine the most appropriate access path for each piece of data. Through actual path adjustment operations, we combine data access frequency with path optimization to generate a detailed statistical report. The report records the position of each type of data in the new access path and the access speed improvement due to path optimization. The optimization results are also compared with the original access records to analyze the changes in retrieval efficiency before and after optimization. This is to verify the effectiveness of the path optimization strategy and generate dynamic retrieval results for high-frequency IoT data. This not only improves the speed of data retrieval, but also optimizes the data storage structure, providing strong support for the dynamic retrieval of high-frequency data.

[0159] The cloud platform-based IoT data processing system is used to execute the above-mentioned cloud platform-based IoT data processing method, and the system includes:

[0160] The node ID allocation module compares the timestamp difference and delay parameters based on the multimodal data uploaded by IoT devices, analyzes the frequency of associated communications between device nodes, extracts high-frequency node pairs, calculates the transmission success rate based on the data packet transmission records, and establishes the node ID allocation results;

[0161] The communication path optimization module analyzes the communication delay and transmission success rate of the device node path based on the node identification allocation results, selects the path according to the loss rate, extracts the optimal path data, rearranges the path bandwidth allocation, adjusts the task data flow transmission order, and generates the task bandwidth allocation result;

[0162] The data storage distribution module counts the device data access frequency and time interval based on the task bandwidth allocation results, filters the access frequency data sets and merges them into high-speed storage media, analyzes the compression rate of low-frequency data and allocates storage locations, and generates data storage optimization results;

[0163] Based on the data storage optimization results, the high-frequency data retrieval module extracts the high-frequency data storage path and original access records, analyzes the path access frequency and index level, optimizes the index order, adjusts the high-frequency data access path, and generates dynamic retrieval results for high-frequency IoT data.

[0164] The above are merely preferred embodiments of the present invention and do not limit the present invention in any other form. Any technician familiar with the profession may use the technical content disclosed above to change or modify it into an equivalent embodiment with equivalent changes and apply it to other fields. However, any simple modification, equivalent change and modification made to the above embodiment based on the technical essence of the present invention without departing from the content of the technical solution of the present invention shall still fall within the scope of protection of the technical solution of the present invention.

Claims

1. A method for processing Internet of Things data based on a cloud platform, characterized in that: The following steps are involved: S1: Based on the multimodal data uploaded by IoT devices, analyze the device node data synchronization delay and timestamp matching, count the node communication frequency and device association, select the optimal communication node pair, assign identifiers, calculate the data packet transmission success rate, and form the node identifier assignment result; S2: Based on the node identifier allocation results, analyze the path communication delay, data transmission success rate, and loss rate, select the path data with the best transmission efficiency, generate a communication path efficiency ranking list, adjust the transmission order of the communication tasks within the group through bandwidth allocation, redistribute the data traffic, and generate a node task bandwidth allocation dataset; S3: Based on the node task bandwidth allocation data set, count the access popularity of device data, identify the access frequency and time interval of the data, store high-popularity data in high-performance storage media, obtain high-popularity data storage distribution data, perform storage compression on low-popularity data, allocate storage space based on the compression rate and remaining capacity, and establish a data storage optimization result; S4: Based on the data storage optimization results, extract high-frequency data access records, count the data sets with high access frequency, optimize the access order through multi-layer index node path distribution, adjust the index level order, optimize the dynamic retrieval of high-frequency data, and establish dynamic retrieval results for high-frequency IoT data.

2. The method for processing Internet of Things data based on a cloud platform according to claim 1, characterized in that: The steps for obtaining the node identification allocation result are specifically as follows: S111: Based on the multimodal data uploaded by the IoT devices, extract the timestamps uploaded by the devices, calculate the time difference between the data uploaded by each device node, compare the time difference with the set synchronization threshold, select the node pairs whose time difference meets the threshold, and generate a time synchronization matching index; S112: Based on the time synchronization matching index, select node pairs whose time synchronization matching meets the requirements, count the communication frequencies between the nodes, sort the communication frequencies of the node pairs, and generate an optimal selection list of node pairs; S113: Based on the optimal selection list of node pairs, combined with the communication frequency and time synchronization matching degree between nodes, the formula is adopted: ； Calculate the transmission success rate and generate the node identification allocation result; in, Represents the transmission success rate, represents the weight factor of synchronization delay, The weight factor representing the communication frequency, represents the correction factor of node association, Represents the synchronization delay value, Represents the communication frequency, Represents the relevance index of the device.

3. The method for processing Internet of Things data based on a cloud platform according to claim 2, characterized in that: The steps for obtaining the communication path efficiency ranking list are specifically as follows: S211: calling the node pair identifier in the node identifier allocation result, using the cloud platform's data processing capabilities to retrieve the communication records associated with the node pair, extracting the communication timestamp data, identifying the communication delay between the nodes, and integrating the node pairs that meet the conditions by comparing with a preset delay threshold to obtain a list of preferred communication routes; S212: Analyze the data transmission success rate and loss rate according to each path in the preferred communication route list, using the formula: ； Calculate the path transmission efficiency score and generate the path transmission efficiency analysis results; in, represents the path transmission efficiency score, Represents the number of packets successfully transmitted, Represents the total number of packets, Represents the rate of packet loss, Represents the communication delay time, is the weighting factor of the effect of delay on efficiency; S213: Calling the path transmission efficiency analysis result, sorting the recorded transmission efficiency data, arranging the paths according to the order of data transmission efficiency, and generating a communication path efficiency sorting list.

4. The method for processing Internet of Things data based on a cloud platform according to claim 3, characterized in that: The steps for obtaining the node task bandwidth allocation data set are specifically as follows: S221: Based on the communication path efficiency ranking list, calculate the efficiency value of each path, compare the relevance of task paths, record the correspondence between tasks and paths, extract path bandwidth and task bandwidth requirements, and obtain a communication task path priority allocation list; S222: Based on the communication task path priority allocation list, adjust according to the difference between the task bandwidth requirement and the path bandwidth, allocate task traffic that exceeds the bandwidth capacity to a low-priority path, verify the path and task data, and obtain a task bandwidth allocation adjustment list; S223: Based on the task bandwidth allocation adjustment list, sequentially adjust the transmission task of each communication node, allocate the adjusted data traffic according to the data traffic occupancy ratio in the bandwidth allocation list, summarize the task data and bandwidth allocation information of the communication nodes, and generate a node task bandwidth allocation data set.

5. The method for processing Internet of Things data based on a cloud platform according to claim 4, characterized in that: The steps for obtaining the high-heat data storage distribution data are specifically as follows: S311: Based on the node task bandwidth allocation data set, extract the access records associated with the device, count the data access frequency of each device, including the corresponding access log, calculate the timestamp difference of each log, analyze the frequency and time interval data, and generate an access popularity index for the device data; S312: Based on the access popularity index of the device data, a popularity threshold is set and the data is classified. Data above the threshold is identified as high popularity data using the formula: ； Calculate the heat value of the data item and generate the classification results of high-heat data; in, Indicates the heat value of the data item, Indicates the access frequency of target data. represents the average access interval, is the heat weight factor, Represents the bandwidth allocation ratio of the associated device; S313: Based on the high-heat data classification result, migrate the data marked as high-heat to a high-performance storage medium, update the storage location information table, including the data item identifier and the new storage medium location, and obtain high-heat data storage distribution data.

6. The method for processing Internet of Things data based on a cloud platform according to claim 5, characterized in that: The steps for obtaining the data storage optimization results are specifically as follows: S321: Based on the high-heat data storage distribution data, extract the heat parameters of the storage units in the storage location, calculate the heat data in segments according to the storage capacity usage rate, mark the high-heat areas and extract the storage capacity distribution, integrate the distribution characteristics and capacity information of the storage units, and obtain the high-heat data storage distribution characteristics; S322: Based on the high-heat data storage distribution characteristics, select storage locations corresponding to low-heat data, analyze the remaining capacity of each location and the storage demand of the low-heat data, match the storage demand with the remaining capacity, adjust the storage location of the low-heat data according to the compression potential, and obtain a low-heat data storage allocation data set; S323: Based on the low-heat data storage allocation data set, a compression operation is performed on the low-heat data according to the compression potential parameter, the compressed data is reallocated to a matching storage location, the status and distribution parameters of the storage unit are updated, the overall storage data is counted, and a data storage optimization result is constructed.

7. The method for processing Internet of Things data based on a cloud platform according to claim 6, characterized in that: The steps for obtaining the dynamic search results of the high-frequency Internet of Things data are specifically as follows: S411: Calling the high-frequency data identifiers and corresponding access records in the data storage optimization result, extracting the access timestamps, associating the timestamp data with the high-frequency identifiers, identifying access intervals based on the access times, and counting access frequencies to generate a high-access frequency data set; S412: Based on the high-access frequency data set, the access sequence is adjusted through distribution optimization, using the formula: ； Optimize the access priority in the index path, dynamically adjust the index level order, and obtain the optimized path priority value; in, Indicates the optimized path priority value. Representative The access frequency of data, Represents the number of nodes in the access path, is the weight factor set according to the device priority; S413: Calling the optimized path priority value, associating the dynamic retrieval logic of high-popularity data with the index optimization order, and optimizing the access order of high-popularity data to generate dynamic retrieval results for high-frequency IoT data.

8. An Internet of Things data processing system based on a cloud platform, characterized in that: The method for processing Internet of Things data based on a cloud platform according to any one of claims 1 to 7, wherein the system comprises: The node ID allocation module compares the timestamp difference and delay parameters based on the multimodal data uploaded by IoT devices, analyzes the frequency of associated communications between device nodes, extracts high-frequency node pairs, calculates the transmission success rate based on the data packet transmission records, and establishes the node ID allocation results; The communication path optimization module analyzes the communication delay and transmission success rate of the device node path based on the node identifier allocation result, selects the path according to the loss rate, extracts the optimal path data, rearranges the path bandwidth allocation, adjusts the task data flow transmission order, and generates the task bandwidth allocation result; The data storage distribution module counts the device data access frequency and time interval based on the task bandwidth allocation result, filters the access frequency data and merges and allocates it to the high-speed storage medium, analyzes the compression rate of the low-frequency data and allocates the storage location, and generates the data storage optimization result; Based on the data storage optimization results, the high-frequency data retrieval module extracts the high-frequency data storage path and original access records, analyzes the path access frequency and index level, optimizes the index order, adjusts the high-frequency data access path, and generates dynamic retrieval results for high-frequency Internet of Things data.

Images