Internet of Things data processing method and system based on cloud platform

Abstract

The invention relates to the technical field of information processing, in particular to an Internet of Things data processing method and system based on a cloud platform, and the method comprises the following steps: based on multi-modal data uploaded by Internet of Things equipment, analyzing equipment node data synchronization delay and timestamp matching degree, carrying out the statistics of node communication frequency and equipment relevance, screening a communication optimal node pair, and carrying out the communication optimal node pair; and carrying out identifier distribution, calculating a data packet transmission success rate, and forming a node identifier distribution result. According to the method, the multi-modal data uploaded by the Internet of Things equipment is analyzed, the node communication efficiency is optimized, accurate allocation of communication resources is realized, the transmission delay and failure rate are reduced, the data transmission efficiency is improved, the data access popularity is identified, the utilization rate of storage resources is improved, the service life of the equipment is prolonged, and a high-frequency data retrieval path is dynamically optimized; the retrieval speed and accuracy are improved, the response time is shortened, and the data processing efficiency and the resource utilization rate are comprehensively improved from collection to storage, transmission and retrieval.

Description

Technical Field

[0001] The present invention relates to the field of information processing technology, and in particular to a cloud platform-based Internet of Things data processing method and system. Background Art

[0002] The field of information processing technology includes the entire process of data collection, transmission, storage, analysis and utilization, and is widely used in many fields such as computer science, communication technology, artificial intelligence and cloud computing. In the field of information processing technology, the core content includes data acquisition and analysis, efficient data storage and management, data analysis and mining, and data visualization. This technical field focuses on the orderly organization and processing of various types of data through technical means to meet the needs of data utilization in different scenarios.

[0003] Among them, the IoT data processing method refers to a specific technical solution for standardized collection, storage, processing and transmission of massive data generated by IoT devices. The technical matters covered by this patent subject include data storage and computing resource management based on cloud computing environment, data interaction optimization between IoT devices, and processing and organization of large-scale heterogeneous data. This method provides a unified data processing architecture for IoT devices through a cloud platform, adopts distributed storage technology to achieve storage management of massive data, combines specific data cleaning and format conversion technologies to standardize the transmitted data, and uses a task allocation mechanism to complete data transmission between devices and reasonable allocation of computing resources, thereby realizing the processing and scheduling of multi-source data in the IoT environment.

[0004] When processing the massive data generated by IoT devices, the existing technology has the problem of insufficient optimization of the communication efficiency of device nodes, and has failed to effectively solve the problems of high communication delays and data transmission failure rates, resulting in transmission bottlenecks in multi-node communications in complex networks. In terms of data transmission, the existing technology has weak dynamic adjustment capabilities for bandwidth resource allocation, adopts fixed rules or simple traffic allocation modes, and fails to flexibly optimize according to the transmission requirements of real-time data, resulting in waste of bandwidth resources or delays in transmission tasks. At the data storage level, there is a lack of effective means for dynamic identification of data access popularity and priority use of high-performance storage media, resulting in low reading efficiency of high-frequency access data, while low-frequency data occupies a large amount of storage space, causing waste of storage resources. The existing technology does not optimize the access path of high-frequency data sufficiently, and has low retrieval efficiency, which can easily lead to extended response time or resource contention in high-concurrency access scenarios, limiting the processing capabilities of multi-source data and the efficiency of overall data utilization in the IoT environment, and failing to fully meet the dynamic needs in complex application scenarios. Summary of the invention

[0005] The purpose of the present invention is to solve the shortcomings of the prior art and to propose a cloud platform-based Internet of Things data processing method and system.

[0006] In order to achieve the above object, the present invention adopts the following technical solution: a method for processing Internet of Things data based on a cloud platform, comprising the following steps:

[0007] S1: Based on the multimodal data uploaded by IoT devices, analyze the device node data synchronization delay and timestamp matching degree, count the node communication frequency and device association, select the optimal communication node pair, and perform identity allocation, calculate the data packet transmission success rate, and form the node identity allocation result;

[0008] S2: Based on the node identification allocation result, analyze the path communication delay, data transmission success rate and loss rate, select the path data with the best transmission efficiency, generate a communication path efficiency ranking list, adjust the transmission order of the communication tasks in the group through bandwidth allocation, reallocate the data flow, and generate a node task bandwidth allocation data set;

[0009] S3: Based on the node task bandwidth allocation data set, count the access popularity of device data, identify the access frequency and time interval of data, store high-heat data in high-performance storage media, obtain high-heat data storage distribution data, perform storage compression on low-heat data, allocate storage space based on compression rate and remaining capacity, and establish data storage optimization results;

[0010] S4: Based on the data storage optimization results, extract high-frequency data access records, count data sets with high access frequencies, optimize access order through multi-layer index node path distribution, adjust index level order, optimize dynamic retrieval of high-frequency data, and establish dynamic retrieval results for high-frequency IoT data.

[0011] As a further solution of the present invention, the step of obtaining the node identifier allocation result is specifically:

[0012] S111: Based on the multimodal data uploaded by the IoT device, extract the timestamp uploaded by the device, calculate the time difference of the data uploaded by each device node, compare the time difference with the set synchronization threshold, select the node pairs whose time difference meets the threshold, and generate the time synchronization matching index;

[0013] S112: Based on the time synchronization matching index, select node pairs whose time synchronization matching degree meets the requirement, count the communication frequency between the nodes, sort the communication frequency of the node pairs, and generate an optimal selection list of node pairs;

[0014] S113: Based on the optimal selection list of node pairs, combined with the communication frequency and time synchronization matching degree between nodes, the formula is adopted:

[0015]

[0016] Calculate the transmission success rate and generate the node identification allocation result;

[0017] Among them, S represents the transmission success rate, α represents the weight factor of synchronization delay, β represents the weight factor of communication frequency, γ represents the correction coefficient of node association, D represents the synchronization delay value, F represents the communication frequency, and L represents the association index of the device.

[0018] As a further solution of the present invention, the step of obtaining the communication path efficiency ranking list is specifically as follows:

[0019] S211: calling the node pair identifier in the node identifier allocation result, using the cloud platform data processing capability to retrieve the communication records associated with the node pair, extracting the communication timestamp data, identifying the communication delay between the nodes, and integrating the node pairs that meet the conditions by comparing with the preset delay threshold to obtain a list of preferred communication routes;

[0020] S212: Analyze the data transmission success rate and loss rate according to each path in the preferred communication route list, using the formula:

[0021]

[0022] Calculate the path transmission efficiency score and generate the path transmission efficiency analysis result;

[0023] Where E represents the path transmission efficiency score, T s Represents the number of successfully transmitted packets, T t Represents the total number of packets, L r represents the rate of lost packets, H represents the communication delay, and λ is the weight factor of the impact of delay on efficiency;

[0024] S213: calling the path transmission efficiency analysis result, sorting the recorded transmission efficiency data, arranging the paths according to the order of data transmission efficiency, and generating a communication path efficiency sorting list.

[0025] As a further solution of the present invention, the step of acquiring the node task bandwidth allocation data set is specifically as follows:

[0026] S221: Based on the communication path efficiency ranking list, calculate the efficiency value of each path, compare the relevance of the task paths, record the correspondence between the tasks and the paths, extract the path bandwidth and the task bandwidth requirements, and obtain a communication task path priority allocation list;

[0027] S222: Based on the communication task path priority allocation list, make adjustments according to the difference between the task bandwidth requirement and the path bandwidth, allocate the task traffic exceeding the bandwidth capacity to the low priority path, check the path and task data, and obtain the task bandwidth allocation adjustment list;

[0028] S223: Based on the task bandwidth allocation adjustment list, sequentially adjust the transmission task of each communication node, allocate the adjusted data traffic according to the data traffic occupancy ratio in the bandwidth allocation list, summarize the task data and bandwidth allocation information of the communication nodes, and generate a node task bandwidth allocation data set.

[0029] As a further solution of the present invention, the step of obtaining the high-heat data storage distribution data is specifically as follows:

[0030] S311: Based on the node task bandwidth allocation data set, extract the access records associated with the device, count the data access frequency of each device, including the corresponding access log, calculate the timestamp difference of each log, analyze the frequency and time interval data, and generate the access heat index of the device data;

[0031] S312: Based on the access heat index of the device data, a heat threshold is set and the data is classified, and data above the threshold is identified as high heat data, using the formula:

[0032]

[0033] Calculate the heat value of the data item and generate high heat data classification results;

[0034] Among them, H represents the heat value of the data item, Y represents the access frequency of the target data, ΔT represents the average access interval, M is the heat weight factor, and R represents the bandwidth allocation ratio of the associated device;

[0035] S313: Based on the high-heat data classification result, migrate the data marked as high-heat to a high-performance storage medium, update the storage location information table, including the data item identifier and the new storage medium location, and obtain high-heat data storage distribution data.

[0036] As a further solution of the present invention, the step of obtaining the data storage optimization result is specifically as follows:

[0037] S321: Based on the high-heat data storage distribution data, extract the heat parameters of the storage unit in the storage location, count the heat data in segments according to the storage capacity usage rate, mark the high-heat area and extract the storage capacity distribution, integrate the distribution characteristics and capacity information of the storage unit, and obtain the high-heat data storage distribution characteristics;

[0038] S322: Based on the high-heat data storage distribution characteristics, select storage locations corresponding to low-heat data, analyze the remaining capacity of each location and the storage demand of low-heat data, match the storage demand with the remaining capacity, adjust the storage location of low-heat data according to compression potential, and obtain a low-heat data storage allocation data set;

[0039] S323: Based on the low-heat data storage allocation data set, a compression operation is performed on the low-heat data according to the compression potential parameter, the compressed data is reallocated to a matching storage location, the state and distribution parameters of the storage unit are updated, the overall storage data is counted, and a data storage optimization result is constructed.

[0040] As a further solution of the present invention, the steps for obtaining the dynamic search results of the high-frequency Internet of Things data are specifically as follows:

[0041] S411: calling the high-frequency data identifier and the corresponding access record in the data storage optimization result, extracting the access timestamp, associating the timestamp data with the high-frequency identifier, identifying the access interval according to the access time and counting the access frequency, and generating a high-access frequency data set;

[0042] S412: Based on the high access frequency data set, the access sequence is adjusted by distribution optimization, using the formula:

[0043]

[0044] Optimize the access priority in the index path, dynamically adjust the index level order, and obtain the optimized path priority value;

[0045] Among them, P opt Indicates the optimized path priority value, Q i represents the access frequency of the i-th data, H i represents the number of nodes in the access path, W i is a weight factor set according to the device priority;

[0046] S413: Call the optimized path priority value, associate the dynamic retrieval logic of high-heat data with the index optimization order, and optimize the access order of high-heat data to generate dynamic retrieval results for high-frequency IoT data.

[0047] A cloud platform-based Internet of Things data processing system, the cloud platform-based Internet of Things data processing system is used to execute the above-mentioned cloud platform-based Internet of Things data processing method, the system comprising:

[0048] The node identification allocation module compares the timestamp difference and delay parameters based on the multimodal data uploaded by the IoT devices, analyzes the frequency of associated communications between device nodes, extracts high-frequency node pairs, calculates the transmission success rate based on the data packet transmission records, and establishes the node identification allocation results;

[0049] The communication path optimization module analyzes the communication delay and transmission success rate of the device node path based on the node identification allocation result, optimizes the path according to the loss rate, extracts the optimal path data, rearranges the path bandwidth allocation, adjusts the task data flow transmission order, and generates the task bandwidth allocation result;

[0050] The data storage distribution module counts the device data access frequency and time interval based on the task bandwidth allocation result, selects the access frequency data set and merges and allocates it to the high-speed storage medium, analyzes the compression rate of the low-frequency data and allocates the storage location, and generates the data storage optimization result;

[0051] Based on the data storage optimization results, the high-frequency data retrieval module extracts the high-frequency data storage path and original access records, analyzes the path access frequency and index level, optimizes the index order, adjusts the high-frequency data access path, and generates dynamic retrieval results for high-frequency IoT data.

[0052] Compared with the prior art, the advantages and positive effects of the present invention are:

[0053] In the present invention, the efficiency of data processing and utilization is improved by conducting in-depth analysis on the multimodal data uploaded by IoT devices. The communication efficiency between nodes is optimized by analyzing the data synchronization delay and timestamp matching degree of device nodes. The optimal node combination is selected by combining the statistical communication frequency and device association, and the accurate allocation of communication resources is realized, thereby reducing the delay and failure rate in data transmission. In the analysis of path communication delay, transmission success rate and loss rate, the priority optimization of transmission tasks and the reasonable allocation of traffic are realized by generating a path efficiency sorting list and adjusting bandwidth allocation, thereby improving the overall data transmission efficiency. In terms of data storage optimization, Identify the access popularity and access frequency of data, store high-temperature data in high-performance media, and compress low-temperature data to achieve efficient allocation and utilization of storage resources, thereby reducing the capacity pressure of storage devices and extending their service life. Dynamic retrieval optimization improves the speed and accuracy of high-frequency data retrieval and reduces the response time of data queries by optimizing and adjusting high-temperature data access records and index node paths. The overall processing logic is hierarchically optimized from data collection to storage, transmission, and retrieval, which significantly improves the efficiency of data processing and resource utilization in the Internet of Things environment, and enhances the processing capabilities of multi-source data and the accuracy of dynamic scheduling. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1It is a schematic diagram of the workflow of the present invention;

[0055] Figure 2 A flowchart of the node identification allocation result in the present invention;

[0056] Figure 3 A flow chart of a communication path efficiency ranking list in the present invention;

[0057] Figure 4 A flowchart of node task bandwidth allocation data set in the present invention;

[0058] Figure 5 A flow chart of high-heat data storage and distribution data in the present invention;

[0059] Figure 6 A flow chart of the data storage optimization results in the present invention;

[0060] Figure 7 This is a flow chart of the dynamic retrieval results of high-frequency Internet of Things data in the present invention. DETAILED DESCRIPTION

[0061] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0062] In the description of the present invention, it should be understood that the terms "length", "width", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside" and the like indicate positions or positional relationships based on the positions or positional relationships shown in the drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present invention. In addition, in the description of the present invention, "multiple" means two or more, unless otherwise clearly and specifically defined.

[0063] Embodiment 1

[0064] See also Figure 1 The present invention provides a technical solution: a method for processing Internet of Things data based on a cloud platform, comprising the following steps:

[0065] S1: Based on the multimodal data uploaded by IoT devices, analyze the device node data synchronization delay and timestamp matching degree, count the node communication frequency and device association, select the optimal communication node pair, and perform identity allocation, calculate the data packet transmission success rate, and form the node identity allocation result;

[0066] S2: Based on the node identification allocation results, analyze the path communication delay, data transmission success rate and loss rate, select the path data with the best transmission efficiency, generate a communication path efficiency ranking list, adjust the transmission order of the communication tasks within the group through bandwidth allocation, reallocate data traffic, and generate a node task bandwidth allocation data set;

[0067] S3: Allocate data sets based on node task bandwidth, count the access popularity of device data, identify the access frequency and time interval of data, store high-heat data in high-performance storage media, obtain high-heat data storage distribution data, perform storage compression on low-heat data, allocate storage space based on compression rate and remaining capacity, and establish data storage optimization results;

[0068] S4: Based on the data storage optimization results, extract the access records of high-frequency data, count the data sets with high access frequency, optimize the access order through multi-layer index node path distribution, adjust the index level order, optimize the dynamic retrieval of high-frequency data, and establish the dynamic retrieval results of high-frequency IoT data.

[0069] The node identification allocation results include device node synchronization delay, timestamp matching, node communication frequency, and device correlation analysis results. The communication path efficiency ranking list includes path delay, data transmission success rate, data loss rate, and path efficiency ranking. The node task bandwidth allocation data set includes bandwidth allocation plan, transmission order adjustment, data traffic reallocation, and node task bandwidth configuration. High-heat data storage distribution data includes access heat statistics, high-heat data storage location, and high-performance storage media configuration. Data storage optimization results include data compression rate, storage space remaining, low-heat data compression plan, and storage space allocation strategy. The dynamic retrieval results of high-frequency IoT data include access frequency, index path optimization, and retrieval order adjustment.

[0070] See also Figure 2 , the specific steps for obtaining the node identification allocation result are:

[0071] S111: Based on the multimodal data uploaded by the IoT device, extract the timestamp uploaded by the device, calculate the time difference of the data uploaded by each device node, compare the time difference with the set synchronization threshold, select the node pairs whose time difference meets the threshold, and generate the time synchronization matching index;

[0072] Call the timestamp record file uploaded by the device, classify each data point in the timestamp record according to the device node, calculate the data upload time difference between each two nodes one by one, establish the time difference matrix of the node pairs based on the time difference data, compare the time difference values ​​in the matrix with the set synchronization threshold item by item, mark the node pairs whose time difference values ​​exceed the threshold range as out of sync, eliminate the out of sync node pairs and rebuild the time difference matrix, take the average of the time difference values ​​of the remaining node pairs to generate a new time difference matrix, calculate the time synchronization matching index of each pair of nodes based on the matrix, and the matching degree is obtained by subtracting the ratio of the time difference value to the synchronization threshold from 1. After the calculation is completed, select the node pairs whose synchronization matching degree is greater than the threshold setting ratio as synchronization nodes to generate the time synchronization matching degree index.

[0073] S112: based on the time synchronization matching index, screening node pairs whose time synchronization matching degree meets the requirement, counting the communication frequency between the nodes, sorting the communication frequency of the node pairs, and generating an optimal selection list of the node pairs;

[0074] The filtered node pairs are imported into the communication frequency, the communication log files of all nodes are classified in chronological order, the communication records between nodes in the log are counted as a communication frequency table, and the communication frequency of each pair of nodes is calculated according to the frequency table. The communication frequency is obtained by dividing the number of communications by the communication cycle. The nodes are sorted from high to low according to the communication frequency, and the top several pairs of node pairs with the highest communication frequency are extracted as the preferred node pair set. A communication frequency distribution diagram is generated based on the node frequency data in the preferred node pair set, the dense distribution range of the node pair communication frequency is determined, the node pairs in the dense distribution range are preferentially marked, and the optimal selection list of node pairs is generated.

[0075] S113: Based on the optimal selection list of nodes, combined with the communication frequency and time synchronization matching degree between nodes, the formula is adopted:

[0076]

[0077] Calculate the transmission success rate and generate the node identification allocation result;

[0078] Among them, S represents the transmission success rate, α represents the weight factor of synchronization delay, β represents the weight factor of communication frequency, γ represents the correction coefficient of node association, D represents the synchronization delay value, F represents the communication frequency, and L represents the association index of the device;

[0079] The benefit of the formula is that by combining the three parameters of synchronization delay D, communication frequency F and device correlation index L, the weight factors α, β and γ of delay, frequency and correlation are reasonably set. By comprehensively adjusting the weights of the three parameters, the dynamic optimization of the transmission success rate is achieved, and the flexibility and accuracy of the model are improved.

[0080] Take a certain node pair, assume that the synchronization delay D = 0.02 seconds, the communication frequency F = 150 times / minute, the device correlation index L = 0.8, and set the weight factors α = 0.5, β = 0.3, and γ = 0.2 according to the actual monitored environment;

[0081] Substitute into the formula to calculate:

[0082] Calculate the first part: exp(-0.5 0.02-0.3 150) = exp(-45.01) ≈ 2.55 × 10 -20 ;

[0083] therefore:

[0084] Calculate the second part:

[0085] Multiply the two results: S≈1·0.862=0.862;

[0086] The result shows that the transmission success rate of the node pair is 86.2%, indicating that the node pair has high reliability in data transmission. Combining the transmission success rate data of the node pair can further optimize the node identification allocation result, and an efficient communication network can be established by selecting node pairs with higher transmission success rates.

[0087] See also Figure 3 , the specific steps for obtaining the communication path efficiency ranking list are:

[0088] S211: calling the node pair identifier in the node identifier allocation result, using the cloud platform data processing capability to retrieve the communication records associated with the node pair, extracting the communication timestamp data, identifying the communication delay between the nodes, and integrating the node pairs that meet the conditions by comparing with the preset delay threshold to obtain a list of preferred communication routes;

[0089] The communication data associated with the node pairs are filtered out from the communication records of the cloud platform. The start and end timestamps of the timestamp information record file are read step by step. The time difference calculation is performed for each node pair to obtain the communication delay value. All communication delay values ​​are accumulated one by one to form the total communication delay value of the path. The total delay result of each path is compared with the set delay threshold in turn, and the paths with total delay values ​​less than the set threshold are filtered out and added to the valid path set. For the paths that fail the screening, the corresponding node pairs are marked as unqualified and removed from the set. After completing the delay calculation and screening of all node pairs, a list of preferred communication routes is generated. At the same time, the filtered path identifiers and corresponding delay values ​​are recorded for subsequent use. The results of the valid path set support the subsequent transmission efficiency analysis and path sorting.

[0090] S212: Analyze the data transmission success rate and loss rate according to each path in the preferred communication route list, using the formula:

[0091]

[0092] Calculate the path transmission efficiency score and generate the path transmission efficiency analysis result;

[0093] Where E represents the path transmission efficiency score, T s Represents the number of successfully transmitted packets, T t Represents the total number of packets, L r represents the rate of lost packets, H represents the communication delay, and λ is the weight factor of the impact of delay on efficiency;

[0094] The benefit of the formula is that by combining the transmission success rate T s / T t , loss rate L r , delay H and adjustment weight factor λ, realize the comprehensive evaluation of path transmission performance, which helps to optimize path selection to improve network communication efficiency and reliability;

[0095] Calculate a path in the valid path set and set the following parameters: the number of successfully transmitted data packets T s =9500, total number of packets T t =10000, loss rate L r =0.05(by 1-T s / T t Calculated), path delay H = 0.02, delay weight factor λ = 0.1;

[0096] The first part of the calculation:

[0097]

[0098] 1-L r =1-0.05=0.95;

[0099]

[0100] The second part of the calculation:

[0101]

[0102]

[0103] Final transmission efficiency: E = 0.9025 + 5 = 5.9025

[0104] The result shows that the transmission efficiency score of this path is 5.9025. The high transmission efficiency indicates that this path is suitable for priority consideration as a communication path. Combined with the efficiency scores of other paths, the path selection and sorting can be further optimized.

[0105] S213: calling the path transmission efficiency analysis result, sorting the recorded transmission efficiency data, arranging the paths according to the order of data transmission efficiency, and generating a communication path efficiency sorting list;

[0106] By reading the transmission efficiency value of each path in turn, sorting them from high to low according to the transmission efficiency, storing the sorted path identification and the corresponding efficiency value in the communication path efficiency sorting table, and combining the sorted data, further correcting the sorting by setting the path stability index, re-marking the path with path stability lower than the set threshold as a secondary priority path and adjusting the sorting, reconfirming the compliance of the transmission efficiency score with the stability index after the sorting adjustment, and selecting the paths with high transmission efficiency and good stability through cross-validation of the efficiency value and the stability score, finally forming a communication path efficiency sorting list, which includes the path identification, transmission efficiency value and stability score, and completing the path selection optimization in combination with the sorting results, to provide support for subsequent communication optimization decisions.

[0107] See also Figure 4 ,The specific steps for obtaining the node task bandwidth allocation data set are:

[0108] S221: Based on the communication path efficiency ranking list, calculate the efficiency value of each path, compare the relevance of the task path, record the correspondence between the task and the path, extract the path bandwidth and task bandwidth requirements, and obtain the communication task path priority allocation list;

[0109] The real-time throughput, delay and packet loss rate of the path are taken as key input parameters. The path efficiency value is calculated by comparing the communication indicators of each path. The efficiency value is obtained by comprehensive weighted calculation after real-time monitoring of the path throughput (measured in Mbps), delay (measured in ms) and packet loss rate (percentage). While calculating the path efficiency value, the relevance of the task path is compared. By retrieving the communication requirements of the task and the characteristics of the path resources, the matching degree is judged, and the corresponding relationship between the task and the path is established. The maximum bandwidth value of the path and the bandwidth requirement of the current communication task are further extracted, and the matching situation of the two is recorded. For the path that cannot meet the bandwidth requirement, it is marked as a secondary path. After obtaining the relevance of the task path, a communication task path priority allocation list is constructed according to the task priority and the path efficiency value.

[0110] S222: Based on the communication task path priority allocation list, make adjustments according to the difference between the task bandwidth requirement and the path bandwidth, allocate the task traffic that exceeds the bandwidth capacity to the low priority path, check the path and task data, and obtain the task bandwidth allocation adjustment list;

[0111] First, extract the task demand bandwidth and the available bandwidth of the corresponding path from the priority allocation result, and classify the difference between the task demand and the path bandwidth into three situations: fully satisfied, partially satisfied, and unsatisfied. For tasks that exceed the path bandwidth capacity, extract the excess bandwidth flow, and re-search the path resources that meet the bandwidth demand in the low-priority path, while ensuring that the path efficiency meets the minimum available standard. After completing the bandwidth adjustment and allocation, re-check all task flows to check whether the adjusted path meets the task requirements and is consistent with the communication task priority. If there is still remaining available bandwidth on the adjusted path, allocate the remaining resources to tasks with lower priority but not fully satisfied. Finally, record the adjusted task and path allocation to form a task bandwidth allocation adjustment list, and mark the total flow and task occupancy ratio of each path.

[0112] S223: based on the task bandwidth allocation adjustment list, sequentially adjust the transmission task of each communication node, allocate the adjusted data traffic according to the data traffic occupancy ratio in the bandwidth allocation list, summarize the task data and bandwidth allocation information of the communication nodes, and generate a node task bandwidth allocation data set;

[0113] First, the tasks of each communication node are classified and sorted, and the data traffic of high-priority tasks is allocated first. The node's bandwidth resources are redistributed according to the task data traffic occupancy ratio to ensure that the bandwidth allocation is consistent with the allocation data in the list. The communication data of each node is checked one by one, and the traffic is reallocated according to the occupancy ratio specified in the list to avoid resource waste or path overload. For nodes with multi-path task transmission, the task traffic distribution between paths must be balanced to ensure that the multi-path bandwidth allocation ratio is consistent with the task requirements. After the allocation is completed, the task data, path allocation information, bandwidth utilization and other contents of each node are summarized to generate a node task bandwidth allocation data set. This data set includes the task traffic details, path allocation results and bandwidth utilization of each communication node for subsequent optimization and monitoring.

[0114] See also Figure 5 ,The specific steps for obtaining high-temperature data storage distribution data are:

[0115] S311: Based on the node task bandwidth allocation data set, extract the access records associated with the device, count the data access frequency of each device, including the corresponding access log, calculate the timestamp difference of each log, analyze the frequency and time interval data, and generate the access heat index of the device data;

[0116] The access records associated with each device ID are extracted from the data set, and the access frequency and timestamp of each record are counted. The data access frequency of each device is determined by calculating the time interval of each access record. The access frequency and time interval are stored in the statistical table with the device ID as the key. The data is further used to analyze the access heat. The device ID and timestamp information are read, and the difference between consecutive access timestamps is stored as the access interval. The number of accesses is accumulated and counted for all records under the same device ID, and the total access time is recorded. The preliminary heat value of each record is calculated through the operational relationship between the access frequency and the time interval, and the heat values ​​are merged into a device access heat statistics table. The extreme heat values ​​are smoothed through normalization and the overall heat distribution is generated, and finally the access heat index of the device data is generated.

[0117] S312: Based on the access heat index of the device data, a heat threshold is set and the data is classified, and data above the threshold is identified as high heat data, using the formula:

[0118]

[0119] Calculate the heat value of the data item and generate high heat data classification results;

[0120] Among them, H represents the heat value of the data item, Y represents the access frequency of the target data, ΔT represents the average access interval, M is the heat weight factor, and R represents the bandwidth allocation ratio of the associated device;

[0121] The benefit of the formula is that by combining the access frequency Y, time interval ΔT, heat weight factor M and bandwidth ratio R, the frequency of device use and its resource occupancy can be evaluated, providing a reliable basis for the accurate identification of high-heat data;

[0122] Detailed explanation of the formula and the process of formula calculation and derivation:

[0123] Take a device data item, and set the parameters as follows: access frequency Y = 100, average access interval ΔT = 10, bandwidth allocation ratio R = 0.2, and heat weight factor M = 0.5. Substitute into the formula:

[0124] The first part of the calculation:

[0125] The second part of the calculation:

[0126]

[0127]

[0128] Final heat value: H = 10 + 0.224 = 10.224;

[0129] The result shows that the access heat value of this device data is 10.224, which is higher than the set heat threshold. This data is marked as high-heat data and enters the next storage optimization process.

[0130] S313: Based on the high-heat data classification result, migrate the data marked as high-heat to a high-performance storage medium, update the storage location information table, including the data item identifier and the new storage medium location, and obtain high-heat data storage distribution data;

[0131] Call the high-heat data marking results, filter data records above the threshold from the device access heat index, match the record identification information with the original storage location one by one, transfer the data content to the preset high-performance storage medium, update the new storage location to the storage location table, and record the identification, original location and current storage medium type of the migrated data in the storage distribution table. Reallocate storage resources in combination with the bandwidth allocation ratio and heat index. The storage distribution table contains the identification of each high-heat data and its corresponding latest storage medium, and finally obtains the high-heat data storage distribution data.

[0132] See also Figure 6 , the specific steps for obtaining data storage optimization results are:

[0133] S321: based on the high-heat data storage distribution data, extract the heat parameters of the storage unit in the storage location, count the heat data in segments according to the storage capacity usage rate, mark the high-heat area and extract the storage capacity distribution, integrate the distribution characteristics and capacity information of the storage unit, and obtain the high-heat data storage distribution characteristics;

[0134] By real-time monitoring of the read and write frequency, data request times and access duration of each storage unit, the access volume of each storage unit within a set period is recorded and its average value is calculated to obtain the heat parameters of the storage unit. According to the storage capacity utilization rate, the heat data of all storage units are segmented and statistically analyzed, and the heat data is divided into three intervals: high, medium and low. Each interval is further refined according to the capacity utilization ratio of the storage unit. When marking high-heat areas, the storage units are classified according to the set heat threshold, and the storage units above the threshold are marked as high-heat areas. At the same time, the storage capacity distribution of high-heat units is extracted, including the distribution of total capacity, used capacity and remaining capacity. The distribution characteristics, capacity information and heat data of each storage unit are integrated to generate high-heat data storage distribution characteristics results, forming a comprehensive view reflecting the resource utilization and data access characteristics of each storage unit, providing a basis for subsequent optimization of storage layout.

[0135] S322: Based on the high-heat data storage distribution characteristics, select the storage location corresponding to the low-heat data, analyze the remaining capacity of each location and the storage demand of the low-heat data, match the storage demand with the remaining capacity, adjust the storage location of the low-heat data according to the compression potential, and obtain the low-heat data storage allocation data set;

[0136] By extracting storage units below the set heat threshold from the storage distribution characteristics, the low-heat data corresponding to the storage units are screened out. When analyzing the remaining capacity of each low-heat storage location, its current remaining capacity parameters are extracted. Combined with the storage demand of the low-heat data, the remaining capacity is matched according to the data volume size. For low-heat data that cannot be directly stored, it is distributed to multiple storage locations after sharding. In order to improve storage efficiency, combined with the analysis of compression potential, the compressibility ratio of the low-heat data is calculated according to data redundancy, structural characteristics and compression ratio parameters, and the storage location of the data is adjusted to make the storage units with higher remaining capacity in the low-heat data storage data set. After the matching is completed, the new storage location of each low-heat data and its corresponding capacity occupancy are recorded to generate a low-heat data storage allocation data set.

[0137] S323: Based on the low-heat data storage allocation data set, a compression operation is performed on the low-heat data according to the compression potential parameter, the compressed data is reallocated to a matching storage location, the state and distribution parameters of the storage unit are updated, the overall storage data is counted, and a data storage optimization result is constructed;

[0138] The compression ratio is calculated for each low-heat data, and the compressible volume is estimated and the compression operation is performed based on its redundancy and compression algorithm characteristics. The compressed data is reallocated to the corresponding storage unit according to the storage location parameters in the data allocation data set. The status of all reallocated storage locations is updated, including the remaining capacity, used capacity and re-statistics of heat data. To ensure data consistency, it is verified whether the compressed data is completely allocated to the new storage location, and the compression ratio and storage location changes are recorded. When counting the overall storage data, the distribution parameters of each storage unit are summarized, including the total capacity, usage rate and heat status. The data storage optimization results are generated based on the reallocated data. The optimization results can reflect the overall heat distribution changes of the storage unit and the compression and re-layout of low-heat data, laying the foundation for further improving the efficiency of the storage system.

[0139] See also Figure 7 ,The specific steps for obtaining dynamic retrieval results of high-frequency IoT data are:

[0140] S411: calling the high-frequency data identifier and the corresponding access record in the data storage optimization result, extracting the access timestamp, associating the timestamp data with the high-frequency identifier, identifying the access interval according to the access time and counting the access frequency, and generating a high-access frequency data set;

[0141] Filter high-frequency data access records and make detailed statistics on the access time of the records. The main purpose is to identify data sets with high access frequency, extract the access timestamp of each data from the optimization results, and calculate the time interval. The time interval is classified and stored according to the unique identifier of the data for the next step of processing. Perform frequency analysis on the classified time interval data, use statistical methods such as standard deviation and mean calculation to find data sets with access frequencies higher than the average level, further mark and store the data sets, ensure that the marks correspond to the statistical results in the original data table, thereby generating a high-access frequency data set. The final data set will be used for subsequent indexing and storage optimization operations.

[0142] S412: Based on the high access frequency data set, the access sequence is adjusted through distribution optimization, using the formula:

[0143]

[0144] Optimize the access priority in the index path, dynamically adjust the index level order, and obtain the optimized path priority value;

[0145] Among them, P opt Indicates the optimized path priority value, Q i represents the access frequency of the i-th data, H i represents the number of nodes in the access path, W i is a weight factor set according to the device priority;

[0146] The usefulness of the formula is that by integrating the access frequency F i , path length H i and data weight W i , improving the efficiency of data access and the responsiveness of dynamic retrieval;

[0147] In the process of determining the optimized access order, firstly, based on the high-heat data set, representative data records are selected and their path priorities are calculated. For example, three data records are selected for calculation, and the parameters of the records are set as follows:

[0148] The access frequency of record 1 is Q1=50, the path length is H1=4, and the data weight is W1=1.2;

[0149] Record 2 has an access frequency of Q2 = 40, a path length of H2 = 5, and a data weight of W2 = 1.0;

[0150] The access frequency of record 3 is Q3=60, the path length is H3=3, and the data weight is W3=1.5;

[0151] The path priority of each record is determined by the following calculation:

[0152] Priority of record 1:

[0153] Priority of record 2:

[0154] Priority of record 3:

[0155] These calculation results are added together to get the total path priority: P opt =15+8+30=53;

[0156] The results show that the optimized path priority value is 53, indicating that the optimized data index and access order can significantly improve the efficiency and response speed of data retrieval, further enhance the performance of the data storage system and the reliability of data access. This optimization method is particularly critical in IoT data processing and can ensure efficient and accurate data processing and analysis.

[0157] S413: calling the optimized path priority value, associating the dynamic retrieval logic of high-heat data with the index optimization order, and optimizing the access order of high-heat data to generate dynamic retrieval results of high-frequency IoT data;

[0158] By dynamically adjusting the index path and optimizing the data retrieval order, we first match the data access records with the new index path, give priority to those data with high access frequency, and apply dynamic retrieval strategies to the data to ensure that the required information can be quickly located during the data retrieval process to determine the most appropriate access path for each data. Through actual path adjustment operations, we combine the data access frequency with path optimization to generate a detailed statistical report. The report records the position of each data in the new access path and the access speed improvement due to path optimization. The optimization results are compared with the original access records, and the changes in retrieval efficiency before and after optimization are analyzed to verify the effectiveness of the path optimization strategy and generate dynamic retrieval results for high-frequency IoT data. This not only improves the speed of data retrieval, but also optimizes the data storage structure, providing strong support for the dynamic retrieval of high-frequency data.

[0159] The Internet of Things data processing system based on a cloud platform is used to execute the above-mentioned Internet of Things data processing method based on a cloud platform, and the system includes:

[0160] The node identification allocation module compares the timestamp difference and delay parameters based on the multimodal data uploaded by the IoT devices, analyzes the frequency of associated communications between device nodes, extracts high-frequency node pairs, calculates the transmission success rate based on the data packet transmission records, and establishes the node identification allocation results;

[0161] The communication path optimization module analyzes the communication delay and transmission success rate of the device node path based on the node identification allocation result, selects the path according to the loss rate, extracts the optimal path data, rearranges the path bandwidth allocation, adjusts the task data flow transmission order, and generates the task bandwidth allocation result;

[0162] The data storage distribution module counts the device data access frequency and time interval based on the task bandwidth allocation results, filters the access frequency data sets and merges and allocates them to high-speed storage media, analyzes the compression rate of low-frequency data and allocates storage locations, and generates data storage optimization results;

[0163] Based on the data storage optimization results, the high-frequency data retrieval module extracts the high-frequency data storage path and original access records, analyzes the path access frequency and index level, optimizes the index order, adjusts the high-frequency data access path, and generates dynamic retrieval results for high-frequency IoT data.

[0164] The above are only preferred embodiments of the present invention and are not intended to limit the present invention in other forms. Any technician familiar with the profession may use the technical contents disclosed above to change or modify them into equivalent embodiments with equivalent changes and apply them to other fields. However, any simple modification, equivalent change and modification made to the above embodiments based on the technical essence of the present invention without departing from the technical solution of the present invention still falls within the protection scope of the technical solution of the present invention.

Claims

1. A method for processing Internet of Things data based on a cloud platform, characterized in that: The following steps are involved: S1: Based on the multimodal data uploaded by IoT devices, analyze the device node data synchronization delay and timestamp matching degree, count the node communication frequency and device association, select the optimal communication node pair, and perform identity allocation, calculate the data packet transmission success rate, and form the node identity allocation result; S2: Based on the node identification allocation result, analyze the path communication delay, data transmission success rate and loss rate, select the path data with the best transmission efficiency, generate a communication path efficiency ranking list, adjust the transmission order of the communication tasks in the group through bandwidth allocation, reallocate the data flow, and generate a node task bandwidth allocation data set; S3: Based on the node task bandwidth allocation data set, count the access popularity of device data, identify the access frequency and time interval of data, store high-heat data in high-performance storage media, obtain high-heat data storage distribution data, perform storage compression on low-heat data, allocate storage space based on compression rate and remaining capacity, and establish data storage optimization results; S4: Based on the data storage optimization results, extract high-frequency data access records, count data sets with high access frequencies, optimize access order through multi-layer index node path distribution, adjust index level order, optimize dynamic retrieval of high-frequency data, and establish dynamic retrieval results for high-frequency IoT data.

2. The method for processing Internet of Things data based on a cloud platform according to claim 1, characterized in that: The steps for obtaining the node identification allocation result are specifically as follows: S111: Based on the multimodal data uploaded by the IoT device, extract the timestamp uploaded by the device, calculate the time difference of the data uploaded by each device node, compare the time difference with the set synchronization threshold, select the node pairs whose time difference meets the threshold, and generate the time synchronization matching index; S112: Based on the time synchronization matching index, select node pairs whose time synchronization matching degree meets the requirement, count the communication frequency between the nodes, sort the communication frequency of the node pairs, and generate an optimal selection list of node pairs; S113: Based on the optimal selection list of node pairs, combined with the communication frequency and time synchronization matching degree between nodes, the formula is adopted: Calculate the transmission success rate and generate the node identification allocation result; Among them, S represents the transmission success rate, α represents the weight factor of synchronization delay, β represents the weight factor of communication frequency, γ represents the correction coefficient of node association, D represents the synchronization delay value, F represents the communication frequency, and L represents the association index of the device.

3. The method for processing Internet of Things data based on a cloud platform according to claim 2, characterized in that: The steps for obtaining the communication path efficiency ranking list are specifically as follows: S211: calling the node pair identifier in the node identifier allocation result, using the cloud platform data processing capability to retrieve the communication records associated with the node pair, extracting the communication timestamp data, identifying the communication delay between the nodes, and integrating the node pairs that meet the conditions by comparing with the preset delay threshold to obtain a list of preferred communication routes; S212: Analyze the data transmission success rate and loss rate according to each path in the preferred communication route list, using the formula: Calculate the path transmission efficiency score and generate the path transmission efficiency analysis result; Where E represents the path transmission efficiency score, T s Represents the number of successfully transmitted packets, T t Represents the total number of packets, L r represents the rate of lost packets, H represents the communication delay, and λ is the weight factor of the impact of delay on efficiency; S213: calling the path transmission efficiency analysis result, sorting the recorded transmission efficiency data, arranging the paths according to the order of data transmission efficiency, and generating a communication path efficiency sorting list.

4. The method for processing Internet of Things data based on a cloud platform according to claim 3, characterized in that: The steps for obtaining the node task bandwidth allocation data set are specifically as follows: S221: Based on the communication path efficiency ranking list, calculate the efficiency value of each path, compare the relevance of the task paths, record the correspondence between the tasks and the paths, extract the path bandwidth and the task bandwidth requirements, and obtain a communication task path priority allocation list; S222: Based on the communication task path priority allocation list, make adjustments according to the difference between the task bandwidth requirement and the path bandwidth, allocate the task traffic exceeding the bandwidth capacity to the low priority path, check the path and task data, and obtain the task bandwidth allocation adjustment list; S223: Based on the task bandwidth allocation adjustment list, sequentially adjust the transmission task of each communication node, allocate the adjusted data traffic according to the data traffic occupancy ratio in the bandwidth allocation list, summarize the task data and bandwidth allocation information of the communication nodes, and generate a node task bandwidth allocation data set.

5. The method for processing Internet of Things data based on a cloud platform according to claim 4, characterized in that: The steps for obtaining the high-temperature data storage distribution data are specifically as follows: S311: Based on the node task bandwidth allocation data set, extract the access records associated with the device, count the data access frequency of each device, including the corresponding access log, calculate the timestamp difference of each log, analyze the frequency and time interval data, and generate the access heat index of the device data; S312: Based on the access heat index of the device data, a heat threshold is set and the data is classified, and data above the threshold is identified as high heat data, using the formula: Calculate the heat value of the data item and generate high heat data classification results; Among them, H represents the heat value of the data item, Y represents the access frequency of the target data, ΔT represents the average access interval, M is the heat weight factor, and R represents the bandwidth allocation ratio of the associated device; S313: Based on the high-heat data classification result, migrate the data marked as high-heat to a high-performance storage medium, update the storage location information table, including the data item identifier and the new storage medium location, and obtain high-heat data storage distribution data.

6. The method for processing Internet of Things data based on a cloud platform according to claim 5, characterized in that: The steps for obtaining the data storage optimization result are specifically as follows: S321: Based on the high-heat data storage distribution data, extract the heat parameters of the storage unit in the storage location, count the heat data in segments according to the storage capacity usage rate, mark the high-heat area and extract the storage capacity distribution, integrate the distribution characteristics and capacity information of the storage unit, and obtain the high-heat data storage distribution characteristics; S322: Based on the high-heat data storage distribution characteristics, select storage locations corresponding to low-heat data, analyze the remaining capacity of each location and the storage demand of low-heat data, match the storage demand with the remaining capacity, adjust the storage location of low-heat data according to compression potential, and obtain a low-heat data storage allocation data set; S323: Based on the low-heat data storage allocation data set, a compression operation is performed on the low-heat data according to the compression potential parameter, the compressed data is reallocated to a matching storage location, the state and distribution parameters of the storage unit are updated, the overall storage data is counted, and a data storage optimization result is constructed.

7. The method for processing Internet of Things data based on a cloud platform according to claim 6, characterized in that: The steps for obtaining the dynamic search results of the high-frequency Internet of Things data are specifically as follows: S411: calling the high-frequency data identifier and the corresponding access record in the data storage optimization result, extracting the access timestamp, associating the timestamp data with the high-frequency identifier, identifying the access interval according to the access time and counting the access frequency, and generating a high-access frequency data set; S412: Based on the high access frequency data set, the access sequence is adjusted by distribution optimization, using the formula: Optimize the access priority in the index path, dynamically adjust the index level order, and obtain the optimized path priority value; Among them, P opt Indicates the optimized path priority value, Q i represents the access frequency of the i-th data, H i represents the number of nodes in the access path, W i is a weight factor set according to the device priority; S413: Call the optimized path priority value, associate the dynamic retrieval logic of high-heat data with the index optimization order, and optimize the access order of high-heat data to generate dynamic retrieval results for high-frequency IoT data.

8. An Internet of Things data processing system based on a cloud platform, characterized in that: According to the method for processing Internet of Things data based on a cloud platform according to any one of claims 1 to 7, the system comprises: The node identification allocation module compares the timestamp difference and delay parameters based on the multimodal data uploaded by the IoT devices, analyzes the frequency of associated communications between device nodes, extracts high-frequency node pairs, calculates the transmission success rate based on the data packet transmission records, and establishes the node identification allocation results; The communication path optimization module analyzes the communication delay and transmission success rate of the device node path based on the node identification allocation result, optimizes the path according to the loss rate, extracts the optimal path data, rearranges the path bandwidth allocation, adjusts the task data flow transmission order, and generates the task bandwidth allocation result; The data storage distribution module counts the device data access frequency and time interval based on the task bandwidth allocation result, selects the access frequency data set and merges and allocates it to the high-speed storage medium, analyzes the compression rate of the low-frequency data and allocates the storage location, and generates the data storage optimization result; Based on the data storage optimization results, the high-frequency data retrieval module extracts the high-frequency data storage path and original access records, analyzes the path access frequency and index level, optimizes the index order, adjusts the high-frequency data access path, and generates dynamic retrieval results for high-frequency IoT data.

Images