A lightweight data collection method for physical environment

Abstract

The present invention relates to the field of environmental information and discloses a lightweight data collection method for a physical environment. The method automatically triggers the collection of physical environment data by setting a preset time interval and a delayed response time, detects abnormal conditions during the collection, monitors the changes in the data flow during data collection in real time, preliminarily analyzes the degree of deviation in the data collection, and detects whether there is a collection error in the data collection. The error detection result is generated, and the abnormal values ​​of the physical environment data are detected and fed back. The data is then lightweight processed. The data transmission process of the lightweight processed physical environment data is analyzed to monitor and optimize the data transmission process in real time. Based on the results of the data transmission optimization, the storage of the physical environment data is optimized, which is conducive to quickly responding to and eliminating data collection failures, realizing fast and accurate collection of multiple parameters of the physical environment, optimizing the data transmission and storage process, and reducing the data maintenance cost.

Description

Technical Field

[0001] The present invention relates to the field of environmental information technology, and more particularly to a method for collecting lightweight data on a physical environment. Background Art

[0002] In many physical environment monitoring scenarios, especially in remote or mobile environments, data acquisition equipment often faces resource constraints, such as limited power, computing power, and bandwidth. These constraints require data acquisition methods to be efficient and resource-saving. Due to the high deployment and maintenance costs of data acquisition equipment, especially in large-scale or long-term monitoring projects, the use of IoT technology and the fusion and processing of data from multiple sensors can reduce costs and improve economic benefits.

[0003] However, the above process still has the following disadvantages:

[0004] First, existing lightweight data collection methods lack a delay detection and analysis process for data collection, making it impossible to promptly detect abnormal problems that arise during the data collection process. This reduces data collection efficiency and is not conducive to rapid response and troubleshooting of data collection failures.

[0005] Second, existing lightweight data collection methods lack the ability to monitor changes in the data stream during data collection in real time, making it impossible to promptly detect errors in the data collection process. This results in an inability to understand the distribution of outliers in the collected data, and thus an inability to automatically determine the type of errors generated by data collection.

[0006] Third, existing lightweight data collection methods lack automatic optimization analysis of data transmission and storage, and are unable to optimize and adjust data transmission and storage based on the analysis results, resulting in increased data maintenance costs. Summary of the Invention

[0007] In order to overcome the above-mentioned defects of the prior art, the present invention provides a lightweight data collection method for a physical environment to solve the problems existing in the above-mentioned background technology.

[0008] The present invention provides the following technical solution: a method for collecting lightweight data about a physical environment, comprising:

[0009] S1: Automatically triggering physical environment data collection by setting a preset time interval on the data collection device and setting a delayed response time for data collection;

[0010] S2: By reading the collected physical environment data from the data acquisition device and monitoring the data flow changes during data acquisition in real time, a preliminary analysis of the deviation degree of data acquisition is performed to obtain the acquisition deviation degree coefficient, which is used to detect whether there is acquisition error in the data acquisition;

[0011] S3: Based on the analysis results of data collection errors, detect abnormal values ​​of physical environment data and provide feedback on the detection results;

[0012] S4: used to pre-process the collected physical environment data and perform lightweight processing on the pre-processed physical environment data;

[0013] S5: Optimize and analyze the data transmission process based on the lightweight physical environment data and calculate the network congestion control coefficient for real-time monitoring and optimization of the data transmission process;

[0014] S6: Based on the results of data transmission optimization, the storage of physical environment data is optimized and analyzed to obtain a storage optimization index, thereby optimizing data storage in real time.

[0015] Preferably, the S1 is used to automatically trigger the collection of physical environment data and detect abnormal delays during data collection. The data collection device selects a multi-channel sensor according to the measurement object to collect physical environment data in parallel, and uses an embedded system to configure a timer based on the collection frequency requirements of the data collection device. A specific time interval is set in the timer, and an interrupt service routine is triggered when the timer overflows, thereby performing regular collection of physical environment data, recording the trigger start time and end time of each data collection, and calculating the actual time of each data collection based on the trigger start time and end time of the data collection. The specific calculation formula is: , by taking the actual time of each data collection and delayed response time Comparison is used to detect whether there are abnormal problems in data collection. When the system fails to complete the data collection within the preset delay time, it will immediately enter the warning state and send a warning reminder of data collection delay to the management personnel, thereby eliminating the abnormal problem of data collection. When , it indicates that there is no abnormal problem caused by delay in data collection, and further analysis of the deviation of data collection is carried out.

[0016] Preferably, the S2 reads the data stream from the data acquisition device in real time and records the timestamp of the data acquisition, and then performs preprocessing operations on the read physical environment data. The specific operation process includes data cleaning, data conversion and data smoothing. By using statistical methods to analyze the change trend of the data stream within a period of acquisition time, and calculating the acquisition deviation coefficient, the acquisition deviation coefficient is used to preliminarily detect whether there is an acquisition error in the data acquisition process;

[0017] The specific analysis method of the acquisition deviation coefficient is as follows:

[0018] Step S211: Read the data points of a certain physical environment data collected within a period of time and calculate the average value of a certain physical environment parameter. ,in, Indicates the The average value of the physical environment parameters, Indicates the The physical environment parameters measurements, Indicates the total number of measurement values ​​read within a period of time;

[0019] Step S212: Based on the average value of a physical environment parameter calculated over a period of time, the standard deviation corresponding to the physical environment parameter is calculated as follows: ;

[0020] Step S213: Based on the analysis of the average value of a certain physical environment parameter and the standard deviation corresponding to the certain physical environment parameter, the acquisition deviation coefficient is comprehensively calculated as follows: ,in, Indicates the The collection deviation coefficient of each physical environment parameter;

[0021] By taking the collection deviation coefficient Deviation threshold from the preset value Compare to determine whether there is a collection error in the data collection of a physical environment parameter. At the same time, it is used to perform parallel error detection on all collected physical environment parameters. If the collection deviation coefficient Preset deviation threshold , indicating that the data collection of the physical environment parameters is normal, and continue to execute S4 to process the data lightweight. If the collection deviation coefficient Preset deviation threshold , indicating that there is an error in the data collection of the physical environment parameter, the collection process of the physical environment data is immediately stopped, and all physical environment parameters with collection errors are screened out, and S3 is further executed to detect the data collection abnormal value of the physical environment parameter.

[0022] Preferably, the S3 is based on analyzing the abnormal values ​​of the physical environment parameters detected to have acquisition errors, and calculating the deviation between the actual measured value and the expected value of each data point of a physical environment parameter. The specific calculation formula is: ,in, Indicates the The physical environment parameters measurements, Indicates the The expected value of a physical environment data; the deviation between the actual measured value of each data point of a physical environment parameter and the expected value , and the acquisition deviation coefficient Compare to detect all the abnormal values ​​of a certain physical environment parameter collected within a period of time. , then the detected data point is not an outlier, and continue to detect the next data point. If , then the detected data point is an outlier, and all detected outliers are marked as outliers. Then, based on the distribution of the deviation value, the error type generated is determined, including systematic error and random error. At the same time, the outlier detection result and the collection error type are warned, and all detected collection outliers are sent to the management terminal.

[0023] Preferably, the operation process of preprocessing the collected physical environment data in S4 is: removing obviously erroneous, incomplete or irrelevant data, checking the consistency and accuracy of the data, and converting the data into a unified format through linear transformation. The specific conversion formula is: , Represents the original data, Represents the converted data, and Represents the transformation coefficient; the pre-processed physical environment data is subjected to dimensionality reduction processing by the principal component analysis method. The specific formula is: , represents the covariance matrix of the data, Represents the eigenvector of the covariance matrix, and then uses Huffman coding to compress the reduced dimensionality data.

[0024] Preferably, the S5 is used to use a network monitoring tool to collect network data during physical environment data transmission in real time, including data delay, packet loss rate and throughput, and calculate the network congestion control coefficient by analyzing the network data during data transmission. The specific calculation formula is: ,in, Indicates the throughput of data transmission. Indicates the maximum throughput of the network, Indicates the data delay during data transmission. Indicates the maximum acceptable delay of the network. Indicates the packet loss rate during data transmission;

[0025] By setting the network congestion control coefficient and the preset congestion threshold Compare to determine whether there is network congestion during data transmission. If , it indicates that the current data transmission network status is good, continue to monitor the data transmission process, if , it indicates that there is a network congestion problem in the current data transmission, and the network congestion problem is adjusted immediately, including reducing the data sending rate, adjusting the size of the congestion window according to the network congestion control coefficient, and changing the dynamic route. At the same time, the network congestion monitoring results are fed back to the management terminal.

[0026] Preferably, S6 analyzes and calculates the storage optimization index by collecting utilization, access logs, costs, and fault records of the current storage system;

[0027] The specific analysis method of the storage optimization index is as follows:

[0028] Step S611: Analyze and calculate the storage utilization , Indicates the actual amount of data stored. Indicates the total storage capacity;

[0029] Step S612: Analyze and calculate the data access efficiency to be , Indicates the number of valid accesses during the current data transmission. Indicates the total number of visits;

[0030] Step S613: Analyze and calculate the storage cost efficiency , represents the storage cost, Indicates the value of data;

[0031] Step S614: Analyze and calculate the data reliability. , Indicates the data loss rate;

[0032] Step S615: Based on the above analysis process, the storage optimization index is calculated as ,in, represents the weight coefficient;

[0033] By setting a storage optimization threshold , the storage optimization index Storage Optimization Thresholds Comparison is used to determine whether data storage needs to be optimized. , it indicates that the current data storage status is good and no storage optimization is required. Continue to monitor the storage status. If , it indicates that the current data storage status is poor, and data storage optimization measures are automatically triggered. At the same time, the data storage optimization results are sent to the management terminal for feedback.

[0034] Technical effects and advantages of the present invention:

[0035] The present invention automatically triggers the collection of physical environment data by setting a preset time interval on the data collection device, and sets a delayed response time for data collection. By reading the collected physical environment data from the data collection device, and monitoring the data flow changes during data collection in real time, and preliminarily analyzing the degree of deviation of data collection, it is detected whether there is a collection error in the data collection. According to the analysis results of the data collection error, the abnormal values ​​of the physical environment data are detected and the detection results are fed back. By preprocessing the collected physical environment data and performing lightweight processing on the preprocessed physical environment data, the data transmission process of the lightweight processed physical environment data is optimized and analyzed, and the network congestion control coefficient is calculated for real-time monitoring and optimization of the data transmission process. Based on the data transmission Based on the optimized results of data transmission, the storage of physical environment data is optimized and analyzed to obtain the storage optimization index, thereby optimizing data storage in real time; by setting the delay threshold of data collection, it is conducive to timely detecting abnormal problems that occur during the data collection process, and can quickly respond to and eliminate data collection failures, realize fast and accurate collection of multiple parameters of the physical environment, and improve the efficiency of data collection. By real-time monitoring of the changes in data flow during data collection, it is possible to timely discover the errors generated in the data collection process, understand the distribution of abnormal values ​​in the collected data, and automatically judge the error type generated by data collection. By automatically optimizing and analyzing data transmission and storage, it is possible to optimize and adjust data transmission and storage based on the analysis results, thereby reducing the cost of data maintenance. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 A diagram showing the steps of the method of the present invention.

[0037] Figure 2 This is a system structure diagram of the present invention. DETAILED DESCRIPTION

[0038] The technical solutions of the present invention will be clearly and completely described below in conjunction with the drawings in the present invention. In addition, the forms of the various structures described in the following embodiments are merely examples. The lightweight data acquisition method for a physical environment involved in the present invention is not limited to the various structures described in the following embodiments. All other implementations obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0039] like Figure 1 This embodiment provides a lightweight physical environment data collection method, including:

[0040] S1: Automatically trigger physical environment data collection by setting a preset time interval on the data collection device and setting a delayed response time for data collection.

[0041] In this embodiment, the S1 is used to automatically trigger the collection of physical environment data and detect abnormal delays during data collection. The data collection device selects multi-channel sensors according to the measurement object to collect physical environment data in parallel, and uses an embedded system to configure a timer based on the collection frequency requirements of the data collection device. A specific time interval is set in the timer, and an interrupt service routine is triggered when the timer overflows, thereby performing regular collection of physical environment data, recording the trigger start time and end time of each data collection, and calculating the actual time of each data collection based on the trigger start time and end time of the data collection. The specific calculation formula is: , by taking the actual time of each data collection and delayed response time Comparison is used to detect whether there are abnormal problems in data collection. When the system fails to complete the data collection within the preset delay time, it will immediately enter the warning state and send a warning reminder of data collection delay to the management personnel, thereby eliminating the abnormal problem of data collection. When , it indicates that there is no abnormal problem caused by delay in data collection, and further analysis of the deviation of data collection is carried out.

[0042] S2: By reading the collected physical environment data from the data acquisition device and monitoring the data flow changes during data acquisition in real time, the deviation degree of data acquisition is preliminarily analyzed to obtain the acquisition deviation degree coefficient, which is used to detect whether there is acquisition error in data acquisition.

[0043] In this embodiment, S2 reads the data stream from the data acquisition device in real time and records the timestamp of the data acquisition. It then performs preprocessing operations on the read physical environment data. The specific operation process includes data cleaning, data conversion and data smoothing. By using statistical methods to analyze the change trend of the data stream within a period of acquisition time and calculate the acquisition deviation coefficient, the acquisition deviation coefficient is used to preliminarily detect whether there is acquisition error in the data acquisition process.

[0044] The specific analysis method of the acquisition deviation coefficient is as follows:

[0045] Step S211: Read the data points of a certain physical environment data collected within a period of time and calculate the average value of a certain physical environment parameter. ,in, Indicates the The average value of the physical environment parameters, Indicates the The physical environment parameters measurements, Indicates the total number of measurement values ​​read within a period of time;

[0046] Step S212: Based on the average value of a physical environment parameter calculated over a period of time, the standard deviation corresponding to the physical environment parameter is calculated as follows: ;

[0047] Step S213: Based on the analysis of the average value of a certain physical environment parameter and the standard deviation corresponding to the certain physical environment parameter, the acquisition deviation coefficient is comprehensively calculated as follows: ,in, Indicates the The collection deviation coefficient of each physical environment parameter;

[0048] By taking the collection deviation coefficient Deviation threshold from the preset value Compare to determine whether there is a collection error in the data collection of a physical environment parameter. At the same time, it is used to perform parallel error detection on all collected physical environment parameters. If the collection deviation coefficient Preset deviation threshold , indicating that the data collection of the physical environment parameters is normal, and continue to execute S4 to process the data lightweight. If the collection deviation coefficient Preset deviation threshold , indicating that there is an error in the data collection of the physical environment parameter, the collection process of the physical environment data is immediately stopped, and all physical environment parameters with collection errors are screened out, and S3 is further executed to detect the data collection abnormal value of the physical environment parameter.

[0049] It should be specifically noted that the type of error in data collection can be determined based on the distribution of the deviation values. If the deviation values ​​of the data points are irregularly distributed, the sizes are evenly distributed, and the sizes of most deviation values ​​are close to CV, this indicates that the error in data collection may be a random error; if the deviation values ​​of the data points show that the deviation values ​​are continuously larger or smaller, and the direction of the deviation is consistent, then this indicates that the error in data collection may be a systematic error. Systematic errors are usually caused by factors such as improper equipment calibration, sensor failure, or incorrect measurement methods.

[0050] S3: Based on the analysis results of data collection errors, detect abnormal values ​​of physical environment data and feedback the detection results.

[0051] In this embodiment, S3 is based on analyzing the abnormal values ​​of the physical environment parameters detected with acquisition errors, and calculating the deviation between the actual measured value and the expected value of each data point of a physical environment parameter. The specific calculation formula is: ,in, Indicates the The physical environment parameters measurements, Indicates the The expected value of a physical environment data; the deviation between the actual measured value of each data point of a physical environment parameter and the expected value , and the acquisition deviation coefficient Compare to detect all the abnormal values ​​of a certain physical environment parameter collected within a period of time. , then the detected data point is not an outlier, and continue to detect the next data point. If , then the detected data point is an outlier, and all detected outliers are marked as outliers. Then, based on the distribution of the deviation value, the error type generated is determined, including systematic error and random error. At the same time, the outlier detection result and the collection error type are warned, and all detected collection outliers are sent to the management terminal.

[0052] S4: used to preprocess the collected physical environment data and perform lightweight processing on the preprocessed physical environment data.

[0053] In this embodiment, the operation process of preprocessing the collected physical environment data in S4 is to remove obviously erroneous, incomplete or irrelevant data, check the consistency and accuracy of the data, and convert the data into a unified format through linear transformation. The specific conversion formula is: , Represents the original data, Represents the converted data, and Represents the transformation coefficient; the pre-processed physical environment data is subjected to dimensionality reduction processing by the principal component analysis method. The specific formula is: , represents the covariance matrix of the data, Represents the eigenvector of the covariance matrix, and then uses Huffman coding to compress the reduced dimensionality data.

[0054] S5: Optimize and analyze the data transmission process based on the lightweight physical environment data, and calculate the network congestion control coefficient for real-time monitoring and optimization of the data transmission process.

[0055] In this embodiment, the S5 is used to use a network monitoring tool to collect network data during physical environment data transmission in real time, including data delay, packet loss rate and throughput, and calculate the network congestion control coefficient by analyzing the network data during data transmission. The specific calculation formula is: ,in, Indicates the throughput of data transmission. Indicates the maximum throughput of the network, Indicates the data delay during data transmission. Indicates the maximum acceptable delay of the network. Indicates the packet loss rate during data transmission;

[0056] By setting the network congestion control coefficient and the preset congestion threshold Compare to determine whether there is network congestion during data transmission. If , it indicates that the current data transmission network status is good, continue to monitor the data transmission process, if , it indicates that there is a network congestion problem in the current data transmission, and the network congestion problem is adjusted immediately, including reducing the data sending rate, adjusting the size of the congestion window according to the network congestion control coefficient, and changing the dynamic route. At the same time, the network congestion monitoring results are fed back to the management terminal.

[0057] S6: Based on the results of data transmission optimization, the storage of physical environment data is optimized and analyzed to obtain a storage optimization index, thereby optimizing data storage in real time.

[0058] In this embodiment, S6 analyzes and calculates the storage optimization index by collecting the utilization, access logs, costs, and fault records of the current storage system;

[0059] The specific analysis method of the storage optimization index is as follows:

[0060] Step S611: Analyze and calculate the storage utilization , Indicates the actual amount of data stored. Indicates the total storage capacity;

[0061] Step S612: Analyze and calculate the data access efficiency to be , Indicates the number of valid accesses during the current data transmission. Indicates the total number of visits;

[0062] Step S613: Analyze and calculate the storage cost efficiency , represents the storage cost, Indicates the value of data;

[0063] Step S614: Analyze and calculate the data reliability. , Indicates the data loss rate;

[0064] Step S615: Based on the above analysis process, the storage optimization index is calculated as ,in, represents the weight coefficient;

[0065] By setting a storage optimization threshold , the storage optimization index Storage Optimization Thresholds Comparison is used to determine whether data storage needs to be optimized. , it indicates that the current data storage status is good and no storage optimization is required. Continue to monitor the storage status. If , it indicates that the current data storage status is poor, and data storage optimization measures are automatically triggered. At the same time, the data storage optimization results are sent to the management terminal for feedback.

[0066] like Figure 2 The embodiment shown provides an implementation system corresponding to a physical environment lightweight data collection method, including a data collection trigger module, a data deviation detection module, a data anomaly detection module, a data lightweight processing module, a data transmission optimization module, and a data storage optimization module. The data collection trigger module is connected to the data deviation detection module, the data deviation detection module is connected to the data anomaly detection module, the data deviation detection module is connected to the data lightweight processing module, the data lightweight processing module is connected to the data transmission optimization module, and the data transmission optimization module is connected to the data storage optimization module.

[0067] The data acquisition trigger module automatically triggers the physical environment data acquisition by setting a preset time interval on the data acquisition device and setting a delayed response time for data acquisition;

[0068] The data deviation detection module reads the collected physical environment data from the data acquisition device and monitors the data flow changes during data acquisition in real time, preliminarily analyzes the deviation degree of data acquisition, and obtains the acquisition deviation degree coefficient for detecting whether there is acquisition error in data acquisition;

[0069] The data anomaly detection module detects abnormal values ​​of the physical environment data based on the analysis results of the data collection errors and feeds back the detection results;

[0070] The data lightweight processing module is used to preprocess the collected physical environment data and perform lightweight processing on the preprocessed physical environment data;

[0071] The data transmission optimization module optimizes and analyzes the data transmission process of the lightweight physical environment data and calculates the network congestion control coefficient for real-time monitoring and optimization of the data transmission process.

[0072] The data storage optimization module optimizes and analyzes the storage of physical environment data based on the results of data transmission optimization, obtains a storage optimization index, and thus optimizes data storage in real time.

[0073] Finally: The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

[0074] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

Claims

1. A lightweight data collection method for a physical environment, characterized in that: include: S1: Automatically triggering physical environment data collection by setting a preset time interval on the data collection device and setting a delayed response time for data collection; S2: By reading the collected physical environment data from the data acquisition device and monitoring the data flow changes during data acquisition in real time, a preliminary analysis of the deviation degree of data acquisition is performed to obtain the acquisition deviation degree coefficient, which is used to detect whether there is acquisition error in the data acquisition; S3: Based on the analysis results of data collection errors, detect abnormal values ​​of physical environment data and provide feedback on the detection results; S4: used to pre-process the collected physical environment data and perform lightweight processing on the pre-processed physical environment data; S5: Optimize and analyze the data transmission process based on the lightweight physical environment data and calculate the network congestion control coefficient for real-time monitoring and optimization of the data transmission process; S6: Based on the results of data transmission optimization, the storage of physical environment data is optimized and analyzed to obtain a storage optimization index, thereby optimizing data storage in real time. The S1 is used to automatically trigger the collection of physical environment data and detect abnormal delays during data collection. The data collection device selects multi-channel sensors according to the measurement object to collect physical environment data in parallel, and uses an embedded system to configure a timer based on the collection frequency requirements of the data collection device. A specific time interval is set in the timer, and an interrupt service routine is triggered when the timer overflows, thereby performing regular collection of physical environment data, recording the trigger start time and end time of each data collection, and calculating the actual time of each data collection based on the trigger start time and end time of the data collection. The specific calculation formula is Δt=t 结束 -t 开始 By comparing the actual time Δt of each data collection with the delayed response time t′, it is used to detect whether there are any abnormal problems in data collection. When Δt ≥ t′, it indicates that the data collection is not completed within the preset delay time, and the system immediately enters the warning state and issues a warning prompt of data collection delay to the management personnel, thereby eliminating the abnormal problems of data collection. When Δt < t′, it indicates that there are no abnormal problems caused by delay in data collection, and further analysis of data collection deviations is carried out; The S2 reads the data stream from the data acquisition device in real time and records the timestamp of the data acquisition. It then performs preprocessing operations on the read physical environment data. The specific operation process includes data cleaning, data conversion and data smoothing. By using statistical methods to analyze the change trend of the data stream within a period of acquisition time, and calculating the acquisition deviation coefficient, the acquisition deviation coefficient is used to preliminarily detect whether there is acquisition error in the data acquisition process; The specific analysis method of the acquisition deviation coefficient is as follows: Step S211: Read the data points of a certain physical environment data collected within a period of time and calculate the average value of a certain physical environment parameter. Among them, μ i represents the average value of the i-th physical environment parameter, x i，j represents the jth measurement value of the i-th physical environment parameter, and m represents the total number of measurement values ​​read within a period of time; Step S212: Based on the average value of a physical environment parameter calculated over a period of time, the standard deviation corresponding to the physical environment parameter is calculated as follows: Step S213: Based on the analysis of the average value of a certain physical environment parameter and the standard deviation corresponding to the certain physical environment parameter, the acquisition deviation coefficient is comprehensively calculated as follows: Among them, CV i Indicates the collection deviation coefficient of the i-th physical environment parameter; By taking the coefficient of variation CV i It is compared with the preset deviation threshold θ to determine whether there is a collection error in the data collection of a physical environment parameter. At the same time, it is used to perform parallel error detection on all collected physical environment parameters. If the collection deviation coefficient CV i ≤ the preset deviation threshold θ, indicating that the data collection of the physical environment parameter is normal, and continue to execute S4 to process the data lightweight. If the collection deviation coefficient CV i > the preset deviation threshold θ, indicating that there is an error in the data collection of the physical environment parameter, the collection process of the physical environment data is immediately stopped, and all physical environment parameters with collection errors are screened out, and S3 is further executed to detect abnormal values ​​in the data collection of the physical environment parameter; The S3 is based on analyzing the abnormal values ​​of the physical environment parameters detected with acquisition errors, and calculating the deviation between the actual measured value and the expected value of each data point of a physical environment parameter. The specific calculation formula is ΔP i =x i，j -x i，预期 , where x i，j represents the jth measurement value of the i-th physical environment parameter, x i，预期 Represents the expected value of the i-th physical environment data; by calculating the deviation ΔP between the actual measured value and the expected value of each data point of a physical environment parameter i , and the acquisition deviation coefficient CV i Compare to detect all the abnormal values ​​of a certain physical environment parameter collected over a period of time. If ΔP i ≤CV i , then the detected data point is not an outlier, and continue to detect the next data point. If ΔP i >CV i , then the detected data point is an outlier, and all detected outliers are marked as outliers. Then, based on the distribution of the deviation value, the error type generated is determined, including systematic error and random error. At the same time, the outlier detection result and the collection error type are warned, and all detected collection outliers are sent to the management terminal; The operation process of S4 for preprocessing the collected physical environment data is as follows: removing obviously erroneous, incomplete or irrelevant data, checking the consistency and accuracy of the data, and converting the data into a unified format through linear transformation. The specific conversion formula is: y=ax+b, x represents the original data, y represents the converted data, a and b represent the transformation coefficients; the preprocessed physical environment data is subjected to dimensionality reduction processing through the principal component analysis method, and the specific formula is C=cov(X)·V, cov(X) represents the covariance matrix of the data, V represents the eigenvector of the covariance matrix, and then Huffman coding is used to compress the dimensionality-reduced data.

2. A method for collecting lightweight physical environment data according to claim 1, characterized in that: The S5 is used to use network monitoring tools to collect network data during physical environment data transmission in real time, including data delay, packet loss rate and throughput, and calculate the network congestion control coefficient by analyzing the network data during data transmission. The specific calculation formula is: Where T represents the throughput of data transmission, T max represents the maximum throughput of the network, D represents the data delay during data transmission, and D max It represents the maximum acceptable delay of the network, and P represents the packet loss rate during data transmission; By comparing the network congestion control coefficient R with the preset congestion threshold λ, it is determined whether there is network congestion during data transmission. If R ≥ λ, it indicates that the current data transmission network status is good, and the data transmission process continues to be monitored. If R < λ, it indicates that the current data transmission has a network congestion problem, and the network congestion problem is immediately adjusted, including adjusting to reduce the data transmission rate, adjusting the size of the congestion window according to the network congestion control coefficient, and changing the dynamic route. At the same time, the network congestion monitoring results are fed back to the management terminal.

3. The method for collecting lightweight physical environment data according to claim 1, characterized in that: S6 analyzes and calculates the storage optimization index by collecting the utilization, access logs, costs, and fault records of the current storage system; The specific analysis method of the storage optimization index is as follows: Step S611: Analyze and calculate the storage utilization s represents the actual amount of stored data, 总 Indicates the total storage capacity; Step S612: Analyze and calculate the data access efficiency to be N represents the number of valid accesses during the current data transmission. 总 Indicates the total number of visits; Step S613: Analyze and calculate the storage cost efficiency Z represents storage cost, and I represents data value; Step S614: Analyze and calculate the data reliability as U=1-h, where h represents the data loss rate; Step S615: Based on the above analysis process, a storage optimization index is comprehensively calculated as Q = α1×S+α2×A+α3×E+α4×U, where α1, α2, α3, and α4 represent weight coefficients; By setting a storage optimization threshold κ, the storage optimization index Q is compared with the storage optimization threshold κ to determine whether data storage needs to be optimized. If Q ≥ κ, it indicates that the current data storage status is good and no storage optimization is required, and the storage status continues to be monitored. If Q < κ, it indicates that the current data storage status is poor, and data storage optimization measures are automatically triggered. At the same time, the data storage optimization results are sent to the management terminal for feedback.

Images