A data processing platform analysis method in big data processing

Abstract

The present invention discloses a data processing platform analysis method in a big data processing process, and belongs to the technical field of data analysis and processing. The present invention uses relevant status data to dynamically correct the standard load threshold, and at the same time dynamically corrects the standard signal strength threshold according to the position data, so that the subsequent load status score and signal strength status score are more accurate, so that the performance status of the sensor can be evaluated more accurately; by establishing an association between the single scoring data and the comprehensive characterization image, various single scorings can be fused and transmitted, making the subsequent performance status evaluation work more convenient, and using a custom association rule, even if the comprehensive characterization image is accidentally leaked, the security of the data can be guaranteed.

Description

Technical Field

[0001] The present invention relates to the technical field of data analysis and processing, and in particular to a data processing platform analysis method in a big data processing process. Background Art

[0002] The Internet of Things refers to connecting any object to the network through information sensing devices according to agreed protocols, and objects exchange and communicate information through information dissemination media to achieve intelligent identification, positioning, tracking, supervision and other functions. IoT devices can generate a large amount of data, especially when there are a large number of these devices and they are running continuously. IoT devices can generate data in real time or near real time. This high speed of data generation and processing is another key feature of big data. The types of data generated by IoT devices are very diverse, including sensor data, text, etc. When processing the big data generated by IoT devices, appropriate technologies and tools need to be used to collect, store, process and analyze this data. This includes technologies and methods such as cloud computing, edge computing, machine learning, and data mining. Through these technologies and methods, valuable information can be extracted from the big data generated by IoT devices to support various applications and business decisions.

[0003] Sensors are an important component of IoT devices. The performance status of sensors themselves often has an important impact on the stable operation of IoT systems. In existing technologies, the performance or operating status of the monitored devices or components is mainly evaluated based on the monitoring parameters obtained by the sensors, and there are few evaluation technologies for the performance status of sensors.

[0004] The above problems need to be solved urgently. Therefore, a data processing platform analysis method in the process of big data processing is proposed. Summary of the invention

[0005] The technical problem to be solved by the present invention is: how to accurately evaluate the performance status of the sensor through the analysis and processing of the data processing platform, and provides a data processing platform analysis method in the process of big data processing.

[0006] The present invention solves the above technical problems through the following technical solutions, and the present invention comprises the following steps:

[0007] S1: Acquire multiple types of data related to sensor performance status;

[0008] S2: Based on the acquired multi-category data, multiple single-item scores are obtained;

[0009] S3: according to the obtained multiple single-item scores, the single-item score data are associated with the comprehensive representation image;

[0010] S4: Read the comprehensive characterization image, and evaluate the performance status of the sensor according to the comprehensive characterization image to obtain a performance status score of the sensor.

[0011] Furthermore, in step S1, the multiple types of data include load status data F, environmental status data, position data J, signal strength status data X, and power status data D, wherein the environmental status data includes temperature status data T and humidity status data H, and the position data J is the communication distance data between the sensor and the wireless network communication module to which it is connected.

[0012] Furthermore, in step S2, the specific processing process is as follows:

[0013] S21: Dynamically modify the standard load threshold according to the temperature state data T and the humidity state data H to obtain a modified load threshold F 阈值 ′, and dynamically correct the standard signal strength threshold according to the position data J to obtain the corrected signal strength threshold X 阈值 ′;

[0014] S22: Calculate multiple single-item scores, as follows:

[0015] According to the load state data F and the corrected load threshold F 阈值 ′, calculate the load status score P F , the calculation formula is as follows:

[0016] P F =[|FF 阈值 ′|];

[0017] According to the temperature state data T, the humidity state data H and the corresponding preset standard temperature threshold T 阈值 , Standard humidity threshold H 阈值 , calculate the environmental status score P S , the calculation formula is as follows:

[0018] P S =[|TT 阈值 |+|HH 阈值 |];

[0019] According to the signal strength status data X and the corrected signal strength threshold X 阈值 Calculate the signal strength status score P X , the calculation formula is as follows:

[0020] P X =[|XX 阈值 ′|];

[0021] The power scoring unit is used to compare the power state data D with the preset power state threshold D阈值 , calculate the power status score P D , the calculation formula is as follows:

[0022] P D =[|DD 阈值 |];

[0023] Among them, [] represents the rounding operation, and || represents the absolute value operation.

[0024] Furthermore, in step S21, the correction formula for dynamically correcting the standard load threshold is as follows:

[0025] F 阈值 ′=(W1*Q T +W2*Q H )*F 阈值

[0026] Among them, F 阈值 ′ is the corrected load threshold, Q T is the temperature correction coefficient obtained according to the temperature state data T, Q H is the humidity correction coefficient obtained according to the humidity state data H, F 阈值 is the preset standard load threshold, W1 and W2 are the weights of the temperature correction coefficient and the humidity correction coefficient respectively, and the sum of W1 and W2 is 1.

[0027] Furthermore, the humidity correction factor Q T According to the temperature state data T, the temperature correction coefficient database is searched and obtained. The temperature state data T and the humidity correction coefficient Q are pre-stored in the temperature correction coefficient database. T The correlation between humidity correction factor Q H According to the humidity state data H, the humidity correction coefficient Q is obtained from the preset humidity correction coefficient database, which pre-stores the humidity state data H and the humidity correction coefficient Q. H The correlation relationship between them.

[0028] Furthermore, in step S21, the correction formula for dynamically correcting the standard signal strength threshold is as follows:

[0029] X 阈值 ′=Q J *X 阈值

[0030] Among them, X 阈值 ′ is the corrected signal strength threshold, Q J is the position correction coefficient obtained according to the position data J, X 阈值 is the preset standard signal strength threshold.

[0031] Furthermore, in step S3, the specific processing process is as follows:

[0032] S31: Read the load status score P respectively and simultaneously F , Environmental status score P S , signal strength status score P X , Power status score P D ;

[0033] S32: Calculate the number of pixels that need to be filled in each filling area frame in the comprehensive representation image template according to each single item score in step S31, and the calculation formula is as follows:

[0034] Pixel 1 = R F* P F

[0035] Pixel 2 = R S* P S

[0036] Pixel 3 = R X* P X

[0037] Pixel 4 = R D* P D

[0038] Among them, Pixel1~Pixel4 represent the number of pixels that need to be filled in the first to fourth filling area boxes, R F , R S , R X , R D The fill quantity correction coefficients respectively represent the number of pixels that need to be filled in the first to fourth fill area boxes;

[0039] S33: Filling each filling area frame with pixels according to the calculated number of pixels that need to be filled in each filling area frame in the comprehensive representation image template, to obtain a comprehensive representation image after the pixels are filled.

[0040] Furthermore, in the step S33, in the comprehensive representation image template, the first to fourth filling area boxes are arranged in sequence from top to bottom; when pixel filling is performed, in a single filling area box, the filling is performed row by row from top to bottom.

[0041] Furthermore, in step S4, the specific processing process is as follows:

[0042] S41: read the comprehensive representation image, and use the pixel detection function in OpenCV to read the number of pixels in the first to fourth filling area boxes in sequence, which are Pixel1, Pixel2, Pixel3, and Pixel4 respectively;

[0043] S42: Calculate the performance status score of the sensor according to the number of pixels in the first to fourth filled area boxes:

[0044] C total =Z1*Pixel1+Z2*Pixel2+Z3*Pixel3+Z4*Pixel4

[0045] Among them, C total is the performance status score of the sensor, Z1, Z2, Z3 and Z4 are the weights corresponding to the number of pixels in each filled area box, and the sum of Z1, Z2, Z3 and Z4 is 1.

[0046] Compared with the prior art, the present invention has the following advantages: the data processing platform analysis method in the big data processing process uses relevant status data to dynamically correct the standard load threshold, and dynamically corrects the standard signal strength threshold according to the position data, so that the subsequent load status score and signal strength status score are more accurate, and the performance status of the sensor can be evaluated more accurately; by associating the single scoring data with the comprehensive characterization image, various single scores can be fused and transmitted, making the subsequent performance status evaluation work more convenient, and adopting customized association rules, even if the comprehensive characterization image is accidentally leaked, the security of the data can be guaranteed. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 It is a structural diagram of a data processing platform analysis system in a big data processing process according to an embodiment of the present invention;

[0048] Figure 2 It is a flow chart of a data processing platform analysis method in a big data processing process in an embodiment of the present invention. DETAILED DESCRIPTION

[0049] The following is a detailed description of an embodiment of the present invention. This embodiment is implemented on the premise of the technical solution of the present invention, and a detailed implementation method and a specific operation process are given, but the protection scope of the present invention is not limited to the following embodiment.

[0050] like Figure 1 As shown, this embodiment provides a technical solution: a data processing platform analysis system in a big data processing process, comprising the following modules: a data acquisition module, a data comparison processing module, a single item processing module and a performance status evaluation module;

[0051] In this embodiment, the data acquisition module is used to acquire multiple types of data related to the sensor performance status.

[0052] As more specifically, the data acquisition module includes a load status data acquisition unit, an environment status data acquisition unit, a position data acquisition unit, a signal strength status data acquisition unit and a power status data acquisition unit; the load status data acquisition unit is used to acquire the load status data F of the sensor, the environment status data acquisition unit is used to acquire the environment status data when the sensor is running, including temperature status data T and humidity status data H, the position data acquisition unit is used to acquire the communication distance data between the sensor and the wireless network communication module to which it is connected, as the position data J of the sensor, the signal strength status data acquisition unit is used to acquire the signal strength status data X between the sensor and the wireless network communication module to which it is connected, and the power status data acquisition unit is used to acquire the power status data D of the sensor battery; after acquiring the above-mentioned various types of data, the above-mentioned various types of data are sent to the data comparison processing module for further processing.

[0053] It should be noted that the load status data F, environmental status data, position data J, signal strength status data X, and power status data D are all data types that are highly correlated with the sensor performance status. Therefore, the present invention selects the above-mentioned multiple types of data as basic data, and the load status data F, environmental status data, position data J, signal strength status data X, and power status data D are all obtained through periodic collection.

[0054] It should be noted that the position data J (communication distance data) of the sensor is calculated through a ranging algorithm, that is, by measuring the signal propagation time between nodes to calculate the communication distance between nodes.

[0055] In this embodiment, the data comparison and processing module is used to obtain multiple single-item scores based on the multiple types of data acquired by the data acquisition module.

[0056] As more specific, the data comparison processing module includes a load threshold dynamic correction unit, a signal strength dynamic correction unit, a load scoring unit, an environment scoring unit, a signal strength scoring unit and a power scoring unit; the load threshold dynamic correction unit is used to dynamically correct the standard load threshold according to the temperature state data T and the humidity state data H, and the correction formula is as follows: F 阈值 ′=(W1*Q T +W2*Q H )*F 阈值 , where F 阈值 ′ is the corrected load threshold, Q T is the temperature correction coefficient obtained according to the temperature state data T, Q H is the humidity correction coefficient obtained according to the humidity state data H, F 阈值is the preset standard load threshold, W1 and W2 are the weights of the temperature correction coefficient and the humidity correction coefficient respectively, the sum of W1 and W2 is 1, which can be set according to the actual application requirements. In this embodiment, W1=0.8, W2=0.2, and the humidity correction coefficient Q T According to the temperature state data T, the temperature correction coefficient database is searched and obtained. The temperature state data T and the humidity correction coefficient Q are pre-stored in the temperature correction coefficient database. T The correlation between humidity correction factor Q H According to the humidity state data H, the humidity correction coefficient Q is obtained from the preset humidity correction coefficient database, which pre-stores the humidity state data H and the humidity correction coefficient Q. H The signal strength dynamic correction unit is used to dynamically correct the standard signal strength threshold according to the position data J, and the correction formula is as follows: X 阈值 ′=Q J *X 阈值 , where X 阈值 ′ is the corrected signal strength threshold, Q J is the position correction coefficient obtained according to the position data J, X 阈值 is a preset standard signal strength threshold; the load scoring unit is used to compare the load state data F with the modified load threshold F 阈值 ′, calculate the load status score P F , the calculation formula is as follows: F =[|FF 阈值 ′|], where [] represents a rounding operation, and || represents an absolute value operation; the environmental scoring unit is used to calculate the temperature state data T, the humidity state data H and the corresponding preset standard temperature threshold T 阈值 , Standard humidity threshold H 阈值 , calculate the environmental status score P S , the calculation formula is as follows: S =[|TT 阈值 |+|HH 阈值 |], where [] represents a rounding operation, and || represents an absolute value operation; the signal strength scoring unit is used to compare the signal strength state data X with the corrected signal strength threshold X 阈值 Calculate the signal strength status score P X , the calculation formula is as follows: X =[|XX 阈值 ′|], where [] represents a rounding operation, and || represents an absolute value operation; the power scoring unit is used to compare the power state data D with the preset power state threshold D 阈值 , calculate the power status score P D, the calculation formula is as follows: D =[|DD 阈值 |], where [] represents the rounding operation and || represents the absolute value operation.

[0057] It should be noted that in this embodiment, in order to obtain more accurate individual scores, the standard load threshold is dynamically corrected using temperature status data and humidity status data, and the standard signal strength threshold is dynamically corrected based on the location data, thereby making subsequent load status scores and signal strength status scores more accurate, and thus being able to more accurately evaluate the performance status of the sensor.

[0058] In this embodiment, the single item processing module is used to associate the single item scoring data with the comprehensive representation image according to the multiple single item scorings obtained by the data comparison processing module. By establishing an association between the single item scoring data and the comprehensive representation image, various single item scorings can be fused and transmitted, making the subsequent performance status evaluation work more convenient. Due to the use of the self-defined association rules in the present invention, the security of the data can be guaranteed even if the comprehensive representation image is accidentally leaked.

[0059] To be more specific, the single item processing module includes a single item score reading unit and a comprehensive representation image generation unit; the single item score reading unit is used to obtain multiple single item scores from the data comparison processing module; the comprehensive representation image generation unit is used to associate multiple single item scores with a comprehensive representation image template to obtain a comprehensive representation image.

[0060] As more specific, the specific processing process of the comprehensive representation image generation unit is as follows:

[0061] S101: Read the load status scores P simultaneously F , Environmental status score P S , signal strength status score P X , Power status score P D ;

[0062] S102: Calculate the number of pixels that need to be filled in each filling area frame in the comprehensive representation image template according to each single item score in step S101, and the calculation formula is as follows:

[0063] Pixel 1 = R F* P F

[0064] Pixel 2 = R S* P S

[0065] Pixel 3 = R X* P X

[0066] Pixel 4 = R D* P D

[0067] Among them, Pixel1~Pixel4 represent the number of pixels that need to be filled in the first to fourth filling area boxes, R F , R S , R X , R D The filling quantity correction coefficients respectively represent the number of pixels to be filled in the first to fourth filling area frames. In this embodiment, R F =6, R S =2, R X =2, R D =4;

[0068] S103: Filling each filling region frame with pixels according to the calculated number of pixels that need to be filled in each filling region frame in the comprehensive representation image template, to obtain a comprehensive representation image after the pixels are filled.

[0069] To be more specific, in step S103, the comprehensive representation image template (with a white background color) is used to set the length and width. In the comprehensive representation image template, the first to fourth filling area boxes (four rectangular boxes of the same size, black lines, and no overlap or intersection with each other) are arranged in sequence from top to bottom; when filling pixels (the pixel color can be set as required, such as red), in a single filling area box, the filling is carried out row by row from top to bottom.

[0070] It should be noted that there are ten sets of image templates from top to bottom.

[0071] In this embodiment, the performance status evaluation module is used to obtain a comprehensive characterization image, and evaluate the performance status of the sensor according to the comprehensive characterization image to obtain a performance status score of the sensor.

[0072] As more specific, the specific processing process of the performance status evaluation module is as follows:

[0073] S201: using the pixel detection function in OpenCV to read the number of pixels in the first to fourth filling area boxes in sequence, which are Pixel1, Pixel2, Pixel3, and Pixel4 respectively;

[0074] S202: Calculate the performance status score of the sensor according to the number of pixels in the first to fourth filled area boxes:

[0075] C total =Z1*Pixel1+Z2*Pixel2+Z3*Pixel3+Z4*Pixel4

[0076] Among them, C total is the performance status score of the sensor, Z1, Z2, Z3 and Z4 are the weights corresponding to the number of pixels in each filling area box, the sum of Z1, Z2, Z3 and Z4 is 1, which can be set according to actual application requirements. In this embodiment, Z1=0.4, Z2=0.1, Z3=0.2, and Z4=0.3.

[0077] like Figure 2 As shown, this embodiment also provides a data processing platform analysis method in the big data processing process, which uses the data processing platform analysis system in the big data processing process to complete the evaluation of the sensor performance status in the Internet of Things device, including the following steps:

[0078] S1: Acquire multiple types of data related to sensor performance status;

[0079] S2: Based on the acquired multi-category data, multiple single-item scores are obtained;

[0080] S3: according to the obtained multiple single-item scores, the single-item score data are associated with the comprehensive representation image;

[0081] S4: Read the comprehensive characterization image, and evaluate the performance status of the sensor according to the comprehensive characterization image to obtain a performance status score of the sensor.

[0082] The specific processing process of each of the above steps refers to the processing process of each module and unit in the data processing platform analysis system in the above big data processing process, which will not be repeated here.

[0083] To summarize, in the data processing platform analysis method and system in the big data processing process of the above-mentioned embodiments, the standard load threshold is dynamically corrected by using relevant status data, and the standard signal strength threshold is dynamically corrected according to the location data, so that the subsequent load status score and signal strength status score are more accurate, and the performance status of the sensor can be evaluated more accurately; by associating the single scoring data with the comprehensive characterization image, various single scores can be fused and transmitted, making the subsequent performance status evaluation work more convenient, and adopting customized association rules, even if the comprehensive characterization image is accidentally leaked, the security of the data can be guaranteed.

[0084] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the description of the present invention, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0085] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0086] Although the embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and are not to be construed as limitations of the present invention. A person skilled in the art may change, modify, replace and vary the above embodiments within the scope of the present invention.

Claims

1. A data processing platform analysis method in a big data processing process, characterized in that: The following steps are involved: S1: Acquire multiple types of data related to sensor performance status; S2: Based on the acquired multi-category data, multiple single-item scores are obtained; S3: according to the obtained multiple single-item scores, the single-item score data are associated with the comprehensive representation image; S4: obtaining a comprehensive characterization image, and evaluating the performance status of the sensor according to the comprehensive characterization image to obtain a performance status score of the sensor; In step S1, the multiple types of data include load status data F, environmental status data, position data J, signal strength status data X, and power status data D, wherein the environmental status data includes temperature status data T and humidity status data H, and the position data J is the communication distance data between the sensor and the wireless network communication module to which it is connected; In step S2, the specific processing process is as follows: S21: Dynamically modify the standard load threshold according to the temperature state data T and the humidity state data H to obtain a modified load threshold F 阈值 ´, and at the same time, the standard signal strength threshold is dynamically corrected according to the position data J to obtain the corrected signal strength threshold X 阈值 ´; S22: Calculate multiple single-item scores, as follows: According to the load state data F and the corrected load threshold F 阈值 ´, calculate the load status score P F , the calculation formula is as follows: P F =[|F-F 阈值 ´|]; According to the temperature state data T, the humidity state data H and the corresponding preset standard temperature threshold T 阈值 , Standard humidity threshold H 阈值 , calculate the environmental status score P S , the calculation formula is as follows: P S =[|T-T 阈值 |+|H-H 阈值 |]; According to the signal strength status data X and the corrected signal strength threshold X 阈值 Calculate the signal strength status score P X , the calculation formula is as follows: P X =[|X-X 阈值 ´|]; The power scoring unit is used to compare the power state data D with the preset power state threshold D 阈值 , calculate the power status score P D , the calculation formula is as follows: P D =[|D-D 阈值 |]; Among them, [ ] represents the rounding operation, | | represents the absolute value operation; In step S21, the correction formula for dynamically correcting the standard load threshold is as follows: F 阈值 ´=(W1*Q T +W2*Q H )*F 阈值 Among them, F 阈值 ´ is the corrected load threshold, Q T is the temperature correction coefficient obtained according to the temperature state data T, Q H is the humidity correction coefficient obtained according to the humidity state data H, F 阈值 is the preset standard load threshold, W1 and W2 are the weights of the temperature correction coefficient and the humidity correction coefficient respectively, and the sum of W1 and W2 is 1; In step S21, the correction formula for dynamically correcting the standard signal strength threshold is as follows: X 阈值 ´=Q J *X 阈值 Among them, X 阈值 ´ is the corrected signal strength threshold, Q J is the position correction coefficient obtained according to the position data J, X 阈值 is the preset standard signal strength threshold; In step S3, the specific processing process is as follows: S31: Read the load status score P respectively and simultaneously F , Environmental status score P S , signal strength status score P X , Power status score P D ; S32: Calculate the number of pixels that need to be filled in each filling area frame in the comprehensive representation image template according to each single item score in step S31, and the calculation formula is as follows: Pixel1=R F* P F Pixel2=R S* P S Pixel3=R X* P X Pixel4=R D* P D Among them, Pixel1~Pixel4 represent the number of pixels that need to be filled in the first to fourth filling area boxes, R F , R S , R X , R D The fill quantity correction coefficients respectively represent the number of pixels that need to be filled in the first to fourth fill area boxes; S33: filling each filling area frame with pixels according to the calculated number of pixels that need to be filled in each filling area frame in the comprehensive representation image template, to obtain a comprehensive representation image after the pixels are filled; In step S4, the specific processing process is as follows: S41: read the comprehensive representation image, and use the pixel detection function in OpenCV to read the number of pixels in the first to fourth filling area boxes in sequence, which are Pixel1, Pixel2, Pixel3, and Pixel4 respectively; S42: Calculate the performance status score of the sensor according to the number of pixels in the first to fourth filled area boxes: C total =Z1*Pixel1+Z2*Pixel2+Z3*Pixel3+Z4*Pixel4 Among them, C total is the performance status score of the sensor, Z1, Z2, Z3 and Z4 are the weights corresponding to the number of pixels in each filled area box, and the sum of Z1, Z2, Z3 and Z4 is 1.

2. The data processing platform analysis method in the process of big data processing according to claim 1 is characterized in that: Temperature correction factor Q T The temperature state data T is retrieved from a preset temperature-correction coefficient database, where the temperature state data T and the temperature correction coefficient Q are pre-stored. T The correlation between humidity correction factor Q H According to the humidity state data H, the humidity correction coefficient Q is obtained from the preset humidity correction coefficient database, which pre-stores the humidity state data H and the humidity correction coefficient Q. H The correlation relationship between them.

3. The data processing platform analysis method in the big data processing process according to claim 2 is characterized in that: In the step S33, in the comprehensive representation image template, the first to fourth filling area boxes are arranged in sequence from top to bottom; when pixel filling is performed, in a single filling area box, the filling is performed row by row from top to bottom.

Images