Rule-based data preprocessing method and device in Internet of Things system and storage medium

Abstract

The invention relates to the field of Internet of Things, in particular to a rule-based data preprocessing method and device in an Internet of Things system and a storage medium. Compared with the prior art, the technical scheme provided by the invention has the following beneficial effects: the method provided by the invention can perform comprehensive and accurate management on the data preprocessing of the object of the Internet of Things in the system of the Internet of Things, and performs data missing, consistency, validity and rationality judgment and processing and accurate marking on the access data, so that the accuracy of the data processing is improved, and the data processing efficiency is improved. An accurate, reliable and complete high-quality data source can be provided for subsequent data processing, analysis and mining, and the training, analysis and optimization efficiency and accuracy are improved; moreover, the data preprocessing management mode can be used for preprocessing the data of the object of the internet of things in various modes, so that the influence of different preprocessing modes on the result can be analyzed subsequently, and support is provided for preferably selecting a proper preprocessing mode.

Description

Technical Field

[0001] The present invention relates to the field of Internet of Things, and in particular to a rule-based data preprocessing method, device and storage medium in an Internet of Things system. Background Art

[0002] Today is the era of the Internet of Everything, and production and operation management cannot be separated from the signal data of networked facilities. These data will cause data anomalies due to various reasons such as reduced accuracy, failures, and jams in sensor collection, transmission, and conversion, which will cause errors and mistakes in subsequent data analysis. Therefore, it has become a consensus in the industry to pre-process the accessed data. Although analysis based on big data can ignore the flaws of a small amount of data, the existence of flaws requires more data for training, which increases the computing load. At the same time, when the trained model is applied to specific data analysis, if there is a lack of data pre-processing, the analysis conclusions will be very different due to the small amount of specific data. Therefore, whether it is mechanism-based analysis or data-driven analysis, the data set as the raw material is crucial to improving analysis efficiency.

[0003] At present, there are various preprocessing methods at home and abroad, but little attention has been paid to the management of complex IoT data preprocessing methods. Summary of the invention

[0004] The purpose of the present invention is to solve the deficiencies of the prior art and propose a rule-based data preprocessing method and storage medium in an Internet of Things system in view of the actual needs of managing a large number of data of Internet of Things objects and performing data preprocessing in an Internet of Things system.

[0005] To achieve the above object, the present invention provides a rule-based data preprocessing method in an Internet of Things system, comprising the following steps:

[0006] S1. Formulate data preprocessing rule options and establish a rule option library;

[0007] S2. Establishing a pre-processing rule option scheme for IoT object data collection;

[0008] S3. Use the rule engine to drive the preprocessing of collected data according to the preprocessing rule option scheme, and store the processed data in the database for subsequent analysis service calls.

[0009] Preferably: the data preprocessing rule options in step S1 include at least one of a missing data processing option, a consistency rule option, a validity rule option, a rationality rule option and a data marking option.

[0010] Preferably: the missing data processing option, consistency rule option, validity rule option, rationality rule option and data marking option are all provided with a default execution option.

[0011] Preferably, the collected data of the IoT object in step S2 includes at least one of the state data, operation physical quantity data and setting data of the IoT object.

[0012] Preferably, the step S2 sets the preprocessing rule options for each collected data of the IoT object, and provides a default preprocessing rule option scheme consisting of preprocessing rule default options.

[0013] Preferably: the rule engine in step S3 can perform data preprocessing on each collected data according to the set rule options according to the selected preprocessing rule option scheme, and the stored data is associated with the preprocessing rule option scheme ID.

[0014] The present invention also provides a computer device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above method when executing the program.

[0015] The present invention also provides a computer-readable storage medium on which a computer program is stored, and the program implements the steps of the above method when executed by a processor.

[0016] Compared with the prior art, the technical solution proposed in this application has the following beneficial effects: the method provided by the present invention can comprehensively and accurately manage the data preprocessing of the IoT objects in the IoT system, and by judging and processing the access data in terms of missing data, consistency, validity, and rationality, as well as accurately marking the access data, it can provide an accurate, reliable, and complete high-quality data source for subsequent data processing, analysis, and mining, thereby improving the efficiency and accuracy of training, analysis, and optimization; moreover, this data preprocessing management method can preprocess the data of the IoT objects in a variety of ways, so that the impact of different preprocessing methods on the results can be analyzed subsequently, providing support for the selection of an appropriate preprocessing method. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Other features, objects and advantages of the present invention will become more apparent from the detailed description of non-limiting embodiments made with reference to the following drawings:

[0018] Figure 1 A flow chart of a rule-based data preprocessing method in an Internet of Things system proposed by the present invention;

[0019] Figure 2 Flow chart of preprocessing single data according to the present invention. DETAILED DESCRIPTION

[0020] The following will clearly and completely describe and discuss the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings of the present invention. Obviously, what is described here is only a part of the examples of the present invention, not all the examples. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0021] It should be noted that when a component is referred to as being "fixed to" another component, it may be directly on the other component or there may also be a component centered. When a component is considered to be "connected to" another component, it may be directly connected to the other component or there may also be a component centered. When a component is considered to be "set on" another component, it may be directly set on the other component or there may also be a component centered. The terms "vertical", "horizontal", "left", "right" and similar expressions used herein are for illustrative purposes only.

[0022] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art of the present invention. The terms used herein in the specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. The term "and / or" used herein includes any and all combinations of one or more related listed items.

[0023] Please refer to Figure 1 ,A rule-based data preprocessing method in the IoT system includes the following three steps:

[0024] Step S1: Formulate data preprocessing rule options and establish a rule option library;

[0025] Step S2: Establishing a pre-processing rule option scheme for IoT object collected data;

[0026] Step S3: The rule engine drives the preprocessing of the collected data according to the preprocessing rule option scheme, and the processed data is stored in the database for subsequent analysis service calls.

[0027] Please refer to Figure 2 ,The preprocessing of a data usually includes multiple aspects, specifically including: missing data processing, ,consistency judgment and processing, validity judgment and processing, rationality judgment and processing, as well as ,adding precise labels to the data.

[0028] The first is data missing processing. When the input data is judged to be empty, missing preprocessing needs to be triggered. The rule options that can be given include: ① assign NULL; ② do no processing; ③ assign 0 according to the target data type (or 0.0, or yes / no, T / F, input is empty, etc.), among which option ① is the default option. And mark the missing data.

[0029] The second is the consistency judgment of the data, that is, the type of the data must be consistent with the target data. If it is inconsistent, the corresponding data preprocessing needs to be performed, so the corresponding processing options must be formulated. For example, when a certain integer data does not meet the type judgment, the rule options that can be given include: ① assign 0 (NULL, or other integer value); ② do not process; ③ convert to integer by rounding; ④ convert to integer by rounding up; ⑤ convert to integer by rounding down, among which option ① is the default option. For another example, when a real number does not meet the type judgment, the rule options that can be given include: ① assign 0.0 (NULL, or other real value); ② do not process; ③ multiply by 1.0, among which option ① is the default option. And mark the data that does not meet the consistency requirements.

[0030] The third is the validity judgment of the data, that is, judging whether the value of the data is within the valid range. Data that does not meet the validity judgment needs to be preprocessed, which is also the most commonly used data preprocessing content, and options for processing methods need to be given. For example, when a real number does not meet the valid range, the rule options that can be given include: ① assign 0.0 (NULL, or other values); ② do not process; ③ according to the maximum value of the valid range; ④ according to the minimum value of the valid range; ⑤ according to the middle value of the valid range (or average value, or mode value, etc.); ⑥ multiply by a coefficient of 0.1 (or other values, which can be used for unit conversion or data correction); ⑦ according to the middle value (or average value, or mode value, etc.) within a month (or other time period); ⑧ the last valid value, etc., among which option ① is the default option. And mark the data that does not meet the validity requirements.

[0031] Fourth, the rationality judgment of the data with rationality requirements is to judge whether the data is within the reasonable range relative to other data. Data that does not meet the rationality judgment needs to be preprocessed, and options for processing methods need to be given. For example, when a real number does not meet the reasonable range, the rule options that can be given include: ① assign 0.0 (NULL, or other values); ② do not process; ③ according to the maximum value of the reasonable range; ④ according to the minimum value of the reasonable range; ⑤ according to the middle value of the reasonable range (or average value, or mode value, etc.); ⑥ multiply by a coefficient of 0.1 (or other value); ⑦ according to the middle value (or average value, or mode value, etc.) within a month (or other time period); ⑧ the last reasonable value, etc., among which option ① is the default option. And mark the data that does not meet the rationality requirements.

[0032] The fifth is the marking of data. In addition to the markings made after the above-mentioned judgments, data can also be specifically marked. For example, the markable rule options include: ① no marking; ② core (or important, or general) parameters; ③ adjustable (or non-adjustable) parameters; ④ target parameters; ⑤ carbon-containing (or non-carbon-containing) raw material (gas, liquid, solid fuel, product, by-product, intermediate product, etc.) parameters; ⑥ consumption (or recycling, or environmental) parameters; etc., among which ① is the default option.

[0033] Each of the above data preprocessing rule options has a default option. This setting can minimize the workload of setting preprocessing rule options for IoT object data collection, and only needs to be set for special data and data processed by special rules.

[0034] After executing the above step S1, formulate data preprocessing rule options and establish a rule option library, so that the preprocessing method can be set for any collected data of any connected IoT object, providing a basis for subsequent preprocessing. Generally, the dynamic data collected from IoT objects include: status data, operating physical quantity data, setting data, etc. Of course, static data such as design data and nameplate data of IoT objects are also necessary for subsequent analysis, but usually only one-time collection is required, and generally no data preprocessing is involved.

[0035] The above step S1 is to establish a preprocessing rule for a single dynamic data. For an IoT object, the data collected at the same time may include multiple data, and the amount of data collected may be increased or decreased due to the needs of the analysis service. Different preprocessing methods may also be used for the same batch of collected data due to the needs of different analysis services. For this reason, this method proposes step S2 to establish a preprocessing rule option scheme for IoT object collection data, which is used for the management of data preprocessing under different collection data sets and different preprocessing setting conditions of IoT objects, and provides an accurate data source for subsequent data applications. Similarly, in order to minimize the workload of setting preprocessing rule options for IoT object collection data sets, for any type of IoT object, a default preprocessing rule option scheme consisting of preprocessing rule default options is given. With the default preprocessing rule option scheme, a new preprocessing rule option scheme can be formed by copying and modifying the default preprocessing rule option scheme, so that a new targeted scheme can be formed, thereby providing support for multi-scheme analysis and comparison, and also minimizing the workload of setting preprocessing rule options for IoT object collection data sets.

[0036] After setting the preprocessing rule options for the IoT object collected data in step S2, step S3 can be executed to drive the rule engine to preprocess the collected data according to the preprocessing rule option scheme, and store the processed data in the database for subsequent analysis service calls. The rule engine can perform data preprocessing on each collected data according to the selected preprocessing rule option scheme.

[0037] Since multiple preprocessing rule options may be set for the same IoT object at the same time, the data preprocessing driven by the rule engine is closely related to the selected preprocessing rule option. Therefore, when storing the processed data in the database, it is necessary to associate the stored data with the ID of the preprocessing rule option. In this way, different preprocessing rule options for the same IoT object can be managed, forming a complete management of rule-based data preprocessing in the IoT system.

[0038] This embodiment also provides a computer device, such as a smart phone, tablet computer, laptop computer, desktop computer, rack server, blade server, tower server or cabinet server (including an independent server or a server cluster composed of multiple servers) that can execute programs. The computer device of this embodiment includes at least but is not limited to: a memory and a processor that can communicate with each other through a system bus.

[0039] In this embodiment, the memory (i.e., readable storage medium) includes flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory may be an internal storage unit of a computer device, such as a hard disk or memory of the computer device. In other embodiments, the memory may also be an external storage device of a computer device, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the computer device. Of course, the memory may also include both the internal storage unit of the computer device and its external storage device. In this embodiment, the memory is generally used to store an operating system and various application software installed on the computer device. In addition, the memory may also be used to temporarily store various types of data that have been output or are to be output.

[0040] The processor may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chip in some embodiments. The processor 22 is generally used to control the overall operation of the computer device. In this embodiment, the processor is used to run the program code stored in the memory or process data.

[0041] This embodiment also provides a computer-readable storage medium, such as a flash memory, a hard disk, a multimedia card, a card-type memory (for example, an SD or DX memory, etc.), a random access memory (RAM), a static random access memory (SRAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a programmable read-only memory (PROM), a magnetic memory, a disk, an optical disk, a server, an App application store, etc., on which a computer program is stored, and the corresponding function is realized when the program is executed by a processor.

[0042] The above describes the specific embodiments of the present invention. It should be understood that the present invention is not limited to the above specific embodiments, and one or more of the above embodiments are combined into one embodiment. Those skilled in the art can make various changes or modifications or combinations within the scope of the claims, which does not affect the essence of the present invention. In the absence of conflict, the embodiments of the present application and the features in the embodiments can be combined with each other at will.

Claims

1. A rule-based data preprocessing method in an Internet of Things system, characterized in that: The following steps are involved: S1. Formulate data preprocessing rule options and establish a rule option library; S2. Establishing a pre-processing rule option scheme for IoT object data collection; S3. Use the rule engine to drive the preprocessing of collected data according to the preprocessing rule option scheme, and store the processed data in the database for subsequent analysis service calls.

2. The rule-based data preprocessing method in an Internet of Things system according to claim 1, characterized in that: The data preprocessing rule options in step S1 include at least one of a missing data processing option, a consistency rule option, a validity rule option, a rationality rule option and a data marking option.

3. The rule-based data preprocessing method in an Internet of Things system according to claim 2, characterized in that: The missing data processing option, consistency rule option, validity rule option, rationality rule option and data marking option are all provided with a default execution option.

4. The rule-based data preprocessing method in an Internet of Things system according to claim 1, characterized in that: The collected data of the IoT object in step S2 includes at least one of the state data, operation physical quantity data, and setting data of the IoT object.

5. The rule-based data preprocessing method in an Internet of Things system according to claim 4, characterized in that: The step S2 sets the preprocessing rule options for each collected data of the IoT object, and provides a default preprocessing rule option scheme consisting of preprocessing rule default options.

6. The rule-based data preprocessing method in an Internet of Things system according to claim 1, characterized in that: The rule engine in step S3 can perform data preprocessing on each collected data according to the set rule options according to the selected preprocessing rule option scheme, and the stored data is associated with the preprocessing rule option scheme ID.

7. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps of the method according to any one of claims 1 to 6 are implemented.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images