A data governance system and method for equipment internet of things devices

Abstract

The present invention provides a data management system and method for equipment IoT devices. The system includes an equipment simulation module, a gateway module, and a data processing module. The equipment simulation module is connected to the gateway module, and the gateway module is connected to the data processing module. The equipment simulation module is used to simulate the equipment IoT device through an equipment IoT device simulator to obtain simulation data. The gateway module is used to select a corresponding gateway based on the simulation data, use the corresponding gateway to convert the simulation data into a data format, obtain converted data, and send the converted data to the data processing module. The data processing module is used to perform data verification, data cleaning, data calculation, and data transmission on the converted data. Through the above-mentioned data management system for equipment IoT devices, data management of equipment IoT devices can be achieved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of Internet of Things, and in particular to a data management system and method for equipment Internet of Things devices. BACKGROUND

[0002] With the continuous progress of social development, the demand for digitalization of equipment Internet of Things devices is also increasing. At present, the equipment Internet of Things devices have the following two characteristics:

[0003] First, the types of equipment Internet of Things devices are various, for example, there are wheeled equipment Internet of Things devices, tracked equipment Internet of Things devices, towed equipment Internet of Things devices, and half-tracked equipment Internet of Things devices, and the corresponding motion parameter characteristics differ greatly.

[0004] Second, the corresponding behaviors of the equipment Internet of Things devices are complex, and the applicable scenarios are complex, resulting in a huge amount of data.

[0005] The above two characteristics result in great difficulty in managing the data of the equipment Internet of Things devices, and therefore, the problem of how to manage the data of the equipment Internet of Things devices is an important issue to be solved in the industry. SUMMARY

[0006] The present application provides a data management system and method for equipment Internet of Things devices to solve the defect that the data of the equipment Internet of Things devices is difficult to manage in the prior art, and to manage the data of the equipment Internet of Things devices.

[0007] The present application provides a data management system for equipment Internet of Things devices, which comprises an equipment simulation module, a gateway module, and a data processing module.

[0008] The equipment simulation module is connected to the gateway module, and the gateway module is connected to the data processing module.

[0009] The equipment simulation module is configured to simulate the equipment Internet of Things devices through an equipment Internet of Things device simulator to obtain simulation data.

[0010] The gateway module is configured to select a corresponding gateway based on the simulation data, convert the simulation data into converted data by using the corresponding gateway, and send the converted data to the data processing module.

[0011] The data processing module is configured to perform data verification, data cleaning, data calculation, and data transmission on the converted data.

[0012] According to a data management system for IoT devices provided by the present invention, the data processing module includes a data verification unit, a data cleaning unit, and a data calculation and transmission unit;

[0013] The data verification unit is used to verify the converted data and obtain the data to be cleaned if the verification passes;

[0014] The data cleaning unit is configured to clean the data to be cleaned using a preset cleaning strategy to obtain cleaned data;

[0015] The data calculation and transmission unit is used to calculate and transmit the cleaned data based on pre-acquired instructions.

[0016] According to a data management system for IoT devices provided by the present invention, the data cleaning unit includes a data detection subunit and a data cleaning subunit;

[0017] The data detection subunit is configured to detect the data to be cleaned based on a preset detection method to obtain a detection result, wherein the preset detection method includes at least one of consistency detection, repeatability detection, missing value detection, and noise value detection;

[0018] The data cleaning subunit is configured to clean the data to be cleaned based on the preset cleaning method corresponding to the preset detection method and the detection result to obtain cleaned data.

[0019] According to a data management system for IoT devices provided by the present invention, when the preset detection method includes the consistency detection, the data detection subunit is specifically used to detect whether the characters of the data to be cleaned meet the preset consistency conditions and obtain a detection result;

[0020] The data cleaning subunit is specifically configured to retain the character if the detection result indicates that the character meets the preset consistency condition; and delete the character if the detection result indicates that the character does not meet the preset consistency condition, so as to obtain cleaned data.

[0021] According to a data management system for IoT devices provided by the present invention, when the preset detection method includes the repeatability detection, the data detection subunit is specifically used to determine whether there is stored data consistent with the data to be cleaned in a preset database, and obtain a detection result;

[0022] The data cleaning subunit is specifically configured to delete data consistent with the stored data in the to-be-cleaned data to obtain cleaned data in a case where the detection result represents that the stored data exists; and use the to-be-cleaned data as the cleaned data in a case where the detection result represents that the stored data does not exist.

[0023] In a case where the preset detection manner includes the missing value detection, the data detection subunit is specifically configured to determine whether there is to-be-filled data based on a preset database in a case where the to-be-cleaned data has missing values, to obtain a detection result.

[0024] The data cleaning subunit is specifically configured to add the to-be-filled data to the to-be-cleaned data to obtain cleaned data in a case where the to-be-detection result represents that the to-be-filled data exists; and perform deletion processing on the to-be-cleaned data in a case where the to-be-detection result represents that the to-be-filled data does not exist.

[0025] In a case where the preset detection manner includes the noise value detection, the data detection subunit is specifically configured to divide the to-be-cleaned data into a plurality of to-be-cleaned sub-data, acquire a to-be-determined value of each to-be-cleaned sub-data by using a preset algorithm, and obtain a detection result.

[0026] The data cleaning subunit is specifically configured to retain the to-be-cleaned sub-data in a case where the detection result represents that the to-be-determined value meets a preset numerical condition; and delete the to-be-cleaned sub-data in a case where the detection result represents that the to-be-determined value meets the preset numerical condition, to obtain cleaned data.

[0027] The data calculation and transmission unit is specifically configured to perform calculation corresponding to a pre-acquired instruction on the cleaned data, to obtain calculated data.

[0028] In a case where the current network is connected, the calculated data is transmitted.

[0029] In a case where the current network is not connected, the calculated data is put into a cache queue until the current network is connected.

[0030] The application further provides a data management method for equipment Internet of Things devices, and the method comprises the following steps.

[0031] Obtain simulation data through the equipment simulation module, wherein the simulation data is data obtained by simulating the equipment Internet of Things device through the equipment Internet of Things device simulator;

[0032] Select a corresponding gateway based on the simulation data through the gateway module, perform data format conversion on the simulation data by using the corresponding gateway, obtain converted data, and send the converted data to the data processing module;

[0033] Perform data verification, data cleaning, data calculation, and data transmission on the converted data through the data processing module.

[0034] According to the data governance method for equipment Internet of Things devices provided by the application, the step of performing data verification, data cleaning, data calculation, and data transmission on the converted data through the data processing module comprises:

[0035] Verify the converted data through the data verification unit, and obtain to-be-cleaned data if the verification is passed;

[0036] Clean the to-be-cleaned data by using a preset cleaning strategy through the data cleaning unit to obtain cleaned data;

[0037] Perform calculation and transmission on the cleaned data based on a pre-acquired instruction through the data calculation and transmission unit.

[0038] The data governance system and method for equipment Internet of Things devices provided by the application, the system comprises an equipment simulation module, a gateway module, and a data processing module, the equipment simulation module is connected with the gateway module, the gateway module is connected with the data processing module, the equipment simulation module is used to simulate the equipment Internet of Things device through the equipment Internet of Things device simulator to obtain simulation data, the gateway module is used to select a corresponding gateway based on the simulation data, perform data format conversion on the simulation data by using the corresponding gateway to obtain converted data, and send the converted data to the data processing module, and the data processing module is used to perform data verification, data cleaning, data calculation, and data transmission on the converted data.

[0039] Through the above-mentioned data governance system for equipment Internet of Things devices, simulation data of the equipment Internet of Things device can be obtained, the simulation data is processed by using a corresponding gateway to obtain converted data, the access of different equipment Internet of Things devices is realized, and then the converted data is verified, cleaned, calculated, and transmitted, so that the data of the equipment Internet of Things device can be governed. BRIEF DESCRIPTION OF DRAWINGS

[0040] In order to make the technical solutions in the present application or the prior art clearer, the accompanying drawings needed in the embodiments or prior art description will be briefly introduced. Obviously, the accompanying drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without any creative effort.

[0041] Figure 1 is a structural schematic diagram of a data governance system for equipment Internet of Things devices provided by the present application;

[0042] Figure 2 is one of the flow schematic diagrams of a data governance method for equipment Internet of Things devices provided by the present application;

[0043] Figure 3 is the second flow schematic diagram of a data governance method for equipment Internet of Things devices provided by the present application;

[0044] Figure 4 is a structural schematic diagram of an electronic device provided by the present application. DETAILED DESCRIPTION

[0045] In order to make the technical solutions in the present application or the prior art clearer, the accompanying drawings needed in the embodiments or prior art description will be briefly introduced. Obviously, the accompanying drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without any creative effort.

[0046] In order to be able to govern the data of equipment Internet of Things devices, the embodiments of the present application provide a data governance system, a method, an electronic device, a non-transitory computer readable storage medium and a computer program product for equipment Internet of Things devices, which will be described below in combination with Figure 1 A data governance system for equipment Internet of Things devices provided by the embodiments of the present application is introduced as follows:

[0047] As shown in Figure 1 A data governance system for equipment Internet of Things devices 110, the data governance system 110 includes an equipment simulation module 120, a gateway module 130 and a data processing module 140.

[0048] The equipment simulation module 120 is connected with the gateway module 130, and the gateway module 130 is connected with the data processing module 140. The equipment simulation module 120 can include a plurality of equipment Internet of Things device simulators, so as to realize the simulation of equipment Internet of Things devices.

[0049] The equipment simulation module 120 is configured to simulate the equipment IoT device by an equipment IoT device simulator to obtain simulation data. The equipment IoT device simulator is a program configured to simulate the equipment IoT device. The equipment IoT device simulator can simulate the actual operation of the equipment IoT device to obtain the simulation data.

[0050] The simulation data can represent the operation data generated by the equipment IoT device during the actual operation. The simulation data can include position information, driving state information, fuel consumption information, and driver driving information of the equipment IoT device. The simulation data is not limited in this regard.

[0051] The gateway module 130 is configured to select a corresponding gateway based on the simulation data, convert the simulation data into converted data by using the corresponding gateway, and send the converted data to the data processing module 140.

[0052] The gateway module 130 can include a plurality of gateways and at least one gateway load regulator. The gateway load regulator can assign a corresponding gateway to the simulation data based on the received simulation data. The corresponding gateway can then convert the simulation data into converted data. The converted data has a data format that can be recognized and processed by the data processing module 140.

[0053] In an embodiment, the gateway load regulator can include a plurality of gateways that can forward data or messages through nginx, which is a high-performance HTTP and reverse proxy web server. In this way, data disaster recovery can be achieved.

[0054] That is, the gateway module 130 can access equipment IoT devices with different data formats and analyze the simulation data of the equipment IoT devices with different data formats to obtain converted data that meets certain data standards.

[0055] As an embodiment, each gateway can report its load state information to the gateway load regulator in real time. The gateway load regulator can obtain the load parameters corresponding to each gateway in real time. The gateway load regulator can include readable data synchronized by the master server to which each gateway belongs, and thus obtain load parameters representing the current running state of the gateway.

[0056] The gateway load regulator can calculate the load parameters of the gateway based on a preset algorithm to obtain load weight values corresponding to the gateways, and then distribute the gateways based on the load weight values, thereby distributing the optimal gateway for the equipment Internet of Things device, dynamically distributing the gateway according to the load capacity of the gateway, and achieving optimal distribution of gateway resources.

[0057] Further, the gateway module 130 can send the converted data to the data processing module 140, that is, the gateway module 130 can standardize the data of different data formats in the front end, thereby transmitting the converted data to the back end (the data processing module 140), and can realize isolation of the production environment and the development environment, and avoid the influence caused by improper configuration.

[0058] The data processing module 140 is configured to perform data verification, data cleaning, data calculation, and data transmission on the converted data.

[0059] After receiving the converted data, the data processing module 140 can perform data verification, data cleaning, data calculation, and data transmission on the converted data to facilitate the management of the converted data.

[0060] In an embodiment, after the data is calculated, the calculated data can be stored in a database for subsequent data cleaning.

[0061] As can be seen, in the embodiment, the simulation data of the equipment Internet of Things device can be obtained, the simulation data is processed by the corresponding gateway to obtain converted data, the access of different equipment Internet of Things devices is realized, and the converted data is verified, cleaned, calculated, and transmitted, thereby realizing the management of the data of the equipment Internet of Things device.

[0062] As an embodiment of the present application, after a new service is added, the data processing module can send relevant data to the gateway load regulator to realize the corresponding service registration of the gateway load regulator and establish a socket connection, so as to realize the function of dynamically expanding the service of the data processing module in the future.

[0063] As an embodiment of the present application, the above-mentioned data processing module can include a data verification unit, a data cleaning unit, and a data calculation and transmission unit.

[0064] The data verification unit is configured to verify the converted data, and obtain to-be-cleaned data if the verification is passed.

[0065] Data verification is the premise and basis of data cleaning. Data verification is to verify whether the converted data is legal. If the converted data is not legal, it means that the verification fails, and thus the converted data can be regarded as illegal data and deleted. If the converted data is legal, it means that the verification passes, and thus the converted data can be regarded as to-be-cleaned data for subsequent data cleaning.

[0066] In an embodiment, the converted data can carry a verification code. After obtaining the converted data, the data verification unit can verify whether the verification code carried by the converted data meets a preset condition. If the verification code meets the preset condition, it means that the verification passes, and thus the converted data can be regarded as to-be-cleaned data. If the verification code does not meet the preset condition, it means that the verification fails, and thus the converted data can be regarded as illegal data and deleted.

[0067] As an embodiment, the verification code can be generated based on a unique ID (Identity document) of each equipment Internet of Things device simulator. For example, a first preset algorithm can be used to process the ID of the equipment Internet of Things device simulator, so as to obtain the verification code. Correspondingly, after receiving the converted data, the data verification unit can use the first preset algorithm to process the verification code, so as to verify whether the verification code meets the preset condition.

[0068] In an embodiment, the verification code can be generated based on the ID of the equipment Internet of Things device simulator and the time when the simulation data is generated, that is, the ID of the equipment Internet of Things device simulator and the time stamp corresponding to the generation time are combined to obtain a to-be-encrypted string, the to-be-encrypted string is encrypted by using a second preset algorithm to obtain an encrypted string, and the encrypted string is subjected to an exclusive OR operation to obtain the verification code.

[0069] Correspondingly, after receiving the converted data, the data verification unit can use the second preset algorithm and the exclusive OR operation to decrypt the verification code, so as to verify whether the verification code meets the preset condition. For example, the second preset algorithm can be a sha256 algorithm.

[0070] After obtaining the to-be-cleaned data, the to-be-cleaned data can be sent to the data cleaning unit.

[0071] The data cleaning unit is configured to clean the to-be-cleaned data by using a preset cleaning strategy to obtain cleaned data.

[0072] The data cleaning unit can receive the to-be-cleaned data that passes the verification, and clean the to-be-cleaned data by using a preset cleaning strategy to obtain cleaned data. The preset cleaning strategy can include at least one of a consistency cleaning strategy, a repetition cleaning strategy, a missing value cleaning strategy, and a noise value cleaning strategy.

[0073] After the cleaning of the to-be-cleaned data, errors in the to-be-cleaned data can be searched and found, and the errors in the to-be-cleaned data can be corrected, so that clean data backflow, that is, cleaned data, can be obtained, and then the cleaned data can be sent to the data calculation and transmission unit.

[0074] The data calculation and transmission unit is configured to calculate and transmit the cleaned data based on a pre-acquired instruction.

[0075] After the data calculation and transmission unit acquires the cleaned data, the user-issued instruction can be acquired, so that the cleaned data is calculated according to the instruction, and the calculated data is transmitted, so that the user can acquire the data corresponding to the instruction. The instruction can include a query instruction, a report generation instruction, a data monitoring instruction, and the like, which are specifically set according to actual use requirements of the user and are not limited herein.

[0076] It can be seen that in the embodiment, data verification, data cleaning, and data calculation and transmission can be implemented on the converted data, so that the data of the equipment Internet of Things device can be managed.

[0077] As an embodiment of the present application, the above-mentioned data cleaning unit includes a data detection sub-unit and a data cleaning sub-unit. The preset cleaning strategy includes a preset detection method and a preset cleaning method, wherein the preset detection method and the preset cleaning method have a corresponding relationship.

[0078] The data detection sub-unit is configured to detect the to-be-cleaned data based on the preset detection method to obtain a detection result.

[0079] After acquiring the to-be-cleaned data, the data detection sub-unit can detect the to-be-cleaned data based on the preset detection method to obtain a detection result, wherein the preset detection method can include at least one of consistency detection, repetition detection, missing value detection, and noise value detection.

[0080] The data cleaning sub-unit is configured to clean the to-be-cleaned data based on the preset cleaning method corresponding to the preset detection method and the detection result to obtain cleaned data.

[0081] The preset cleaning method can include at least one of consistency cleaning method, repetition cleaning method, missing value cleaning method, and noise value cleaning method.

[0082] For different preset detection methods, the preset cleaning method used is also different, and the data cleaning sub-unit cleaning the to-be-cleaned data has the following situations:

[0083] In a first case, when the preset detection manner is consistency detection, repetitive detection, missing value detection, or noise value detection, the to-be-cleaned data can be cleaned based on a preset cleaning manner corresponding to the preset detection manner and the detection result, to obtain cleaned data.

[0084] In a second case, when the preset detection manner is any two of consistency detection, repetitive detection, missing value detection, and noise value detection, the to-be-cleaned data can be cleaned based on a preset cleaning manner corresponding to the preset detection manner and the detection result, to obtain cleaned data. The order of the two preset detection manners can be set according to actual use requirements.

[0085] In a third case, when the preset detection manner is any three of consistency detection, repetitive detection, missing value detection, and noise value detection, the to-be-cleaned data can be cleaned based on a preset cleaning manner corresponding to the preset detection manner and the detection result, to obtain cleaned data. The order of the three preset detection manners can be set according to actual use requirements.

[0086] In a fourth case, when the preset detection manner is consistency detection, repetitive detection, missing value detection, and noise value detection, the to-be-cleaned data can be cleaned based on a consistency cleaning manner, a repetitive cleaning manner, a missing value cleaning manner, and a noise value cleaning manner, and the detection result, to obtain cleaned data. The order of the consistency detection, the repetitive detection, the missing value detection, and the noise value detection can be set according to actual use requirements. This is reasonable and is not limited here.

[0087] As can be seen, in this embodiment, the to-be-cleaned data can be detected based on a preset detection manner to obtain a detection result, and then the to-be-cleaned data can be cleaned based on a preset cleaning manner corresponding to the preset detection manner and the detection result, to obtain cleaned data. In this way, data cleaning can be more accurately completed, so as to govern data of equipment Internet of Things devices.

[0088] As an implementation manner of an embodiment of the present application, in the case where the preset detection manner includes the consistency detection, the data detection subunit is specifically configured to detect whether a character of the to-be-cleaned data satisfies a preset consistency condition, to obtain a detection result.

[0089] The data cleaning subunit is specifically configured to, in the case where the detection result represents that the character satisfies the preset consistency condition, retain the character; and in the case where the detection result represents that the character does not satisfy the preset consistency condition, delete the character, to obtain cleaned data.

[0090] The preset consistency condition may include at least one of a preset format condition, a preset content condition, a preset attribute condition, and a preset requirement condition.

[0091] The data detection subunit is specifically used to verify whether the characters with a fixed format included in the data to be cleaned meet the preset format conditions and obtain a detection result.

[0092] The data cleaning subunit is specifically used to retain the characters with a fixed format when the detection result indicates that the characters with a fixed format meet the preset format conditions; and delete the characters with a fixed format when the detection result indicates that the characters with a fixed format do not meet the preset format conditions, thereby obtaining cleaned data.

[0093] For example, character A with a fixed format may be a date, and the data detection subunit may verify whether character A meets the date format requirements, specifically whether it is arranged in the order of year / month / day, thereby obtaining detection result A. Detection result A may indicate whether character A is arranged in the order of year / month / day.

[0094] When character A is arranged in the order of year / month / day, the data cleaning subunit may retain character A. When character A is not arranged in the order of year / month / day, the data cleaning subunit may delete character A, thereby obtaining cleaned data.

[0095] For another example, the character B with a fixed format may be a character B displayed in a specific format. Specifically, the data detection subunit may verify whether the character B is displayed in half-width, thereby obtaining a detection result B, which may indicate whether the character B is displayed in half-width.

[0096] In the case where character B is displayed in half-width format, the data cleaning subunit may retain character B. In the case where character A is not displayed in half-width format, the data cleaning subunit may delete character B, thereby obtaining cleaned data.

[0097] For another example, a character C with a fixed format may be a numeric value. The data detection subunit may verify whether the character C meets the numeric format requirements, thereby obtaining a detection result C. The detection result C may indicate whether the character C meets the numeric format requirements. If the character C meets the numeric format requirements, it indicates that there are no redundant characters in the character C. Therefore, the data cleaning subunit may retain the character C.

[0098] When character C does not meet the numerical format requirements, it indicates that there are redundant characters in character C, that is, there are characters that should not appear in the numerical format. Therefore, the data cleaning subunit can delete character C to obtain cleaned data.

[0099] The data detection subunit is specifically configured to verify whether the content corresponding to the character of the data to be cleaned meets a preset content condition, and obtain a detection result. The data cleaning subunit is specifically configured to retain the character in a case where the detection result represents that the content corresponding to the character meets the preset content condition, and delete the character in a case where the detection result represents that the content corresponding to the character does not meet the preset content condition, thereby obtaining cleaned data.

[0100] For example, the data detection subunit can detect whether the content corresponding to the character D is reasonable, thereby obtaining a detection result D, which can represent whether the content corresponding to the character D is reasonable. In a case where the content corresponding to the character D is reasonable, the character D can be retained, and in a case where the content corresponding to the character D is unreasonable, the character D can be deleted, thereby obtaining cleaned data.

[0101] The data detection subunit is specifically configured to verify whether the attribute corresponding to the character of the data to be cleaned meets a preset attribute condition, and obtain a detection result. The attribute corresponding to the character can be verified for conflict with the attribute of another character, and the dependency relationship of the character can also be verified for reasonableness, which are all reasonable and are not limited here.

[0102] The data cleaning subunit is specifically configured to retain the character in a case where the detection result represents that the attribute corresponding to the character meets the preset attribute condition, and delete the character in a case where the detection result represents that the attribute corresponding to the character does not meet the preset attribute condition, thereby obtaining cleaned data.

[0103] The data detection subunit is specifically configured to verify whether the character meets a preset requirement condition, and obtain a detection result. The data cleaning subunit is specifically configured to retain the character in a case where the detection result represents that the character is a requirement character, and delete the character in a case where the detection result represents that the character is a non-requirement character, thereby obtaining cleaned data.

[0104] It can be seen that, in the embodiment, the character of the data to be cleaned can be detected in different ways, and then the character is retained or deleted based on the detection result, thereby obtaining accurate cleaned data. In this way, the accuracy of data cleaning can be improved.

[0105] As an implementation manner of the embodiment, in a case where the preset detection manner includes the repetitive detection, the data detection subunit is specifically configured to determine whether there is stored data consistent with the data to be cleaned in a preset database, and obtain a detection result.

[0106] The data cleaning subunit is specifically configured to, in a case where the detection result indicates that the stored data exists, delete data consistent with the stored data from the to-be-cleaned data to obtain cleaned data; and in a case where the detection result indicates that the stored data does not exist, take the to-be-cleaned data as the cleaned data.

[0107] In a case where the detection result indicates that the stored data exists, it is indicated that at least part of the to-be-cleaned data has been stored in the database, and it is not necessary to process again. Therefore, in order to prevent repeated processing of at least part of the to-be-cleaned data, data consistent with the stored data can be deleted from the to-be-cleaned data to obtain cleaned data. In a case where all data of the to-be-cleaned data has corresponding stored data, all data of the to-be-cleaned data corresponding to the stored data can be deleted.

[0108] In a case where the detection result indicates that the stored data does not exist, it is indicated that the to-be-cleaned data does not include data stored in the database, and it is necessary to process the to-be-cleaned data. Therefore, in order to ensure the integrity of the data, the to-be-cleaned data can be taken as the cleaned data.

[0109] It can be seen that, in the embodiment, repeated detection can be performed on the to-be-cleaned data, so that the data management system does not process the data repeatedly and does not waste computing resources.

[0110] As an implementation manner of the embodiment, in a case where the preset detection manner includes the missing value detection, the data detection subunit is specifically configured to, in a case where the to-be-detected data has missing values, determine whether there is to-be-filled data based on a preset database to obtain a detection result.

[0111] The data detection subunit can first detect whether the to-be-cleaned data has missing values, and in a case where it is determined that the to-be-cleaned data has missing values, determine whether there is to-be-filled data corresponding to the missing data in the to-be-cleaned data based on a preset database to obtain a detection result.

[0112] The data cleaning subunit is specifically configured to, in a case where the detection result indicates that the to-be-filled data exists, add the to-be-filled data to the to-be-cleaned data to obtain cleaned data; and in a case where the detection result indicates that the to-be-filled data does not exist, delete the to-be-cleaned data.

[0113] In a case where the to-be-filled data exists in the to-be-detected result representation, the to-be-filled data can be added to the to-be-cleaned data, so as to ensure the integrity of the to-be-cleaned data. In a case where the to-be-filled data does not exist in the to-be-detected result representation, it can be verified whether the missing data in the to-be-cleaned data is necessary data. In a case where the missing data is necessary data, it is indicated that the to-be-cleaned data can be illegal data or dirty data, and thus the to-be-cleaned data can be deleted.

[0114] In a case where the missing data is non-essential data, it is indicated that the missing data is not important, and the missing data can be understood as auxiliary data that can be present or absent. Therefore, the missing data can be ignored or the part of the to-be-cleaned data with the missing data is deleted. The processing is specifically performed according to actual use requirements. No specific limitation is made herein.

[0115] It can be seen that in the embodiment, the missing value detection can be performed on the to-be-cleaned data, so as to ensure the safety of the data governance system.

[0116] As an implementation manner of the embodiment of the present application, in a case where the preset detection manner includes the noise value detection, the noise value refers to a value corresponding to part of data in the to-be-cleaned data, which is greatly different from values corresponding to other data in the to-be-cleaned data. It can also be referred to as existing outlier data.

[0117] The data detection subunit is specifically configured to divide the to-be-cleaned data into a plurality of to-be-cleaned sub-data, and for each to-be-cleaned sub-data, a preset algorithm is used to obtain a to-be-determined value of the to-be-cleaned sub-data, to obtain a detection result.

[0118] The preset algorithm can be an algorithm corresponding to a single-variable capped method, an algorithm corresponding to a binning method, an algorithm corresponding to a multi-variable clustering method, and the like, which are all reasonable, and no specific limitation is made herein.

[0119] For example, in a case where the preset algorithm is an algorithm corresponding to a binning method, the preset algorithm can be an average value algorithm and a median algorithm. Specifically, after the to-be-cleaned data is divided into a plurality of to-be-cleaned sub-data, the average value and the median corresponding to each to-be-cleaned sub-data can be calculated, to obtain a detection result 1.

[0120] The data cleaning subunit is specifically configured to retain the to-be-cleaned sub-data in a case where the detection result represents that the to-be-determined value meets a preset numerical condition, and delete the to-be-cleaned sub-data in a case where the detection result represents that the to-be-determined value meets the preset numerical condition, to obtain cleaned data.

[0121] In the above example, after the average value and the median value corresponding to each piece of to-be-cleaned sub-data are calculated, in a case where the detection result 1 indicates that the average value and the median value of the to-be-cleaned sub-data do not satisfy the preset numerical condition, the to-be-cleaned sub-data is deleted, and in a case where the detection result 1 indicates that the average value and the median value of the to-be-cleaned sub-data satisfy the preset numerical condition, the to-be-cleaned sub-data is retained.

[0122] It can be seen that, in the embodiment, the to-be-cleaned data can be detected based on noise value detection, so that unreasonable to-be-cleaned sub-data can be deleted, so as to ensure the rationality of the to-be-cleaned data, so as to improve the accuracy of data management of the equipment Internet of Things device in the future.

[0123] As an implementation manner of the embodiment of the present application, the data calculation and transmission unit is specifically configured to perform calculation corresponding to the instruction on the cleaned data based on the pre-acquired instruction, to obtain calculated data.

[0124] After the cleaned data is acquired, the data calculation and transmission unit can perform calculation corresponding to the instruction on the cleaned data based on the real-time calculation cluster and the pre-acquired instruction, to obtain calculated data. The real-time calculation cluster relies on storm and can complete various functions of big data instructions.

[0125] For example, the pre-acquired instruction is an oil consumption common feature acquisition instruction, and the data calculation and transmission unit can perform common feature calculation on the cleaned data, to obtain common features of oil consumption corresponding to years, months or days.

[0126] In a case where the current network is connected, the calculated data is transmitted.

[0127] In a case where the current network is not connected, the calculated data is put into a cache queue until the current network is connected.

[0128] As an implementation manner of the embodiment of the present application, after the calculated data is transmitted, the calculated data can be displayed on a corresponding interface, so that data information of the instruction on the data of the equipment Internet of Things device can be displayed to the user, so as to manage the equipment Internet of Things device and the driver in the future.

[0129] As an implementation manner of the embodiment of the present application, the data processing module is in a cluster mode, that is, the data verification unit, the data cleaning unit, and the data calculation and transmission unit are in a cluster mode. When there is a hardware problem such as high CPU (Central Processing Unit) occupation or insufficient memory in the data governance system, the data verification unit, the data cleaning unit, and the data calculation and transmission unit can be expanded in a horizontal expansion manner by increasing nodes, so as to support the business of the data governance system quickly and ensure that the business is not affected. Of course, the capacity can also be reduced according to actual use requirements to reduce costs.

[0130] It can be seen that the data governance system provided by the embodiment of the present application has strict verification, cleaning, calculation, and transmission strategies, can improve the credibility of the simulation data of the equipment Internet of Things device under the double verification of data legality and equipment Internet of Things device identity, and thus realizes efficient processing of the cleaned data. For example, the data governance system provided by the embodiment of the present application can reach a maximum of 5w equipment Internet of Things devices simultaneously accessing, and the limit throughput of parallel computing can reach 100,000 messages / s.

[0131] The data governance method for equipment Internet of Things devices provided by the present application is described below. The data governance method for equipment Internet of Things devices described below can be correspondingly referred to the data governance system for equipment Internet of Things devices described above.

[0132] As shown in Figure 2 A data governance method for equipment Internet of Things devices is applied to any of the data governance systems for equipment Internet of Things devices described above. The method comprises the following steps:

[0133] S201, obtaining simulation data by an equipment simulation module.

[0134] The simulation data is data obtained by simulating the equipment Internet of Things device by an equipment Internet of Things device simulator.

[0135] S202, selecting a corresponding gateway based on the simulation data by a gateway module, performing data format conversion on the simulation data by the corresponding gateway, obtaining converted data, and sending the converted data to the data processing module.

[0136] S203, performing data verification, data cleaning, data calculation, and data transmission on the converted data by the data processing module.

[0137] As an implementation manner of the embodiment of the present application, the step of performing data verification, data cleaning, data calculation, and data transmission on the converted data by the data processing module can comprise the following steps:

[0138] verify the converted data by a data verification unit, and acquire data to be cleaned in a case of passing verification;

[0139] clean the data to be cleaned by a data cleaning unit by using a preset cleaning strategy, and obtain cleaned data;

[0140] perform calculation and transmission on the cleaned data based on a pre-acquired instruction by a data calculation and transmission unit.

[0141] As an implementation of an embodiment of the present application, the step of cleaning the data to be cleaned by the data cleaning unit by using the preset cleaning strategy, and obtaining the cleaned data can include:

[0142] detect the data to be cleaned by a data detection sub-unit based on a preset detection method, and obtain a detection result.

[0143] The preset detection method includes at least one of consistency detection, repetitiveness detection, missing value detection, and noise value detection.

[0144] clean the data to be cleaned by a data cleaning sub-unit based on a preset cleaning method corresponding to the preset detection method and the detection result, and obtain the cleaned data.

[0145] As an implementation of an embodiment of the present application, in a case where the preset detection method includes the consistency detection, the step of detecting the data to be cleaned by the data detection sub-unit based on the preset detection method, and obtaining the detection result can include:

[0146] detect, by the data detection sub-unit, whether a character of the data to be cleaned satisfies a preset consistency condition, and obtain the detection result.

[0147] The step of cleaning the data to be cleaned by the data cleaning sub-unit based on the preset cleaning method corresponding to the preset detection method and the detection result, and obtaining the cleaned data can include:

[0148] In a case where the detection result represents that the character satisfies the preset consistency condition, retain the character by the data cleaning sub-unit; and in a case where the detection result represents that the character does not satisfy the preset consistency condition, delete the character, and obtain the cleaned data.

[0149] As an implementation of an embodiment of the present application, in a case where the preset detection method includes the repetitiveness detection, the step of detecting the data to be cleaned by the data detection sub-unit based on the preset detection method, and obtaining the detection result can include:

[0150] The data detection subunit determines whether there is stored data consistent with the data to be cleaned in a preset database, and obtains a detection result.

[0151] The step of cleaning the data to be cleaned by the data cleaning subunit based on the preset cleaning method corresponding to the preset detection method and the detection result to obtain the cleaned data may include:

[0152] When the detection result indicates that the stored data exists, the data cleaning subunit deletes the data to be cleaned that is consistent with the stored data to obtain cleaned data; when the detection result indicates that the stored data does not exist, the data to be cleaned is used as the cleaned data.

[0153] As an implementation manner of an embodiment of the present invention, when the above-mentioned preset detection method includes the missing value detection, the step of detecting the data to be cleaned based on the preset detection method by the data detection subunit to obtain the detection result may include:

[0154] When the data to be cleaned is missing, the data detection subunit determines whether there is data to be filled based on a preset database to obtain a detection result.

[0155] The step of cleaning the data to be cleaned by the data cleaning subunit based on the preset cleaning method corresponding to the preset detection method and the detection result to obtain the cleaned data may include:

[0156] The data cleaning subunit adds the data to be filled to the data to be cleaned to obtain cleaned data when the result to be detected indicates that the data to be filled exists; and deletes the data to be cleaned when the result to be detected indicates that the data to be filled does not exist.

[0157] As an implementation manner of an embodiment of the present invention, when the above-mentioned preset detection method includes the noise value detection, the step of detecting the data to be cleaned based on the preset detection method by the data detection subunit to obtain the detection result may include:

[0158] The data to be cleaned is divided into a plurality of sub-data to be cleaned by the data detection sub-unit. For each sub-data to be cleaned, a preset algorithm is used to obtain the to-be-determined value of the sub-data to be cleaned, and a detection result is obtained.

[0159] The step of cleaning the data to be cleaned by the data cleaning subunit based on the preset cleaning method corresponding to the preset detection method and the detection result to obtain the cleaned data may include:

[0160] In a case where the detection result indicates that the to-be-determined value satisfies the preset numerical condition, the data cleaning subunit retains the to-be-cleaned sub-data, and in a case where the detection result indicates that the to-be-determined value satisfies the preset numerical condition, the data cleaning subunit deletes the to-be-cleaned sub-data, to obtain cleaned data.

[0161] As an implementation of an embodiment of the present application, the step of calculating and transmitting the cleaned data by the data calculating and transmitting unit based on the pre-acquired instruction can include:

[0162] The data calculating and transmitting unit calculates the cleaned data based on the pre-acquired instruction, to obtain calculated data.

[0163] In a case where the current network is connected, the calculated data is transmitted.

[0164] In a case where the current network is not connected, the calculated data is put into a cache queue until the current network is connected.

[0165] The following will be described in combination with Figure 3 The data management method for equipment Internet of Things devices provided by the embodiment of the present application will be introduced, as shown in Figure 3

[0166] S301, acquiring simulation data by an equipment simulation module.

[0167] S302, selecting a corresponding gateway based on the simulation data by a gateway module, performing data format conversion on the simulation data by using the corresponding gateway, obtaining converted data, and sending the converted data to a data verification unit.

[0168] S303, verifying the converted data by the data verification unit, and obtaining to-be-cleaned data in a case where the verification is passed.

[0169] S304, cleaning the to-be-cleaned data by using a preset cleaning strategy by a data cleaning unit, to obtain cleaned data.

[0170] S305, calculating the cleaned data by using the pre-acquired instruction by a data calculating and transmitting unit, to obtain calculated data, and in a case where the current network is connected, transmitting the calculated data, and in a case where the current network is not connected, putting the calculated data into a cache queue until the current network is connected.

[0171] ​It can be seen that in this way, the simulation data of the equipment Internet of Things device can be obtained, the simulation data is processed by the corresponding gateway to obtain converted data, the access of different equipment Internet of Things devices is realized, and then the converted data is verified, cleaned, calculated and transmitted, so that the data of the equipment Internet of Things device can be governed.

[0172] Figure 4 An example of an entity structure diagram of an electronic device is shown in Figure 4 As shown, the electronic device can include a processor 410, a communication interface 420, a memory 430 and a communication bus 440, wherein the processor 410, the communication interface 420 and the memory 430 complete mutual communication through the communication bus 440. The processor 410 can call the logical instructions in the memory 430 to execute the data governance method for the equipment Internet of Things device, which includes: obtaining simulation data through an equipment simulation module, wherein the simulation data is data obtained by simulating the equipment Internet of Things device through an equipment Internet of Things device simulator, selecting a corresponding gateway based on the simulation data through a gateway module, using the corresponding gateway to convert the data format of the simulation data to obtain converted data, and sending the converted data to a data processing module, and performing data verification, data cleaning, data calculation and transmission on the converted data through the data processing module.

[0173] In addition, the logical instructions in the memory 430 described above can be implemented in the form of a software function unit and sold or used as an independent product, which can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application essentially or the part that contributes to the prior art or part of the technical solutions can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a plurality of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the method described in various embodiments of the present application. The foregoing storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various media that can store program codes.

[0174] In another aspect, the present application also provides a computer program product comprising a computer program, the computer program being stored in a non-transitory computer-readable storage medium, and the computer program being executable by a processor to enable a computer to perform the data management method for equipment Internet of Things devices provided by the above method, the method comprising: obtaining simulation data by an equipment simulation module, wherein the simulation data is data obtained by simulating the equipment Internet of Things devices by an equipment Internet of Things device simulator, selecting a corresponding gateway based on the simulation data by a gateway module, performing data format conversion on the simulation data by the corresponding gateway to obtain converted data, and sending the converted data to a data processing module, and performing data verification, data cleaning, data calculation and transmission on the converted data by the data processing module.

[0175] In another aspect, the present application also provides a non-transitory computer-readable storage medium having a computer program stored thereon, and the computer program being executable by a processor to implement the data management method for equipment Internet of Things devices provided by the above method, the method comprising: obtaining simulation data by an equipment simulation module, wherein the simulation data is data obtained by simulating the equipment Internet of Things devices by an equipment Internet of Things device simulator, selecting a corresponding gateway based on the simulation data by a gateway module, performing data format conversion on the simulation data by the corresponding gateway to obtain converted data, and sending the converted data to a data processing module, and performing data verification, data cleaning, data calculation and transmission on the converted data by the data processing module.

[0176] The device embodiments described above are merely illustrative, wherein the units described as separate components can or can not be physically separate, and the components displayed as units can or can not be physical units, i.e., they can be located in one place, or distributed on multiple network units. Part or all of the modules can be selected to achieve the purpose of the present embodiment scheme according to actual needs. Those skilled in the art can understand and implement without creative labor.

[0177] From the above description of the embodiments, those skilled in the art can clearly understand that the embodiments can be realized by means of software plus necessary general hardware platforms, and of course can also be realized by hardware. Based on such understanding, the above technical solutions can be embodied in the form of a software product, which can be stored in a computer readable storage medium, such as a ROM / RAM, a magnetic disk, an optical disk, etc., and includes a plurality of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute the methods described in each embodiment or some parts of the embodiments.

[0178] It should be pointed out finally that the above embodiments are only used to illustrate the technical solutions of the present application, but not to limit the same; and although the present application has been described in detail with reference to the foregoing embodiments, it should be appreciated by those skilled in the art that the technical solutions recorded in the foregoing embodiments can be modified, or some technical features thereof can be replaced equivalently; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A data management system for IoT devices, characterized in that: The system includes: an equipment simulation module, a gateway module, and a data processing module; The equipment simulation module is connected to the gateway module, and the gateway module is connected to the data processing module; The equipment simulation module is used to simulate the actual operation of the equipment IoT device through the equipment IoT device simulator to obtain simulation data, wherein the simulation data represents the operation data generated by the equipment IoT device during the actual operation, including the location information, driving status information, fuel consumption information, and driver driving information of the equipment IoT device; The gateway module includes a load regulator and multiple gateways, wherein the load regulator is used to dynamically allocate a corresponding gateway to the analog data based on the received analog data, use the corresponding gateway to convert the data format of the analog data to obtain converted data, and send the converted data to the data processing module; The data processing module is used to perform data verification, data cleaning, data calculation and transmission on the converted data, wherein the converted data carries a verification code. If the verification code meets a preset condition, the converted data is used as data to be cleaned; otherwise, the converted data is deleted; The verification code is generated based on a unique ID for each IoT device simulator.

2. The data management system for IoT devices according to claim 1 is characterized in that: The data processing module includes a data verification unit, a data cleaning unit, and a data calculation and transmission unit; The data verification unit is used to verify the converted data and obtain the data to be cleaned if the verification passes; The data cleaning unit is configured to clean the data to be cleaned using a preset cleaning strategy to obtain cleaned data; The data calculation and transmission unit is used to calculate and transmit the cleaned data based on pre-acquired instructions.

3. The data management system for IoT devices according to claim 2, characterized in that: The data cleaning unit includes a data detection subunit and a data cleaning subunit; The data detection subunit is configured to detect the data to be cleaned based on a preset detection method to obtain a detection result, wherein the preset detection method includes at least one of consistency detection, repeatability detection, missing value detection, and noise value detection; The data cleaning subunit is configured to clean the data to be cleaned based on the preset cleaning method corresponding to the preset detection method and the detection result to obtain cleaned data.

4. The data management system for IoT devices according to claim 3 is characterized in that: In the case where the preset detection method includes the consistency detection, the data detection subunit is specifically used to detect whether the characters of the data to be cleaned meet the preset consistency conditions and obtain a detection result; The data cleaning subunit is specifically configured to retain the character if the detection result indicates that the character meets the preset consistency condition; If the detection result indicates that the character does not meet the preset consistency condition, the character is deleted to obtain cleaned data.

5. The data management system for IoT devices according to claim 3 is characterized in that: In the case where the preset detection method includes the repeatability detection, the data detection subunit is specifically used to determine whether there is stored data consistent with the data to be cleaned in a preset database, and obtain a detection result; The data cleaning subunit is specifically configured to delete data in the data to be cleaned that is consistent with the stored data, to obtain cleaned data, if the detection result indicates that the stored data exists; When the detection result indicates that the stored data does not exist, the data to be cleaned is regarded as cleaned data.

6. The data management system for IoT devices according to claim 3, characterized in that: In the case where the preset detection method includes the missing value detection, the data detection subunit is specifically used to determine whether there is data to be filled based on a preset database when there is a missing value in the data to be cleaned, and obtain a detection result; The data cleaning subunit is specifically configured to add the data to be filled to the data to be cleaned to obtain cleaned data when the detection result indicates that the data to be filled exists; When the detection result indicates that the data to be filled does not exist, the data to be cleaned is deleted.

7. The data management system for IoT devices according to claim 3, characterized in that: In the case where the preset detection method includes the noise value detection, the data detection subunit is specifically configured to divide the data to be cleaned into a plurality of sub-data to be cleaned, and for each sub-data to be cleaned, adopt a preset algorithm to obtain a value to be determined of the sub-data to be cleaned, thereby obtaining a detection result; The data cleaning subunit is specifically configured to retain the sub-data to be cleaned if the detection result indicates that the value to be determined meets a preset numerical condition, and to delete the sub-data to be cleaned if the detection result indicates that the value to be determined does not meet the preset numerical condition, so as to obtain cleaned data.

8. The data management system for IoT devices according to any one of claims 2 to 7, characterized in that: The data calculation and transmission unit is specifically configured to perform calculations corresponding to the pre-acquired instructions on the cleaned data to obtain calculated data; When the current network is connected, transmitting the calculated data; When the current network is not connected, the calculated data is placed in a cache queue until the current network is connected.

9. A data management method for IoT devices, characterized in that: The method is applied to the data governance system for IoT devices according to any one of claims 1 to 8, and the method includes: Acquiring simulation data through an equipment simulation module, wherein the simulation data is data obtained by simulating the actual operation of an equipment IoT device through an equipment IoT device simulator, and the simulation data represents the operating data generated by the equipment IoT device during the actual operation, including location information, driving status information, fuel consumption information, and driver driving information of the equipment IoT device; The gateway module includes a load regulator and multiple gateways, the load regulator dynamically assigns a corresponding gateway to the analog data based on the received analog data, uses the corresponding gateway to convert the data format of the analog data to obtain converted data, and sends the converted data to the data processing module; The converted data is subjected to data verification, data cleaning, data calculation and transmission by a data processing module. The converted data carries a verification code. If the verification code meets a preset condition, the converted data is used as data to be cleaned; otherwise, the converted data is deleted; The verification code is generated based on a unique ID for each IoT device simulator.

10. The data management method for IoT devices according to claim 9 is characterized in that: The step of performing data verification, data cleaning, data calculation and transmission on the converted data by the data processing module includes: Verifying the converted data by a data verification unit, and obtaining the data to be cleaned if the verification passes; Cleaning the data to be cleaned by a data cleaning unit using a preset cleaning strategy to obtain cleaned data; The cleaned data is calculated and transmitted based on pre-acquired instructions by a data calculation and transmission unit.

Images