Data quality inspection method and system of Internet of Things data, electronic equipment and medium
By performing protocol adaptation and model mapping on the Internet of Things platform, unified processing of heterogeneous device data, and quality inspection based on preset rules, the complexity and manual intervention problems in the quality inspection of government IoT data are solved, and efficient Data cataloging and real-time quality inspection.
Patent Information
- Application Number
- CN202411223279.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-02
- Publication Date
- 2024-11-22
AI Technical Summary
The existing data quality inspection technology of government IoT data has problems such as heterogeneous data increasing the complexity of quality inspection, highly reliance on manual intervention and real-time data quality monitoring, and it is difficult to effectively deal with quality problems in large-scale data and real-time data flows.
By connecting all devices to the Internet of Things platform, performing protocol adaptation and model mapping, the unity of data models is achieved, and quality inspection is carried out based on the unified data model and preset IoT data quality inspection rules, reducing manual intervention and providing real-time Data quality inspection and convenient attachment tools improve data cataloging attachment efficiency.
It realizes unified processing of heterogeneous data, reduces manual intervention, improves data cataloging and attaches efficiency, and supports real-time data quality inspection, solving the complexity of heterogeneous data, manual intervention and real-time data monitoring.
Smart Images

Figure CN119003511A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the technical field of data processing, and in particular to a data quality inspection method for Internet of Things data, a data quality inspection system for Internet of Things data, an electronic device, and a computer-readable storage medium. Background Art
[0002] Data quality inspection is an important means to ensure data accuracy, completeness, consistency and timeliness. It is widely used in various industries. Data quality inspection usually includes data integrity, accuracy, consistency, timeliness and uniqueness.
[0003] Common technologies for data quality checks on the market include ETL tools such as Informatica and Talend, which perform quality checks and cleansing during data extraction, transformation, and loading. Specialized data quality tools such as IBM InfoSphere Quality Stage and SAS Data Quality provide comprehensive data quality management capabilities. Real-time stream processing technologies such as Apache Kafka and Apache Flink are used to perform quality checks on real-time data streams to promptly detect and handle data quality issues. Data visualization tools such as Tableau and Power BI are used to create data quality dashboards to monitor data quality status in real time.
[0004] Although various data quality inspection tools already exist, the current data quality inspection technology for government IoT data still has limitations, mainly including:
[0005] 1. Heterogeneous data increases the complexity of quality inspection: Differences in data formats and structures brought about by heterogeneous data sources make data integration and quality inspection complicated. Different data systems and standards may lead to increased difficulty in data consistency and integration.
[0006] 2. High reliance on manual intervention: Although the data quality inspection process has been improved in terms of automation, a lot of manual intervention is still required in key links such as rule setting and data repair. This is not only time-consuming but also may introduce human errors.
[0007] 3. Difficulty in processing large-scale data: With the dramatic increase in data volume, traditional data quality inspection tools may be inefficient in processing large-scale data sets.
[0008] 4. Complexity of real-time data quality monitoring: Quality monitoring and inspection of real-time data streams requires the system to be able to identify and handle data problems immediately, but many existing data quality tools have limited support for real-time processing and may not be able to effectively handle immediate problems in data streams. Summary of the invention
[0009] In order to at least solve the problems in the prior art that the current data quality inspection technology for IoT data still has heterogeneous data that increases the complexity of quality inspection, relies on manual intervention, and has complex real-time data quality monitoring, the present disclosure provides a data quality inspection method for IoT data, a data quality inspection system for IoT data, an electronic device, and a computer-readable storage medium, which can solve the problem of inconsistent data sources, provide stronger openness for devices, realize real-time data quality inspection of connected IoT data, reduce manual intervention in data quality inspection and data cataloging, and improve data cataloging efficiency.
[0010] In a first aspect, the present disclosure provides a data quality inspection method for IoT data, which is applied to an IoT platform, and the method comprises:
[0011] Connect all devices to the IoT platform, perform protocol adaptation and model mapping on the IoT platform, clean various multi-object models into standard object models of the platform, and unify data models;
[0012] Based on the unified data model, the data of various connected devices are matched with the preset IoT data quality inspection rules for quality inspection;
[0013] After data quality inspection, the data sharing platform is provided with the equipment's mounting point table and mounting interface information, so that the mounting points and mounting interfaces can be imported into the data sharing platform to realize the cataloging and mounting of the quality-inspected data on the data sharing platform.
[0014] Furthermore, connecting all devices to the Internet of Things platform includes:
[0015] For incremental devices, they are connected to the IoT platform through direct device connection;
[0016] Existing devices that already exist on other platforms can be connected to the IoT platform through cloud-to-cloud docking.
[0017] Furthermore, the method further comprises:
[0018] According to the subordinate relationship of equipment data, equipment is divided according to administrative divisions and industry units;
[0019] Provide resource directory function to enable the platform to view the affiliation, equipment type and equipment quantity information of equipment data according to administrative divisions and industry units.
[0020] Furthermore, the preset IoT data quality inspection rules include:
[0021] Quality inspection rules set according to device name, device manufacturer, device type, device model, and device code;
[0022] Each quality inspection rule includes major rules and minor rules defined according to the data sharing platform. The major rules include data integrity, uniqueness, validity, consistency, accuracy and timeliness, and the minor rules include non-null constraints, uniqueness constraints, code value domain constraints, length constraints, value range constraints, existence consistency constraints, logical consistency constraints, value accuracy constraints and timeliness constraints.
[0023] Furthermore, the method further comprises:
[0024] Receive device data supplemented by the user. The user downloads the data table of each type of device on the IoT platform according to administrative divisions and industry units based on the data quality inspection results, supplements and corrects the data in a table form, and then re-uploads the table file to the IoT platform;
[0025] Perform quality check on the supplemented equipment data.
[0026] Furthermore, the method further comprises:
[0027] The data after quality inspection and equipment division is stored to form various equipment libraries, equipment relationship libraries, object model libraries and business subject libraries.
[0028] Furthermore, the data sharing platform is provided with a table of device attachment points and attachment interface information after data quality inspection, so as to import the attachment points and attachment interfaces into the data sharing platform, and catalog the data after quality inspection on the data sharing platform, including:
[0029] Provide the data sharing platform with the equipment's mounting point table and mounting interface information, so that the data sharing platform can fill in the mounting point and mounting interface information of the Internet of Things platform. When the user unit applies for data on the data sharing platform and it is approved, the data sharing platform sends the call point or interface method to the user unit, and the user unit calls the mounting point or mounting interface of the Internet of Things platform to obtain the corresponding equipment data information.
[0030] In a second aspect, the present disclosure provides a data quality inspection system for IoT data, the system comprising an IoT platform, the IoT platform comprising:
[0031] The access module is configured to connect all devices to the IoT platform, perform protocol adaptation and model mapping on the IoT platform, clean various multi-object models into standard object models of the platform, and unify the data model;
[0032] The quality inspection module is configured to perform quality inspection and matching of the data of various connected devices with the preset IoT data quality inspection rules based on the unified data model;
[0033] The business function module is configured to provide the data sharing platform with the equipment's mounting point table and mounting interface information after data quality inspection, so as to import the mounting points and mounting interfaces into the data sharing platform and realize the cataloging and mounting of the quality-inspected data on the data sharing platform.
[0034] In a third aspect, the present disclosure provides an electronic device, comprising a memory and a processor, wherein the memory stores a computer program, and when the processor runs the computer program stored in the memory, the processor executes the data quality inspection method for Internet of Things data as described in any one of the first aspects.
[0035] In a fourth aspect, the present disclosure provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the data quality inspection method for IoT data described in any one of the first aspects above is implemented.
[0036] Beneficial effects:
[0037] The data quality inspection method, data quality inspection system, electronic device and storage medium of the Internet of Things data provided by the present invention provide a unified object model capability based on the Internet of Things platform after all devices are connected to the Internet of Things platform, reduce the coupling between the object model and the device manufacturer through protocol adaptation and model mapping, provide stronger openness for the device, are not restricted by various types of devices and manufacturers, and realize data unification; they can also perform real-time data quality inspection on the connected Internet of Things data, and for the Internet of Things data that passes the data quality inspection, the platform provides a convenient mounting tool that can quickly provide equipment point table and interface information, so that the staff can conveniently and quickly mount the Internet of Things data information that has passed the quality inspection to the data public platform; reduce the manual intervention of data quality inspection and data catalog mounting, and improve the efficiency of data catalog mounting. BRIEF DESCRIPTION OF THE DRAWINGS
[0038] Figure 1 A flowchart of a data quality inspection method for IoT data provided in the first embodiment of the present disclosure;
[0039] Figure 2 A specific architecture diagram of an Internet of Things platform provided in Embodiment 2 of the present disclosure;
[0040] Figure 3 A business process diagram of a data quality inspection method for IoT data provided in Embodiment 2 of the present disclosure;
[0041] Figure 4 A platform interface diagram of a resource directory viewing result provided in the second embodiment of the present disclosure;
[0042] Figure 5 This is a platform interface diagram for viewing data quality inspection results provided in the second embodiment of the present disclosure;
[0043] Figure 6 A platform interface diagram for supplementing data provided in the second embodiment of the present disclosure;
[0044] Figure 7 A schematic diagram of cataloging and connecting IoT sensing device data on a data sharing platform provided in the second embodiment of the present disclosure;
[0045] Figure 8 An Internet of Things platform provided in the second embodiment of the present disclosure provides a table of device attachment points and an interface diagram of attachment interface information;
[0046] Fig. 9 A schematic diagram of an interface mounting and point mounting service provided in the second embodiment of the present disclosure;
[0047] Fig.10 This is an architecture diagram of an Internet of Things platform provided in Embodiment 3 of the present disclosure;
[0048] Fig.11 This is an architecture diagram of an electronic device provided in Embodiment 4 of the present disclosure. DETAILED DESCRIPTION
[0049] In order to enable those skilled in the art to better understand the technical solution of the present disclosure, the present disclosure is further described in detail below in conjunction with the drawings and embodiments. It should be understood that the specific embodiments and drawings described herein are only used to explain the present invention, rather than to limit the present invention.
[0050] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence; and, in the absence of conflict, the embodiments in the present disclosure and the features in the embodiments can be arbitrarily combined with each other.
[0051] The terms used in the embodiments of the present disclosure are only for the purpose of describing specific embodiments and are not intended to limit the present disclosure. The singular forms of "a", "said" and "the" used in the embodiments of the present disclosure and the appended claims are also intended to include plural forms unless the context clearly indicates other meanings.
[0052] In the subsequent description, the suffixes such as "module", "component" or "unit" used to represent elements are only used to facilitate the description of the present disclosure, and have no specific meanings. Therefore, "module", "component" or "unit" can be used in a mixed manner.
[0053] Data quality inspection is an important means to ensure data accuracy, completeness, consistency and timeliness, and is widely used in various industries. Data quality inspection usually includes aspects such as data integrity, accuracy, consistency, timeliness and uniqueness. In order to achieve these goals, related technologies may involve multiple fields such as rule engines, machine learning and artificial intelligence, database technology, data governance and management. The rule engine checks through predefined business rules, machine learning algorithms are used for anomaly detection and data repair, databases use triggers and stored procedures for real-time checks, and the data governance framework ensures comprehensive control from source to application. In addition, metadata management tools help track data sources and changes, and improve data transparency and credibility.
[0054] Data quality checks in IoT platforms rely on edge computing, standardized protocols, and real-time monitoring technologies to ensure the high quality and consistency of large amounts of heterogeneous data generated by sensors and devices.
[0055] Data quality inspection is mostly achieved through the following steps:
[0056] 1. Define data quality standards: Before data quality inspection, define clear data quality standards and indicators. These standards generally include data integrity, accuracy, consistency, timeliness and reliability.
[0057] 2. Data collection and organization: Collect data that needs to be quality checked. This includes collecting data from different data sources (such as databases, APIs, file systems, etc.). Preprocess the collected data, including data cleaning (such as removing duplicates and correcting format errors), data standardization (ensuring consistent data formats), and data fusion (integrating data from different sources).
[0058] 3. Data review and validation: Apply predefined data quality rules to review the data. These rules may include logic checks (such as value ranges, date validity), dependency checks (such as foreign key constraints), and pattern matching (such as regular expressions for checking email formats). Use automated data quality tools or software to enforce these rules to quickly identify and flag problem data.
[0059] 4. Data quality results: Generate data quality results, which describe the nature, location, and severity of data quality issues, and provide statistical analysis of the impact on data quality, such as error rate, percentage of missing values, etc.
[0060] 5. Data repair and improvement: Based on the findings in the report, fix the identified issues. This may involve manually correcting the data, applying algorithms to estimate missing values, or using automated tools to adjust the data. When necessary, update or optimize the rules and processes for data quality checks to avoid future quality issues.
[0061] 6. Continuous monitoring and maintenance: Implement continuous data quality monitoring mechanisms to regularly check and maintain the quality of data. This can be achieved through automated monitoring tools that can track data quality in real time and trigger alerts when a downward trend is detected. Continuously adjust and improve data quality standards and inspection processes based on changes in business needs and new data governance policies.
[0062] However, the current data quality inspection technology for government IoT data still has limitations. The current data quality inspection technology for IoT data still has problems such as heterogeneous data that increases the complexity of quality inspection, reliance on manual intervention, and complex real-time data quality monitoring.
[0063] The technical solution of the present invention and how the technical solution of the present invention solves the above-mentioned problems existing in the prior art are described in detail below with specific embodiments. It is understandable that in the embodiments of the present application, the execution subject can perform some or all of the steps in the embodiments of the present application, and these steps or operations are only examples. The embodiments of the present application can also perform other operations or deformations of various operations. In addition, each step can be performed in a different order presented in the embodiments of the present application, and it is possible not to perform all the operations in the embodiments of the present application. And, the following several specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments.
[0064] Figure 1 A flow chart of a data quality inspection method for IoT data provided in the first embodiment of the present disclosure is applied to an IoT platform, such as Figure 1 As shown, the method includes:
[0065] Step S101: Connect all devices to the IoT platform, perform protocol adaptation and model mapping on the IoT platform, clean various multi-object models into standard object models of the platform, and unify the data models;
[0066] Step S102: Based on the unified data model, the connected data of various devices are matched with the preset IoT data quality inspection rules for quality inspection;
[0067] Step S103: After data quality inspection, the data sharing platform is provided with the equipment's mounting point table and mounting interface information, so as to import the mounting points and mounting interfaces into the data sharing platform, and realize the cataloging and mounting of the quality inspected data on the data sharing platform.
[0068] The purpose of the disclosed embodiment is to collect multiple types and multiple sources of data based on the Internet of Things platform, and after mapping, converting and storing the data in a unified object model, perform data quality inspection according to the Internet of Things data quality inspection rules, and finally act on the data sharing platform to catalog and connect the Internet of Things data. The Internet of Things data can be government data or equipment data of various enterprises or industries.
[0069] After all devices are connected to the IoT platform, various data such as JSON and binary data from multiple networks and devices with different protocols are collected through the IoT platform. After the platform's protocol adaptation and object model mapping, the multiple object models of multiple platforms and different manufacturers can be cleaned into the platform's standard object model, thereby achieving the unification of the data model.
[0070] Protocol adaptation can solve many communication problems between heterogeneous devices. In the IoT environment, due to the wide variety of devices and the different communication protocols they follow, protocol adaptation is required to ensure that different devices can communicate with each other. It involves converting the original data format of the device into a unified or more general data format to facilitate data transmission and processing. It can be achieved through the adaptation method of the custom object model and the application of digital twin technology. The adaptation method of the custom object model is to solve the protocol differences of heterogeneous devices, the coupling of software and hardware development, and other problems through steps such as object model customization, object model information synchronization, object model mapping definition, adaptation access, and object model data output, so as to achieve flexible and standardized access of each device. Digital twin technology describes and models the characteristics, behaviors, states, and performance of physical entities by using digital technology, and establishes a virtual model that is completely corresponding and consistent with the physical entities in the real world, so as to realize real-time interaction between the physical world and the digital world. Model mapping is the process of mapping entities or processes in the physical world to the digital world. Through model mapping, the state, behavior, and other characteristics of physical devices can be converted into digital models, so as to be simulated and analyzed in the digital space. This process helps to unify the physical model and the data model, so that the monitoring, control and optimization of physical equipment can be carried out in a digital environment. Since the protocol adaptation and model mapping technologies are relatively mature, the specific implementation process will not be described here.
[0071] After the data model is unified, the data of various connected devices can be combined into a large-scale data set based on the unified data model, and the data can be transformed into a form that can be processed uniformly. The large-scale data set is matched with the preset quality inspection rules (data sharing platform IoT data requirements and standards) to complete the data quality inspection; a comprehensive quality inspection is performed on the data to ensure that the data's accuracy, consistency, completeness, standardization, timeliness, and accessibility meet specific requirements and standards, and specific inspection conditions are set according to business needs, such as text inspection, keyword inspection, etc., to ensure that the data meets the preset standards.
[0072] After data quality inspection, the cataloging and mounting function is provided, that is, the mounting point table and mounting interface information of the equipment are provided to the data sharing platform, so that the IoT device data can be cataloged and mounted on the data sharing platform conveniently and quickly.
[0073] After all devices are connected to the Internet of Things platform, the disclosed embodiment provides a unified object model capability based on the Internet of Things platform, reduces the coupling between the object model and the device manufacturer through protocol adaptation and model mapping, provides stronger openness for the device, is not restricted by various types of devices and manufacturers, and realizes data unification; it can also perform real-time data quality inspection on the connected Internet of Things data, and for the Internet of Things data that passes the data quality inspection, the platform provides a convenient attachment tool that can quickly provide equipment point tables and interface information, so that the staff can conveniently and quickly attach the Internet of Things data information that has passed the quality inspection to the data public platform; reduce the manual intervention of data quality inspection and data catalog attachment, and improve the efficiency of data catalog attachment.
[0074] Furthermore, connecting all devices to the Internet of Things platform includes:
[0075] For incremental devices, they are connected to the IoT platform through direct device connection;
[0076] Existing devices that already exist on other platforms can be connected to the IoT platform through cloud-to-cloud docking.
[0077] For incremental devices, they are directly connected to the IoT platform, while for devices on other platforms, the flexibility, stability and security of device access are improved by connecting platforms to each other, so that all devices can be connected to the IoT platform.
[0078] Furthermore, the method further comprises:
[0079] According to the subordinate relationship of equipment data, equipment is divided according to administrative divisions and industry units;
[0080] Provide resource directory function to enable the platform to view the affiliation, equipment type and equipment quantity information of equipment data according to administrative divisions and industry units.
[0081] The equipment is divided according to administrative divisions and industry units, and users can view basic equipment information and search equipment through administrative divisions and industry units. In addition, by providing a quality inspection function and supporting the viewing of quality inspection results, users can view the data quality inspection results of each type of equipment on the platform after data quality inspection, and grasp the data quality inspection status in real time.
[0082] Furthermore, the preset IoT data quality inspection rules include:
[0083] Quality inspection rules set according to device name, device manufacturer, device type, device model, and device code;
[0084] Each quality inspection rule includes major rules and minor rules defined according to the data sharing platform. The major rules include data integrity, uniqueness, validity, consistency, accuracy and timeliness, and the minor rules include non-null constraints, uniqueness constraints, code value domain constraints, length constraints, value range constraints, existence consistency constraints, logical consistency constraints, value accuracy constraints and timeliness constraints.
[0085] According to the requirements of the data sharing platform, quality inspection rules can be set for each field such as device name, device manufacturer, device type, device model, device code, etc. For each field, quality inspection is performed according to the major and minor rules of the set quality inspection rules.
[0086] Furthermore, the method further comprises:
[0087] Receive device data supplemented by the user. The user downloads the data table of each type of device on the IoT platform according to administrative divisions and industry units based on the data quality inspection results, supplements and corrects the data in a table form, and then re-uploads the table file to the IoT platform;
[0088] Perform quality check on the supplemented equipment data.
[0089] Based on the data quality inspection results, users can download data tables for each type of equipment on the IoT platform according to administrative divisions and industry units, and supplement the data. They can download data detail tables for each type of equipment, supplement and correct the data in table form, and then re-upload the table file. The system will perform quality inspection tasks asynchronously again.
[0090] Furthermore, the method further comprises:
[0091] The data after quality inspection and equipment division is stored to form various equipment libraries, equipment relationship libraries, object model libraries and business subject libraries.
[0092] Perform data matching and data update on the quality-inspected data, and store the data in the corresponding database.
[0093] Furthermore, the data sharing platform is provided with a table of device attachment points and attachment interface information after data quality inspection, so as to import the attachment points and attachment interfaces into the data sharing platform, and catalog the data after quality inspection on the data sharing platform, including:
[0094] Provide the data sharing platform with the equipment's mounting point table and mounting interface information, so that the data sharing platform can fill in the mounting point and mounting interface information of the Internet of Things platform. When the user unit applies for data on the data sharing platform and it is approved, the data sharing platform sends the call point or interface method to the user unit, and the user unit calls the mounting point or mounting interface of the Internet of Things platform to obtain the corresponding equipment data information.
[0095] According to the business process of the data sharing platform, the staff of each industry unit needs to log in to the data sharing platform to catalog (add new data) and connect the IoT sensing devices under the unit through point connection (add point) or interface connection. The IoT platform provides the equipment connection point table (corresponding to the "add point" of the data sharing platform) and connection interface information (corresponding to the "batch import" of the data sharing platform) through the catalog connection function. The staff can download it on the IoT platform and then upload it to the data sharing platform, which improves the staff's cataloging and connection efficiency on the data sharing platform.
[0096] The disclosed embodiments are based on the unified data model capability, data governance capability, and data storage capability of the Internet of Things platform, and fully describe the business processes for accessing, data quality inspection, cataloging, and attaching IoT sensing data, providing users with convenient data quality inspection functions and cataloging attachment tools, and integrating the services of the IoT platform and the data sharing platform: allowing users to obtain data quality inspection results in advance on the IoT platform and providing complete cataloging attachment tools. When subsequent users perform data cataloging and attachment on the data sharing platform, they no longer need to manually count device data for attachment, thereby assisting in IoT data sharing and exchange.
[0097] The second embodiment of the present disclosure also provides a data quality inspection method for IoT data, which is applied to government IoT data processing. The method is implemented through an IoT platform. The IoT platform is mainly composed of an IoT access module, a data governance module, a data storage module and a business function module. The specific architecture is as follows Figure 2 As shown,
[0098] 1) IoT access module: Based on the IoT platform, multiple networks, different protocol devices and various data such as JSON, binary, etc. are collected. After the platform's protocol adaptation and object model mapping, the multiple object models of multiple platforms and different manufacturers can be cleaned into the platform's standard object model to achieve the unification of the data model.
[0099] 2) Data governance module: After the data model is unified, a series of business data governance will be carried out in advance for large-scale data sets, such as:
[0100] a) Match the data from various connected devices with the IoT data quality inspection rules of the digital government’s data sharing platform (“One Network Sharing Platform”).
[0101] b) The equipment will be divided according to the subordinate relationship of the equipment data, administrative divisions and industry units.
[0102] c) Data governance can be further customized according to customer needs. Here, we can define major categories of data quality inspection rules. The major categories mainly involve data integrity, uniqueness, validity, consistency, accuracy, and timeliness; the minor categories mainly include non-null constraints, uniqueness constraints, code value domain constraints, length constraints, value range constraints, existence consistency constraints, logical consistency constraints, value accuracy constraints, timeliness constraints, etc.
[0103] 3) Data storage module: After passing through the data governance module, the governed data is stored to form various types of equipment libraries, equipment relationship libraries, object model libraries, and business subject libraries.
[0104] 4) Business function module: It has the functions of resource directory, data pre-quality inspection, data supplement and cataloging connection. Through the data supplement function of the business function module, the data quality inspection function of the data governance module is used, and it acts on the update of data storage to form a closed loop:
[0105] a) The resource directory function divides the equipment according to administrative divisions and industry units. Users can view basic information and search for equipment by administrative divisions and industry units.
[0106] b) The data pre-quality inspection function conducts quality inspection on each type of equipment based on the IoT data quality inspection rules of the data sharing platform ("One Network Sharing Platform"). This function supports viewing of quality inspection results.
[0107] c) The data supplement function means that users can manually make up for the data by re-uploading the equipment data sheet, and the system will asynchronously perform a second quality inspection task.
[0108] d) The cataloging and mounting function refers to the function that can quickly catalog and mount the IoT device data on the data sharing platform ("one-network sharing platform").
[0109] Main business processes such as Figure 3 As shown, including:
[0110] 1. Incremental devices are connected to the Internet of Things platform (also known as the Internet of Things perception platform) through direct device connection.
[0111] 2. Existing devices that already exist on other platforms are connected to the IoT platform through cloud-to-cloud docking.
[0112] 3. At this point, all devices are connected to the IoT platform, and protocol adaptation and model mapping are performed through the IoT access module of the IoT platform to achieve the unification of the object model and data model.
[0113] 4. The platform will match the data of various devices connected to it with the IoT data quality inspection rules of the digital government's data sharing platform ("One Network Sharing Platform"). Currently, the data sharing platform has set quality inspection rules for 16 fields such as device name, device manufacturer, device type, device model, and device code.
[0114] 5. Users can use the resource directory function on the platform to view the subordinate relationship, device type, device quantity and other information of device data according to administrative divisions and industry units. The results of a resource directory are as follows: Figure 4 shown.
[0115] 6. Users can use the data quality inspection function on the platform to view the data quality inspection results of each type of equipment based on the platform after point 4, such as Figure 5 The following is a data quality check result.
[0116] 7. Based on the data quality inspection results, users can download the data sheet of each type of equipment according to administrative divisions and industry units on the platform and manually supplement the data, such as Figure 6 As shown, the data details table of each type of equipment can be downloaded through the corresponding function of the platform, and the data can be supplemented and corrected in the form of a table, and then the table file can be re-uploaded, and the system will perform the quality inspection task again.
[0117] 8. According to the business process of the digital government data sharing platform ("One Network Sharing Platform"), government staff of various industry units need to log in to the data sharing platform to catalog (add new data) and connect the IoT sensing devices under the unit through point connection (adding points) or interface connection (download the template, fill in the form and upload it). The specific cataloging and connection process is as follows: Figure 7 shown.
[0118] 9. The cataloging and mounting function of this platform provides a table of mounting points for the equipment (corresponding to the "add points" of the one-network sharing platform) and mounting interface information (corresponding to the "batch import" of the one-network sharing platform), such as Figure 8 As shown in the Internet of Things platform page, government workers can download data from this platform and then upload it to the data sharing platform (one-network sharing platform), which improves the efficiency of cataloging and connection of government workers on the data sharing platform.
[0119] Interface mounting business process and point mounting business process such as Fig. 9 shown.
[0120] The IoT perception platform encapsulates the data interface, the One-Network Sharing Platform interface is attached, the One-Network Sharing Platform fills in the interface information encapsulated by the IoT perception platform, and the user unit applies for data on the One-Network Sharing Platform. After the data application is approved, the One-Network Sharing Platform will send the interface call method to the user unit, and the user unit will call the IoT perception platform encapsulated interface to obtain the corresponding device data information.
[0121] The disclosed embodiments are based on the unified data model capability, data governance capability, and data storage capability of the Internet of Things platform, and fully describe the business processes for accessing, data quality inspection, cataloging, and attaching IoT perception data, providing government users with convenient data quality inspection functions and cataloging attachment tools, integrating the services of the IoT perception platform and the One-Network Sharing Platform: allowing government users to obtain data quality inspection results in advance on the IoT perception platform and provide complete cataloging attachment tools. When government users subsequently perform data cataloging and attachment on the One-Network Sharing Platform, they no longer need to manually count device data for attachment, thereby assisting government IoT data sharing and exchange.
[0122] The third embodiment of the present disclosure also provides a data quality inspection system for IoT data, the system includes an IoT platform, such as Fig.10 As shown, the Internet of Things platform includes:
[0123] The access module 11 is configured to connect all devices to the IoT platform, perform protocol adaptation and model mapping on the IoT platform, clean various multi-object models into standard object models of the platform, and achieve data model unification;
[0124] The quality inspection module 12 is configured to perform quality inspection and matching of the data of various types of connected devices with the preset IoT data quality inspection rules based on the unified data model;
[0125] The business function module 13 is configured to provide the data sharing platform with the equipment's mounting point table and mounting interface information after data quality inspection, so as to import the mounting points and mounting interfaces into the data sharing platform and realize the cataloging and mounting of the quality inspected data on the data sharing platform.
[0126] Furthermore, the access module 11 is specifically configured as follows:
[0127] For incremental devices, they are connected to the IoT platform through direct device connection;
[0128] Existing devices that already exist on other platforms can be connected to the IoT platform through cloud-to-cloud docking.
[0129] Furthermore, the Internet of Things platform also includes a data partitioning and viewing module 14:
[0130] The data division and viewing module 14 is configured to divide the devices according to the subordinate relationship of the device data, administrative divisions, and industry units; and
[0131] Provide resource directory function to enable the platform to view the affiliation, equipment type and equipment quantity information of equipment data according to administrative divisions and industry units.
[0132] Furthermore, the preset IoT data quality inspection rules include:
[0133] Quality inspection rules set according to device name, device manufacturer, device type, device model, and device code;
[0134] Each quality inspection rule includes major rules and minor rules defined according to the data sharing platform. The major rules include data integrity, uniqueness, validity, consistency, accuracy and timeliness, and the minor rules include non-null constraints, uniqueness constraints, code value domain constraints, length constraints, value range constraints, existence consistency constraints, logical consistency constraints, value accuracy constraints and timeliness constraints.
[0135] Furthermore, the Internet of Things platform also includes a supplementary module 15:
[0136] The supplement module 15 is configured to receive device data supplemented by the user. The device data is downloaded by the user from the IoT platform by administrative divisions and industry units according to the data quality inspection results, and the data is supplemented and corrected in a table form, and then the table file is re-uploaded to the IoT platform; and
[0137] The quality inspection module 12 performs quality inspection on the supplemented device data.
[0138] Furthermore, the Internet of Things platform also includes a storage module 16;
[0139] The storage module 16 is configured to store data after quality inspection and equipment division, forming various equipment libraries, equipment relationship libraries, object model libraries and business subject libraries.
[0140] Furthermore, the business function module 13 is specifically configured as follows:
[0141] Provide the data sharing platform with the equipment's mounting point table and mounting interface information, so that the data sharing platform can fill in the mounting point and mounting interface information of the Internet of Things platform. When the user unit applies for data on the data sharing platform and it is approved, the data sharing platform sends the call point or interface method to the user unit, and the user unit calls the mounting point or mounting interface of the Internet of Things platform to obtain the corresponding equipment data information.
[0142] The data quality inspection system for IoT data in the disclosed embodiment is used to implement the data quality inspection method for IoT data in method embodiment 1 and embodiment 2, so the description is relatively simple. For details, please refer to the relevant description in the previous method embodiment, which will not be repeated here.
[0143] In addition, if Fig.11 As shown, the fourth embodiment of the present disclosure further provides an electronic device, including a memory 100 and a processor 200, wherein the memory 100 stores a computer program, and when the processor 200 runs the computer program stored in the memory 100, the processor 200 executes the above-mentioned various possible methods.
[0144] The memory 100 is connected to the processor 200. The memory 100 may be a flash memory, a read-only memory or other memory, and the processor 200 may be a central processing unit or a single-chip microcomputer.
[0145] In addition, an embodiment of the present disclosure further provides a computer-readable storage medium, on which a computer program is stored, and the computer program is executed by a processor to perform the above-mentioned various possible methods.
[0146] The computer-readable storage medium includes volatile or non-volatile, removable or non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, computer program modules or other data). Computer-readable storage media include, but are not limited to, RAM (Random Access Memory), ROM (Read-Only Memory), EEPROM (Electrically Erasable Programmable read only memory), flash memory or other memory technology, CD-ROM (Compact Disc Read-Only Memory), Digital Video Disc (DVD) or other optical disk storage, magnetic cassettes, magnetic tapes, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and can be accessed by a computer.
[0147] It is to be understood that the above embodiments are merely exemplary embodiments used to illustrate the principles of the present disclosure, but the present disclosure is not limited thereto. For those of ordinary skill in the art, various modifications and improvements can be made without departing from the spirit and substance of the present disclosure, and these modifications and improvements are also considered to be within the scope of protection of the present disclosure.
Claims
1. A data quality inspection method for IoT data, characterized in that: Applied to the Internet of Things platform, the method includes: Connect all devices to the IoT platform, perform protocol adaptation and model mapping on the IoT platform, clean various multi-object models into standard object models of the platform, and unify data models; Based on the unified data model, the data of various connected devices are matched with the preset IoT data quality inspection rules for quality inspection; After data quality inspection, the data sharing platform is provided with the equipment's mounting point table and mounting interface information, so that the mounting points and mounting interfaces can be imported into the data sharing platform to realize the cataloging and mounting of the quality-inspected data on the data sharing platform.
2. The method according to claim 1, characterized in that Connecting all devices to the IoT platform includes: For incremental devices, they are connected to the IoT platform through direct device connection; Existing devices that already exist on other platforms can be connected to the IoT platform through cloud-to-cloud docking.
3. The method according to claim 1, characterized in that: The method further comprises: According to the subordinate relationship of equipment data, equipment is divided according to administrative divisions and industry units; Provide resource directory function to enable the platform to view the affiliation, equipment type and equipment quantity information of equipment data according to administrative divisions and industry units.
4. The method according to claim 1, characterized in that: The preset IoT data quality inspection rules include: Quality inspection rules set according to device name, device manufacturer, device type, device model, and device code; Each quality inspection rule includes major rules and minor rules defined according to the data sharing platform. The major rules include data integrity, uniqueness, validity, consistency, accuracy and timeliness, and the minor rules include non-null constraints, uniqueness constraints, code value domain constraints, length constraints, value range constraints, existence consistency constraints, logical consistency constraints, value accuracy constraints and timeliness constraints.
5. The method according to claim 1, characterized in that The method further comprises: Receive device data supplemented by the user. The user downloads the data table of each type of device on the IoT platform according to administrative divisions and industry units based on the data quality inspection results, supplements and corrects the data in a table form, and then re-uploads the table file to the IoT platform; Perform quality inspection on the supplemented equipment data.
6. The method according to claim 3, characterized in that The method further comprises: The data after quality inspection and equipment division is stored to form various equipment libraries, equipment relationship libraries, object model libraries and business subject libraries.
7. The method according to claim 1, characterized in that After the data quality inspection, the data sharing platform is provided with the equipment mounting point table and mounting interface information, so as to import the mounting points and mounting interfaces into the data sharing platform, and realize the cataloging and mounting of the data after the quality inspection on the data sharing platform, including: Provide the data sharing platform with the equipment's mounting point table and mounting interface information, so that the data sharing platform can fill in the mounting point and mounting interface information of the Internet of Things platform. When the user unit applies for data on the data sharing platform and it is approved, the data sharing platform sends the call point or interface method to the user unit, and the user unit calls the mounting point or mounting interface of the Internet of Things platform to obtain the corresponding equipment data information.
8. A data quality inspection system for IoT data, characterized in that: The system includes an Internet of Things platform, and the Internet of Things platform includes: The access module is configured to connect all devices to the IoT platform, perform protocol adaptation and model mapping on the IoT platform, clean various multi-object models into standard object models of the platform, and unify the data model; The quality inspection module is configured to perform quality inspection and matching of the data of various connected devices with the preset IoT data quality inspection rules based on the unified data model; The business function module is configured to provide the data sharing platform with the equipment's mounting point table and mounting interface information after data quality inspection, so as to import the mounting points and mounting interfaces into the data sharing platform and realize the cataloging and mounting of the quality-inspected data on the data sharing platform.
9. An electronic device, characterized in that: It includes a memory and a processor, wherein the memory stores a computer program, and when the processor runs the computer program stored in the memory, the processor executes the data quality inspection method for Internet of Things data according to any one of claims 1 to 7.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the data quality inspection method for IoT data according to any one of claims 1 to 7 is implemented.
Citation Information
Patent Citations
Data quality inspection system and method based on EPDM data model
CN114490615A
Industrial object model based on middleware of Internet of Things and implementation method of industrial object model
CN114510219A
One-stop big data management service platform for environmental industry
CN117371945A
Cited By
Government-affair-oriented Internet of Things data sharing and data quality inspection method, system and device based on large model, and storage medium
CN120670415A
A government affair based on large model internet of things data sharing and data quality inspection method, system, device and storage medium
CN120670415B
