A multi-source heterogeneous data access system and method for the Internet of Things

Abstract

The application discloses a kind of multi-source heterogeneous data access systems and methods for Internet of Things, system includes: equipment access management module, for shielding multi-source equipment data heterogeneity, unified management and standardized access to Internet of Things equipment and data;Data analysis fusion module is used to different sources, different formats, various structures, fragmented perception data are summarized, processed, aggregated according to business needs;Data storage module is used to guide the efficient storage and classified management of unstructured data and structured data generated by equipment.This application solves the problem of how to integrate control the massive equipment in the field of Internet of Things, how to real-time and fast guide the massive multi-source heterogeneous data, how to clean up and integrate the scattered chaotic data and quality management.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of data connection and processing of the Internet of Things, and particularly relates to a multi-source heterogeneous data connection system and method for the Internet of Things. BACKGROUND

[0002] As the core field of a new round of information industry revolution, the Internet of Things has attracted more business demands, and a large number of Internet of Things devices are connected to the Internet of Things platform. A large amount of unstructured data and semi-structured data such as XML, JSON, audio, picture and video information needs to be able to store data types in multiple formats. Meanwhile, a large number of Internet of Things devices lack unified communication standards and security standards, and the platform needs to be coded and connected for each type of device, resulting in high complexity and coupling of the platform. In addition, the sensing data comes from devices of different manufacturers, is obtained by various means, has different data formats, and has problems such as conflict, repetition and loss.

[0003] The Internet of Things data has the characteristics of massiveness, complexity, multi-source and heterogeneity, and higher requirements for connection, storage and analysis of the Internet of Things data connection system. Therefore, how to quickly connect, analyze, fuse and efficiently store multi-source heterogeneous data with massiveness, wide sources, uncertainty and weak association is a technical problem that needs to be solved for the practicality, universality and efficiency of the Internet of Things platform. SUMMARY

[0004] One of the purposes of the application is to provide a multi-source heterogeneous data connection system for the Internet of Things, which solves the problems of multi-source connection, cleaning and conversion, and efficient storage of sensing data in view of the characteristics of wide sources, few connection means, various data formats and low analysis degree.

[0005] To achieve the above purpose, the technical solution adopted by the application is as follows:

[0006] A multi-source heterogeneous data connection system for the Internet of Things, comprising a device connection management module, a data analysis and fusion module and a data storage module, wherein:

[0007] The device connection management module is used to uniformly connect a large number of multi-source heterogeneous sensing devices, and provides a development data service bus for the application layer through a unified Internet of Things data protocol. The device connection management module comprises a device management unit, an interface protocol connection unit and a text data connection unit, wherein:

[0008] The device management unit is configured to manage and perform security access authentication on the accessed sensing device, and determine whether the device meets a standard protocol access, and to pre-establish a data structure conversion model for the sensing device that is incompatible with the standard protocol;

[0009] The interface protocol connection unit is configured to formulate a metadata model, a data format, and a data acquisition standard protocol interface for various types of sensing data of the multi-source heterogeneous sensing device;

[0010] The text data connection unit is configured to receive text data of different formats transmitted by the sensing device through a file interface;

[0011] The data analysis and fusion module is configured to convert a large amount of structured data and semi-structured data received by the device access management module into structured data of the same type and isomorphism, and transmit the structured data to the data storage module;

[0012] The data storage module is configured to store and manage the semi-structured data and unstructured data received by the device access management module and the structured data transmitted by the data analysis and fusion module.

[0013] Several optional modes are also provided below, but are not additional limitations on the above overall scheme, but are only further supplements or preferences, and each optional mode can be combined with the above overall scheme alone, and can also be combined between multiple optional modes, without technical or logical contradiction.

[0014] Preferably, the interface protocol connection unit comprises a standard protocol service unit and a private interface adaptation unit;

[0015] The standard protocol service unit is configured to provide a unified standard interface service for the same attribute sensing device to connect multi-source heterogeneous data; the multi-source heterogeneous data comprises structured data, semi-structured data, and unstructured data generated by different devices;

[0016] The private interface adaptation unit is configured to convert the access data into standard interface data by using an extended interface and a data structure conversion model for the sensing device that is incompatible with the standard protocol.

[0017] Preferably, the standard protocol service unit defines the multi-source heterogeneous data as four types of standard protocol data, i.e., reporting data, state data, control instruction data, and linkage instruction data, according to the device data attribute;

[0018] The standard protocol service unit defines a corresponding interface according to each type of standard protocol data, including a device acquisition data interface, a device state reporting interface, a device control interface, and a device linkage interface, wherein:

[0019] The device collection data interface is used for receiving the reported data collected by the sensing device, including target data, alarm information and current working parameters thereof;

[0020] The device state reporting interface is used for receiving the state data reported by the sensing device in a timely manner according to the interface protocol, including heartbeat state data, time setting information and power-on / off information;

[0021] The device control interface is used for converting the control instruction data of the sensing device into interface parameters and transmitting the interface parameters to the sensing device, so as to control the working parameters or state of the sensing device;

[0022] The device linkage interface is used for issuing a linkage instruction data request of tracking target information to a disposal device, wherein the target information is the target information of the received detection device.

[0023] Preferably, the data analysis and fusion module comprises a data analysis and classification unit, a data cleaning unit and a data conversion unit, wherein:

[0024] The data analysis and classification unit is used for converting semi-structured text data in the data storage module into structured data and transmitting the structured data to the data cleaning unit; the semi-structured text data includes binary text data, XML text data and JSON text data, and the semi-structured text data is sent by the device access management module to the data storage module for caching;

[0025] The data cleaning unit is used for performing repeated elimination, missing supplement and abnormal correction operations on the structured data processed by the data analysis and classification unit and the structured data directly accessed by the device access management module, and then transmitting the structured data to the data conversion unit; the structured data includes structured text data, and the structured text data includes CSV and XLS format text data;

[0026] The data conversion unit is used for transmitting the structured data sent by the data cleaning unit to the data storage module after data extraction by an ETL tool.

[0027] Preferably, the data analysis and classification unit uses an improved distributed parallel computing framework MapReduce to perform parallel processing and analysis on a large amount of semi-structured text data stored and cached in the data storage module, and quickly and real-timely converts the semi-structured text data into structured data of a single device, and performs the following operations:

[0028] Step a: reading semi-structured text data to be operated from a storage area, and dividing the data into a plurality of logical fragments, wherein the size of the fragments = total memory size of the read file / hdfs block default size;

[0029] Step b: calling the Map end to parse the shard data into a series of key-value pairs <key, value> for subsequent processing, and the shard data of the same text type has the same key key;

[0030] Step c: according to the business requirements of the sensing device and the text type, the key-value pairs processed by the Map end are divided into n partition processing;

[0031] Step d: sorting the shard data of each partition from small to large, and merging the value with the same key key in the sorted partition;

[0032] Step e: calling the Reduce end to classify the merged value in the partition, and sorting the shard data of the same text type into the same type of text;

[0033] Step f: reading the classified text data, dividing it into structured data streams of single devices according to the sensing device type and the field length of the sensing device, and transmitting it to the data cleaning unit.

[0034] The multi-source heterogeneous data access system for the Internet of Things provided by the application has the following beneficial effects compared with the prior art:

[0035] (1) The device access management module can realize the compatibility of multiple sensing device communication standards and security protocols, quickly and automatically collect massive sensing data, adaptively adapt to the private interface protocol of the sensing device, and reduce the complexity and coupling degree of the platform.

[0036] (2) The data analysis and fusion module provides a distributed parallel computing framework to quickly process large amounts of device sensing data in real time, preventing data packet loss and congestion.

[0037] (3) The data storage module provides a distributed, multi-format, and high-security storage architecture to ensure classified storage of sensing data and high computing power requirements.

[0038] The second purpose of the application is to provide a multi-source heterogeneous data access method for the Internet of Things, which solves the problems of multi-source connection, cleaning and conversion, and efficient storage of sensing data.

[0039] To achieve the above purpose, the technical scheme adopted by the application is:

[0040] A multi-source heterogeneous data access method for the Internet of Things, the multi-source heterogeneous data access method for the Internet of Things, comprising:

[0041] Step 1, add information of access sensing device, and judge whether the sensing device meets the standard interface protocol, add data structure conversion model for sensing device incompatible with standard protocol;

[0042] Step 2, judge the data type transmitted by the sensing device, execute step 6 if the data type is text data, and execute step 3 if the data type is interface protocol data;

[0043] Step 3, judge whether the sensing device is compatible with the standard interface service, execute step 4 if compatible, otherwise execute step 5;

[0044] Step 4, for sensing device compatible with standard protocol, perform multi-source heterogeneous data connection according to standard interface protocol, and execute step 6;

[0045] Step 5, for sensing device incompatible with standard protocol, use extended interface and data structure conversion model to convert access data into standard interface data, realize multi-source heterogeneous data connection of this kind of sensing device, and execute step 6;

[0046] Step 6, convert the accessed structured data and semi-structured text data into structured data of the same type and structure, and execute step 7;

[0047] Step 7, store and manage the semi-structured data and unstructured data in the directly connected multi-source heterogeneous data and the structured data obtained in step 6 independently.

[0048] As a preferred, the conversion of the accessed structured data and semi-structured data into structured data of the same type and structure comprises:

[0049] Step 6.1, convert the accessed semi-structured text data into structured data;

[0050] Step 6.2, perform repeated elimination, missing supplement and exception correction operations on the structured data processed in step 6.1 and the directly accessed structured data;

[0051] Step 6.3, use ETL tool to extract data from the structured data processed in step 6.2 to obtain structured data of the same type and structure.

[0052] As a preferred, the conversion of the accessed semi-structured text data into structured data comprises using improved distributed parallel computing framework MapReduce to perform parallel processing and analysis on large batches of semi-structured text data stored and cached in a distributed manner, and quickly and real-time convert them into structured data of a single device, the specific steps are as follows:

[0053] Step a: reading semi-structured text data to be operated from a storage area, splitting the data into a plurality of logical fragments, wherein the size of the fragments = the total memory size of the read file / the default size of the hdfs block;

[0054] Step b: calling the Map end to parse the fragment data into a series of key-value pairs <key, value> for subsequent processing, and the fragment data of the same text type has the same key key;

[0055] Step c: dividing the key-value pairs processed by the Map end into n partitions according to the business requirements of the sensing device and the text type;

[0056] Step d: sorting each partition of the fragment data from small to large, and merging the values with the same key key in the sorted partition;

[0057] Step e: calling the Reduce end to classify the merged values in the partition, and sorting the fragment data of the same text type into the same type of text;

[0058] Step f: reading the classified text data, and dividing it into single device structured data streams according to the sensing device type and the field length of the sensing device, and completing the data conversion.

[0059] The multi-source heterogeneous data access method for the Internet of Things provided by the application has the following beneficial effects compared with the prior art:

[0060] (1) The device access management module can realize the compatibility of multiple sensing device communication standards and security protocols, quickly and automatically collect massive sensing data, adaptively adapt to the sensing device private interface protocol, and reduce the complexity and coupling degree of the platform.

[0061] (2) The data analysis and fusion module realizes multi-source heterogeneous data fusion and provides accurate, standard and high-quality data for the platform.

[0062] (3) The data storage module provides a distributed, multi-format and high-security storage architecture to ensure the classification storage of sensing data and high computing power requirements. BRIEF DESCRIPTION OF DRAWINGS

[0063] Figure 1 The structure diagram of the multi-source heterogeneous data access system for the Internet of Things of the application;

[0064] Figure 2 The flowchart of the data analysis and classification unit of the application for converting semi-structured data into structured data;

[0065] Figure 3 The flowchart of the multi-source heterogeneous data access method for the Internet of Things of the application. DETAILED DESCRIPTION

[0066] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative effort belong to the scope of protection of the present application.

[0067] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present application belongs. The terminology used in the description herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the present application.

[0068] In order to solve the problem that it is difficult to realize multi-source connection, cleaning conversion and efficient storage for multi-source heterogeneous perception data in the prior art, the embodiment provides a multi-source heterogeneous data access system for Internet of Things.

[0069] As shown in the figure, the multi-source heterogeneous data access system for Internet of Things in the embodiment mainly comprises a device access management module, a data analysis and fusion module and a data storage module. Figure 1

[0070] 1) The device access management module of the multi-source heterogeneous data access system, used for uniformly accessing multi-data structure, cross-unit and cross-system perception devices, and providing a development data service bus for an application layer through a unified Internet of Things data protocol.

[0071] The embodiment unifies management and access of Internet of Things perception devices and data through the device access management module, shields multi-source device data heterogeneity, realizes full-source information connection through open multi-type protocols for multi-source heterogeneous devices, and supports access to real-time stream data, audio, video, pictures, binary data, files and other types of data.

[0072] In the embodiment, the Internet of Things perception devices can be cameras, radars, photoelectric devices, electronic fences, unmanned aerial vehicles, vibration optical fibers, anti-unmanned aerial vehicle devices, weather monitoring sensors and the like. Of course, they can also be other Internet of Things perception devices or sensors, and the embodiment is not limited in particular.

[0073] In order to improve the adaptability of the multi-source heterogeneous data access system in the embodiment, the device access management module in the embodiment defines interface standard and sets standard protocol interface, and simultaneously adapts to private interfaces. Specifically, the device access management module comprises a device management unit, an interface protocol connection unit and a text data connection unit.

[0074] ​1-1) Device management unit for managing and securely accessing the connected sensing device and determining whether the device meets the standard protocol access, and pre-establishing a data structure conversion model for sensing devices that do not comply with the standard protocol.

[0075] 1-2) Interface protocol connection unit for the system to formulate metadata models, data formats, and data collection standard protocol interfaces for various types of sensing data of multi-source heterogeneous sensing devices, and to realize data collection level interconnection and intercommunication for data access.

[0076] The interface protocol connection unit of the embodiment formulates data receiving protocol standards to realize real-time, concurrent, and rapid collection of multi-source heterogeneous data of different sources, means, and formats, and to realize dynamic access of various forms of sensing information in the Internet of Things. Specifically, in one embodiment, the interface protocol connection unit includes a standard protocol service unit and a private interface adaptation unit.

[0077] 1-2-1) Standard protocol service unit for providing uniform standard interface services for multi-source heterogeneous data connection for sensing devices of the same attribute; multi-source heterogeneous data includes structured data, semi-structured data, and unstructured data generated by different devices. The structured data here includes structured interface protocol data and structured text data, and the semi-structured data includes semi-structured text data.

[0078] The embodiment formulates self-defined open standard protocols for data items, data structures, data storage, and data interfaces transmitted by sensing devices that comply with the system's self-defined standard interface, to ensure standardized access and management of isomorphic data of the same type of device.

[0079] To improve the flexibility of data acquisition and the adaptability of the standard interface of the embodiment, the embodiment defines multi-source heterogeneous data as four types of standard protocol data, namely, reported data, state data, control instruction data, and linkage instruction data, according to the device data attributes, and provides standard interface services including device collection data interface, device state reporting interface, device control interface, and device linkage interface.

[0080] A, Device collection data interface for the system to receive the target data collected by the connected device, alarm information, and current working parameters, etc. reported data, to provide data support for the business layer.

[0081] B, Device state reporting interface for receiving device state data reported by sensing devices according to the interface protocol; that is, the interface is used for state data reporting between the system and the connected device. The device reports state data, heartbeat state data, time setting information, and power-on / off information to the system according to the interface protocol.

[0082] C. Device control interface, for converting perception device control instruction data into interface parameters and transmitting to the perception device to control the working parameters or state of the perception device. That is, the interface is used for the system to control some device working parameters or state, and the system receives device control instructions and converts them into interface parameters and transmits them to the device.

[0083] D. Device linkage interface, for issuing a data request for tracking target information to a treatment device, the target information being the target information received by the probe device. That is, the interface is used for the system to receive the target information of the probe device, and after analysis by the upper layer system, to issue a data request for tracking target information to the treatment device.

[0084] 1-2-2) Private interface adaptation unit, using an extended interface and a data structure conversion model to convert the perception data of perception devices that cannot be upgraded to compatible standard interface services into standard interface data.

[0085] This embodiment is aimed at the case where some devices cannot be upgraded and can only be connected through device private protocols. The adaptive extended private interface and information conversion model are used to quickly connect such devices, so as to ensure that the multi-source heterogeneous data access system of the embodiment has high adaptability.

[0086] 1-3) Text data connection unit, for the system to receive text data of different formats such as binary, JSON, XML, CSV, XLS, etc. transmitted by perception devices through a file interface. The text data introduced by the text data connection unit includes structured text data and semi-structured text data.

[0087] 2) Data analysis and fusion module of the multi-source heterogeneous data access system, for converting the structured data and semi-structured data received by the device access management module into high-quality structured data of the same type and structure.

[0088] This embodiment converts a large amount of data of different sources and various structures (structured and semi-structured) into the same type and structure in real time and quickly according to business needs through summarization, verification, and aggregation. Since direct interface parsing of semi-structured data may cause data congestion, the semi-structured data to be parsed is first cached, and then processed by the data analysis and fusion module to realize real-time data parsing.

[0089] Specifically, in one embodiment, the data analysis and fusion module includes a data parsing and classification unit, a data cleaning unit, and a data conversion unit.

[0090] 2-1) Data parsing and classification unit, for obtaining semi-structured text data from the data storage module and converting it into structured data and transmitting it to the data cleaning unit. The semi-structured text data includes binary text data, XML text data, and JSON text data.

[0091] The data analysis and classification unit of the embodiment converts binary, JSON, XML and other format data obtained by standard interface service and private interface adaptation into structured data. In order to prevent data packet loss blocking and ensure data real-time performance, the improved distributed parallel computing framework MapReduce is used to quickly convert semi-structured text data into structured data.

[0092] As shown in Figure 2 The data analysis and classification unit uses the distributed parallel computing framework MapReduce to analyze binary, JSON, XML and other format files, and performs the following operations:

[0093] Step a: read the semi-structured text data to be operated from the storage area, and divide the data into a plurality of logical fragments, wherein the fragment size is defined according to the data size and the distributed server block memory, and the default size = the total memory size of the read file / hdfs block default size, and the hdfs block size of the embodiment is 128M;

[0094] Step b: call the Map end to parse the fragment data into a series of key-value pairs <key, value> for subsequent processing, and the fragment data of the same text type has the same key;

[0095] Step c: according to the business requirements and data types of the sensing device, the key-value pairs processed by the Map end are divided into n partitions for processing, and the value of n in the embodiment is 3;

[0096] Step d: sort the fragment data of each partition from small to large, and merge the values with the same key in the sorted partition;

[0097] Step e: call the Reduce end to classify the merged values in the partition, and arrange the fragment data of the same text type to the same type of text;

[0098] Step f: call the InputFormat method of MapReduce to read the classified text data, divide it into single device sensing data streams according to the sensing device type and field length in the data, and transmit it to the data cleaning unit.

[0099] 2-2) Data cleaning unit, used for repeated elimination, missing supplement and abnormal correction operation on the structured data processed by the data analysis and classification unit and the structured data directly accessed by the device access management module (including the structured data accessed by the interface protocol adapter unit and the structured text data accessed by the text data adapter unit), and then transmitted to the data conversion unit.

[0100] In order to obtain high-quality data, the embodiment carries out data consistency check, repeated data elimination, missing data supplement, abnormal data correction and other operations on the classified files or directly accessed structured data by the data cleaning unit, and generates data meeting the quality requirements and subsequent data application requirements.

[0101] 2-3) Data conversion unit, for transmitting the structured data sent by the data cleaning unit to the data storage module after data extraction by the ETL tool.

[0102] The embodiment carries out standard conversion, data code translation, data field normalization and other operations on the data according to the formulated data type standard by the ETL tool, obtains standard and high-quality data, and transmits the data to the data storage module.

[0103] 3) Multi-source heterogeneous data access system data storage module, for independently storing and managing the semi-structured data and unstructured data received by the device access management module and the structured data sent by the data analysis and fusion module.

[0104] The embodiment carries out targeted classification storage and management on the unstructured data, semi-structured data and structured data generated by the access device. The data storage module is specifically set to include an object storage unit, a block storage unit and a distributed text storage unit.

[0105] 3-1) Block storage unit, for storing the structured data sent by the data analysis and fusion module. In actual implementation, the object storage can use a relational database MySQL to store the standard complete structured data processed by the data analysis and fusion module.

[0106] 3-2) Object storage unit, for storing the unstructured data connected by the device access management module; the block storage uses CVR to store the unstructured video and image sensing data connected by the device access management module.

[0107] 3-3) Distributed text storage unit, for high-speed storage of the semi-structured data connected by the device access management module by using the distributed text storage HDFS. The distributed file storage of the embodiment uses the distributed file system service HDFS based on the Hadoop architecture to store the semi-structured file data connected by the device access management module at high speed, including static text data, pictures and other semi-structured data.

[0108] The system provided by the embodiment introduces an equipment access management module, which is used for shielding the data heterogeneity of multiple sources, and uniformly managing and standardizing the access of Internet of Things equipment and data; introduces a data analysis and fusion module, which is used for collecting, processing and aggregating the perception data of different sources, different formats, various structures and fragmentation according to business needs; and introduces a data storage module, which is used for efficiently storing and classifiedly managing the unstructured data and structured data generated by the equipment. The problems of how to integrally control the massive equipment in the Internet of Things field, how to quickly and real-timely introduce the massive multi-source heterogeneous data, how to clean and integrate the scattered and chaotic data and how to manage the quality are solved.

[0109] The modules of the multi-source heterogeneous data access system provided by the embodiment can be all or partially implemented by software, hardware and a combination thereof. The modules can be embedded in or independent of the processor in the computer device in hardware form, or can be stored in the memory in the computer device in software form, so as to be called and executed by the processor to perform the operations corresponding to the modules. The memory and the processor are directly or indirectly electrically connected to realize the transmission or interaction of data.

[0110] The memory can be, but is not limited to, a random access memory (RAM), a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM) and the like. The memory is used to store a program, and the processor executes the program after receiving an execution instruction.

[0111] The processor can be an integrated circuit chip with data processing capability. The processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP) and the like. The processor can realize or execute the functions of the modules disclosed in the embodiments of the application. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor.

[0112] In another embodiment, as Figure 3As shown, a multi-source heterogeneous data access method for Internet of Things is provided, which is implemented in any of the foregoing multi-source heterogeneous data access systems for Internet of Things. The multi-source heterogeneous data access method for Internet of Things comprises the following steps:

[0113] Step 1, adding information of the access-aware device, and judging whether the aware device meets the standard interface protocol. For the aware device incompatible with the standard protocol, a data structure conversion model is added;

[0114] Step 2, judging the data type transmitted by the connected aware device. For the text data, step 6 is executed. For the interface protocol data, step 3 is executed.

[0115] Step 3, judging whether the connected aware device is compatible with the standard interface service. If yes, step 4 is executed. Otherwise, step 5 is executed.

[0116] Step 4, for the aware device compatible with the standard protocol, the multi-source heterogeneous data connection is performed according to the standard interface protocol, and step 6 is executed.

[0117] Step 5, for the aware device incompatible with the standard protocol, the access data is converted into the standard interface data by using the extended interface and the data structure conversion model, the multi-source heterogeneous data connection of the aware device is realized, and step 6 is executed.

[0118] Step 6, converting the accessed structured data and semi-structured text data into the same type and same structure structured data, and executing step 7.

[0119] Step 7, independently storing and managing the semi-structured data and unstructured data in the directly connected multi-source heterogeneous data and the structured data obtained by step 6.

[0120] The conversion of the accessed structured data and semi-structured data into the same type and same structure structured data comprises:

[0121] Step 6.1, converting the accessed semi-structured text data into structured data.

[0122] Step 6.2, performing the repeated elimination, missing supplement and abnormal correction operations on the structured data processed by step 6.1 and the directly accessed structured data.

[0123] Step 6.3, using the ETL tool to perform data extraction on the structured data processed by step 6.2, to obtain the same type and same structure structured data.

[0124] Wherein the accessed semi-structured text data is converted into structured data, including using the improved distributed parallel computing framework MapReduce to analyze and process the distributed storage cache bulk semi-structured text data in parallel, and quickly convert it into structured data of a single device in real time, the specific steps are as follows:

[0125] Step a: read the semi-structured text data to be operated from the storage area, and divide the data into several logical fragments, wherein the size of the fragment = the total memory size of the read file / hdfs block default size;

[0126] Step b: call the Map end to parse the fragment data into a series of key-value pairs <key, value> for subsequent processing, and the same text type of fragment data has the same key key;

[0127] Step c: according to the business requirements of the sensing device and the text type, the key-value pairs processed by the Map end are divided into n partitions for processing;

[0128] Step d: sort the fragment data in each partition from small to large, and merge the values with the same key key in the sorted partition;

[0129] Step e: call the Reduce end to classify the merged values in the partition, and arrange the same text type of fragment data to the same type of text;

[0130] Step f: call the InputFormat method of MapReduce to read the classified text data, and divide it into structured data streams of a single device according to the sensing device type and the field length of the sensing device, and complete the data conversion.

[0131] The specific limitations of the multi-source heterogeneous data access method for the Internet of Things can be referred to the limitations of the multi-source heterogeneous data access system for the Internet of Things described above, which will not be repeated here.

[0132] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by a computer program instructing related hardware, and the computer program can be stored in a non-volatile computer readable storage medium. When the computer program is executed, it can include the processes of the above-mentioned embodiments.

[0133] The technical features of the above-described embodiments can be combined in any way. In order to make the description concise, all possible combinations of the technical features in the above-described embodiments are not described, but as long as the combination of the technical features does not exist contradictory, it should be considered as the scope of the present application.

[0134] The above embodiments only express several implementation manners of the present application, and the description is relatively specific and detailed, but should not be understood as a limitation on the scope of the application. It should be noted that for ordinary skilled persons in the art, several modifications and improvements can be made without departing from the concept of the present application, which are within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the appended claims.

Claims

1. A multi-source heterogeneous data access system for Internet of Things, characterized in that, The Internet of Things-oriented multi-source heterogeneous data access system comprises a device access management module, a data analysis and fusion module and a data storage module, wherein: The device access management module is configured to uniformly access a large number of multi-source heterogeneous sensing devices, and provide a development data service bus for an application layer through a unified Internet of Things data protocol; the device access management module comprises a device management unit, an interface protocol connection unit and a text data connection unit, wherein: The device management unit is configured to manage and securely access the sensing devices, and determine whether the sensing devices meet a standard protocol access, and pre-establish a data structure conversion model for sensing devices that are incompatible with the standard protocol; The interface protocol connection unit is configured to formulate a metadata model, a data format and a data acquisition standard protocol interface for various types of sensing data of multi-source heterogeneous sensing devices; the interface protocol connection unit comprises a standard protocol service unit and a private interface adaptation unit; The standard protocol service unit is configured to provide a uniform standard interface service for multi-source heterogeneous data connection of sensing devices with the same attribute; the multi-source heterogeneous data comprises structured data, semi-structured data and unstructured data generated by different devices; The private interface adaptation unit is configured to convert the access data into standard interface data by using an extended interface and a data structure conversion model for sensing devices that are incompatible with the standard protocol; The text data connection unit is configured to receive different formats of text data transmitted by the sensing devices through a file interface; The data analysis and fusion module is configured to convert a large amount of structured data and semi-structured data received by the device access management module into the same type and structure of structured data, and transmit the structured data to the data storage module; The data storage module is configured to classify and store and manage the semi-structured data and unstructured data received by the device access management module and the structured data sent by the data analysis and fusion module. 2.The Internet of Things oriented multi-source heterogeneous data access system of claim 1, wherein, The standard protocol service unit defines the multi-source heterogeneous data as four types of standard protocol data, i.e., reporting data, state data, control instruction data and linkage instruction data, according to the device data attributes; The standard protocol service unit defines corresponding interfaces according to each type of standard protocol data, including a device acquisition data interface, a device state reporting interface, a device control interface and a device linkage interface, wherein: The device acquisition data interface is configured to receive reporting data collected by the sensing device, including target data, alarm information and current working parameters; The device state reporting interface is configured to receive state data reported by the sensing device according to the interface protocol, including heartbeat state data, time setting information and power-on / off information; The device control interface is configured to convert the control instruction data of the sensing device into interface parameters and transmit the interface parameters to the sensing device to control the working parameters or state of the sensing device; The device linkage interface is configured to send a linkage instruction data request for tracking target information to a disposal device, wherein the target information is target information received by a detection device. 3.The Internet of Things oriented multi-source heterogeneous data access system of claim 1, wherein, The data analysis and fusion module comprises a data analysis classification unit, a data cleaning unit and a data conversion unit, wherein: The data analysis and classification unit is configured to obtain semi-structured text data converted from the data storage module to structured data and transmit the structured data to the data cleaning unit; the semi-structured text data includes binary text data, XML text data and JSON text data, and the semi-structured text data is sent by the device access management module to the data storage module for caching; The data cleaning unit is configured to perform repeated elimination, missing supplement and abnormal correction operations on the structured data processed by the data analysis and classification unit and the structured data directly accessed by the device access management module, and transmit the structured data to the data conversion unit; the structured data includes structured text data, and the structured text data includes CSV and XLS format text data; The data conversion unit is configured to perform data extraction on the structured data sent by the data cleaning unit by using an ETL tool and transmit the structured data to the data storage module. 4.The Internet of Things oriented multi-source heterogeneous data access system of claim 3, wherein, The data analysis and classification unit uses an improved distributed parallel computing framework MapReduce to perform parallel processing and analysis on a large amount of semi-structured text data stored and cached in the data storage module, and quickly and real-timely converts the semi-structured text data into structured data of a single device, and performs the following operations: Step a: reading semi-structured text data to be operated from a storage area, and dividing the data into a plurality of logical segments, wherein the size of the segment = the total memory size of the read file / hdfs block default size; Step b: calling the Map end to parse the segment data into a series of key-value pairs <key, value> for subsequent processing, and the segment data of the same text type has the same key; Step c: dividing the key-value pairs processed by the Map end into n partitions according to the business requirements of the sensing device and the text type; Step d: sorting the segment data of each partition from small to large, and merging the values with the same key in the sorted partition; Step e: calling the Reduce end to classify the merged values of the partition, and arranging the segment data of the same text type to the same type of text; Step f: reading the classified text data, dividing the text data into structured data streams of a single device according to the sensing device type and the field length of the sensing device, and transmitting the structured data streams to the data cleaning unit.

5. A multi-source heterogeneous data access method for Internet of Things, characterized in that, The multi-source heterogeneous data access method for the Internet of Things includes: Step 1, adding information of an access sensing device, and judging whether the sensing device meets a standard interface protocol, and adding a data structure conversion model for a sensing device incompatible with the standard protocol; Step 2, judging the data type transmitted by the connected sensing device, executing step 6 if the data type is text data, and executing step 3 if the data type is interface protocol data; Step 3, judging whether the connected sensing device is compatible with a standard interface service, executing step 4 if the sensing device is compatible, and executing step 5 otherwise; Step 4, for a sensing device compatible with the standard protocol, performing multi-source heterogeneous data connection according to the standard interface protocol, and executing step 6; Step 4, for a sensing device compatible with the standard protocol, performing multi-source heterogeneous data connection according to the standard interface protocol, and executing step 6; Step 5, for the sensing device that cannot be compatible with the standard protocol, an extended interface and data structure conversion model is used to convert the access data into standard interface data, realizing the multi-source heterogeneous data access of the sensing device, and executing step 6; Step 6, converting the accessed structured data and semi-structured text data into the same type and same structure structured data, and executing step 7; Step 7, independently storing and managing the semi-structured data and unstructured data in the directly accessed multi-source heterogeneous data and the structured data obtained by processing step 6. 6.The Internet of Things oriented multi-source heterogeneous data access method of claim 5, wherein, The conversion of the accessed structured data and semi-structured data into the same type and same structure structured data comprises: Step 6.1, converting the accessed semi-structured text data into structured data; Step 6.2, performing the repeated elimination, missing supplement and exception correction operations on the structured data processed by step 6.1 and the directly accessed structured data; Step 6.3, using the ETL tool to perform data extraction on the structured data processed by step 6.2 to obtain the same type and same structure structured data.

7. The Internet of Things oriented multi-source heterogeneous data access method of claim 6, wherein, The conversion of the accessed semi-structured text data into structured data comprises using the improved distributed parallel computing framework MapReduce to perform parallel processing and analysis on the large batch of semi-structured text data stored and cached in a distributed manner, and quickly and real-timely converting the semi-structured text data into structured data of a single device, and the specific steps are as follows: Step a: reading the semi-structured text data to be operated from a storage area, and dividing the data into a plurality of logical fragments, wherein the size of the fragment = the total memory size of the read file / hdfs block default size; Step b: calling the Map end to parse the fragment data into a series of key-value pairs <key, value> for subsequent processing, and the fragment data of the same text type has the same key; Step c: according to the business requirements of the sensing device and the text type, the key-value pairs processed by the Map end are divided into n partitions for processing; Step d: sorting the fragment data of each partition from small to large, and merging the values with the same key in the sorted partition; Step e: calling the Reduce end to classify the values after partition merging, and arranging the fragment data of the same text type to the same type of text; Step f: reading the classified text data, and according to the sensing device type in the data and the field length of the sensing device, dividing the text data into a structured data stream of a single device to complete the data conversion.

Images