Data self-driven processing system, method and equipment based on big data internet of things, and medium

Through the big data IoT data self-driven processing system, data preprocessing, homework and feature engineering modules are used to generate multi-dimensional features, dynamically adjust the alarm threshold, and solve the problem of false alarm system caused by the variability of IoT data, and realize the implementation of the problem of false alarm system errors and omissions in alarm system. Agile monitoring and accurate abnormal identification of IoT systems.

CN118964855APending Publication Date: 2024-11-15ZHENGJIANG PUBLIC INFORMATION
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411167405.4
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-23
Publication Date
2024-11-15

AI Technical Summary

Technical Problem

The variability of IoT data leads to false alarm or missed alarm systems. It is difficult for the existing technology to dynamically adjust alarm rules and thresholds, and it is impossible to effectively respond to changes in the environment and equipment status.

Method used

A self-driven processing system based on big data IoT data is designed, including data preprocessing, data reunification and calibration, feature engineering and real-time alarm modules, and process historical data through custom data models and clustering algorithms to generate multi-dimensional data Features and dynamically adjust the alarm threshold according to feature changes to realize real-time data prediction and alarm.

Benefits of technology

It improves the intelligent monitoring and operation and maintenance capabilities of the Internet of Things system, reduces false alarms and missed reports, and ensures accurate identification and timely response to abnormal data.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118964855A_ABST
    Figure CN118964855A_ABST
Patent Text Reader

Abstract

The invention provides an Internet of Things data self-driven processing system, method and device based on big data and a medium, and relates to the technical field of Internet of Things. Real-time data comparison and alarm rule dynamic adjustment based on streaming processing can respond to data abnormal conditions in real time and carry out agile monitoring on the Internet of Things system; alarm rules are intelligently adjusted according to data change characteristics, monitoring of a big data model algorithm is achieved, and the alarm accuracy is improved; through multi-dimensional feature extraction, the data state can be depicted in a three-dimensional manner, abnormal data can be accurately identified, and the intelligent monitoring and operation and maintenance capability of the Internet of Things system is improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence, and in particular to a self-driven processing system, method, device and medium based on big data Internet of Things data. Background Art

[0002] With the advent of the big data era, the scale and complexity of IoT data have exploded. These data cover various types of sensor data, device status data, user behavior data, etc. The growth and diversity of IoT data have brought new challenges and opportunities to data processing and alarm systems. As external factors such as environment, time, geography, and climate change, the normal and abnormal conditions of IoT data may change. For example, the normal range of temperature and humidity may vary in different seasons or climate conditions. Similarly, the normal and abnormal behaviors of devices may vary in different geographical locations or working environments.

[0003] This variability poses a challenge to the alarm system, because the alarm rules and thresholds need to be dynamically adjusted according to different factors. If the alarm rules are fixed, false positives or false negatives may occur. Summary of the invention

[0004] The present disclosure provides a self-driven processing system, method, device and medium based on big data Internet of Things data to at least solve the above technical problems existing in the prior art.

[0005] According to a first aspect of the present disclosure, a self-driving processing system based on big data Internet of Things is provided, comprising: A data preprocessing module is used to select data type flushing execution capabilities according to a custom data model, and gradually perform data flushing capability encapsulation on historical data to obtain encapsulated data; A data clustering and calibration module is used to cluster the encapsulated data using a clustering algorithm, establish a data family system and load it into a big data concentration warehouse for data calibration to obtain calibrated data; A feature engineering module is used to perform multi-dimensional aggregation analysis on the calibrated data, generate multi-dimensional data features, and store them in a warehouse model. The multi-dimensional data features reflect the trend of historical data changes from multiple dimensions; The real-time alarm module is used to call the warehouse model, predict the real-time data according to the multi-dimensional data characteristics, obtain suspected abnormal data, and trigger the alarm program according to the alarm threshold. The alarm threshold is dynamically and automatically adjusted as the multi-dimensional data characteristics change.

[0006] In one possible implementation, the data preprocessing module includes: A data extraction unit is used to connect to the data source and extract the historical data that needs to be flushed; The quality inspection unit is used to perform quality inspection according to predefined quality rules and remove data that does not meet the quality standards; The conversion unit is used to clean, deduplicate, complete, and label the data to obtain encapsulated data.

[0007] In one embodiment, the feature engineering module includes: Data warehouse and measurement unit, used to store the calibrated data in Elasticsearch, use the full-text search and distributed computing capabilities of ElasticSearch to group or bucket the calibrated data according to different dimensions, and calculate the data features of each group or bucket; The feature generation unit is used to form multi-dimensional data features according to the data features of each group or bucket, and store them in the warehouse model.

[0008] In one embodiment, the real-time alarm module further includes: A rule matching and threshold adjustment unit, configured to call a corresponding alarm rule according to a target data type of the suspected abnormal data, and adjust an alarm threshold according to the multi-dimensional data feature to generate a new alarm threshold; The alarm event generating unit is used to automatically trigger a predefined alarm program, generate an alarm event and perform equipment alarm management and control when the suspected abnormal data exceeds the new alarm threshold.

[0009] According to a second aspect of the present disclosure, a method for self-driving processing of data based on big data Internet of Things is provided, comprising: Select the data type flushing execution capability according to the custom data model, and gradually perform data flushing capability encapsulation on the historical data to obtain encapsulated data; The encapsulated data is grouped, a data family system is established, and the data is loaded into a big data warehouse for data calibration to obtain calibrated data; Performing multi-dimensional aggregation analysis on the calibrated data to generate multi-dimensional data features, which are stored in a warehouse model. The multi-dimensional data features reflect the trend of historical data changes from multiple dimensions. The warehouse model is called to predict the real-time data according to the multi-dimensional data features to obtain suspected abnormal data, and an alarm program is triggered according to an alarm threshold. The alarm threshold is dynamically and automatically adjusted as the multi-dimensional data features change.

[0010] In one possible implementation, the step of selecting data type flushing execution capability according to the custom data model, gradually performing data flushing capability encapsulation on historical data, and obtaining encapsulated data includes: Connect to data sources and extract historical data that needs to be flushed; Perform quality checks based on predefined quality rules and remove data that does not meet quality standards; Clean, deduplicate, complete, and label the data to obtain encapsulated data.

[0011] In one possible implementation, the step of performing multi-dimensional aggregation analysis on the calibrated data, generating multi-dimensional data features, and storing them in a warehouse model includes: The calibrated data is stored in Elasticsearch. The full-text search and distributed computing capabilities of ElasticSearch are used to group or bucket the calibrated data according to different dimensions, and the data features of each group or bucket are calculated. Based on the data features of each group or bucket, multi-dimensional data features are formed and stored in the warehouse model.

[0012] In one possible implementation, the step of predicting the real-time data according to the multi-dimensional data features to obtain suspected abnormal data, and triggering an alarm program according to an alarm threshold, includes: Calling a corresponding alarm rule according to the target data type of the suspected abnormal data, and adjusting the alarm threshold according to the multi-dimensional data characteristics to generate a new alarm threshold; When the suspected abnormal data exceeds the new alarm threshold, a predefined alarm program is automatically triggered to generate an alarm event and perform equipment alarm management and control.

[0013] According to a third aspect of the present disclosure, there is provided an electronic device, including: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the self-driven processing method based on big data Internet of Things data described in the present disclosure.

[0014] According to a fourth aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute the self-driven processing method based on big data Internet of Things data described in the present disclosure.

[0015] The disclosed self-driven processing system, method, device and medium based on big data Internet of Things data can respond to data anomalies in real time and perform agile monitoring of the Internet of Things system based on real-time data comparison and dynamic adjustment of alarm rules based on streaming processing; it can intelligently adjust the alarm rules according to the characteristics of data changes, realize the monitoring of big data model algorithms, and improve the accuracy of alarms; through multi-dimensional feature extraction, it can three-dimensionally portray the data status, accurately identify abnormal data, and enhance the intelligent monitoring and operation and maintenance capabilities of the Internet of Things system.

[0016] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The above and other objects, features and advantages of the exemplary embodiments of the present disclosure will become readily understood by reading the detailed description below with reference to the accompanying drawings. In the accompanying drawings, several embodiments of the present disclosure are shown in an exemplary and non-limiting manner, in which: In the drawings, the same or corresponding reference numerals represent the same or corresponding parts.

[0018] Figure 1 A schematic diagram of the structure of a self-driving processing system based on big data IoT data provided by the first embodiment of the present disclosure is shown; Figure 2 A schematic diagram of the implementation process of a method for self-driving processing of big data IoT data provided in the second embodiment of the present disclosure is shown; Figure 3 A schematic diagram of the structure of an electronic device provided by an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0019] In order to make the purpose, features, and advantages of the present disclosure more obvious and easy to understand, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only part of the embodiments of the present disclosure, not all of the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present disclosure.

[0020] Embodiment 1: like Figure 1 As shown, Figure 1 This is a schematic diagram of the structure of a self-driven data processing system based on big data IoT provided in the first embodiment of the present disclosure. The system mainly includes a data preprocessing module, a data clustering and calibration module, a feature engineering module and a real-time alarm module.

[0021] The data preprocessing module is responsible for selecting data type flushing execution capabilities according to the custom data model, and gradually encapsulating the data flushing capabilities on the historical data to finally obtain the encapsulated data. This module specifically includes the following units: The data extraction unit first connects to the data source through the API interface or database connection technology to extract the historical data that needs to be flushed. The above database connection technology can be JDBC (Java DataBase Connectivity) or ODBC (Open Database Connectivity). The data source can be an IoT device database, a sensor data warehouse, etc. During the data extraction process, the integrity and consistency of the data must be ensured.

[0022] The quality inspection unit performs quality checks based on predefined quality rules (such as data format, range, completeness, etc.). Data that does not meet quality standards, such as too many missing values ​​or incorrect formats, are marked or removed. This process can be achieved by writing SQL queries, using data cleaning tools (such as DataCleaner), or writing custom scripts.

[0023] The conversion unit cleans, de-duplicates, completes, and labels the data that has passed the quality inspection. Cleaning operations include removing invalid characters and correcting erroneous values; de-duplicate operations ensure that there are no duplicate records in the data set; completion operations make reasonable estimates for missing values; and labeling operations add classification labels to the data according to business needs. The converted data is encapsulated data and is ready to enter the next processing stage.

[0024] The data grouping and calibration module groups the encapsulated data, establishes a data family system, and loads it into the big data warehouse for data calibration, ultimately obtaining the calibrated data. The specific steps are as follows: Data clustering is to use clustering algorithms (such as K-means, DBSCAN, etc.) to cluster encapsulated data. First, select appropriate clustering algorithms and parameters (such as 2 to 8 K values) based on business needs and data characteristics. Then, execute the clustering algorithm to group similar data into a family to form a data family system.

[0025] Specifically, K cluster centers are randomly selected as initial points, and the distance between each sample point and the cluster center of each data family is calculated.

[0026] According to the calculated distance, each sample point is assigned to the cluster represented by the closest cluster center, and then the cluster center of the cluster is recalculated. After the cluster center of the family is updated, the distance between the sample point and the cluster center of the family to which it belongs meets the preset threshold to determine whether the sample is abnormal data.

[0027] Data calibration is to load the grouped data into a large data warehouse (such as Hadoop HDFS, HBase, etc.), and perform basic calibration (such as entity, attribute, and relationship definition) and deep calibration (such as thing authentication, logic authentication, and performance structure authentication) according to the algorithm mechanism corresponding to the data model. During the deep calibration process, a data classification and lineage system is established to ensure the accuracy and traceability of the data.

[0028] Store the calibrated data in a high-speed distributed memory database (such as Redis, Memcached) to support fast access and analysis. At the same time, keep it synchronized with the large data warehouse to ensure data integrity and consistency.

[0029] The feature engineering module performs multi-dimensional aggregation analysis on the calibrated data, generates multi-dimensional data features, and stores them in the warehouse model. Specifically, it may include: The data partitioning and measurement unit stores the calibrated data in Elasticsearch and uses its full-text retrieval and distributed computing capabilities for data processing. First, create appropriate indexes and field mappings based on business needs and data characteristics. Then, use data partitioning operations to group or bucket the data according to different dimensions. Finally, use measurement operations to calculate the data features of each group or bucket, such as the mean, median, standard deviation, etc.

[0030] The feature generation unit organizes and generates multi-dimensional data features based on the multi-dimensional data extraction results. The data features of each dimension are taken as an item to form a number of multi-dimensional data features. These features reflect the changing trends and laws of historical data in different dimensions, providing a basis for subsequent data analysis and model prediction.

[0031] The real-time alarm module calls the warehouse model, predicts the real-time data based on the multi-dimensional data characteristics, obtains suspected abnormal data, and triggers the alarm program according to the alarm threshold. Specifically, it may include: The rule matching and threshold adjustment unit calls the corresponding alarm rules according to the target data type of the suspected abnormal data. At the same time, the alarm threshold is dynamically adjusted according to the multi-dimensional data characteristics to generate new sensitive alarm rules. This process can achieve automatic adjustment and optimization of the threshold through machine learning algorithms (such as support vector machines, random forests, etc.).

[0032] The alarm event generation unit automatically triggers the predefined alarm program when the prediction result of real-time data exceeds the new alarm threshold. The alarm program generates an alarm event and notifies relevant personnel through email, SMS, message push, etc. At the same time, alarm control is performed on abnormal devices, such as automatic isolation, restart, etc., to ensure the stable operation of the IoT system.

[0033] The self-driving processing system based on big data Internet of Things data provided by the present invention can also be extended to different Internet of Things systems and has good applicability.

[0034] For example, in a data center, the stable operation of the cooling system is critical to maintaining the performance of servers and other equipment. However, as the ambient temperature rises (such as in summer), the temperature inside the data center will also rise accordingly. If you still use fixed temperature thresholds to monitor the performance of the cooling system, it may cause frequent alarms in the summer, even if these alarms are caused by temperature increases within the normal range.

[0035] In order to solve this problem, the present invention can be applied to the monitoring of the cooling system of the data center. By collecting historical data (including the internal temperature of the data center in different seasons, the efficiency of the cooling system, etc.), and combining the current environmental conditions (such as the outside temperature, humidity, etc.), the temperature alarm threshold is dynamically adjusted. For example, in summer, the temperature alarm threshold is automatically increased to adapt to the normal rise in the temperature inside the data center, thereby reducing unnecessary alarms. At the same time, the system can also monitor the performance of the cooling system in real time to ensure that the alarm can be triggered in time when an abnormal situation really occurs.

[0036] In a large network environment, fluctuations in network traffic are normal, but in some cases, abnormal growth in traffic may indicate a potential security threat or performance bottleneck. If a network monitoring system uses a fixed traffic threshold to detect anomalies, it may frequently trigger false alarms due to normal fluctuations in network traffic.

[0037] In order to more accurately detect network traffic anomalies, the present invention can be applied to a network traffic monitoring system. The system can collect and analyze historical network traffic data to understand the normal fluctuation range and pattern of traffic. At the same time, combined with the current network environment (such as holidays, special events, and other factors that may cause traffic surges), the traffic alarm threshold is dynamically adjusted. For example, the traffic alarm threshold is automatically increased before holidays to adapt to possible traffic peaks. In addition, the system can also use machine learning algorithms to predict future traffic trends, and further adjust the alarm threshold accordingly to improve the accuracy and timeliness of anomaly detection.

[0038] Embodiment 2: like Figure 2 As shown, Figure 2 A schematic diagram of the implementation flow of a method for self-driving data processing based on big data IoT provided in the second embodiment of the present disclosure. The method includes: Select the data type flushing execution capability according to the custom data model, and gradually perform data flushing capability encapsulation on the historical data to obtain encapsulated data. The specific steps are as follows: Connect to data sources and extract historical data that needs to be flushed; Perform quality checks based on predefined quality rules and remove data that does not meet quality standards; Clean, deduplicate, complete, and label the data to obtain encapsulated data.

[0039] The encapsulated data is grouped, a data family system is established and loaded into the big data warehouse for data calibration to obtain the calibrated data.

[0040] Perform multi-dimensional aggregation analysis on the calibrated data to generate multi-dimensional data features and store them in the warehouse model. The multi-dimensional data features reflect the trend of historical data changes from multiple dimensions. The specific steps are as follows: The calibrated data is stored in Elasticsearch. The full-text search and distributed computing capabilities of ElasticSearch are used to group or bucket the calibrated data according to different dimensions, and the data features of each group or bucket are calculated. Based on the data characteristics of each group or bucket, multi-dimensional data features are formed and stored in the warehouse model.

[0041] Call the warehouse model to predict the real-time data based on the multi-dimensional data characteristics, obtain suspected abnormal data, and trigger the alarm program according to the alarm threshold. The alarm threshold is dynamically and automatically adjusted as the multi-dimensional data characteristics change. The specific steps are as follows: Call the corresponding alarm rules according to the target data type of the suspected abnormal data, adjust the alarm threshold according to the multi-dimensional data characteristics, and generate a new alarm threshold; When suspected abnormal data exceeds the new alarm threshold, the predefined alarm program is automatically triggered to generate an alarm event and perform device alarm management and control.

[0042] The implementation principle and technical effects of the self-driven processing method based on big data Internet of Things provided in the second embodiment of the present disclosure are the same as those of the aforementioned system embodiment 1. For the sake of brief description, for matters not mentioned in the method embodiment, reference can be made to the corresponding contents in the aforementioned system embodiment 1.

[0043] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device and a readable storage medium.

[0044] Figure 3A schematic block diagram of an example electronic device 300 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0045] like Figure 3 As shown, the device 300 includes a computing unit 301, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 302 or a computer program loaded from a storage unit 308 into a random access memory (RAM) 303. In the RAM 303, various programs and data required for the operation of the device 300 can also be stored. The computing unit 301, the ROM 302, and the RAM 803 are connected to each other via a bus 304. An input / output (I / O) interface 305 is also connected to the bus 304.

[0046] A number of components in the device 300 are connected to the I / O interface 305, including: an input unit 306, such as a keyboard, a mouse, etc.; an output unit 307, such as various types of displays, speakers, etc.; a storage unit 308, such as a disk, an optical disk, etc.; and a communication unit 309, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 309 allows the device 300 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0047] The computing unit 301 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 301 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 301 performs the various methods and processes described above, such as a deep learning-based application service self-inspection method. For example, in some embodiments, the deep learning-based application service self-inspection method may be implemented as a computer software program, which is tangibly included in a machine-readable medium, such as a storage unit 308. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 300 via the ROM 302 and / or the communication unit 309. When the computer program is loaded into the RAM 303 and executed by the computing unit 301, one or more steps of the deep learning-based application service self-inspection method described above may be performed. Alternatively, in other embodiments, the computing unit 301 may be configured to execute the deep learning-based application service self-inspection method in any other appropriate manner (for example, by means of firmware).

[0048] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0049] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0050] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0051] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0052] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0053] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0054] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0055] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of the features. In the description of the present disclosure, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0056] The above is only a specific embodiment of the present disclosure, but the protection scope of the present disclosure is not limited thereto. Any person skilled in the art who is familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present disclosure, which should be included in the protection scope of the present disclosure. Therefore, the protection scope of the present disclosure should be based on the protection scope of the claims.

Claims

1. A self-driving processing system based on big data Internet of Things, characterized in that: include: A data preprocessing module is used to select data type flushing execution capabilities according to a custom data model, and gradually perform data flushing capability encapsulation on historical data to obtain encapsulated data; A data clustering and calibration module is used to cluster the encapsulated data using a clustering algorithm, establish a data family system, and load the data into a big data warehouse for data calibration to obtain calibrated data; A feature engineering module is used to perform multi-dimensional aggregation analysis on the calibrated data, generate multi-dimensional data features, and store them in a warehouse model. The multi-dimensional data features reflect the trend of historical data changes from multiple dimensions. The real-time alarm module is used to call the warehouse model, predict the real-time data according to the multi-dimensional data characteristics, obtain suspected abnormal data, and trigger the alarm program according to the alarm threshold. The alarm threshold is dynamically and automatically adjusted as the multi-dimensional data characteristics change.

2. The system according to claim 1, characterized in that The data preprocessing module comprises: A data extraction unit is used to connect to the data source and extract the historical data that needs to be flushed; The quality inspection unit is used to perform quality inspection according to predefined quality rules and remove data that does not meet the quality standards; The conversion unit is used to clean, deduplicate, complete, and label the data to obtain encapsulated data.

3. The system according to claim 1, characterized in that The feature engineering module includes: Data warehouse and measurement unit, used to store the calibrated data in Elasticsearch, use the full-text search and distributed computing capabilities of ElasticSearch to group or bucket the calibrated data according to different dimensions, and calculate the data features of each group or bucket; The feature generation unit is used to form multi-dimensional data features according to the data features of each group or bucket, and store them in the warehouse model.

4. The system according to claim 1, characterized in that The real-time alarm module further comprises: A rule matching and threshold adjustment unit, configured to call a corresponding alarm rule according to a target data type of the suspected abnormal data, and adjust an alarm threshold according to the multi-dimensional data feature to generate a new alarm threshold; The alarm event generating unit is used to automatically trigger a predefined alarm program, generate an alarm event and perform equipment alarm management and control when the suspected abnormal data exceeds the new alarm threshold.

5. A method for self-driving data processing based on big data Internet of Things, characterized in that: include: Select the data type flushing execution capability according to the custom data model, and gradually perform data flushing capability encapsulation on the historical data to obtain encapsulated data; The encapsulated data is grouped, a data family system is established, and the data is loaded into a big data warehouse for data calibration to obtain calibrated data; Performing multi-dimensional aggregation analysis on the calibrated data to generate multi-dimensional data features, which are stored in a warehouse model. The multi-dimensional data features reflect the trend of historical data changes from multiple dimensions. The warehouse model is called to predict the real-time data according to the multi-dimensional data features to obtain suspected abnormal data, and an alarm program is triggered according to an alarm threshold. The alarm threshold is dynamically and automatically adjusted as the multi-dimensional data features change.

6. The method according to claim 5, characterized in that The step of selecting data type flushing execution capability according to the custom data model, and gradually encapsulating the historical data by executing the data flushing capability to obtain encapsulated data includes: Connect to data sources and extract historical data that needs to be flushed; Perform quality checks based on predefined quality rules and remove data that does not meet quality standards; Clean, deduplicate, complete, and label the data to obtain encapsulated data.

7. The method according to claim 5, characterized in that The step of performing multi-dimensional aggregation analysis on the calibrated data, generating multi-dimensional data features, and storing them in the warehouse model includes: The calibrated data is stored in Elasticsearch. The full-text search and distributed computing capabilities of ElasticSearch are used to group or bucket the calibrated data according to different dimensions, and the data features of each group or bucket are calculated. Based on the data features of each group or bucket, multi-dimensional data features are formed and stored in the warehouse model.

8. The method according to claim 5, characterized in that The step of predicting the real-time data according to the multi-dimensional data features to obtain suspected abnormal data, and triggering an alarm program according to an alarm threshold, includes: Calling a corresponding alarm rule according to the target data type of the suspected abnormal data, and adjusting the alarm threshold according to the multi-dimensional data characteristics to generate a new alarm threshold; When the suspected abnormal data exceeds the new alarm threshold, a predefined alarm program is automatically triggered to generate an alarm event and perform equipment alarm management and control.

9. An electronic device, characterized in that: include: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 5 to 8.

10. A non-transitory computer-readable storage medium storing computer instructions, characterized in that: The computer instructions are used to cause the computer to execute the method according to any one of claims 5-8.