Mass data mining system based on DMA model

Abstract

The invention relates to the technical field of data processing, in particular to a mass data mining system based on a DMA (direct memory access) model, which comprises an acquisition module for acquiring data content generated by a data source in real time; the preprocessing module is used for receiving the data content acquired by the acquisition module and preprocessing the data content; the storage module is used for receiving the preprocessed data content in the preprocessing module and storing the data content; the mining module is used for editing keywords and executing data mining operation in the data content stored in the storage module based on the keywords; according to the method, data of different data sources are collected, the collected data content is comprehensively managed, meanwhile, the stored data content is mined in the mode that keywords are edited in cooperation with multiple data mining algorithms, data calling convenience is provided for a data management end user, and the user experience is improved. And it is ensured that the user at the data management end can quickly mine the data content required by the user from the massive data content.

Description

Technical Field

[0001] The invention relates to the technical field of data processing, and in particular to a mass data mining system based on a DMA model. Background Art

[0002] Traditional BI (Business Intelligence) analysis reports need to be used in combination with third-party BI software, and can only be provided to corporate management for use after steps such as demand analysis, modeling, data extraction and conversion, and analysis report design to assist in their management decision-making.

[0003] The invention patent with application number 202410775200.8 discloses a method for constructing a data mining model based on machine learning, which is characterized in that it includes the following steps: collecting and preprocessing data through a data acquisition module and a data preprocessing module, the data acquisition module uses an application access interface and a data sensor network to collect multi-source data in real time, and the data preprocessing module uses a real-time stream processing engine to clean and convert the collected data, and the multi-source data at least includes enterprise file data, database data and business system data; the preprocessed data is evaluated for data quality through a data quality assessment module, and the data quality assessment module uses a multivariate comprehensive analysis model to perform a comprehensive assessment of data integrity, accuracy, consistency and correlation, and performs secondary processing on the evaluated residual data through a self-supervised representation learning model, and the data quality assessment module uses a multi-threaded parallel computing method to improve the efficiency of data quality assessment.

[0004] This application aims to solve the problem that: "the data mining models constructed by existing methods and systems often have poor adaptability and reliability. The main reasons are as follows: the existing data mining model construction methods and systems only rely on a single or limited data source, which makes it difficult for the constructed data mining models to adapt to this complex data environment; and as the data is constantly updated, the data mining model also needs to be constantly updated. The existing model construction methods and systems require manual intervention, resulting in weak generalization capabilities of the model, which cannot be applied to data mining tasks in different scenarios, thereby reducing the adaptability of the model; the data often has problems such as noise, missing values, outliers, etc., and incomplete processing will seriously affect the reliability of the data mining model. And there is a lack of methods for performance simulation testing and feedback optimization of the constructed model."

[0005] However, the above model often involves periodic batch processing, in which timeliness cannot be guaranteed and the focus is only on the analysis and reporting of historical data. Summary of the invention

[0006] In view of the above shortcomings of the prior art, the present invention provides a massive data mining system based on a DMA model, which solves the technical problems raised in the above background technology.

[0007] To achieve the above objectives, the present invention is implemented through the following technical solutions: A massive data mining system based on a DMA model, comprising: The acquisition module is used to acquire data content generated in real time by the data source; the preprocessing module is used to receive the data content acquired by the acquisition module and preprocess the data content; the storage module is used to receive the data content that has been preprocessed in the preprocessing module and store the data content; the mining module is used to edit keywords and perform data mining operations on the data content stored in the storage module based on the keywords; the visualization module is used to receive the data content mined by the mining module and display the data content; the monitoring module is used to edit the keywords edited during the operation of the mining module, monitor the popularity of the keywords, and coordinate the keywords edited in the mining module based on the keyword popularity monitoring results.

[0008] Furthermore, the data source of the acquisition module for collecting data content includes: a database or a file management system. After collecting the data content, the acquisition module synchronously marks the data content, and the marking information of the data content is the source data source name of the data content; Among them, when the collection module collects the data content generated in real time by the data source, it synchronously identifies the format of each data content, selects the data format of the type of data content with the largest proportion as the conversion target, and converts the format of the data content generated in real time by the data source collected by the collection module, so that all data content collected by the collection module is represented in the same data format. The collection module is integrated by Sqoop or Flume.

[0009] Furthermore, during the data content collection stage, the collection module synchronously performs a data content buffering operation, the collection module continuously transmits the collected data content to the preprocessing module, and the data content that is not transmitted to the preprocessing module is buffered and stored in the data content transmission queue; Before the acquisition module transmits data content to the preprocessing module each time, it simultaneously performs data cleaning operations on the data content that performs the transmission operation: ; Where: The consistency between data content A and data content B; is the set of vectors in the data content; is the value of the i-th vector in the data content A; is the value of the i-th vector in the data content B; in, represents the decision function, When the value of is zero, =0, When the value of is non-zero, =1, data content A is consistent with data content B When the value of is zero, it means that any one of the two groups of data contents that perform consistency calculation is used as the data cleaning target. Otherwise, the two groups of data contents that perform consistency calculation do not perform cleaning operations. Based on the above formula, all data contents transmitted from the acquisition module to the preprocessing module are grouped together for consistency calculation, and then a global cleaning operation is performed on the data contents.

[0010] Furthermore, the preprocessing module performs a preprocessing operation on the received data content, namely, a denoising process; Among them, the preprocessing operation on the data content in the preprocessing module adopts any one of the mean filtering algorithm, wavelet transform denoising algorithm, Kalman filtering algorithm or PCA denoising algorithm, and the preprocessing target of the data content in the preprocessing module is the incomplete data content in the data content.

[0011] Furthermore, the storage module is provided with submodules at the lower level, including: A classification unit, used to traverse the data content received by the storage module, identify the tag information of each data content, and distinguish the data content based on the data content tag information; The marking unit is used to obtain the data content that has been distinguished in the classification unit, create the same number of distinguished storage intervals as the number of distinctions in the storage module, use the distinguished storage intervals to distinguish and store the data content that has been distinguished, and mark the distinguished storage intervals; The marking of the distinguishing interval for storing data content in the marking unit is the marking information for distinguishing the data content stored in the storage interval.

[0012] Furthermore, during the operation phase of the mining module, the data content stored in the storage module is synchronously acquired, and during the acquisition phase of the data content in the storage module, the differentiated storage intervals in the storage module are used as the minimum unit for acquiring the data content, and each time a group of data content stored in the differentiated storage intervals is acquired. After the data mining operation is performed and the data content stored in each differentiated storage interval is completed, a group of data content stored in the differentiated storage intervals is acquired again in the storage module, and the data content mining operation is performed again, and this is repeated until each group of differentiated storage intervals in the storage module has been applied to the data mining operation.

[0013] Furthermore, the keywords edited during the operation of the mining module are phrases consisting of 1 to 6 characters, and the keywords edited during the operation of the mining module are manually edited by the system end user and stored with the mining module. When the mining module is run a second time, the keywords applied are manually edited by the system end user or selected from the stored keywords; The mining module stores the running programs of the classification algorithm, clustering algorithm, association rule mining algorithm and anomaly detection algorithm. The mining module runs the data mining operation phase based on keywords, and applies the algorithm programs stored internally to repeat the data mining operation until all the algorithm programs stored in the mining module are applied.

[0014] Furthermore, after receiving the mined data content, the visualization module synchronously obtains the time taken for the data content to be mined, further accumulates the obtained time to the timestamp when the data content is collected by the collection module, and finally sorts the mined data content by the timestamp to generate a data queue; The visualization module is integrated by a computer device with a display function. When the visualization module stores the mined data content, it is displayed in the format of a data queue.

[0015] Furthermore, the keyword popularity monitored by the monitoring module during the running stage is the number of occurrences of the keywords stored in the mining module. The monitoring module coordinates the keywords edited in the mining module, that is, generates a keyword queue based on the keyword popularity, and places the keywords stored in the mining module in the order of keywords in the keyword queue.

[0016] Furthermore, the acquisition module is interactively connected to a preprocessing module and a storage module via a wireless network, the storage module is interactively connected to a classification unit and a marking unit via a wireless network, the storage module is interactively connected to a mining module via a wireless network, the mining module is interactively connected to the marking unit via a wireless network, and the mining module is interactively connected to a visualization module and a monitoring module via a wireless network.

[0017] Compared with the known public technology, the technical solution provided by the present invention has the following beneficial effects: The present invention provides a massive data mining system based on a DMA model. During operation, the system collects data from different data sources and manages the collected data content comprehensively. Under the comprehensive management of the data content, the system mines the stored data content by editing keywords and coordinating multiple data mining algorithms, providing data retrieval convenience for data management end users, ensuring that the data management end users can quickly mine the data content they need from the massive data content. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the prior art descriptions. Obviously, the drawings described below are only some embodiments of the present invention, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0019] Figure 1 It is a structural diagram of a massive data mining system based on the DMA model; Figure 2 An example of a data operation interface for developing an APP according to the system of the present invention; Figure 3 Figure 2 is an example of a data operation interface for developing an APP according to the system of the present invention; Figure 4 Three figures are examples of data operation interfaces for developing an APP according to the system of the present invention; Figure 5 Four figures are examples of data operation interfaces for developing an APP according to the system of the present invention; Figure 6 This is an example diagram of the data comprehensive data display interface of the APP developed according to the system of the present invention. DETAILED DESCRIPTION

[0020] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0021] The present invention will be further described below in conjunction with the embodiments.

[0022] Embodiment 1: A massive data mining system based on a DMA model in this embodiment, such as Figure 1 As shown, including: The acquisition module is used to collect the data content generated by the data source in real time; During the data content collection phase of the collection module, the data content buffering operation is synchronously performed, and the collection module continuously transmits the collected data content to the preprocessing module, and the data content that has not been transmitted to the preprocessing module is buffered and stored in the data content transmission queue; Before the acquisition module transmits data content to the preprocessing module each time, it simultaneously performs data cleaning operations on the data content that performs the transmission operation: ; Where: The consistency between data content A and data content B; is the set of vectors in the data content; is the value of the i-th vector in the data content A; is the value of the i-th vector in the data content B; in, represents the decision function, When the value of is zero, =0, When the value of is non-zero, =1, data content A is consistent with data content B When the value of is zero, it means that any one of the two groups of data contents that perform consistency calculation is used as the data cleaning target. Otherwise, the two groups of data contents that perform consistency calculation do not perform cleaning operations. Based on the above formula, all data contents transmitted from the acquisition module to the preprocessing module are grouped together to perform consistency calculations, and then perform global cleaning operations on the data contents. A preprocessing module is used to receive the data content collected by the collection module and preprocess the data content; A storage module is used to receive the data content preprocessed by the preprocessing module and store the data content; The storage module is provided with submodules at the lower level, including: A classification unit, used to traverse the data content received by the storage module, identify the tag information of each data content, and distinguish the data content based on the data content tag information; The marking unit is used to obtain the data content that has been distinguished in the classification unit, create the same number of distinguished storage intervals as the number of distinctions in the storage module, use the distinguished storage intervals to distinguish and store the data content that has been distinguished, and mark the distinguished storage intervals; The marking of the distinguishing intervals for storing data content in the marking unit is: marking information for distinguishing the data content stored in the storage interval; A mining module, for editing keywords and performing data mining operations on the data content stored in the storage module based on the keywords; A visualization module is used to receive the data content mined by the mining module and display the data content; The monitoring module is used to monitor the keywords edited during the operation of the mining module, monitor the popularity of the keywords, and coordinate the keywords edited in the mining module based on the keyword popularity monitoring results; The acquisition module is interactively connected to the preprocessing module and the storage module through a wireless network. The storage module is interactively connected to the classification unit and the marking unit through a wireless network. The storage module is interactively connected to the mining module through a wireless network. The mining module is interactively connected to the marking unit through a wireless network. The mining module is interactively connected to the visualization module and the monitoring module through a wireless network.

[0023] In this embodiment, the collection module runs to collect data content generated in real time by the data source, the preprocessing module runs in the post-position to receive the data content collected by the collection module, and preprocesses the data content. The storage module further receives the data content preprocessed in the preprocessing module and stores the data content. The classification unit synchronously traverses the data content received by the storage module, identifies the marking information of each data content, and distinguishes the data content based on the data content marking information. The marking unit obtains the data content that has been distinguished in the classification unit in real time, creates a distinguished storage interval with the same number of distinctions in the storage module, applies the distinguished storage interval to distinguish and store the data content that has been distinguished, and marks the distinguished storage interval. The mining module then edits keywords and performs data mining operations on the data content stored in the storage module based on the keywords. The visualization module further receives the data content mined by the mining module and displays the data content. Finally, the monitoring module mines the keywords edited during the operation phase of the module, monitors the popularity of the keywords, and coordinates the keywords edited in the mining module based on the keyword popularity monitoring results.

[0024] Through the operation of the system in the above embodiment, a new data mining system is provided for the data management end user, so as to facilitate the query and mining of the data required by the user in the massive data content.

[0025] Embodiment 2: In terms of specific implementation, based on Example 1, this example refers to Figure 1 The massive data mining system based on the DMA model in Example 1 is further described in detail: The data source of the collection module running the collection data content application includes: a database or a file management system. After the collection module collects the data content, it synchronously marks the data content. The marking information of the data content is the source data source name of the data content; When the acquisition module collects the data content generated in real time by the data source, it synchronously identifies the format of each data content, selects the data format of the data content with the largest proportion as the conversion target, and performs format conversion on the data content generated in real time by the data source collected by the acquisition module, so that all the data content collected by the acquisition module is represented in the same data format. The acquisition module is integrated by Sqoop or Flume; The preprocessing module performs a preprocessing operation on the received data content, namely, denoising; Among them, the preprocessing operation for the data content in the preprocessing module adopts any one of the mean filtering algorithm, wavelet transform denoising algorithm, Kalman filtering algorithm or PCA denoising algorithm, and the preprocessing target of the data content by the preprocessing module is the incomplete data content in the data content; In the mining module operation phase, the data content stored in the storage module is synchronously acquired, and in the data content acquisition phase in the storage module, the differentiated storage intervals in the storage module are used as the minimum unit for acquiring the data content, and each time a group of data content stored in the differentiated storage intervals is acquired, and after the data mining operation is performed and the data content stored in each differentiated storage interval is completed, a group of data content stored in the differentiated storage intervals is acquired again in the storage module, and the data content mining operation is performed again, and this is repeated until each group of differentiated storage intervals in the storage module is applied to the data mining operation and then ends; The keywords edited during the mining module operation phase are phrases consisting of 1 to 6 characters. The keywords edited during the mining module operation phase are manually edited by the system end user and stored with the mining module. When the mining module is run a second time, the keywords applied are manually edited by the system end user or selected from the stored keywords. The mining module stores the running programs of the classification algorithm, clustering algorithm, association rule mining algorithm and anomaly detection algorithm. The mining module runs the data mining operation phase based on keywords, and applies the algorithm program stored internally to repeat the data mining operation until all the algorithm programs stored in the mining module are applied and then terminated; after the visualization module receives the mined data content, it synchronously obtains the time taken for the data content to be mined, and further accumulates the obtained time to the timestamp when the data content is collected by the collection module, and finally sorts the mined data content by the timestamp to generate a data queue; Wherein, the visualization module is integrated by a computer device with a display function, and when the visualization module stores the mined data content, it is displayed in the format of a data queue; The keyword popularity monitored by the monitoring module during its operation phase is the number of occurrences of the keywords stored in the mining module. The monitoring module coordinates the keywords edited in the mining module, that is, generates a keyword queue based on the keyword popularity, and places the keywords stored in the mining module in the order of the keywords in the keyword queue.

[0026] In this embodiment, through the above settings, the specified operation logic and data support are provided for the operation of the system in Embodiment 1, ensuring the stability of the operation of the system in Embodiment 1.

[0027] In summary, during the operation of the system in the above embodiment, the system collects data from different data sources and comprehensively manages the collected data content. Under the condition of comprehensive management of the data content, the stored data content is mined by editing keywords and coordinating multiple data mining algorithms, providing data retrieval convenience for data management users, ensuring that data management users can quickly mine the data content they need from massive data content.

[0028] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements will not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A massive data mining system based on DMA model, characterized in that: include: The acquisition module is used to collect the data content generated by the data source in real time; A preprocessing module is used to receive the data content collected by the collection module and preprocess the data content; A storage module is used to receive the data content preprocessed by the preprocessing module and store the data content; A mining module, for editing keywords and performing data mining operations on the data content stored in the storage module based on the keywords; A visualization module is used to receive the data content mined by the mining module and display the data content; The monitoring module is used to monitor the keywords edited during the operation phase of the mining module, monitor the popularity of keywords, and coordinate the keywords edited in the mining module based on the keyword popularity monitoring results.

2. A massive data mining system based on a DMA model according to claim 1, characterized in that: The data source of the data content application run by the acquisition module includes: a database or a file management system. After the acquisition module acquires the data content, it synchronously marks the data content. The marking information of the data content is the source data source name of the data content; Among them, when the collection module collects the data content generated in real time by the data source, it synchronously identifies the format of each data content, selects the data format of the type of data content with the largest proportion as the conversion target, and converts the format of the data content generated in real time by the data source collected by the collection module, so that all data content collected by the collection module is represented in the same data format. The collection module is integrated by Sqoop or Flume.

3. A massive data mining system based on a DMA model according to claim 1, characterized in that: During the data content collection phase of the collection module, a data content buffering operation is synchronously performed, the collection module continuously transmits the collected data content to the preprocessing module, and the data content that is not transmitted to the preprocessing module is buffered and stored in the data content transmission queue; Before the acquisition module transmits data content to the preprocessing module each time, it simultaneously performs data cleaning operations on the data content that performs the transmission operation: ； Where: The consistency between data content A and data content B; is the set of vectors in the data content; is the value of the i-th vector in the data content A; is the value of the i-th vector in the data content B; in, represents the decision function, When the value of is zero, =0, When the value of is non-zero, =1, data content A is consistent with data content B When the value of is zero, it means that any one of the two groups of data contents that perform consistency calculation is used as the data cleaning target. Otherwise, the two groups of data contents that perform consistency calculation do not perform cleaning operations. Based on the above formula, all data contents transmitted from the acquisition module to the preprocessing module are grouped together for consistency calculation, and then a global cleaning operation is performed on the data contents.

4. A massive data mining system based on a DMA model according to claim 1, characterized in that: The preprocessing module performs a preprocessing operation on the received data content, namely, a denoising process; Among them, the preprocessing operation on the data content in the preprocessing module adopts any one of the mean filtering algorithm, wavelet transform denoising algorithm, Kalman filtering algorithm or PCA denoising algorithm, and the preprocessing target of the data content in the preprocessing module is the incomplete data content in the data content.

5. The massive data mining system based on the DMA model according to claim 1, characterized in that: The storage module is provided with submodules at the lower level, including: A classification unit, used to traverse the data content received by the storage module, identify the tag information of each data content, and distinguish the data content based on the data content tag information; The marking unit is used to obtain the data content that has been distinguished in the classification unit, create the same number of distinguished storage intervals as the number of distinctions in the storage module, use the distinguished storage intervals to distinguish and store the data content that has been distinguished, and mark the distinguished storage intervals; The marking of the distinguishing interval for storing data content in the marking unit is the marking information for distinguishing the data content stored in the storage interval.

6. A massive data mining system based on a DMA model according to claim 1, characterized in that: During the operation phase of the mining module, the data content stored in the storage module is synchronously acquired. In the acquisition phase of the data content in the storage module, the differentiated storage intervals in the storage module are used as the minimum unit for acquiring the data content. Each time, a group of data content stored in the differentiated storage intervals is acquired. After the data mining operation is performed and the data content stored in each differentiated storage interval is completed, a group of data content stored in the differentiated storage intervals is acquired again in the storage module, and the data content mining operation is performed again. This is repeated until each group of differentiated storage intervals in the storage module has been applied to the data mining operation.

7. A massive data mining system based on a DMA model according to claim 1, characterized in that: The keywords edited during the operation phase of the mining module are phrases consisting of 1 to 6 characters. The keywords edited during the operation phase of the mining module are manually edited by the system end user and stored with the mining module. When the mining module is run a second time, the keywords applied are manually edited by the system end user or selected from the stored keywords. The mining module stores the running programs of the classification algorithm, clustering algorithm, association rule mining algorithm and anomaly detection algorithm. The mining module runs the data mining operation phase based on keywords, and applies the algorithm programs stored internally to repeat the data mining operation until all the algorithm programs stored in the mining module are applied.

8. The massive data mining system based on the DMA model according to claim 1, characterized in that: After receiving the mined data content, the visualization module synchronously obtains the time taken for the data content to be mined, further accumulates the obtained time to the timestamp when the data content is collected by the collection module, and finally sorts the mined data content by the timestamp to generate a data queue; The visualization module is integrated by a computer device with a display function. When the visualization module stores the mined data content, it is displayed in the format of a data queue.

9. The massive data mining system based on the DMA model according to claim 1, characterized in that: The keyword popularity monitored by the monitoring module during the running phase is the number of occurrences of the keywords stored in the mining module. The monitoring module coordinates the keywords edited in the mining module, that is, generates a keyword queue based on the keyword popularity, and places the keywords stored in the mining module in the order of the keywords in the keyword queue.

10. The massive data mining system based on the DMA model according to claim 1, characterized in that: The acquisition module is interactively connected to a preprocessing module and a storage module via a wireless network, the storage module is interactively connected to a classification unit and a marking unit via a wireless network, the storage module is interactively connected to a mining module via a wireless network, the mining module is interactively connected to the marking unit via a wireless network, and the mining module is interactively connected to a visualization module and a monitoring module via a wireless network.

Images