A data management method of a data lake

Abstract

A data management method of a data lake comprises the following steps: S1: collecting data information and uploading to an initial data pool module to classify a complex data system, and then delivering the classified data to a data pool processing module; S2: the different types of data pool processing modules comprise an analog signal data pool processing module, an application program data pool processing module and a text data processing module, each module processes data information of its own type to realize unified data management; S3: a thread pool occupancy rate model is constructed, and data containing a removal mark and an archiving mark are managed and scheduled to an archiving data pool module. Through the above scheme, the internal rules of numerous data are found and corresponding storage is realized, and the technical problem that current data complexity is not easy to arrange is solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data management, in particular to a data management method of data lake. BACKGROUND

[0002] With the advent of the era of big data, a huge amount of data fragments are generated every day in various industries. Although the collection of data is no longer a technical problem, it is still a key problem to find the internal law and manage and store the data in the face of so many data. The data lake architecture is oriented to multi-data source information storage, including the Internet of Things, big data analysis or archiving can process or deliver data subsets to requesting users by accessing the data lake, but the data lake architecture is not just a huge storage disk, and the data persistence and security of the data lake are also its excellent storage features.

[0003] The present application is based on the data lake architecture, and is aimed at screening and reducing a huge amount of data fragments, long-term storing and retaining the effective data with analysis value, and periodically storing and deleting the meaningless data fragments within a certain time to release the data lake space. The present application can find the internal law of the data and store it accordingly, solving the technical problem of current data complexity that is not easy to organize, which is not possessed by the prior art. SUMMARY

[0004] The present application provides a data management method of data lake to solve the technical problems in the background art.

[0005] To achieve the above-mentioned purpose, the present application adopts the following technical scheme:

[0006] A data management method of data lake, comprising the following steps:

[0007] S1: collecting data information of physical devices, and then uploading to an initial data pool module in the data lake; the initial data pool module is used as a storage unit for the data, and classifies the complex data system according to the data collection object and the data characteristics, and then delivers the classified data to the corresponding type of data pool processing module; wherein each data has its own meta-process data, and after the data is classified, the meta-process data corresponding to each data is also delivered to the corresponding data pool processing module;

[0008] S2: The different types of data pool processing modules include: an analog signal data pool processing module, configured to reduce analog signal data to a certain amount of data that is easy to operate and manage, and to reorganize the reduced analog signal data with an archive mark or a removal mark; an application data pool processing module, configured to process application data from multiple applications to solve the inconsistency of data structures in different applications, and to achieve unified management of data, and to reorganize the application data with an archive mark or a removal mark; and a text data processing module, configured to process text data from multiple applications to solve the dispersion of text data in different applications, to achieve unified management of data, and to reorganize the text data with an archive mark or a removal mark.

[0009] S3: A thread pool occupancy model is constructed, and data with a removal mark or an archive mark is managed and scheduled to the archive data pool module.

[0010] To optimize the above technical solution, the following specific measures are taken:

[0011] Further, the specific content of the step S1 of classifying the complex data system according to the collection object of the data and the characteristics of the data, and then transporting the classified data to the data pool processing module of the corresponding type is as follows:

[0012] The data information collected from physical devices includes three types: analog signal data, application data, and text data.

[0013] For analog signal data, the object of collection is network devices and network traffic, and the characteristic of the data is that the data volume is repetitive and complex.

[0014] For application data, the object of collection is application programs and business processing, and the characteristic of the data is that it is repetitive and consistent structured data.

[0015] For text data, the object of collection is application programs, and the characteristic of the data is that it is unstructured data.

[0016] Each data is classified according to the collection object of the data and the corresponding characteristics of the data, and the analog signal data is transported to the analog signal data pool processing module, the application data is transported to the application data pool processing module, and the text data is transported to the text data pool processing module.

[0017] Further, each data has its own corresponding meta-process data in the step S1, and the meta-process data contains more information content for analysis, including the time, place, responsible person, and recording device of the corresponding data.

[0018] Further, the analog signal data pool processing module in step S2 is used to reduce the analog signal data to a certain amount of data that is easy to operate and manage, and the specific content of the reorganization of the reduced analog signal data is that:

[0019] Firstly, a large amount of repeated analog signal data is sorted, and a quadratic difference method is used to perform interpolation operation on every 3 adjacent points to reduce the amount of analog signal data;

[0020] Then, the analog signal data after the quadratic interpolation reduction processing is put into the trained Bayesian prediction model to obtain the abnormal occurrence probability of each analog signal data; wherein the feature of the abnormal occurrence of the analog signal data is that the data value of the analog signal exceeds the set standard value, and the Bayesian prediction model is a network model trained by a certain amount of analog signal data;

[0021] Finally, the data with an abnormal occurrence probability less than or equal to 50% of each analog signal data is retained, that is, the effective analog signal data is retained, and at the same time, it is archived, and the analog signal data greater than 50% is removed, so as to realize the reorganization of the analog signal.

[0022] Further, the application data processing module in step S2 is used to process the application data from multiple applications to solve the inconsistent data structure in different application programs, and to realize unified data management, and the specific content of the reorganization of the application data is that:

[0023] Firstly, the application data is put into the trained Bayesian prediction model to obtain the abnormal occurrence probability of each application data; wherein the feature of the abnormal occurrence of the application data is that the application data appears inconsistent data structure, and the Bayesian prediction model is a network model trained by a certain amount of application data;

[0024] Secondly, the application data with an abnormal occurrence probability less than or equal to 50% of each application data is retained, that is, the effective text data is retained, and the data greater than 50% is removed;

[0025] Then, the application data is classified according to the application program associated with the application program, so that the application data and the application program are one-to-one corresponding;

[0026] Finally, the meta-process data corresponding to different application data is found, the database mapped in the corresponding application is found according to the configuration file stored in the meta-process data, and the parameters of the structure, field and index in the database are put into the meta-process data, so as to realize the data management taking the meta-process data as the main body, solve the problem that the data cannot be sorted due to inconsistent data structures in different applications, and mark the processed meta-process data for archiving, mark the application data that cannot find the mapped database for removal, and realize the re-sorting of the application data.

[0027] Further, the text data processing module in step S2 is used to process text data from multiple applications to solve the problem of scattered text data in different applications, realize unified data management, and further realize the re-sorting of the text data by archiving or removing the specific content of the text data.

[0028] Firstly, the text data is put into the trained Bayesian prediction model to obtain the abnormal occurrence probability of each text data; wherein the feature of the abnormal occurrence of the text data is that the text data has a consistent data structure, and the Bayesian prediction model is a network model trained by a certain amount of text data;

[0029] Secondly, the data with an abnormal occurrence probability less than or equal to 50% of each text data is retained, that is, the effective text data is retained, and the data greater than 50% is marked for removal;

[0030] Then, the text data is classified according to the application program associated with the text data, so that the text data and the application program are one-to-one corresponding;

[0031] Finally, the meta-process data corresponding to different text data is found, the database mapped in the corresponding application is found according to the configuration file stored in the meta-process data, and the parameters of the structure, field and index in the database are put into the meta-process data, so as to realize the data management taking the meta-process data as the main body, solve the problem of scattered text data in different applications, and mark the processed meta-process data for archiving, mark the text data that cannot find the mapped database for removal, and realize the re-sorting of the text data.

[0032] Further, the specific content of step S3 is:

[0033] The thread pool occupancy rate model calculation formula is:

[0034]

[0035] In the formula, ω represents the thread pool load index, N is the number of working threads when the thread pool runs, N max is the maximum number of threads set, Tcur is the number of tasks in the current collection time window, T pre is the number of tasks in the previous collection time window, Q is the task buffer queue size, and ξ1, ξ2, and ξ3 all represent weight coefficients.

[0036] The thread pool occupancy model is used to collect and process data in the simulated signal data pool processing module, the application data processing module, and the text data processing module, and the thread pool load ω is calculated during processing. If ω is greater than the preset thread pool load ω', the adaptive weight coefficient parameters are triggered to adjust and calculate. If ω is still greater than ω', the current collection time window is skipped. If ω is less than or equal to the preset thread pool load ω', the archiving mark data is preferentially transferred to the archiving data pool module, and the removal mark data is secondarily transferred to the archiving data pool module.

[0037] Further, the archiving data pool module is used to periodically clean and release the removal mark data.

[0038] The application has the following advantages: based on the data lake management platform, the initial data pool module is designed, and the complex data system is classified through the initial data pool module. The complex and diverse data is preliminarily sorted according to its collection source and characteristics, avoiding the complexity of subsequent module processing. Then different types of data pool processing modules are designed, and different types of data are processed through these modules, thereby solving the problem of large amount of simulated signal data, the problem of inconsistent data structure in different application programs, and the problem of scattered text data in different application programs, and realizing the optimized management of data through different data processing modules. BRIEF DESCRIPTION OF DRAWINGS

[0039] Figure 1 is the overall technical scheme flowchart of the application. DETAILED DESCRIPTION

[0040] Now, the application will be described in detail. Figure 1 The application will be described in detail.

[0041] The overall technical scheme of the application is as follows:

[0042] A data management method of a data lake, comprising the following steps:

[0043] S1: Collect data information of physical devices, and then upload to an initial data pool module in a data lake; the initial data pool module is used as a storage unit for the data, and classifies the complex data system according to the collection object of the data and the characteristics of the data, and then delivers the classified data to a data pool processing module of a corresponding type; each data has its own meta-process data, and after the data is classified, the meta-process data corresponding to each data is also delivered to the corresponding data pool processing module;

[0044] S2: Different types of data pool processing modules include: an analog signal data pool processing module, which is used to reduce analog signal data to a certain amount of data that is easy to operate and manage, and to reorganize the reduced analog signal data by archiving or removing labels; an application data pool processing module, which is used to process application data from multiple applications to solve the inconsistency of data structures in different applications, and to achieve unified management of data, and then reorganize the application data by archiving or removing labels; a text data processing module, which is used to process text data from multiple applications to solve the dispersion of text data in different applications, achieve unified management of data, and then reorganize the text data by archiving or removing labels;

[0045] S3: Build a thread pool occupancy model, and manage and schedule data containing removal labels and archive labels to an archive data pool module.

[0046] Further, the specific content of the step S1 of classifying the complex data system according to the collection object of the data and the characteristics of the data, and then delivering the classified data to a data pool processing module of a corresponding type is:

[0047] The collection of data information of physical devices includes three categories: analog signal data, application data, and text data.

[0048] For analog signal data, the object collected is network devices and network traffic, and the characteristic of the data is that the data volume is repetitive and complex.

[0049] For application data, the object collected is application programs and business processing, and the characteristic of the data is that it is repetitive and consistent structured data.

[0050] For text data, the object collected is application programs, and the characteristic of the data is that it is unstructured data.

[0051] The data are classified by the collection object of the data information and the corresponding characteristics of the data, and the analog signal data are transmitted to the analog signal data pool processing module, the application program data are transmitted to the application program data pool processing module, and the text data are transmitted to the text data pool processing module.

[0052] Further, each data in step S1 has its own corresponding meta-process data, and the meta-process data contains more information content for analysis, including the time, place, responsible person and recording equipment of the generation of the corresponding data.

[0053] Further, the analog signal data pool processing module in step S2 is used to reduce the analog signal data to a certain amount of data that is easy to operate and manage, and the specific content of the reorganization of the archived and removed marked analog signal data after reduction is:

[0054] Firstly, a large number of repeated analog signal data are sorted, and a quadratic difference method is used to perform interpolation operation on every 3 adjacent points to realize the reduction of the amount of analog signal data.

[0055] Then, the analog signal data after the quadratic interpolation reduction processing are put into the trained Bayesian prediction model to obtain the abnormal occurrence probability of each analog signal data; wherein the feature of the abnormal occurrence of the analog signal data is that the data value of the analog signal exceeds the set standard value, and the Bayesian prediction model is a network model trained by a certain amount of analog signal data.

[0056] Finally, the data with an abnormal occurrence probability less than or equal to 50% of each analog signal data are retained, that is, the effective analog signal data are retained and archived, while the analog signal data greater than 50% are removed and marked, to realize the reorganization of the analog signal.

[0057] Further, the application program data processing module in step S2 is used to process the application program data from multiple applications to solve the inconsistent data structure in different application programs, and to realize unified management of the data, and the specific content of the reorganization of the archived and removed marked application program data is:

[0058] Firstly, the application program data are put into the trained Bayesian prediction model to obtain the abnormal occurrence probability of each application program data; wherein the feature of the abnormal occurrence of the application program data is that the application program data appear inconsistent data structure, and the Bayesian prediction model is a network model trained by a certain amount of application program data.

[0059] Secondly, the application data with the probability of abnormal occurrence of each piece of application data less than or equal to 50% is retained, that is, the valid text data is retained, and the data greater than 50% is marked for removal;

[0060] Then, the application data is classified according to the application programs associated with the application data, so that the application data and the application programs are one-to-one corresponding;

[0061] Finally, the meta-process data corresponding to different application data is found, the database mapped in the corresponding application program is found according to the configuration file stored in the meta-process data, and the parameters of the structure, field and index in the database are put into the meta-process data, so as to realize the data management taking the meta-process data as the main body, solve the problem that the data cannot be sorted due to inconsistent data structures in different application programs, and mark the processed meta-process data for archiving, mark the application data that cannot find the mapped database for removal, and realize the reorganization of the application data.

[0062] Further, the text data processing module in step S2 is used to process the text data from multiple applications to solve the problem of scattered text data in different applications, realize unified management of the data, and further realize the reorganization of the text data by archiving or marking for removal.

[0063] Firstly, the text data is put into the trained Bayesian prediction model to obtain the probability of abnormal occurrence of each piece of text data; wherein the feature of the abnormal occurrence of the text data is that the text data has a consistent data structure, and the Bayesian prediction model is a network model trained by a certain amount of text data;

[0064] Secondly, the data with the probability of abnormal occurrence of each piece of text data less than or equal to 50% is retained, that is, the valid text data is retained, and the data greater than 50% is marked for removal;

[0065] Then, the text data is classified according to the application programs associated with the text data, so that the text data and the application programs are one-to-one corresponding;

[0066] Finally, the meta-process data corresponding to different text data is found, the database mapped in the corresponding application program is found according to the configuration file stored in the meta-process data, and the parameters of the structure, field and index in the database are put into the meta-process data, so as to realize the data management taking the meta-process data as the main body, solve the problem of scattered text data in different application programs, and mark the processed meta-process data for archiving, mark the text data that cannot find the mapped database for removal, and realize the reorganization of the text data.

[0067] Further, the specific content of step S3 is:

[0068] The thread pool occupancy rate model calculation formula is:

[0069]

[0070] In the formula, ω represents the thread pool load index, N is the number of working threads in the thread pool at runtime, N max is the maximum number of threads set, T cur is the number of tasks in the current collection time window, T pre is the number of tasks in the last collection time window, Q is the size of the task buffer queue, and ξ1, ξ2 and ξ3 all represent weight coefficients.

[0071] The thread pool occupancy rate model is used to collect and process data in the analog signal data pool processing module, the application data processing module and the text data processing module, and the thread pool load ω is calculated during processing. If ω is greater than the preset thread pool load ω', the adaptive weight coefficient parameters are triggered to adjust and calculate. If ω is still greater than ω', the current collection time window is skipped. If ω is less than or equal to the preset thread pool load ω', the archiving mark data is preferentially transferred to the archiving data pool module, and the removal mark data is secondarily transferred to the archiving data pool module.

[0072] Further, the archiving data pool module is used to periodically clean and release the removal mark data.

[0073] It should be noted that:

[0074] 1. About meta-process data: more valuable than collected data, usually contains more information. (Record, date, place, responsible person, recording equipment, other auxiliary information).

[0075] 2. Classification of data types (three categories):

[0076] · Analog signal data (analog data)

[0077] Source: collected object (physical and network equipment, network traffic)

[0078] Characteristics: huge and repetitive, data values are similar

[0079] Abnormal characteristics: data values have large abnormalities

[0080] · Application data (application data)

[0081] Source: application, business processing

[0082] Characteristics: consistent structure, uniform format in the form of records

[0083] Abnormal feature: data structure inconsistency, and data from one data source

[0084] • Text data

[0085] Source: associated with an application

[0086] Feature: unstructured data

[0087] Abnormal feature: repetition consistency structure, data values are similar.

[0088] 3. The advantages of using the second difference value in the present application:

[0089] 1) Uniform interval, and more matching with transformer timing processing.

[0090] 2) More realistic restoration of missing data in simulation data scenarios.

[0091] The formula is as follows:

[0092]

[0093] x = the current value of the classification object, y = the 3 adjacent points of the classification object, i = the sequence number of the classification object.

[0094] 4. The present application creates a corresponding data processing pool, and rests the stored data, specifically removes the valueless data and archives the effective data. At the same time, for the massive data with removal marks and archive marks, the constructed

thread pool occupancy model

[0095] It should be noted that the terms such as "up", "down", "left", "right", "front", "back" and the like cited in the invention are only for the convenience of clear description, and are not used to limit the scope of the invention. The change or adjustment of the relative relationship is also considered as the scope of the invention without substantial change of the technical content.

[0096] The above is only a preferred embodiment of the present application, and the protection scope of the present application is not limited to the above-mentioned embodiments. Any technical solution falling within the scope of the present application is within the protection scope of the present application. It should be noted that for ordinary skilled persons in the art, some improvements and refinements without departing from the principles of the present application are considered as the protection scope of the present application.

Claims

1. A data management method of a data lake, characterized by, Comprise the following steps: S1: collect the data information of physical equipment, and then upload to the initial data pool module in the data lake; the initial data pool module is used as a storage unit for these data, and classifies the complex data system according to the collection object of the data and the characteristics of the data, and then delivers the classified data to the corresponding type of data pool processing module; wherein each data has its own meta-process data, and after the data classification is completed, the meta-process data corresponding to each data is also delivered to the corresponding data pool processing module; S2: different types of data pool processing modules include: analog signal data pool processing module, used to reduce analog signal data to a certain amount of data easy to operate and manage, and to reorganize the archived marked or removed marked reduced analog signal data; application data pool processing module, used to process application data from multiple applications to solve the inconsistent data structure in different applications, and to realize unified management of data, and then reorganize the archived marked or removed marked application data; text data processing module, used to process text data from multiple applications to solve the scattered text data in different applications, realize unified management of data, and then reorganize the archived marked or removed marked text data; S3: build a thread pool occupancy rate model, and manage and schedule the data containing removal marks and archive marks to the archive data pool module; The data information of the physical equipment includes three types: analog signal data, application data and text data; Step S2 includes: Firstly, sort a large number of repeated analog signal data, and use the quadratic difference method to perform interpolation operation on every 3 adjacent points to reduce the amount of analog signal data; Then, put the analog signal data processed by the quadratic interpolation reduction into the trained Bayesian prediction model to obtain the abnormal occurrence probability of each analog signal data; wherein the feature of analog signal data abnormality is that the data value of the analog signal exceeds the set standard value, and the Bayesian prediction model is a network model trained by a certain number of analog signal data; Finally, retain the data whose abnormal occurrence probability is less than or equal to 50%, that is, retain the effective analog signal data and mark it as archived, and mark the analog signal data greater than 50% as removed to realize the reorganization of the analog signal; Firstly, put the application data into the trained Bayesian prediction model to obtain the abnormal occurrence probability of each application data; wherein the feature of application data abnormality is that the application data appears inconsistent data structure, and the Bayesian prediction model is a network model trained by a certain number of application data; Secondly, retain the application data whose abnormal occurrence probability is less than or equal to 50%, that is, retain the effective text data and mark the data greater than 50% as removed; Then, the application data is classified according to the application associated with the application data, so that the application data and the application are one-to-one corresponding; Finally, the meta-process data corresponding to different application data is found, the database mapped in the corresponding application is found according to the configuration file stored in the meta-process data, and the parameters of the structure, field and index in the database are put into the meta-process data, so as to realize the data management taking the meta-process data as the main body, solve the problem that the data cannot be sorted due to inconsistent data structures in different applications, and mark the processed meta-process data for archiving, remove the application data that cannot find the mapped database, and realize the reorganization of the application data. Firstly, the text data is put into the trained Bayesian prediction model to obtain the abnormal occurrence probability of each text data; wherein the feature of the abnormal occurrence of the text data is that the text data has a consistent data structure, and the Bayesian prediction model is a network model trained by a certain amount of text data; Secondly, the data with an abnormal occurrence probability less than or equal to 50% is reserved, that is, the effective text data is reserved, and the data greater than 50% is marked for removal; Then, the text data is classified according to the application associated with the text data, so that the text data and the application are one-to-one corresponding; Finally, the meta-process data corresponding to different text data is found, the database mapped in the corresponding application is found according to the configuration file stored in the meta-process data, and the parameters of the structure, field and index in the database are put into the meta-process data, so as to realize the data management taking the meta-process data as the main body, solve the problem that the data cannot be sorted due to inconsistent data structures in different applications, and mark the processed meta-process data for archiving, remove the application data that cannot find the mapped database, and realize the reorganization of the text data.

2. The data management method of a data lake according to claim 1, characterized in that, The specific content of step S1 is that the data is classified according to the collection object of the data and the characteristics of the data, and then the classified data is transmitted to the corresponding type of data pool processing module: For analog signal data, the object collected is network equipment and network traffic, and the characteristic of the data is that the data amount is repeated and complex; For application data, the object collected is application and business processing, and the characteristic of the data is that it is repeated and consistent structured data; For text data, the object collected is application, and the characteristic of the data is that it is unstructured data; Each data is classified according to the collection object of the data and the corresponding characteristics of the data, and transmitted to the analog signal data pool processing module, the application data pool processing module and the text data pool processing module.

3. The data management method of a data lake according to claim 1, characterized in that, Each data in step S1 has its corresponding meta-process data, and the meta-process data contains more information content for analysis, including the time, place, responsible person and recording equipment of the corresponding data.

4. The data management method of a data lake according to claim 2, characterized in that, The specific content of step S3 is: The thread pool occupancy rate model calculation formula is: In the formula, represents the thread pool load degree index, is the number of working threads in the thread pool at runtime, is the maximum number of threads set, is the number of tasks in the current collection time window, is the number of tasks in the last collection time window, is the task buffer queue size, , , all represent weight coefficients; The thread pool occupancy model is used to collect and process data in the simulated signal data pool processing module, the application data processing module and the text data processing module, and the thread pool load is calculated during the processing , if is greater than a preset thread pool load , adaptive weight coefficient parameters are triggered to adjust and calculate, if is still greater than , the current collection time window is skipped, if is less than or equal to the preset thread pool load , the archiving mark data is preferentially transmitted to the archiving data pool module, and the removal mark data is secondarily transmitted to the archiving data pool module.

5. The data management method of a data lake according to claim 2, characterized in that, The archiving data pool module is used for periodically cleaning and releasing the removal marked data.

Images