Data lake entering demand analysis method and system, terminal and storage medium

Abstract

The invention discloses a data lake-entering demand analysis method and system, a terminal and a storage medium, and the method comprises the steps: obtaining lake-entering original data, and carrying out the preprocessing of the lake-entering original data, and obtaining target lake-entering data; performing standard degree detection processing on the target lake entering data to obtain standard lake entering data; performing quality evaluation processing and identification processing on the standard lake entering data to obtain quality lake entering data, and importing the quality lake entering data into a preset target data lake; and when a data calling request is received, determining target storage data in the preset target data lake according to the data calling request, and calling the target storage data. According to the method, the data needing to enter the data lake is preprocessed and subjected to quality evaluation, so that the quality of the data entering the data lake can be effectively ensured, and meanwhile, the efficiency of calling the business data in the data lake by the user is also effectively improved.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a method, system, terminal and computer-readable storage medium for analyzing data entry requirements. Background Art

[0002] At present, all gas companies have basically built their own multiple business systems and data middle platforms. Through the data middle platform, the business data of multiple business systems can be processed into the data lake to achieve the management and call of business data. However, in the process of data entering the lake, the source of the data often has exceptions or errors, which leads to the data quality of the business data entering the data lake cannot be guaranteed and cannot meet the user's demand for data call.

[0003] Therefore, the prior art still needs to be improved and developed. Summary of the invention

[0004] The main purpose of the present invention is to provide a data entry demand analysis method, system, terminal and computer-readable storage medium, aiming to solve the problem in the prior art that in the process of data entry into the data lake, the source of data may be abnormal or erroneous, resulting in the data quality of the business data entering the data lake cannot be guaranteed and the user's data call needs cannot be met.

[0005] To achieve the above object, the present invention provides a method for analyzing data entry requirements, which comprises the following steps:

[0006] Acquire original lake entry data, and pre-process the original lake entry data to obtain target lake entry data;

[0007] Performing a standardization detection process on the target lake entry data to obtain standard lake entry data;

[0008] Performing quality assessment and identification processing on the standardized data entering the lake to obtain quality data entering the lake, and importing the quality data entering the lake into a preset target data lake;

[0009] When a data call request is received, the target storage data in the preset target data lake is determined according to the data call request, and the target storage data is called.

[0010] Optionally, the method for analyzing data entry requirements, wherein the step of obtaining the original data to be entered into the lake and preprocessing the original data to be entered into the lake to obtain target data to be entered into the lake, specifically includes:

[0011] Obtain multiple business data tables of each business system in the data center system, and perform data extraction and processing on the multiple business data tables to obtain the original data entering the lake;

[0012] The raw data entering the lake are preprocessed to obtain target data entering the lake, wherein the preprocessing includes data cleaning processing, data conversion processing and data standardization processing.

[0013] Optionally, the data entry demand analysis method, wherein the step of performing a standardization detection process on the target entry data to obtain the standardized entry data, specifically includes:

[0014] Obtaining a data dictionary and a data model of the data middle platform system, performing a standardization detection process on the target lake-entering data according to the data dictionary and the data model, and obtaining a standard value corresponding to each lake-entering data in the target lake-entering data;

[0015] A preset standard threshold is determined, and the lake entry data whose standard value in the target lake entry data is less than the preset standard threshold is eliminated to obtain standardized lake entry data.

[0016] Optionally, the data entry demand analysis method, wherein the quality assessment and identification processing is performed on the standard entry data to obtain quality entry data, and the quality entry data is imported into a preset target data lake, specifically including:

[0017] Acquire the field type of the standardized data entering the lake, and perform quality assessment processing on the standardized data entering the lake according to the field type to obtain a quality assessment result;

[0018] The standardized inbound data is identified according to the quality assessment result to obtain quality inbound data, and the quality inbound data is imported into a preset target data lake.

[0019] Optionally, in the data entry demand analysis method, the field type includes field duplicate values ​​and field null values; the quality assessment result includes low-quality data, medium-quality data, and high-quality data;

[0020] The obtaining of the field type of the standardized data entering the lake, and performing quality assessment processing on the standardized data entering the lake according to the field type to obtain a quality assessment result specifically includes:

[0021] Obtaining field duplicate values ​​and field null values ​​of the standardized data entering the lake, comparing the field duplicate values ​​with a first preset threshold, and comparing the field null values ​​with a second preset threshold;

[0022] If the field duplication value is greater than a first preset threshold and the field null value is greater than a second preset threshold, the standardized data entering the lake is determined to be low-quality data;

[0023] If the field repetition value is less than or equal to the first preset threshold and the field null value is greater than the second preset threshold, or the field repetition value is greater than the first preset threshold and the field null value is less than or equal to the second preset threshold, the standardized entry data is determined to be medium quality data;

[0024] If the field repetition value is less than or equal to the first preset threshold and the field empty value is less than or equal to the second preset threshold, the standardized inbound data is determined to be high-quality data.

[0025] Optionally, the data entry demand analysis method, wherein the step of marking the standard entry data according to the quality assessment result to obtain quality entry data, and importing the quality entry data into a preset target data lake, specifically includes:

[0026] The low-quality data, the medium-quality data, and the high-quality data are set as data labels, and the standard in-lake data are marked according to the data labels to obtain quality in-lake data;

[0027] The quality data entering the lake is compared with the standard data entering the lake to obtain a comparison result. If the comparison result meets the preset requirements, the quality data entering the lake is imported into the preset target data lake.

[0028] Optionally, the method for analyzing data entry requirements into a data lake, wherein when a data call request is received, determining target storage data in the preset target data lake according to the data call request, and calling the target storage data, specifically includes:

[0029] When receiving a data call request sent by a user, determining the user call authority of the user according to the data call request;

[0030] Performing authority authentication in the preset target data lake according to the user's calling authority, and when the authority authentication passes, determining a data query scope of the user's calling authority in the preset target data lake;

[0031] According to the data call request, a data query call search is performed within the data query range to obtain target storage data, and the target storage data is sent to the user.

[0032] In addition, to achieve the above-mentioned purpose, the present invention also provides a data entry demand analysis system, wherein the data entry demand analysis system comprises:

[0033] A data preprocessing module is used to obtain the original lake entry data and preprocess the original lake entry data to obtain target lake entry data;

[0034] A standardization detection module is used to perform standardization detection processing on the target lake entry data to obtain standard lake entry data;

[0035] A data import module is used to perform quality assessment and identification processing on the standardized data entering the lake, obtain quality data entering the lake, and import the quality data entering the lake into a preset target data lake;

[0036] The data calling module is used to determine the target storage data in the preset target data lake according to the data calling request when a data calling request is received, and call the target storage data.

[0037] In addition, to achieve the above-mentioned purpose, the present invention also provides a terminal, wherein the terminal includes: a memory, a processor, and a data entry into the lake demand analysis program stored in the memory and executable on the processor, and when the data entry into the lake demand analysis program is executed by the processor, the steps of the data entry into the lake demand analysis method as described above are implemented.

[0038] In addition, to achieve the above-mentioned purpose, the present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a data entry into the lake demand analysis program, and when the data entry into the lake demand analysis program is executed by a processor, the steps of the data entry into the lake demand analysis method as described above are implemented.

[0039] In the present invention, the original data entering the lake is obtained, and the original data entering the lake is preprocessed to obtain the target data entering the lake; the target data entering the lake is subjected to a standardization detection process to obtain the standardized data entering the lake; the standardized data entering the lake is subjected to a quality assessment process and an identification process to obtain the quality data entering the lake, and the quality data entering the lake is imported into the preset target data lake; when a data call request is received, the target storage data in the preset target data lake is determined according to the data call request, and the target storage data is called. The present invention can effectively ensure the data quality of the business data entering the data lake by preprocessing, standardization detection and quality assessment of the data that needs to enter the data lake. At the same time, by identifying the data in the data lake, the data call efficiency of the data lake is also effectively improved, which can help users quickly obtain the target storage data in the data lake and improve the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 It is a flow chart of a preferred embodiment of the method for analyzing data entry requirements of the present invention;

[0041] Figure 2 It is a structural diagram of a preferred embodiment of the data entry demand analysis system of the present invention;

[0042] Figure 3It is a structural diagram of a preferred embodiment of the terminal of the present invention. DETAILED DESCRIPTION

[0043] In order to make the purpose, technical solution and advantages of the present invention clearer and more specific, the present invention is further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0044] Data lake is a flexible and scalable data storage and management system that receives and stores raw data from heterogeneous data sources in their original format, and provides query processing and data analysis in a dynamic manner. Generally, data lake data storage and analysis include three links: the first is raw data storage, which stores data in the databases of the business systems; the second is data entry, which ingests raw data into the centralized data storage and analysis engine through tasks. Data ingestion can be batch processing tasks or stream processing; the third is data processing, which converts and analyzes data in the centralized data storage and analysis engine.

[0045] At present, all gas companies have basically built their own multiple business systems and data middle platforms. Through the data middle platform, the business data of multiple business systems can be processed into the data lake to achieve the management and call of business data. However, in the process of data entering the lake, the source of the data often has exceptions or errors, which leads to the data quality of the business data entering the data lake cannot be guaranteed and cannot meet the user's demand for data call.

[0046] The data entry demand analysis method described in the preferred embodiment of the present invention is as follows: Figure 1 As shown, the data entry demand analysis method includes the following steps:

[0047] Step S10: obtaining original lake entry data, and preprocessing the original lake entry data to obtain target lake entry data.

[0048] Data entry demand analysis refers to the management and screening of data that needs to enter the data lake in a planning-driven or demand-driven scenario. Taking the gas data middle-end system as an example, the present invention needs to sort out the data sources including CIS, GIS, PS, pipeline integrity, and school business systems, identify the core data that needs to be accessed, and design data collection content and data quality requirements.

[0049] Among them, before the data enters the lake, some data preprocessing work needs to be done, including data cleaning, conversion and standardization. For the collection process of the original data entering the lake, the present invention is achieved by expanding the scope of data collection and supporting multi-source data collection, wherein the multi-source data includes unstructured and structured data, relational and non-relational databases and real-time data. The business system data in the present invention include CIS system, GIS system, PS system, pipeline integrity system, and school system data (i.e., the original data entering the lake in the present invention).

[0050] Specifically, multiple business data tables of each business system in the data middle platform system are obtained, and data extraction processing is performed on the multiple business data tables to obtain original data entering the lake; the original data entering the lake is pre-processed to obtain target data entering the lake, wherein the pre-processing includes data cleaning processing, data conversion processing and data standardization processing.

[0051] It can be understood that the original data entering the lake in the present invention refers to the records in a table. For example, 100 transaction records are saved in the transaction order table, and these 100 records are the data entering the lake. In the present invention, the data of each business system under the data middle-end system is used as the data entering the lake.

[0052] Since the data sources of the original database are relatively complex, and there is no unified planning for the data between the various business systems that serve as data sources, before loading these data into the data lake, it is necessary to fully understand the situation and characteristics of these data and form a data quality statistical report. Among them, all source data need to be cleaned and processed, including data standard management, data quality assessment, and data cleaning and processing.

[0053] The monitoring and management process of data quality includes: 1. Data fusion tasks and configurations can be defined through the interface; 2. The association relationship of data source tables can be defined through the interface, and data can be stored after rule processing, calculation, sorting and other operations. 3. It can support the verification of data quality during the data fusion process, statistics and location of invalid data, and realize the monitoring and management of data quality.

[0054] For the data conversion process, a data format conversion unit is provided in the present invention, wherein the data format conversion unit is used to convert the format of the data so that it complies with the data specification written into the data lake. The data format conversion unit can be selected according to the table type corresponding to the original data entering the lake, so as to realize parallel processing of incremental data by multiple data format conversion units.

[0055] Regarding data standardization processing, the present invention performs standardization processing on the original data entering the lake by obtaining user needs and data lake entry needs, which is conducive to building standardized data entering the lake.

[0056] Step S20: performing a standardization detection process on the target lake entry data to obtain standardized lake entry data.

[0057] Before data enters the lake, it is necessary to check the readiness of the data source, which includes providing the data dictionary and data model of the data middle platform system and checking the standardization of the physical tables of the data middle platform system. The present invention obtains the data dictionary and data model of the data middle platform system, performs a standardization check on the pre-processed target data entering the lake according to the data dictionary and the data model, and removes the data that does not meet the requirements, so as to obtain standardized data entering the lake.

[0058] Specifically, the data dictionary and the data model of the data middle platform system are obtained, and the target lake-entering data are subjected to standardization detection and processing according to the data dictionary and the data model to obtain the standard value corresponding to each lake-entering data in the target lake-entering data; a preset standard threshold is determined, and the lake-entering data whose standard value in the target lake-entering data is less than the preset standard threshold is eliminated to obtain the standardized lake-entering data.

[0059] Among them, the data dictionary refers to the definition and description of the data items, data structure, data flow, data storage, processing logic and external entities of the target data entering the lake. Its purpose is to provide a detailed description of each element in the target data entering the lake.

[0060] Furthermore, in the present invention, the catalog, metadata, data application and data quality, data standards, and data security of all data entering the lake must be managed in a unified manner. Among them, data model standards, dynamic maintenance of master data, centralized data quality governance, data problem tracing processing of native systems, and production system builders need to participate in data management throughout the process, and their responsibilities and rights must remain consistent. The production system management department is responsible for the standardization of system models and master data within its jurisdiction; the data lake uniformly manages the data model and master data of the production system; for production system data models that have not yet been standardized, the management department of the corresponding system is responsible for the conversion and operation of the data model, and coordinates and promotes the production system data standardization process.

[0061] In the present invention, the target data entering the lake is first defined and described through a data dictionary, and then the corresponding definition and description are input into a pre-trained data model to obtain the standardization of each target data entering the lake and the corresponding standard value.

[0062] Step S30: perform quality assessment and identification processing on the standardized data entering the lake to obtain quality data entering the lake, and import the quality data entering the lake into a preset target data lake.

[0063] In the present invention, the criteria for evaluating data entering the lake include clarifying the data and authenticating the data source, and also evaluating the quality of the data entering the lake and labeling the data entering the lake with quality labels. After that, it is necessary to use specific tools or write code to import data from the data middle platform system into the data lake.

[0064] For data storage and computing needs: urban gas, upstream resources, integrated energy, and smart services, these topics are important future development directions for data services, and these fields generally have large amounts of data and complex data types. There are a large number of unstructured data types such as office documents, texts, GIS, pictures, audio and video data, so it is necessary to implement a complete relational database capability to support the storage and computing of structured data, while supporting traditional relational databases and columnar databases. The database has distributed storage characteristics and uses a high-performance distributed computing engine. Among them, the storage database set by the present invention has the following characteristics: 1. Distributed storage: It is designed using a distributed system infrastructure to meet the massive data access of structured and unstructured data, including office documents, texts, pictures, GIS, audio and video, etc. For data with retention value, it can support the storage of massive historical data. 2. Distributed computing: The system supports automatically dividing the big data to be processed by a job into many data blocks, each data block corresponds to a computing task, and automatically schedules computing nodes to process the corresponding data blocks. 3. Large-scale parallel computing: including large-scale parallel computing technology and row-column hybrid storage: including shared-nothing / MPP architecture, support for row / column storage database, in-database compression, MapReduce, online expansion, multi-level fault tolerance, etc. 4. In-memory computing engine: supports in-memory computing, uses Spark as a real-time analysis engine, and is compatible with Hadoop data sources.

[0065] Specifically, the field duplication value and field empty value of the standardized data entering the lake are obtained, the field duplication value is compared with a first preset threshold, and the field empty value is compared with a second preset threshold; if the field duplication value is greater than the first preset threshold and the field empty value is greater than the second preset threshold, the standardized data entering the lake is determined to be low-quality data; if the field duplication value is less than or equal to the first preset threshold and the field empty value is greater than the second preset threshold, or the field duplication value is greater than the first preset threshold and the field empty value is less than or equal to the second preset threshold, the standardized data entering the lake is determined to be medium-quality data; if the field duplication value is less than or equal to the first preset threshold and the field empty value is less than or equal to the second preset threshold, the standardized data entering the lake is determined to be high-quality data.

[0066] It can be understood that the present invention performs verification based on the field duplication values ​​and field empty values ​​of the standardized data entering the lake (wherein the number of field duplication values ​​and field empty values ​​represents the input quality of the data entering the lake) to ensure data quality. For example, if the field duplication value of the standardized data entering the lake is greater than the first preset threshold and the field empty value is greater than the second preset threshold, the standardized data entering the lake is determined to be low-quality data.

[0067] In the present invention, after the quality data entering the lake is imported into the preset target data lake, the quality of the quality data entering the lake needs to be checked again to ensure that no problems occur during the import process of the quality data entering the lake.

[0068] The low-quality data, the medium-quality data and the high-quality data are set as data labels, and the standardized data entering the lake are identified according to the data labels to obtain quality data entering the lake; the quality data entering the lake is compared with the standardized data entering the lake to obtain a comparison result. If the comparison result meets the preset requirements, the quality data entering the lake is imported into the preset target data lake.

[0069] It can be understood that after the present invention performs quality assessment processing on the standardized data entering the lake and obtains the quality assessment results, it will once again perform data quality inspection and data quality rectification on the quality assessment results to ensure that the data entering the lake can automatically perform preliminary data cleaning and interpretation recognition, and associate the quality assessment results as labels with the standardized data entering the lake. The updated standardized data entering the lake will not be overwritten, but will be labeled with a version date number (i.e., the quality assessment results in the present invention). Data process automation tools are used to automatically run decision-making processes or actions to manage the data ecosystem, and tools for building, deploying and managing workflow automation and simplification are used to streamline and automate data quality improvement work and data labeling workflows.

[0070] After obtaining the quality assessment results of the standardized lake-entering data, the present invention will use the quality assessment results as data labels, and identify each lake-entering data in the standardized lake-entering data, and determine the lake-entering data through the labels, thereby realizing the comparison between the quality lake-entering data and the standardized lake-entering data. This step aims to verify the integrity and accuracy of the lake-entering data and determine whether the data loss degree meets the lake-entering requirements.

[0071] Furthermore, before importing the quality data into the preset target data lake, each piece of data in the quality data will be assigned an ordered and unique number, and then the quality data will be globally ordered and deduplicated according to the number, and finally written into the data lake table in order. Among them, the present invention uses the API interface of the data lake technology to convert the data into the Hudi table format and write it into the distributed file system of the data lake.

[0072] Furthermore, after the quality data entering the lake is imported into the preset target data lake, the quality data entering the lake will be compared with the standard data entering the lake. If they are consistent, it means that the data quality requirements are met and the quality data entering the lake can be stored.

[0073] Step S40: When a data call request is received, the target storage data in the preset target data lake is determined according to the data call request, and the target storage data is called.

[0074] When the data is successfully imported into the data lake, it needs to be published to the appropriate repository and appropriate access control is set. The present invention provides that when the quality of the data entering the lake is determined to meet the detection requirements, the data entering the lake is published and access control is set, and only users who meet the preset permissions can access it.

[0075] Specifically, when a data call request issued by a user is received, the user's user call authority is determined according to the data call request; authority authentication is performed in the preset target data lake according to the user call authority, and when the authority authentication passes, the data query scope of the user call authority in the preset target data lake is determined; data query call search is performed within the data query scope according to the data call request to obtain the target storage data, and the target storage data is sent to the user.

[0076] The present invention sets calling permissions for users' data calls. The data calling permissions of each user are different. When a user needs to call data in a preset target data lake, identity authentication and permission identification are required. Only after identity authentication can the queryable scope corresponding to the user in the preset target data lake be determined according to the user's corresponding permissions. The user calls data based on the queryable scope, which effectively ensures the accuracy and security of data calls.

[0077] In addition, the present invention can also set up the asset catalog that needs to be shared in the selected preset target data lake, configure filtering, desensitizing and encryption rules according to the classification and grading information, and then check the fields that need to be shared to form a service interface, and conduct interface approval for the formed service interface. After approval, a data service security window is formed and opened to the outside.

[0078] Furthermore, if Figure 2 As shown, based on the above-mentioned data entry demand analysis method, the present invention also provides a data entry demand analysis system, wherein the data entry demand analysis system includes:

[0079] The data preprocessing module 51 is used to obtain the original lake entry data and preprocess the original lake entry data to obtain the target lake entry data;

[0080] A standardization detection module 52 is used to perform standardization detection processing on the target lake entry data to obtain standard lake entry data;

[0081] The data import module 53 is used to perform quality assessment and identification processing on the standard data entering the lake, obtain quality data entering the lake, and import the quality data entering the lake into a preset target data lake;

[0082] The data calling module 54 is used to determine the target storage data in the preset target data lake according to the data calling request when receiving the data calling request, and call the target storage data.

[0083] Furthermore, if Figure 3 As shown, based on the above-mentioned data entry demand analysis method and system, the present invention also provides a terminal accordingly, and the terminal includes a processor 10, a memory 20 and a display 30. Figure 3 Only some components of the terminal are shown, but it should be understood that it is not required to implement all of the components shown, and more or fewer components may be implemented instead.

[0084] In some embodiments, the memory 20 may be an internal storage unit of the terminal, such as a hard disk or memory of the terminal. In other embodiments, the memory 20 may also be an external storage device of the terminal, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (SecureDigital, SD) card, a flash card (Flash Card), etc. equipped on the terminal. Furthermore, the memory 20 may also include both an internal storage unit of the terminal and an external storage device. The memory 20 is used to store application software and various types of data installed on the terminal, such as the program code of the installation terminal. The memory 20 can also be used to temporarily store data that has been output or is to be output. In one embodiment, a data entry demand analysis program 40 is stored on the memory 20, and the data entry demand analysis program 40 can be executed by the processor 10, thereby realizing the data entry demand analysis method in the present application.

[0085] In some embodiments, the processor 10 may be a central processing unit (CPU), a microprocessor or other data processing chip, used to run the program code stored in the memory 20 or process data, such as executing the data entry demand analysis method.

[0086] In some embodiments, the display 30 may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, an OLED (Organic Light-Emitting Diode) touch device, etc. The display 30 is used to display information on the terminal and to display a visual user interface. The components 10-30 of the terminal communicate with each other via a system bus.

[0087] In one embodiment, when the processor 10 executes the data entry requirement analysis program 40 in the memory 20, the following steps are implemented:

[0088] Acquire original lake entry data, and pre-process the original lake entry data to obtain target lake entry data;

[0089] Performing a standardization detection process on the target lake entry data to obtain standard lake entry data;

[0090] Performing quality assessment and identification processing on the standardized data entering the lake to obtain quality data entering the lake, and importing the quality data entering the lake into a preset target data lake;

[0091] When a data call request is received, the target storage data in the preset target data lake is determined according to the data call request, and the target storage data is called.

[0092] The obtaining of the original lake entry data and preprocessing of the original lake entry data to obtain the target lake entry data specifically includes:

[0093] Obtain multiple business data tables of each business system in the data center system, and perform data extraction and processing on the multiple business data tables to obtain the original data entering the lake;

[0094] The raw data entering the lake are preprocessed to obtain target data entering the lake, wherein the preprocessing includes data cleaning processing, data conversion processing and data standardization processing.

[0095] The process of performing a standardization detection process on the target lake entry data to obtain the standard lake entry data specifically includes:

[0096] Obtaining a data dictionary and a data model of the data middle platform system, performing a standardization detection process on the target lake-entering data according to the data dictionary and the data model, and obtaining a standard value corresponding to each lake-entering data in the target lake-entering data;

[0097] A preset standard threshold is determined, and the lake entry data whose standard value in the target lake entry data is less than the preset standard threshold is eliminated to obtain standardized lake entry data.

[0098] The step of performing quality assessment and identification processing on the standardized data entering the lake to obtain quality data entering the lake, and importing the quality data entering the lake into a preset target data lake specifically includes:

[0099] Acquire the field type of the standardized data entering the lake, and perform quality assessment processing on the standardized data entering the lake according to the field type to obtain a quality assessment result;

[0100] The standardized inbound data is identified according to the quality assessment result to obtain quality inbound data, and the quality inbound data is imported into a preset target data lake.

[0101] Wherein, the field type includes field duplicate values ​​and field null values; the quality assessment result includes low-quality data, medium-quality data and high-quality data;

[0102] The obtaining of the field type of the standardized data entering the lake, and performing quality assessment processing on the standardized data entering the lake according to the field type to obtain a quality assessment result specifically includes:

[0103] Obtaining field duplicate values ​​and field null values ​​of the standardized data entering the lake, comparing the field duplicate values ​​with a first preset threshold, and comparing the field null values ​​with a second preset threshold;

[0104] If the field duplication value is greater than a first preset threshold and the field null value is greater than a second preset threshold, the standardized data entering the lake is determined to be low-quality data;

[0105] If the field repetition value is less than or equal to the first preset threshold and the field null value is greater than the second preset threshold, or the field repetition value is greater than the first preset threshold and the field null value is less than or equal to the second preset threshold, the standardized entry data is determined to be medium quality data;

[0106] If the field repetition value is less than or equal to the first preset threshold and the field empty value is less than or equal to the second preset threshold, the standardized inbound data is determined to be high-quality data.

[0107] The step of marking the standard data entering the lake according to the quality assessment result to obtain quality data entering the lake, and importing the quality data entering the lake into a preset target data lake specifically includes:

[0108] The low-quality data, the medium-quality data, and the high-quality data are set as data labels, and the standard in-lake data are marked according to the data labels to obtain quality in-lake data;

[0109] The quality data entering the lake is compared with the standard data entering the lake to obtain a comparison result. If the comparison result meets the preset requirements, the quality data entering the lake is imported into the preset target data lake.

[0110] When a data call request is received, determining the target storage data in the preset target data lake according to the data call request, and calling the target storage data specifically includes:

[0111] When receiving a data call request sent by a user, determining the user call authority of the user according to the data call request;

[0112] Performing authority authentication in the preset target data lake according to the user's calling authority, and when the authority authentication passes, determining a data query scope of the user's calling authority in the preset target data lake;

[0113] According to the data call request, a data query call search is performed within the data query range to obtain target storage data, and the target storage data is sent to the user.

[0114] The present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a data entry into the lake demand analysis program, and when the data entry into the lake demand analysis program is executed by a processor, the steps of the data entry into the lake demand analysis method as described above are implemented.

[0115] In summary, the present invention provides a method for analyzing data entry requirements and related equipment, the method comprising: obtaining the original data entering the lake, and preprocessing the original data entering the lake to obtain target data entering the lake; performing standardization detection processing on the target data entering the lake to obtain standardized data entering the lake; performing quality assessment processing and identification processing on the standardized data entering the lake to obtain quality data entering the lake, and importing the quality data entering the lake into a preset target data lake; when a data call request is received, determining the target storage data in the preset target data lake according to the data call request, and calling the target storage data. The present invention can effectively ensure the data quality of business data entering the data lake by preprocessing, standardization detection processing, and quality assessment processing on the data that needs to enter the data lake. At the same time, by identifying the data in the data lake, it also effectively improves the data call efficiency of the data lake, which can help users quickly obtain the target storage data in the data lake and improve the user experience.

[0116] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the presence of other identical elements in the process, method, article or terminal including the element.

[0117] Of course, those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing related hardware (such as a processor, a controller, etc.) through a computer program, and the program can be stored in a computer-readable storage medium that can be read by a computer, and the program can include the processes of the above-mentioned method embodiments when executed. The computer-readable storage medium can be a memory, a disk, an optical disk, etc.

[0118] It should be understood that the application of the present invention is not limited to the above examples. For ordinary technicians in this field, improvements or changes can be made based on the above description. All these improvements and changes should fall within the scope of protection of the claims attached to the present invention.

Claims

1. A method for analyzing data entry requirements, characterized in that: The data entry demand analysis method includes: Acquire original lake entry data, and pre-process the original lake entry data to obtain target lake entry data; Performing a standardization detection process on the target lake entry data to obtain standard lake entry data; Performing quality assessment and identification processing on the standardized data entering the lake to obtain quality data entering the lake, and importing the quality data entering the lake into a preset target data lake; When a data call request is received, the target storage data in the preset target data lake is determined according to the data call request, and the target storage data is called.

2. The data lake demand analysis method according to claim 1 is characterized in that: The obtaining of the original lake entry data and preprocessing of the original lake entry data to obtain target lake entry data specifically includes: Obtain multiple business data tables of each business system in the data center system, and perform data extraction and processing on the multiple business data tables to obtain the original data entering the lake; The raw data entering the lake are preprocessed to obtain target data entering the lake, wherein the preprocessing includes data cleaning processing, data conversion processing and data standardization processing.

3. The data entry demand analysis method according to claim 2 is characterized in that: The process of performing a standardization detection process on the target lake entry data to obtain the standard lake entry data specifically includes: Obtaining a data dictionary and a data model of the data middle platform system, performing a standardization detection process on the target lake-entering data according to the data dictionary and the data model, and obtaining a standard value corresponding to each lake-entering data in the target lake-entering data; A preset standard threshold is determined, and the lake entry data whose standard value in the target lake entry data is less than the preset standard threshold is eliminated to obtain standardized lake entry data.

4. The method for analyzing data entry requirements according to claim 1 is characterized in that: The step of performing quality assessment and identification processing on the standard data entering the lake to obtain quality data entering the lake, and importing the quality data entering the lake into a preset target data lake specifically includes: Acquire the field type of the standardized data entering the lake, and perform quality assessment processing on the standardized data entering the lake according to the field type to obtain a quality assessment result; The standardized inbound data is identified according to the quality assessment result to obtain quality inbound data, and the quality inbound data is imported into a preset target data lake.

5. The method for analyzing data entry requirements according to claim 4 is characterized in that: The field types include field duplicate values ​​and field null values; the quality assessment results include low-quality data, medium-quality data, and high-quality data; The obtaining of the field type of the standardized data entering the lake, and performing quality assessment processing on the standardized data entering the lake according to the field type to obtain a quality assessment result specifically includes: Obtaining field duplicate values ​​and field null values ​​of the standardized data entering the lake, comparing the field duplicate values ​​with a first preset threshold, and comparing the field null values ​​with a second preset threshold; If the field duplication value is greater than a first preset threshold and the field null value is greater than a second preset threshold, the standardized data entering the lake is determined to be low-quality data; If the field repetition value is less than or equal to the first preset threshold and the field null value is greater than the second preset threshold, or the field repetition value is greater than the first preset threshold and the field null value is less than or equal to the second preset threshold, the standardized entry data is determined to be medium quality data; If the field repetition value is less than or equal to the first preset threshold and the field empty value is less than or equal to the second preset threshold, the standardized inbound data is determined to be high-quality data.

6. The data entry demand analysis method according to claim 5 is characterized in that: The step of marking the standard data entering the lake according to the quality assessment result to obtain quality data entering the lake, and importing the quality data entering the lake into a preset target data lake specifically includes: The low-quality data, the medium-quality data, and the high-quality data are set as data labels, and the standard in-lake data are marked according to the data labels to obtain quality in-lake data; The quality data entering the lake is compared with the standard data entering the lake to obtain a comparison result. If the comparison result meets the preset requirements, the quality data entering the lake is imported into the preset target data lake.

7. The method for analyzing data entry requirements according to claim 1 is characterized in that: When receiving the data call request, determining the target storage data in the preset target data lake according to the data call request, and calling the target storage data specifically includes: When receiving a data call request sent by a user, determining the user call authority of the user according to the data call request; Performing authority authentication in the preset target data lake according to the user's calling authority, and when the authority authentication passes, determining a data query scope of the user's calling authority in the preset target data lake; According to the data call request, a data query call search is performed within the data query range to obtain target storage data, and the target storage data is sent to the user.

8. A data lake demand analysis system, characterized in that: The data entry lake demand analysis system includes: A data preprocessing module is used to obtain the original lake entry data and preprocess the original lake entry data to obtain target lake entry data; A standardization detection module is used to perform standardization detection processing on the target lake entry data to obtain standard lake entry data; A data import module is used to perform quality assessment and identification processing on the standardized data entering the lake, obtain quality data entering the lake, and import the quality data entering the lake into a preset target data lake; The data calling module is used to determine the target storage data in the preset target data lake according to the data calling request when a data calling request is received, and call the target storage data.

9. A terminal, characterized in that: The terminal includes: a memory, a processor, and a data entry demand analysis program stored in the memory and executable on the processor. When the data entry demand analysis program is executed by the processor, the steps of the data entry demand analysis method as described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a data entry demand analysis program, and when the data entry demand analysis program is executed by a processor, the steps of the data entry demand analysis method according to any one of claims 1 to 7 are implemented.

Images