Method, device and equipment for realizing lake entering data quality detection based on sentry, and medium

Abstract

The invention relates to the field of data processing and financial science and technology, and discloses a method for realizing lake entering data quality detection based on a sentry, which comprises the following steps: creating a data sentry corresponding to a data lake; scheduling a cluster resource corresponding to the main server, analyzing a resource consumption situation corresponding to the cluster resource, and setting a distribution mechanism corresponding to the data sentry; querying a communication center corresponding to the main server, analyzing a center theme corresponding to the communication center, performing distribution processing on data sentries in a data lake to obtain a data sentry flow, performing sampling inspection processing on the data sentry flow to obtain a sampling inspection report, and analyzing the stability corresponding to the data sentry flow; the data sentry flow is identified to obtain sentry identities, the data sentry flow is subjected to sentry sorting to obtain a sorted sentry flow, and the sorted sentry flow is subjected to lake entering treatment to obtain a lake entering result. The invention aims to improve the accuracy of lake entry data quality detection based on the sentry.

Description

Technical Field

[0001] The present invention relates to the fields of data processing and financial technology, and in particular to a method, device, equipment and medium for detecting the quality of incoming data based on Sentinel. Background Art

[0002] The integration of data warehouse and data lake is a new data architecture that has emerged in various industries in recent years. It combines the advantages of data warehouse and data lake very well. In the financial field, the accuracy of data in the integration of data warehouse and data lake is crucial. For example, in the credit business of banks, the data entering the lake in real time includes the customer's credit record, asset information, transaction flow, etc. If the data quality monitoring is inaccurate, the wrong data will flow into the integration of data warehouse and lake, which may cause the risk assessment model to give wrong assessment results. However, the existing processing of entering the lake still lacks a relatively mature data quality monitoring technology system. Data quality is one of the most core elements to stimulate the potential of data elements. If the accuracy of the data entering the lake cannot be ensured, it will limit the effective promotion of the integration of data architecture in various industries.

[0003] Traditional real-time quality monitoring of data entering the lake mainly uses two methods: task monitoring and data comparison. Both methods have pain points such as monitoring delay, poor data quality, low coverage, and high monitoring resource consumption. Task monitoring needs to be implemented through offline and real-time scheduling systems. It mainly monitors whether the data tasks are completed normally and cannot determine data loss. For the comparison between source data and lake target data, it is more necessary to formulate judgment rules and deploy offline tasks separately to compare data differences. This is a post-comparison with poor real-time performance. In addition, the data that enters the lake incorrectly has been consumed by the downstream in real time, affecting data analysis, decision-making and data application in real time. Therefore, a method is needed to improve the accuracy of quality detection of data entering the lake based on Sentinel. Summary of the invention

[0004] The present invention provides a method, device, equipment and medium for detecting the quality of data entering a lake based on a sentinel, and its main purpose is to improve the accuracy of detecting the quality of data entering a lake based on a sentinel.

[0005] To achieve the above purpose, the present invention provides a method for detecting the quality of lake data based on Sentinel, including:

[0006] Obtain a data lake to be monitored and its corresponding master server, analyze the data business scenario corresponding to the data in the data lake, and create a data sentinel corresponding to the data lake based on the data business scenario;

[0007] Scheduling cluster resources corresponding to the master server, analyzing the resource consumption situation corresponding to the cluster resources, extracting sentinel attributes corresponding to the data sentinel, calculating the sentinel hash value corresponding to the data sentinel based on the sentinel attributes, and setting the distribution mechanism corresponding to the data sentinel according to the resource consumption situation and the sentinel hash value;

[0008] Query the communication hub corresponding to the master server, parse the hub topic corresponding to the communication hub, distribute the data sentinel in the data lake in combination with the distribution mechanism and the hub topic, obtain the data sentinel flow, perform random inspection on the data sentinel flow, obtain the random inspection report, and analyze the stability corresponding to the data sentinel flow based on the random inspection report;

[0009] The data sentinel stream is identified and processed to obtain a sentinel identity; the data sentinel stream is sentinel sorted in combination with the stability and the sentinel identity to obtain a sorted sentinel stream; the sorted sentinel stream is sent to a sentinel receiving center in the main server; and the sorted sentinel stream is early-warning processed by the sentinel receiving center to obtain an early-warning result; and according to the early-warning result, the sorted sentinel stream is entered into a lake to obtain an entry into a lake result.

[0010] In order to solve the above problems, the present invention also provides a device for detecting the quality of lake entry data based on Sentinel, the device comprising:

[0011] A sentinel creation module is used to obtain a data lake to be monitored and its corresponding master server, analyze the data business scenario corresponding to the data in the data lake, and create a data sentinel corresponding to the data lake based on the data business scenario;

[0012] A distribution mechanism setting module, used for scheduling cluster resources corresponding to the master server, analyzing the resource consumption situation corresponding to the cluster resources, extracting sentinel attributes corresponding to the data sentinel, calculating the sentinel hash value corresponding to the data sentinel based on the sentinel attributes, and setting the distribution mechanism corresponding to the data sentinel according to the resource consumption situation and the sentinel hash value;

[0013] A stability analysis module, used to query the communication hub corresponding to the master server, parse the hub theme corresponding to the communication hub, distribute the data sentinel in the data lake in combination with the distribution mechanism and the hub theme, obtain the data sentinel flow, perform random inspection on the data sentinel flow, obtain the random inspection report, and analyze the stability corresponding to the data sentinel flow based on the random inspection report;

[0014] The lake entry processing module is used to identify and process the data sentinel stream to obtain the sentinel identity, perform sentinel sorting processing on the data sentinel stream in combination with the stability and the sentinel identity to obtain a sorted sentinel stream, send the sorted sentinel stream to the sentinel receiving center in the main server, and perform early warning processing on the sorted sentinel stream through the sentinel receiving center to obtain an early warning result, and according to the early warning result, perform lake entry processing on the sorted sentinel stream to obtain a lake entry result.

[0015] In order to solve the above problem, the present invention further provides a computer device, the computer device comprising:

[0016] at least one processor; and,

[0017] a memory communicatively connected to the at least one processor; wherein,

[0018] The memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the above-mentioned method for detecting the quality of lake entry data based on Sentinel.

[0019] In order to solve the above problems, the present invention also provides a computer-readable medium, in which at least one computer program is stored. The at least one computer program is executed by a processor in a computer device to implement the above-mentioned method for detecting the quality of lake entry data based on Sentinel.

[0020] By analyzing the data business context corresponding to the data in the data lake, the present invention can understand the business application scenario corresponding to the data in the data lake, and provide a basis for the subsequent creation of data sentinels. By analyzing the resource consumption situation corresponding to the cluster resources, the present invention can understand the consumption of the cluster resources, and extract the sentinel attributes corresponding to the data sentinels to obtain the sentinel features corresponding to the data sentinels, thereby providing a basis for the subsequent calculation of sentinel hash values. By combining the distribution mechanism and the central theme, the present invention distributes and processes the data sentinels in the data lake, which is convenient for efficient data monitoring and management of the data lake, ensuring the accuracy, completeness and timeliness of the data, and at the same time can better adapt to different data characteristics and business needs, and improve the overall performance and value output of the data lake; by identifying and processing the data sentinel stream, the present invention can obtain the sentinel identity information corresponding to the data sentinel stream, and by combining the stability and the sentinel identity, the data sentinel stream is sentinel sorted, so that data sentinels with higher stability can be sorted out, and sentinels with low stability in the data sentinels can be removed. Therefore, the method, device, equipment and medium for detecting the quality of data entering the lake based on Sentinel provided in the embodiments of the present invention can improve the accuracy of detecting the quality of data entering the lake based on Sentinel. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 A schematic diagram of an application environment for a method for detecting the quality of incoming lake data based on Sentinel provided in an embodiment of the present invention;

[0022] Figure 2A flow chart of a method for detecting lake entry data quality based on sentinel provided in an embodiment of the present invention

[0023] Figure 3 A functional module diagram of a device for detecting the quality of lake entry data based on sentinels provided in one embodiment of the present invention;

[0024] Figure 4 is a schematic diagram of a structure of a computer device in one embodiment of the present invention;

[0025] Figure 5 It is another structural schematic diagram of a computer device in one embodiment of the present invention.

[0026] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0027] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0028] The method for detecting the quality of lake data based on sentinel provided in the embodiment of the present invention can be applied in the following aspects: Figure 1In an application environment, the client communicates with the server through a network. The server can obtain the data lake to be monitored and its corresponding main server through the client, analyze the data business scenario corresponding to the data in the data lake, and create a data sentinel corresponding to the data lake based on the data business scenario; schedule the cluster resources corresponding to the main server, analyze the resource consumption situation corresponding to the cluster resources, extract the sentinel attributes corresponding to the data sentinel, calculate the sentinel hash value corresponding to the data sentinel based on the sentinel attributes, and set the distribution mechanism corresponding to the data sentinel according to the resource consumption situation and the sentinel hash value; query the communication hub corresponding to the main server, parse the hub theme corresponding to the communication hub, and combine the distribution mechanism and the central The pivotal theme is to distribute the data sentinels in the data lake to obtain a data sentinel stream, perform random sampling on the data sentinel stream to obtain a random sampling report, and analyze the stability corresponding to the data sentinel stream based on the random sampling report; perform identification on the data sentinel stream to obtain a sentinel identity, perform sentinel sorting on the data sentinel stream in combination with the stability and the sentinel identity to obtain a sorted sentinel stream, send the sorted sentinel stream to the sentinel receiving center in the main server, and perform early warning processing on the sorted sentinel stream through the sentinel receiving center to obtain an early warning result, and perform entry processing on the sorted sentinel stream according to the early warning result to obtain an entry result.

[0029] Reference Figure 2 As shown, it is a flow chart of a method for detecting the quality of data entering the lake based on sentinel provided in one embodiment of the present invention. In this embodiment, the method for detecting the quality of data entering the lake based on sentinel includes steps S1-S4:

[0030] S1. Obtain a data lake to be monitored and its corresponding master server, analyze the data business scenario corresponding to the data in the data lake, and create a data sentinel corresponding to the data lake based on the data business scenario.

[0031] By analyzing the data business context corresponding to the data in the data lake, the present invention can understand the business application scenario corresponding to the data in the data lake, and provide a basis for the subsequent creation of data sentinels.

[0032] The data lake is a centralized repository, the master server is a core server responsible for managing and coordinating operations related to the data lake, and the data business scenario is an application scenario corresponding to the data in the data lake.

[0033] As an embodiment of the present invention, the analyzing the data business context corresponding to the data in the data lake includes: performing identity resolution on the data in the data lake to obtain data identifiers, mining identification information corresponding to the data identifiers, calculating identification information entropy corresponding to the identification information, extracting key data identifiers from the data identifiers based on the identification information entropy, performing contextual analysis on the key data identifiers, and obtaining the data business context corresponding to the data in the data lake.

[0034] Among them, the data identifier is the feature label corresponding to the data in the data lake, the identification information is the label description content corresponding to the data identifier, and the identification information entropy represents the degree of chaos corresponding to the identification information. The higher the degree of chaos, the lower the regularity of the information and the lower the importance of the corresponding identification information. The key data identifier is a representative identifier in the identification information.

[0035] Furthermore, the identification resolution of the data in the data lake can be achieved through natural language processing technology; the mining of identification information corresponding to the data identification can be achieved through the principal component analysis method; the identification information entropy corresponding to the identification information can be realized through the Shannon entropy algorithm; the data identification corresponding to the maximum value in the identification information entropy is used as the key data identification; and the contextual semantics can be extracted from the identification semantics by performing semantic analysis on the key data identification, so as to obtain the data business context corresponding to the data in the data lake.

[0036] The present invention creates a data sentinel corresponding to the data lake based on the data business scenario, which can facilitate subsequent monitoring and processing of the data in the data lake and provide a basis for ensuring the data quality of the data lake, wherein the data sentinel is a tool for monitoring the data in the data lake.

[0037] As an embodiment of the present invention, creating a data sentinel corresponding to the data lake based on the data business scenario includes: performing field extraction on the data business scenario to obtain scenario fields, determining the monitoring domain corresponding to the data lake based on the scenario fields, querying the data source corresponding to the data in the data lake, drawing a data flow diagram corresponding to the data lake based on the data source, setting data indicators corresponding to the data lake based on the data flow diagram, generating a monitoring strategy corresponding to the data lake based on the data indicators and the monitoring domain, performing data preprocessing on the data in the data lake to obtain target data, performing feature extraction on the target data to obtain data features, and creating a data sentinel corresponding to the data lake in combination with the data features and the monitoring strategy.

[0038] Among them, the context field is a descriptive word in the data business context, the monitoring domain is the monitoring scope corresponding to the data lake, the data traceability is a record of the origin and development of the data in the data lake throughout its life cycle, the data flow diagram is a visual representation corresponding to the data lake, the data indicator is the evaluation standard corresponding to the data lake, the monitoring strategy is the monitoring plan corresponding to the data lake, the target data is the data in the data lake obtained after operations such as cleaning, format conversion, and data aggregation, and the data feature is the data feature corresponding to the target data.

[0039] Furthermore, the field extraction of the data business context can be achieved through text segmentation technology; based on the context field, the monitoring domain corresponding to the data lake is determined, and the key data sets most relevant to the context field are selected from the found data tables and fields. These data sets will constitute the core part of the monitoring domain. For example, if the context field is "high-value customer orders", then a table containing high-value customer information and related order data can be selected as part of the monitoring domain; the data source corresponding to the data in the data lake can be obtained through system log query; the drawing of the data flow diagram corresponding to the data lake can be obtained through Microsoft Visio drawing tool is used to implement the data lake. Based on the data flow diagram, the data indicators corresponding to the data lake are set. The business requirements and goals of each key node and process in the flow diagram can be analyzed first to determine the indicators that can reflect the data quality, performance and business effects. For example, data accuracy can be set by comparing the difference rate between the data source and the corresponding data in the data lake. Indicators such as data processing time can also be set according to the timeliness requirements of data processing. Based on the data indicators and the monitoring domain, a monitoring strategy corresponding to the data lake is generated. For example, if the data indicators include data accuracy and timeliness, and the monitoring domain is a specific business data set, the monitoring strategy can be to regularly check the data accuracy of the data set, and at the same time monitor in real time whether the update time of the data meets the timeliness requirements. The feature extraction of the target data can be achieved through a convolutional neural network. In combination with the data features and the monitoring strategy, a data sentinel corresponding to the data lake can be created through ETL tools, such as Extract, Transform, Load and other tools.

[0040] S2. Schedule the cluster resources corresponding to the main server, analyze the resource consumption situation corresponding to the cluster resources, extract the sentinel attributes corresponding to the data sentinel, calculate the sentinel hash value corresponding to the data sentinel based on the sentinel attributes, and set the distribution mechanism corresponding to the data sentinel according to the resource consumption situation and the sentinel hash value.

[0041] By analyzing the resource consumption situation corresponding to the cluster resources, the present invention can understand the consumption situation corresponding to the cluster resources, and extract the sentinel attributes corresponding to the data sentinel to obtain the sentinel features corresponding to the data sentinel, thereby providing a basis for the subsequent calculation of the sentinel hash value.

[0042] Among them, the cluster resources are composed of multiple independent computing nodes in the main server, such as high-performance server nodes, storage nodes with strong storage capabilities, etc. The resource consumption situation is a description of the consumption situation corresponding to the cluster resources, and the sentinel attributes are the sentinel-specific properties corresponding to the data sentinel. Furthermore, the scheduling of the cluster resources corresponding to the main server can be achieved through a polling scheduling algorithm; the resource usage corresponding to the cluster resources can be monitored through monitoring tools, including CPU usage, memory occupancy, disk I / O, network traffic, etc., and the resource usage can be visualized and analyzed to obtain the resource consumption situation corresponding to the cluster resources; the extraction of the sentinel attributes corresponding to the data sentinel can be achieved through an independent component analysis method.

[0043] The present invention calculates the sentinel hash value corresponding to the data sentinel based on the sentinel attributes, so as to obtain a unique identifier corresponding to the data sentinel, which is convenient for rapid identification and management and provides a basis for the subsequent distribution mechanism setting, wherein the sentinel hash value is a fixed-length value obtained by calculating the attributes of the data sentinel through a specific hash algorithm.

[0044] As an embodiment of the present invention, the sentinel hash value corresponding to the data sentinel is calculated based on the sentinel attribute, including: vectorizing the sentinel attribute to obtain a sentinel attribute vector, calculating the sentinel attribute mean corresponding to the sentinel attribute based on the sentinel attribute vector, and combining the sentinel attribute vector and the sentinel attribute mean to calculate the attribute standard value corresponding to the sentinel attribute by the following formula:

[0045]

[0046] Among them, A represents the attribute standard value corresponding to the sentinel attribute, B a represents the ath vector in the sentinel attribute vector, represents the mean value of sentinel attributes, B max represents the maximum value of the vector in the sentinel attribute vector, B min represents the minimum value of the sentinel attribute vector, a represents the sequence number corresponding to the sentinel attribute vector,

[0047] The sentinel hash value corresponding to the data sentinel is calculated by combining the attribute standard value and the sentinel attribute vector.

[0048] The sentinel attribute vector is an expression vector corresponding to the sentinel attribute, the sentinel attribute mean value represents an average value corresponding to the sentinel attribute, and the attribute standard value represents a reference value corresponding to the sentinel attribute.

[0049] Furthermore, the vectorization processing of the sentinel attribute can be implemented through the word2vec algorithm; the calculation of the sentinel attribute mean corresponding to the sentinel attribute can be implemented through an average function; based on the attribute standard value, the sentinel hash value corresponding to the data sentinel can be calculated through a hash function.

[0050] Further, as an optional embodiment of the present invention, the attribute standard value and the sentinel attribute vector are combined to calculate the sentinel hash value corresponding to the data sentinel:

[0051] The sentinel hash value corresponding to the data sentinel can be calculated by the following formula:

[0052]

[0053] Among them, E represents the sentinel hash value corresponding to the data sentinel, hash() represents the hash function, B1 represents the starting vector in the sentinel attribute vector, A1 represents the starting standard value in the attribute standard value, n1 represents the number corresponding to the sentinel attribute vector, n2 represents the number corresponding to the attribute standard value, mod P represents the remainder operation, and P represents a prime number.

[0054] The present invention sets a distribution mechanism corresponding to the data sentinel according to the resource consumption situation and the sentinel hash value, which can realize more accurate and efficient data monitoring and allocation of the data sentinel, and lays a foundation for the subsequent distribution and processing of the data sentinel in the main server, wherein the distribution mechanism is the allocation strategy corresponding to the data sentinel. Furthermore, according to the resource consumption situation and the sentinel hash value, the distribution mechanism corresponding to the data sentinel is set, such as: when the resource consumption situation is in a low load, the data sentinel can be evenly distributed to each node to achieve comprehensive monitoring coverage, according to the sentinel hash value, the data sentinel can be allocated in a polling or random manner to ensure that each node has the opportunity to undertake the monitoring task, medium load state: under medium load conditions, the data sentinel is preferentially allocated to the resource utilization node. On nodes with lower utilization rates to balance the system load, weighted allocation can be performed based on the resource consumption situation and the sentinel hash value. For example, some combination of hash value and node resource utilization is calculated to select the most appropriate node to allocate the data sentinel. High load state: When the resource consumption situation is at a high load, reduce the number of data sentinels and only allocate them to key nodes or important data. According to the sentinel hash value and the urgency of resource consumption, determine which data sentinels need to be retained and allocated to the most important positions to ensure the stability of the system and the normal operation of key businesses. Dynamic adjustment: Continuously monitor changes in resource consumption trends and sentinel hash values, and dynamically adjust the data sentinel distribution mechanism according to the changes. For example, if the resource consumption situation changes from low load to high load, adjust the data sentinel allocation strategy in time to adapt to the new situation.

[0055] S3. Query the communication hub corresponding to the main server, parse the central theme corresponding to the communication hub, combine the distribution mechanism and the central theme, distribute the data sentinel in the data lake, obtain the data sentinel flow, perform random sampling on the data sentinel flow, obtain the random sampling report, and analyze the stability corresponding to the data sentinel flow based on the random sampling report.

[0056] The present invention distributes and processes the data sentinels in the data lake by combining the distribution mechanism and the central theme, thereby facilitating efficient data monitoring and management of the data lake and ensuring the accuracy, completeness and timeliness of the data. At the same time, it can better adapt to different data characteristics and business needs, and improve the overall performance and value output of the data lake. The communication hub is the message bus center of the main server, the central theme is the core theme corresponding to the communication hub, and the data sentinel stream is the corresponding dynamic monitoring system set after the data sentinels are distributed in the data lake.

[0057] As an embodiment of the present invention, the distribution mechanism and the central theme are combined to distribute the data sentinels in the data lake to obtain a data sentinel flow, including: identifying the data architecture and data elements in the data lake, combining the data architecture and the data elements, performing data labeling on the data in the data lake to obtain data categories, calculating the strength of association between the data sentinels and the data categories, matching the data sentinels with the data in the data lake based on the association strength to obtain a matching set, combining the matching set and the distribution mechanism, distributing the data sentinels in the data lake to obtain a data sentinel flow.

[0058] Among them, the data architecture is the data framework in the data lake, the data elements are the components corresponding to the data in the data lake, the data categories are the data types corresponding to the data in the data lake, the association strength represents the degree of association between the data sentinel and the data category, and the matching set is the result obtained after the data sentinel and the data in the data lake are matched with each other.

[0059] Furthermore, the identification of the data architecture and data elements in the data lake can be achieved through tools such as ERwin and PowerDesigner; in combination with the data architecture and the data elements, the data labeling processing of the data in the data lake can be achieved through a support vector machine, for example, in combination with the data architecture and the data elements, a support vector machine (SVM) algorithm is used to classify the data, the relevant attributes of the data are determined in combination with the data architecture, the data elements are extracted according to the data architecture, and the support vector machine algorithm is used to accurately classify and label the data according to the relevant attributes and data elements, and a corresponding label is assigned to each category; when the correlation strength is greater than a preset value, the data elements are classified into the corresponding categories. When the threshold is reached, the data sentinel is matched with the data in the data lake to obtain a matching set. The preset threshold can be set to 0.8, or it can be set according to the actual business scenario. In combination with the matching set and the distribution mechanism, the data sentinel is distributed in the data lake to obtain a data sentinel stream. For example, the key data areas in the data lake are determined according to the matching set, and the data sentinels are distributed to these areas in a targeted manner in combination with the distribution mechanism. For example, in an e-commerce data lake, for high-traffic commodity transaction data, sentinels that are good at processing such data are distributed according to the distribution mechanism to form an efficient data sentinel stream to monitor the accuracy and security of transactions in real time.

[0060] Further, as an optional embodiment of the present invention, the calculation of the association strength between the data sentinel and the data category includes: respectively calculating the probabilities corresponding to the data sentinel and the data category to obtain the sentinel probability and the category probability, calculating the similarity index between the data sentinel and the data category, and combining the sentinel probability, the category probability and the similarity index to calculate the association strength between the data sentinel and the data category by the following formula:

[0061]

[0062] Among them, F represents the strength of association between data sentinels and data categories, and H b represents the sentinel probability corresponding to the bth sentinel in the data sentinel, G d Represents the category probability corresponding to the dth category in the data category, S b，d represents the similarity index between the bth sentinel in the data sentinel and the dth category in the data category, where b and d represent the serial numbers corresponding to the data sentinel and the data category, respectively.

[0063] Among them, the sentinel probability and the category probability respectively represent the probability values ​​of the occurrence of the data sentinel and the data category, and the similarity index represents the degree of similarity between the data sentinel and the data category. Furthermore, the probability calculation corresponding to the data sentinel and the data category can be achieved by statistical methods; the calculation of the similarity index between the data sentinel and the data category can be achieved by a cosine similarity algorithm.

[0064] The present invention analyzes the stability corresponding to the data sentinel stream based on the sampling report, thereby understanding the stability of the data sentinel stream and then timely discovering potential unstable factors of the data sentinel stream to facilitate subsequent sentinel sorting processing, wherein the sampling report is the operating status feedback corresponding to the data sentinel stream, and the stability represents the stability corresponding to the data sentinel stream. Furthermore, the sampling processing of the data sentinel stream can be achieved by a simple random sampling method.

[0065] As an embodiment of the present invention, the analysis of the stability corresponding to the data sentinel stream based on the sampling report includes: querying the stability index corresponding to the data sentinel stream, extracting the indicator report corresponding to the stability index from the sampling report, calculating the indicator value corresponding to the stability index based on the indicator report, and calculating the indicator weight corresponding to the stability index, calculating the stability coefficient corresponding to the data sentinel stream based on the indicator value and the indicator weight, and analyzing the stability corresponding to the data sentinel stream based on the stability coefficient.

[0066] Among them, the stability index is the stability evaluation standard corresponding to the data sentinel stream, such as error rate, abnormality rate, etc., the index report is the report corresponding to the stability index in the sampling report, the index value is the specific expression value corresponding to the stability index, the index weight indicates the importance of the stability index, and the stability coefficient indicates the stability quantitative value corresponding to the data sentinel stream.

[0067] Furthermore, the query of the stability index corresponding to the data sentinel stream can be obtained from the Internet through human-computer interaction; the index report corresponding to the stability index can be extracted from the sampling report through an extraction function, and the extraction function is compiled by a script language; based on the index report, the index value corresponding to the stability index is calculated, for example, the error rate of the data sentinel stream within a certain period of time can be calculated, which can be obtained by counting the ratio of the number of monitoring errors to the total number of monitoring errors; the calculation of the index weight corresponding to the stability index can be achieved through the hierarchical analysis method; the stability coefficient corresponding to the data sentinel stream can be achieved through the weighted synthesis method; based on the stability coefficient, the stability corresponding to the data sentinel stream is analyzed, for example, if the stability coefficient is high, it means that the data sentinel stream shows high stability in the process of monitoring data, such as being able to continuously and accurately feedback data anomalies to ensure the reliability of data processing. For example, when the stability coefficient is low, it may mean that the data sentinel stream is insufficient in the timeliness and accuracy of data monitoring, and needs to be further optimized to improve its stability.

[0068] S4. Identify and process the data sentinel stream to obtain a sentinel identity; perform sentinel sorting on the data sentinel stream in combination with the stability and the sentinel identity to obtain a sorted sentinel stream; send the sorted sentinel stream to a sentinel receiving center in the main server; and perform early warning processing on the sorted sentinel stream through the sentinel receiving center to obtain an early warning result; and perform lake entry processing on the sorted sentinel stream according to the early warning result to obtain a lake entry result.

[0069] The present invention can obtain the sentinel identity information corresponding to the data sentinel stream by identifying and processing the data sentinel stream, and can perform sentinel sorting processing on the data sentinel stream in combination with the stability and the sentinel identity, so that data sentinels with higher stability can be sorted out and sentinels with low stability in the data sentinels can be removed, wherein the sentinel identity is the identity information corresponding to the data sentinel stream, and the sorted sentinel stream is the sentinel stream obtained after the data sentinel stream is sorted in combination with the stability and the sentinel identity. Furthermore, the identification processing of the data sentinel stream can be implemented by a deep learning model, such as a recurrent neural network; a decision tree model can be constructed by combining the stability and the sentinel identity, and the data sentinel stream can be sentinel sorted by the decision tree model.

[0070] The present invention sends the sorting sentinel stream to the sentinel receiving center in the main server, and performs early warning processing on the sorting sentinel stream through the sentinel receiving center, so as to timely discover potential problems or abnormal situations, such as a sudden increase in data traffic, frequent abnormal signals from a specific type of sentinel, etc., so that relevant personnel can quickly take measures to avoid the problem from expanding, ensure the stable operation of the system and the security of data, and according to the early warning results, perform lake entry processing on the sorting sentinel stream, thereby identifying the data quality of the data entering the lake in real time, and reducing the problem of large-scale pollution caused by erroneous data.

[0071] By analyzing the data business context corresponding to the data in the data lake, the present invention can understand the business application scenario corresponding to the data in the data lake, and provide a basis for the subsequent creation of data sentinels. By analyzing the resource consumption situation corresponding to the cluster resources, the present invention can understand the consumption of the cluster resources, and extract the sentinel attributes corresponding to the data sentinels to obtain the sentinel features corresponding to the data sentinels, thereby providing a basis for the subsequent calculation of sentinel hash values. By combining the distribution mechanism and the central theme, the present invention distributes and processes the data sentinels in the data lake, which is convenient for efficient data monitoring and management of the data lake, ensuring the accuracy, completeness and timeliness of the data, and at the same time can better adapt to different data characteristics and business needs, and improve the overall performance and value output of the data lake; by identifying and processing the data sentinel stream, the present invention can obtain the sentinel identity information corresponding to the data sentinel stream, and by combining the stability and the sentinel identity, the data sentinel stream is sentinel sorted, so that data sentinels with higher stability can be sorted out, and sentinels with low stability in the data sentinels can be removed. Therefore, the method for detecting the quality of data entering the lake based on Sentinel provided in the embodiment of the present invention can improve the accuracy of detecting the quality of data entering the lake based on Sentinel.

[0072] like Figure 3 , which is a functional module diagram of a device for detecting the quality of lake entry data based on Sentinel provided in one embodiment of the present invention.

[0073] The device 100 for detecting the quality of lake entry data based on sentinels of the present invention can be installed in a computer device. According to the functions implemented, the device 100 for detecting the quality of lake entry data based on sentinels can include a sentinel creation module 101, a distribution mechanism setting module 102, a stability analysis module 103 and a lake entry processing module 104. The module of the present invention can also be referred to as a unit, which refers to a series of computer program segments that can be executed by a processor of a computer device and can complete fixed functions, which are stored in the memory of the computer device.

[0074] In this embodiment, the functions of each module / unit are as follows:

[0075] The sentinel creation module 101 is used to obtain a data lake to be monitored and its corresponding master server, analyze the data business scenario corresponding to the data in the data lake, and create a data sentinel corresponding to the data lake based on the data business scenario;

[0076] The distribution mechanism setting module 102 is used to schedule the cluster resources corresponding to the master server, analyze the resource consumption situation corresponding to the cluster resources, extract the sentinel attributes corresponding to the data sentinel, calculate the sentinel hash value corresponding to the data sentinel based on the sentinel attributes, and set the distribution mechanism corresponding to the data sentinel according to the resource consumption situation and the sentinel hash value;

[0077] The stability analysis module 103 is used to query the communication hub corresponding to the main server, parse the hub theme corresponding to the communication hub, distribute the data sentinel in the data lake in combination with the distribution mechanism and the hub theme to obtain a data sentinel flow, perform random inspection on the data sentinel flow to obtain a random inspection report, and analyze the stability corresponding to the data sentinel flow based on the random inspection report;

[0078] The lake entry processing module 104 is used to identify and process the data sentinel stream to obtain the sentinel identity, and perform sentinel sorting processing on the data sentinel stream in combination with the stability and the sentinel identity. When the sorted sentinel stream is obtained, the sorted sentinel stream is sent to the sentinel receiving center in the main server, and the sorted sentinel stream is early-warning processed by the sentinel receiving center to obtain an early-warning result. According to the early-warning result, the sorted sentinel stream is entered into the lake to obtain an entry into the lake result.

[0079] In one embodiment, when the sentinel creation module 101 performs analysis to obtain the data business context corresponding to the data in the data lake, it is used to: perform identification analysis on the data in the data lake to obtain the data identification; mine the identification information corresponding to the data identification, and calculate the identification information entropy corresponding to the identification information; extract the key data identification from the data identification based on the identification information entropy; perform context analysis on the key data identification to obtain the data business context corresponding to the data in the data lake

[0080] In one embodiment, when the sentinel creation module 101 is executed to create a data sentinel corresponding to the data lake based on the data business scenario, it is used to: perform field extraction on the data business scenario to obtain scenario fields; based on the scenario fields, determine the monitoring domain corresponding to the data lake, and query the data traceability corresponding to the data in the data lake; based on the data traceability, draw a data flow diagram corresponding to the data lake; based on the data flow diagram, set data indicators corresponding to the data lake; based on the data indicators and the monitoring domain, generate a monitoring strategy corresponding to the data lake; perform data preprocessing on the data in the data lake to obtain target data; perform feature extraction on the target data to obtain data features; and create a data sentinel corresponding to the data lake in combination with the data features and the monitoring strategy.

[0081] In one embodiment, when the distribution mechanism setting module 102 calculates the sentinel hash value corresponding to the data sentinel based on the sentinel attribute, it is used to: vectorize the sentinel attribute to obtain a sentinel attribute vector; calculate the sentinel attribute mean corresponding to the sentinel attribute based on the sentinel attribute vector; calculate the attribute standard value corresponding to the sentinel attribute by combining the sentinel attribute vector and the sentinel attribute mean; calculate the sentinel hash value corresponding to the data sentinel by combining the attribute standard value and the sentinel attribute vector.

[0082] In one embodiment, when the lake entry processing module 104 executes the distribution process of the data sentinel in the data lake in combination with the distribution mechanism and the central theme to obtain a data sentinel stream, it is used to: identify the data architecture and data elements in the data lake; perform data tagging process on the data in the data lake in combination with the data architecture and the data elements to obtain data categories; calculate the strength of association between the data sentinel and the data category; based on the strength of association, match the data sentinel with the data in the data lake to obtain a matching set; distribute the data sentinel in the data lake in combination with the matching set and the distribution mechanism to obtain a data sentinel stream.

[0083] In one embodiment, when executing the calculation of the association strength between the data sentinel and the data category, the lake entry processing module 104 is used to: respectively calculate the probabilities corresponding to the data sentinel and the data category to obtain the sentinel probability and the category probability; calculate the similarity index between the data sentinel and the data category; and calculate the association strength between the data sentinel and the data category by combining the sentinel probability, the category probability and the similarity index.

[0084] In one embodiment, when analyzing the stability corresponding to the data sentinel stream based on the sampling report, the lake entry processing module 104 is used to: query the stability index corresponding to the data sentinel stream, and extract the index report corresponding to the stability index from the sampling report; based on the index report, calculate the index value corresponding to the stability index, and calculate the index weight corresponding to the stability index; calculate the stability coefficient corresponding to the data sentinel stream by combining the index value and the index weight; and analyze the stability corresponding to the data sentinel stream based on the stability coefficient.

[0085] In detail, each module described in the device 100 for detecting lake entry data quality based on sentinel in the embodiment of the present application is used in the same manner as described above. Figure 2 The same technical means as described in the method for detecting the quality of lake data based on sentinels are used and can produce the same technical effects, so they will not be repeated here.

[0086] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 4 As shown. The computer device includes a processor, a memory, a network interface and a database connected via a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes non-volatile and / or volatile storage media and internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external client through a network connection. When the computer program is executed by the processor, it implements a function or step on the server side of a method for detecting the quality of incoming lake data based on Sentinel.

[0087] In one embodiment, a computer device is provided. The computer device may be a client, and its internal structure diagram may be as follows: Figure 5 As shown. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected via a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external server via a network connection. When the computer program is executed by the processor, it implements a function or step on the client side of a method for detecting the quality of incoming lake data based on Sentinel.

[0088] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the following steps when executing the computer program:

[0089] Obtain a data lake to be monitored and its corresponding master server, analyze the data business scenario corresponding to the data in the data lake, and create a data sentinel corresponding to the data lake based on the data business scenario;

[0090] Scheduling cluster resources corresponding to the master server, analyzing the resource consumption situation corresponding to the cluster resources, extracting sentinel attributes corresponding to the data sentinel, calculating the sentinel hash value corresponding to the data sentinel based on the sentinel attributes, and setting the distribution mechanism corresponding to the data sentinel according to the resource consumption situation and the sentinel hash value;

[0091] Query the communication hub corresponding to the master server, parse the hub topic corresponding to the communication hub, distribute the data sentinel in the data lake in combination with the distribution mechanism and the hub topic, obtain the data sentinel flow, perform random inspection on the data sentinel flow, obtain the random inspection report, and analyze the stability corresponding to the data sentinel flow based on the random inspection report;

[0092] The data sentinel stream is identified and processed to obtain a sentinel identity; the data sentinel stream is sentinel sorted in combination with the stability and the sentinel identity to obtain a sorted sentinel stream; the sorted sentinel stream is sent to a sentinel receiving center in the main server; and the sorted sentinel stream is early-warning processed by the sentinel receiving center to obtain an early-warning result; and according to the early-warning result, the sorted sentinel stream is entered into a lake to obtain an entry into a lake result.

[0093] In one embodiment, a computer readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0094] Obtain a data lake to be monitored and its corresponding master server, analyze the data business scenario corresponding to the data in the data lake, and create a data sentinel corresponding to the data lake based on the data business scenario;

[0095] Scheduling cluster resources corresponding to the master server, analyzing the resource consumption situation corresponding to the cluster resources, extracting sentinel attributes corresponding to the data sentinel, calculating the sentinel hash value corresponding to the data sentinel based on the sentinel attributes, and setting the distribution mechanism corresponding to the data sentinel according to the resource consumption situation and the sentinel hash value;

[0096] Query the communication hub corresponding to the master server, parse the hub topic corresponding to the communication hub, distribute the data sentinel in the data lake in combination with the distribution mechanism and the hub topic, obtain the data sentinel flow, perform random inspection on the data sentinel flow, obtain the random inspection report, and analyze the stability corresponding to the data sentinel flow based on the random inspection report;

[0097] The data sentinel stream is identified and processed to obtain a sentinel identity; the data sentinel stream is sentinel sorted in combination with the stability and the sentinel identity to obtain a sorted sentinel stream; the sorted sentinel stream is sent to a sentinel receiving center in the main server; and the sorted sentinel stream is early-warning processed by the sentinel receiving center to obtain an early-warning result; and according to the early-warning result, the sorted sentinel stream is entered into a lake to obtain an entry into a lake result.

[0098] It should be noted that the above functions or steps that can be implemented by the computer-readable storage medium or computer device can refer to the relevant descriptions on the server side and the client side in the aforementioned method embodiment. To avoid repetition, they will not be described one by one here.

[0099] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0100] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0101] The above-described embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit them; although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some of the technical features therein by equivalents; and these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention, and should be included in the protection scope of the present invention. It should be noted that if software tools or components other than those of the company appear in the embodiments of this application, they are only used for illustration and do not represent actual use.

Claims

1. A method for detecting the quality of lake data based on Sentinel, characterized in that: The method comprises: Obtain a data lake to be monitored and its corresponding master server, analyze the data business scenario corresponding to the data in the data lake, and create a data sentinel corresponding to the data lake based on the data business scenario; Scheduling cluster resources corresponding to the master server, analyzing the resource consumption situation corresponding to the cluster resources, extracting sentinel attributes corresponding to the data sentinel, calculating the sentinel hash value corresponding to the data sentinel based on the sentinel attributes, and setting the distribution mechanism corresponding to the data sentinel according to the resource consumption situation and the sentinel hash value; Query the communication hub corresponding to the master server, parse the hub topic corresponding to the communication hub, distribute the data sentinel in the data lake in combination with the distribution mechanism and the hub topic, obtain the data sentinel flow, perform random inspection on the data sentinel flow, obtain the random inspection report, and analyze the stability corresponding to the data sentinel flow based on the random inspection report; The data sentinel stream is identified and processed to obtain a sentinel identity; the data sentinel stream is sentinel sorted in combination with the stability and the sentinel identity to obtain a sorted sentinel stream; the sorted sentinel stream is sent to a sentinel receiving center in the main server; and the sorted sentinel stream is early-warning processed by the sentinel receiving center to obtain an early-warning result; and according to the early-warning result, the sorted sentinel stream is entered into a lake to obtain an entry into a lake result.

2. The method for detecting the quality of lake entry data based on sentinel according to claim 1, characterized in that: The analyzing the data business scenarios corresponding to the data in the data lake includes: Performing identity resolution on the data in the data lake to obtain a data identity; Mining identification information corresponding to the data identification, and calculating identification information entropy corresponding to the identification information; Extracting key data identifiers from the data identifiers based on the identifier information entropy; A context analysis is performed on the key data identifier to obtain a data business context corresponding to the data in the data lake.

3. The method for detecting the quality of lake entry data based on sentinels as claimed in claim 1, characterized in that: The creating a data sentinel corresponding to the data lake based on the data business scenario includes: Extracting fields from the data business context to obtain context fields; Based on the context field, determine the monitoring domain corresponding to the data lake, and query the data source corresponding to the data in the data lake; Based on the data tracing, draw a data flow diagram corresponding to the data lake; Based on the data flow diagram, set data indicators corresponding to the data lake; Based on the data indicators and the monitoring domain, generating a monitoring strategy corresponding to the data lake; Preprocess the data in the data lake to obtain target data; Extracting features from the target data to obtain data features; In combination with the data features and the monitoring strategy, a data sentinel corresponding to the data lake is created.

4. The method for detecting the quality of lake entry data based on sentinels as claimed in claim 1, characterized in that: The calculating the sentinel hash value corresponding to the data sentinel based on the sentinel attribute includes: Vectorizing the sentinel attribute to obtain a sentinel attribute vector; Based on the sentinel attribute vector, calculating the sentinel attribute mean corresponding to the sentinel attribute; Calculate the attribute standard value corresponding to the sentinel attribute by combining the sentinel attribute vector and the sentinel attribute mean; The sentinel hash value corresponding to the data sentinel is calculated by combining the attribute standard value and the sentinel attribute vector.

5. The method for detecting the quality of lake entry data based on sentinel according to claim 1, characterized in that: The combining of the distribution mechanism and the central theme to distribute the data sentinel in the data lake to obtain a data sentinel stream includes: Identify data architecture and data elements in the data lake; In combination with the data architecture and the data elements, data labeling processing is performed on the data in the data lake to obtain data categories; calculating the strength of association between the data sentinel and the data category; Based on the association strength, matching the data sentinel with the data in the data lake to obtain a matching set; In combination with the matching set and the distribution mechanism, the data sentinel is distributed in the data lake to obtain a data sentinel stream.

6. The method for detecting the quality of lake entry data based on sentinels as claimed in claim 5, characterized in that: The calculating the association strength between the data sentinel and the data category comprises: Calculating the probabilities corresponding to the data sentinels and the data categories respectively to obtain sentinel probabilities and category probabilities; calculating a similarity index between the data sentinel and the data category; The strength of association between the data sentinel and the data category is calculated by combining the sentinel probability, the category probability and the similarity index.

7. The method for detecting the quality of lake entry data based on sentinel according to claim 1, characterized in that: The analyzing the stability corresponding to the data sentinel flow based on the sampling report includes: Querying the stability index corresponding to the data sentinel stream, and extracting the index report corresponding to the stability index from the sampling report; Based on the indicator report, calculating the indicator value corresponding to the stability indicator, and calculating the indicator weight corresponding to the stability indicator; Calculate the stability coefficient corresponding to the data sentinel flow by combining the indicator value and the indicator weight; Based on the stability coefficient, the stability corresponding to the data sentinel stream is analyzed.

8. A device for detecting the quality of lake entry data based on Sentinel, characterized in that: The device comprises: A sentinel creation module is used to obtain a data lake to be monitored and its corresponding master server, analyze the data business scenario corresponding to the data in the data lake, and create a data sentinel corresponding to the data lake based on the data business scenario; A distribution mechanism setting module, used for scheduling cluster resources corresponding to the master server, analyzing the resource consumption situation corresponding to the cluster resources, extracting sentinel attributes corresponding to the data sentinel, calculating the sentinel hash value corresponding to the data sentinel based on the sentinel attributes, and setting the distribution mechanism corresponding to the data sentinel according to the resource consumption situation and the sentinel hash value; A stability analysis module, used to query the communication hub corresponding to the master server, parse the hub theme corresponding to the communication hub, distribute the data sentinel in the data lake in combination with the distribution mechanism and the hub theme, obtain the data sentinel flow, perform random inspection on the data sentinel flow, obtain the random inspection report, and analyze the stability corresponding to the data sentinel flow based on the random inspection report; The lake entry processing module is used to identify and process the data sentinel stream to obtain the sentinel identity, perform sentinel sorting processing on the data sentinel stream in combination with the stability and the sentinel identity to obtain a sorted sentinel stream, send the sorted sentinel stream to the sentinel receiving center in the main server, and perform early warning processing on the sorted sentinel stream through the sentinel receiving center to obtain an early warning result, and according to the early warning result, perform lake entry processing on the sorted sentinel stream to obtain a lake entry result.

9. A computer device, characterized in that: The computer device comprises: at least one processor; and, a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the method for detecting lake entry data quality based on Sentinel as described in any one of claims 1 to 7.

10. A computer readable medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method for detecting the quality of lake entry data based on Sentinel is implemented as described in any one of claims 1 to 7.

Images