Data management method and device and storage medium

Abstract

The invention discloses a data governance method and device and a storage medium, belongs to the technical field of data processing, and is used for solving the technical problems that a data governance tool cannot solve the special data problem in actual business and cannot process unstructured data. The method comprises the following steps: reading original data in a data source; for the structured data, after the verification process based on the general verification rule is passed, secondary verification is carried out by using the service verification rule, and a data quality report is generated according to a verification result; and aiming at the unstructured data, checking by utilizing an image checking rule, and generating a data quality report according to a checking result. According to the technical scheme, the technical effect that the special data problem in the actual service can be solved on the basis that the unstructured data can be processed can be achieved.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a data management method, device and storage medium. Background Art

[0002] With the advent of the big data era, data has become one of the most important assets of enterprises and organizations. Through the effective management and utilization of massive data, enterprises can make more accurate decisions, improve operational efficiency, and gain advantages in market competition.

[0003] However, in the face of the growing amount of data and increasingly complex relationships between data, without effective data governance, data quality, such as data integrity and data consistency, will be difficult to guarantee, which may lead to decision-making errors, reduced operational efficiency and even legal compliance issues. Data governance refers to the use of a series of standards and technical means to delete, modify and annotate data to ensure that data quality meets requirements. Although there are many data governance tools on the market, these tools still have some significant problems in practical applications: First, the current common data governance tools are only applicable to solving basic problems such as missing data and inconsistent data formats, and cannot deeply solve the special data problems encountered by users in actual business scenarios. Second, the current data governance tools are only applicable to processing structured data, but cannot process unstructured data such as images and videos. Summary of the invention

[0004] The present invention provides a data management method, device and storage medium for solving at least one of the above-mentioned technical problems.

[0005] The present invention adopts the following technical solutions:

[0006] In a first aspect, the present invention provides a data governance method, comprising: reading the original data in the data source; for structured data, after the verification process based on the general verification rules is passed, performing a secondary verification using business verification rules, and generating a data quality report based on the verification results; for unstructured data, performing verification using image verification rules, and generating a data quality report based on the verification results.

[0007] In a feasible implementation, secondary verification is performed using business verification rules, including: parsing structured data of the verification process corresponding to the general verification rules to determine the data type; obtaining the business type corresponding to the data type; and performing secondary verification on the structured data according to the business verification rules corresponding to the business type.

[0008] In a feasible implementation, verification is performed using image verification rules, including: parsing unstructured data, obtaining an image type, and determining whether the image type belongs to a specific type, the specific type including at least one of an ID card image, a license plate image, and a face image; if it belongs to a specific type, verification is performed using a first image verification rule; if it does not belong to a specific type, verification is performed using a second image verification rule.

[0009] In a feasible implementation, verification is performed using a first image verification rule, including: extracting image features of unstructured data according to a specific type corresponding to the unstructured data; calculating the clarity of the image features; if the clarity reaches a preset clarity standard, calculating the information completeness of the image features, otherwise, recording the unstructured data as not meeting the clarity standard; if the information completeness reaches a preset information completeness standard, recording the unstructured data as passing verification, otherwise, recording the unstructured data as not meeting the information completeness standard.

[0010] In a feasible implementation, verification is performed using a second image verification rule, including: loading an image verification standard library, the standard library including at least one of an image brightness standard, an image contrast standard, and an image size standard, the image size being used to indicate the size of the storage space occupied by the image; verifying whether the unstructured data meets all the standards in the standard library; if so, recording that the unstructured data has passed the verification, otherwise recording that the unstructured data does not meet the standards.

[0011] In a feasible implementation, after generating a data quality report based on the verification results, a data governance method also includes: if the structured data fails the verification process, normalizing the structured data to obtain standardized data; performing a secondary verification on the standardized data using business rules, and generating a data quality report based on the verification results.

[0012] In a feasible implementation, structured data is normalized, including: extracting and deleting duplicate values ​​in the structured data; searching for missing values ​​in the structured data and processing them using a preset method, which includes at least one of deleting missing values, filling missing values ​​with interpolation, and filling with default values; identifying abnormal values ​​in the structured data and deleting them; searching for data in the structured data that belongs to a preset data type and converting it into a corresponding data format, the preset data type includes at least date and currency; extracting text data in the structured data and performing standardization processing, which includes removing redundant spaces, converting letters to uppercase and lowercase, and replacing special characters.

[0013] In a feasible implementation, a data governance method also includes: obtaining multiple data quality reports and performing data analysis; and generating a data governance effectiveness report based on the results of the data analysis.

[0014] In a second aspect, the present invention also provides a data governance device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions that can be executed by the at least one processor so that the at least one processor can execute a data governance method as described in any of the above embodiments.

[0015] In a third aspect, the present invention further provides a non-volatile computer storage medium storing computer executable instructions, wherein the computer executable instructions are configured to be able to execute a data governance method as described in any of the above-mentioned embodiments.

[0016] The data governance method provided by the present invention has the following beneficial effects:

[0017] 1. The present invention uses business verification rules for secondary verification. The business verification rules here are formulated according to specific business needs and logic, and can more accurately identify and process business-related data problems. Therefore, this method can significantly improve the business relevance of data governance. The image verification rules generated by the AI ​​server are used to verify unstructured data, so that unstructured data can also be processed, which solves the technical problem that data governance tools cannot process unstructured data. Furthermore, AI technology can automatically process large amounts of data and perform complex analysis and verification. At the same time, it can run around the clock without being restricted by time and energy. Compared with manual processing, it can not only improve the effect of data governance, but also increase the speed of data processing.

[0018] 2. The present invention divides unstructured data into specific types and non-specific types, extracts data containing specific and important image features such as identity card images, license plate images and face images from the unstructured data, and verifies the clarity and completeness of the image features of the specific type of data. Through this quality verification method, the unique features of each specific type of data can be evaluated, thereby improving the accuracy of verification of specific types of data. In addition, the present invention also uses a verification standard library to verify non-specific types of data. On the one hand, the use of multiple standards ensures the comprehensiveness of the verification, and multiple standards can cover different verification requirements and data features, ensuring that all aspects of the data governance process are fully checked and verified; on the other hand, the standard library can change dynamically, allowing standards to be added or modified according to different data, so that data governance tools can flexibly respond to various data changes and emerging data types, thereby enhancing the pertinence of data governance.

[0019] 3. In the case that the structured data fails the verification process, the present invention adopts a variety of processing methods to perform comprehensive normalization processing on the structured data. This multi-level, multi-step processing method can identify and correct different types of errors and inconsistencies in each step by verifying the data layer by layer, thereby ensuring that the final data can meet the requirements of the general verification rules.

[0020] 4. The present invention obtains multiple data quality reports, analyzes these data quality reports, identifies common problems and trends in the data, and generates data governance reports accordingly. This approach can provide a comprehensive assessment of the overall data quality and improvement suggestions, making it easier for users to improve the data governance process. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the prior art descriptions. Obviously, the drawings described below are only some embodiments recorded in the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. In the drawings:

[0022] Figure 1 A flow chart of a data governance method provided by the present invention;

[0023] Figure 2 A structural schematic diagram of a data management device provided by the present invention. DETAILED DESCRIPTION

[0024] In order to enable those skilled in the art to better understand the technical solutions in the present invention, the technical solutions in the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of this specification, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0025] The present invention provides a data governance method, which solves the technical problems that data governance tools cannot solve special data problems in actual business and cannot process unstructured data by using different data verification rules for structured data and unstructured data, and performing secondary verification on structured data using business verification rules.

[0026] The method of the present invention is described in detail below with reference to the accompanying drawings.

[0027] Figure 1 A flowchart of a data governance method provided by the present invention, such as Figure 1As shown, the method in the present invention at least includes the following execution steps:

[0028] Step 101: Read the original data in the data source.

[0029] Raw data includes structured data and unstructured data. Structured data refers to data organized and stored according to a predefined format or model, usually in a table form, such as digital data. Unstructured data refers to data that does not conform to a predefined data model or has no fixed structure, such as image data such as images and videos.

[0030] Step 102: After the structured data passes the verification process based on the general verification rules, a secondary verification is performed using the business verification rules, and a data quality report is generated based on the verification results.

[0031] General validation rules are a set of basic rules applicable to a wide range of data sets. These rules do not rely on specific business logic or domain knowledge, but are based on the general characteristics of the data itself for validation, including: data type validation, format validation, non-empty validation, consistency validation and other validation rules. Business validation rules refer to validation rules based on specific business logic and domain knowledge. These rules usually rely on specific business needs and contexts. Unlike general validation rules, business validation rules are more targeted and are designed to ensure that data meets the requirements of business processes and business rules. A data quality report is a document that records and evaluates the quality of a data set in detail, aiming to reveal potential problems in the data, the results of data cleaning and validation, and an assessment of overall data quality.

[0032] Specifically, first, parse the structured data verified by the general verification rules to extract the basic features and attributes of the data, such as field type, format and content. This step can be implemented using regular expressions or predefined pattern matching algorithms. Then, based on the extracted data features, the data type can be determined by querying the preset mapping relationship table. Then, the data type is mapped to the corresponding business type, which can be implemented by database query. Finally, according to the business type, call the corresponding business verification rules to perform secondary verification on the structured data. This step can be achieved by writing a specific verification function or using a rule engine. This method of secondary verification of data can perform more refined and targeted verification on different types of data according to specific business needs, while realizing business verification of structured data, it can also improve the accuracy of data verification.

[0033] In one example, the original data is the relevant data of the user's account registration. The user needs to fill in the name, email, phone number, age and ID number and other related information. For underage users, they also need to fill in the guardian's name and guardian's ID number and other information. First, use the general verification rules to perform preliminary verification on these data. The general verification rules include: the name cannot be empty and the length does not exceed 50 characters, the email must conform to the standard format, the phone number must be in a format that conforms to international standards, the age must be an integer between 0 and 120, and the ID number must conform to the standard format. After the data passes the general verification, it is determined that this group of data types is user registration account related data based on the data content, and then the business type corresponding to this group of data is obtained as user registration business. Finally, the original data is double-checked according to the verification rules corresponding to the user registration business. The business verification rules corresponding to this business are: for adult users, verify whether their ID card information corresponds to their name. For underage users, in addition to verifying whether the ID card information corresponds to their name, it is also necessary to verify whether the data contains guardian information, whether the guardian is an adult, and whether the guardian's name and ID number correspond. Using the above business verification rules, it is possible to verify whether the identity information submitted by the user is authentic, whether the underage user has submitted the guardian information, and whether the guardian information is authentic.

[0034] In a possible implementation of the present invention, after generating a data quality report according to the verification results, the data that has not passed the general rule verification is processed. First, the structured data that has not passed the general rule verification is normalized to obtain standard data. The normalization processing here includes multiple steps, specifically: extracting and deleting duplicate values ​​in the structured data, finding missing values ​​in the structured data, and processing them using a preset method, where the preset method includes deleting missing values, filling missing values ​​with interpolation methods, and filling default values, identifying abnormal values ​​in the structured data and deleting them, finding data belonging to a preset data type in the structured data, and converting them into a corresponding data format, where the preset data types include at least date and currency types, extracting text data in the structured data, and performing standardization processing, and the standardization processing includes removing redundant spaces, converting the uppercase and lowercase forms of letters, and replacing special characters. At least one of the following. Then, the above-mentioned standard data is secondary verified using business rules, and a data quality report is generated according to the verification results. The above-mentioned normalization processing is a staged, multi-step processing method that can identify and correct different types of errors and inconsistencies in each step by verifying the data layer by layer, thereby ensuring that the final data meets the general verification rules.

[0035] Step 103: Use image verification rules to verify the unstructured data, and generate a data quality report based on the verification results.

[0036] The image verification rules here are generated by the AI ​​server and can be used to verify image data as well as images in video data.

[0037] Specifically, first, the unstructured data is parsed to obtain the image type, and it is determined whether the image type belongs to a specific type, where the specific type includes at least one of an ID card image, a license plate image, and a face image. The unstructured data parsed here can be classified by using an image classification algorithm. The image classification algorithm can use a deep learning model such as CNN, or other machine learning models. To determine whether the image type belongs to a specific type, a method of querying a preset mapping relationship table can be used. For example, for an ID card image data, CNN is first used to classify it, and the type is obtained as an ID card image, and then the mapping relationship table is queried to obtain that the ID card image belongs to a specific type. Dividing the unstructured data into specific types and non-specific types facilitates the use of specific verification standards for specific types of images. The specific verification standards can be verified for the properties unique to the specific types of images, which can improve the pertinence of the verification, and thus improve the verification effect for the specific types of images.

[0038] Furthermore, different image verification rules are used to verify the unstructured data according to whether it belongs to a specific type. For a specific type of unstructured data, the clarity and information integrity of its image features are verified. The specific steps are as follows: First, based on the specific type to which the unstructured data belongs, the image features of the data are extracted. For example, for an ID card image, features such as face and ID card number can be extracted. Next, the clarity of the image features is calculated, and algorithms such as Laplace transform are used to evaluate the sharpness of the image. If the clarity meets the standard, the information integrity is calculated. Information integrity can be achieved by detecting the existence and integrity of specific key areas, such as whether the ID card image contains a complete ID card number and face area. If these key areas are complete and clear, the unstructured data is recorded as passing the verification, otherwise it is recorded as not meeting the information integrity standard. If the clarity does not meet the standard, it is directly recorded that the unstructured data does not meet the clarity standard. The main information contained in a specific type of image, which is often the most important information, is usually contained in its image features. Therefore, by verifying the image features, it is possible to detect whether a specific type of image can clearly present the important information it contains and whether the information presented in the image is complete, thereby determining whether the quality of the specific type of image meets the requirements.

[0039] For unstructured data that does not belong to a specific type, use the image verification standard library to verify it. The specific steps are: first load the image verification standard library, which includes at least one of the image brightness standard, image contrast standard and image size standard. For example, the image brightness standard can set the image average brightness value to be within a certain range, the image contrast standard can stipulate that the image brightness contrast should reach a certain threshold, and the image size standard stipulates that the storage space size of the image file should be within a certain range. Then, check whether the unstructured data meets these standards in turn. For image brightness, the average brightness of the image pixels can be calculated and checked whether the average brightness of the image is within the standard range; for contrast, the grayscale histogram of the image can be calculated to evaluate whether its contrast meets the standard. If the image meets all the standards in the standard library, the unstructured data is recorded as having passed the verification; if any one item does not meet the standard, for example, the image size exceeds the specified size in the standard library, the image size standard that the unstructured data does not meet is recorded. Using a verification standard library to verify unstructured data that does not belong to a specific type can, on the one hand, ensure the use of multiple standards to fully cover the characteristics of the data that need to be verified. On the other hand, there are many types of unstructured data that do not belong to a specific type. Using a dynamically changing standard library for verification can adopt different verification standards for different types of data, thereby improving the effectiveness of data governance.

[0040] In one possible implementation of the present invention, a single data report is difficult to reflect the overall quality of the data in the data source, and it is difficult to reflect the pros and cons of the data governance method itself. Therefore, the present invention further obtains multiple data quality reports generated previously, and aggregates these reports for data analysis. Data analysis includes trends, fluctuations, and identification of potential problems for various data quality indicators. By analyzing multiple reports, the effectiveness of the data governance method can be more accurately reflected. For example, assuming that it is found in multiple reports that the data accuracy indicator is gradually improving, this indicates that the data governance measures have been effective in improving data accuracy. Finally, based on these analysis results, a data governance effectiveness report is generated, which records in detail the improvement of data quality and the effectiveness of the data governance method, thereby providing a scientific basis for further optimizing the data governance strategy.

[0041] Based on the same inventive concept, the present invention also provides a data management device, the structure of which is as follows: Figure 2 shown.

[0042] Figure 2 This is a schematic diagram of the structure of a data management device provided by the present invention. Figure 2As shown, a data governance device 200 in the present invention specifically includes: at least one processor 201; and a memory 203 that is communicatively connected to the at least one processor (connected via a bus 202); wherein the memory 203 stores instructions that can be executed by the at least one processor 201, so that the at least one processor 201 can execute a data governance method as described in the above embodiment.

[0043] In one or more possible implementations of the present invention, the aforementioned processor is used to execute, reading the original data in the data source; for structured data, after the verification process based on the general verification rules is passed, a secondary verification is performed using the business verification rules, and a data quality report is generated based on the verification results; for unstructured data, verification is performed using image verification rules, and a data quality report is generated based on the verification results.

[0044] In addition, the present invention also provides a non-volatile computer storage medium storing computer executable instructions, and the computer executable instructions are configured to execute a data governance method as described in any of the above embodiments.

[0045] In one or more possible implementations of the present invention, the aforementioned computer executable instructions are configured to be executed to read the original data in the data source; for structured data, after the verification process based on the general verification rules is passed, a secondary verification is performed using the business verification rules, and a data quality report is generated based on the verification results; for unstructured data, verification is performed using image verification rules, and a data quality report is generated based on the verification results.

[0046] Each embodiment of the present invention is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the IoT device and medium embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiments.

[0047] The device and medium provided by the present invention correspond one-to-one to the method, so the device and medium also have similar beneficial technical effects to the corresponding method. Since the beneficial technical effects of the method have been described in detail above, the beneficial technical effects of the device and medium will not be repeated here.

[0048] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0049] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0050] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0051] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0052] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0053] The above are only embodiments of the present invention and are not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A data governance method, characterized in that: The method comprises: Reading raw data from a data source, where the raw data includes at least structured data and unstructured data; For the structured data, after the verification process based on the general verification rules is passed, a secondary verification is performed using the business verification rules, and a data quality report is generated according to the verification results; The unstructured data is verified using image verification rules, and a data quality report is generated based on the verification results, wherein the image verification rules are generated by the AI ​​server.

2. A data management method according to claim 1, characterized in that: Use business verification rules to perform secondary verification, including: Parsing structured data of the verification process corresponding to the general verification rule to determine the data type; Obtaining the business type corresponding to the data type; The structured data is secondarily verified according to the business verification rules corresponding to the business type.

3. A data management method according to claim 1, characterized in that: Use image verification rules to verify, including: Parsing the unstructured data to obtain an image type, and determining whether the image type belongs to a specific type, wherein the specific type includes at least one of an ID card image, a license plate image, and a face image; If it belongs to the specific type, verify it using the first image verification rule; If it does not belong to the specific type, the second image verification rule is used for verification.

4. A data governance method according to claim 3, characterized in that: Verification is performed using the first image verification rule, including: Extracting image features of the unstructured data according to the specific type corresponding to the unstructured data; Calculating the clarity of the image feature; If the clarity meets the preset clarity standard, the information completeness of the image feature is calculated, otherwise, the unstructured data is recorded as not meeting the clarity standard; If the information integrity meets a preset information integrity standard, then the unstructured data is recorded as passing the verification; otherwise, the unstructured data is recorded as not meeting the information integrity standard.

5. A data governance method according to claim 3, characterized in that: Verification is performed using the second image verification rule, including: Loading an image verification standard library, the standard library including at least one of an image brightness standard, an image contrast standard, and an image size standard, the image size being used to indicate the size of a storage space occupied by the image; Verifying whether the unstructured data meets all standards in the standard library; If it meets the criteria, it records that the unstructured data has passed the verification, otherwise it records that the unstructured data does not meet the criteria.

6. A data governance method according to claim 1, characterized in that: After generating a data quality report according to the verification result, the method further includes: If the structured data fails the verification process, normalizing the structured data to obtain normalized data; The standard data is double-checked using business rules, and a data quality report is generated based on the check results.

7. A data governance method according to claim 6, characterized in that: The structured data is normalized, including: Extracting and deleting duplicate values ​​in the structured data; Find missing values ​​in the structured data and process them using a preset method, wherein the preset method includes at least one of deleting the missing values, filling the missing values ​​with interpolation, and filling with default values; Identifying and removing outliers in the structured data; Searching for data of a preset data type in the structured data and converting the data into a corresponding data format, wherein the preset data type includes at least date and currency; The text data in the structured data is extracted and standardized, wherein the standardization includes at least one of removing redundant spaces, converting uppercase and lowercase letters, and replacing special characters.

8. A data governance method according to claim 1, characterized in that: The method further comprises: Obtain multiple data quality reports and perform data analysis; Generate a data governance effectiveness report based on the results of the data analysis.

9. A data management device, characterized in that: The device comprises: at least one processor; and, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor so that the at least one processor can execute a data governance method according to any one of claims 1-8.

10. A non-volatile computer storage medium storing computer executable instructions, characterized in that: The computer executable instructions are configured to be able to execute a data governance method according to any one of claims 1-8.

Images