A data management method, device, equipment and storage medium for data model

Abstract

The present application relates to a data management method, device, computer equipment and storage medium for a data model, including: when constructing a data model, determining data standard information; generating data type mapping relationships and data quality rules based on the data standard information and binding them with the data model; when the data model is published, prompting the number of data construction times based on the number of times the data standard information is cited; when the data model is published, reading data from the target database and converting the read data according to the data type mapping relationship and storing it to provide data support for the operation of the data model; after the data model is published, performing a data quality check on the data model according to the data quality rules. The above prompting of the number of data construction times based on the number of times the data standard information is cited can reduce repeated data construction and data redundancy, and uniformly manage the data read from the database through the data type mapping relationship to reduce data quality issues of the data source.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a data management method, device, computer equipment and storage medium for a data model. Background Art

[0002] The Internet industry mainly uses dimensional modeling for data modeling. Dimensional modeling is a method for building data warehouses. It is a subject-based, business-oriented data modeling method. The advantages of dimensional modeling are that it is simple to understand, easy to maintain and expand, and can quickly respond to business needs. At the same time, dimensional modeling can also improve the comprehensibility and usability of data, allowing users to more conveniently analyze and make decisions about data.

[0003] During the implementation process, the conventional data overall architecture design mainly divides the data according to the data domain, and then constructs the bus matrix according to the dimensional modeling theory to abstract the business process and dimension. The main function of the bus matrix is ​​to help data warehouse designers understand the relationship between business processes and business dimensions, so as to better design the data model of the data warehouse.

[0004] Traditionally, without the constraints of data governance, data quality is often uncontrollable, which directly affects the accuracy and reliability of the data model obtained by dimensional modeling. In other words, in the process of dimensional modeling, if there are no constraints of data governance, data redundancy may occur, which will increase the complexity of data storage and management. In addition, the data source that dimensional modeling relies on may have quality problems, such as inaccurate, incomplete or inconsistent data formats, which will directly affect the effect and accuracy of dimensional modeling. Summary of the invention

[0005] Based on this, it is necessary to provide a data management method, device, computer equipment and storage medium for a data model to address the above technical problems. The number of data construction times is prompted by the number of times the data standard information is referenced to reduce repeated data construction and data redundancy. In addition, the data read from the database is uniformly managed through data type mapping relationships to reduce data quality issues of the data source.

[0006] A data management method for a data model comprises: determining data standard information when constructing a data model; generating data type mapping relationships and data quality rules according to the data standard information, and binding the data type mapping relationships and data quality rules to the data model; providing a data construction number prompt according to the number of times the data standard information is cited when the data model is published; reading data from a target database and converting and storing the read data according to the data type mapping relationship when the data model is published, so as to provide data support for the operation of the data model; and performing a data quality check on the data model according to the data quality rules after the data model is published.

[0007] In one embodiment, a data management method for a data model also includes: generating a controlled vocabulary based on keywords, keyword relationships, and regular relationships of controlled data; configuring a security level for the controlled vocabulary based on data security standard information; configuring a data processing method for the controlled vocabulary based on the data security management system information and based on the security level; reading data from a target database and converting the read data according to a data type mapping relationship and then storing it, including: if the data read from the target database is a controlled vocabulary, converting the read data according to the data type mapping relationship, and then processing the converted controlled vocabulary based on the data processing method of the controlled vocabulary and then storing it.

[0008] In one of the embodiments, a data management method for a data model also includes: collecting system specification information, business standard information and data control information; generating data standard information and controlled data based on the system specification information, business standard information and data control information.

[0009] In one of the embodiments, a prompt for the number of data construction times is provided based on the number of times the data standard information is cited, including: when the number of times the data standard information is cited is less than a first threshold number, displaying a prompt message indicating whether reconstruction is required; when the number of times the data standard information is cited is greater than a second threshold number, providing a prompt for manual intervention; wherein the second threshold is greater than or equal to the first threshold.

[0010] In one embodiment, generating a data type mapping relationship according to data standard information includes: identifying parameter types and data types of a target database; and generating a data type mapping relationship according to the parameter types and data types of the target database and the data standard information.

[0011] In one of the embodiments, the data quality rules include a data stability rule, a data integrity rule, and a data uniqueness rule, the data stability rule is configured as a strong rule or a weak rule, the data integrity rule is configured as a strong rule or a weak rule, and the data uniqueness rule is configured as a strong rule or a weak rule; performing a data quality check on the data model according to the data quality rules, including: when the check result of the data quality check on the data model according to the data stability rule is that the data is unstable, if the data stability rule is configured as a strong rule, a first alarm processing is performed, and if the data stability rule is configured as a weak rule, a second alarm processing is performed. processing; when the result of the data quality check on the data model according to the data integrity rule is that the data is incomplete, if the data integrity rule is configured as a strong rule, a first alarm processing is performed, and if the data integrity rule is configured as a weak rule, a second alarm processing is performed; when the result of the data quality check on the data model according to the data uniqueness rule is that the data is not unique, if the data uniqueness rule is configured as a strong rule, a first alarm processing is performed, and if the data uniqueness rule is configured as a weak rule, a second alarm processing is performed; wherein the alarm urgency of the first alarm processing is greater than the alarm urgency of the second alarm processing.

[0012] In one of the embodiments, the data standard information includes data-related description information and data constraint information, and data quality rules are generated based on the data standard information, including: generating data stability rules, data integrity rules and data uniqueness rules based on the data-related description information and data constraint information.

[0013] A data management device for a data model comprises: a determination module for determining data standard information when constructing a data model; a generation module for generating data type mapping relationships and data quality rules according to the data standard information, and binding the data type mapping relationships and data quality rules to the data model; a prompt module for providing a data construction number prompt according to the number of times the data standard information is cited when the data model is published; a storage module for reading data from a target database and converting and storing the read data according to the data type mapping relationship when the data model is published, so as to provide data support for the operation of the data model; and a quality inspection module for performing a data quality inspection on the data model according to the data quality rules after the data model is published.

[0014] A computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of any of the above-mentioned embodiments are implemented.

[0015] A computer-readable storage medium stores a computer program, which implements the steps of any of the above-mentioned method embodiments when the computer program is executed by a processor.

[0016] The data management method, device, computer equipment and storage medium of the above-mentioned data model determine data standard information when constructing a data model; generate data type mapping relationships and data quality rules based on the data standard information, and bind the data type mapping relationships and data quality rules to the data model; when the data model is published, prompt the number of data construction times based on the number of times the data standard information is cited; when the data model is published, read data from the target database and convert the read data according to the data type mapping relationship and store it to provide data support for the operation of the data model; after the data model is published, perform a data quality check on the data model according to the data quality rules.

[0017] Therefore, when the data model refers to data from any database, the format of the referenced data can be guaranteed to be unified and complete. In addition, the number of references to the data standard information can be used to indicate the number of data constructions, thereby reducing repeated data construction and data redundancy. In addition, after the data model is released, the data quality is checked according to the data quality rules to ensure the accuracy of the data in the data model. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 An application environment diagram of a data management method for a data model in an embodiment;

[0019] Figure 2 A schematic diagram of a flow chart of a data management method for a data model in an embodiment;

[0020] Figure 3 A schematic diagram of an interface of a field definition stage of a data model in one embodiment;

[0021] Figure 4 A schematic diagram of an interface for automatically matching data stability rules of a data model in an embodiment;

[0022] Figure 5 A schematic diagram of an interface for automatically matching data integrity rules of a data model in an embodiment;

[0023] Figure 6 A schematic diagram of an interface for automatically matching data uniqueness rules of a data model in one embodiment;

[0024] Figure 7 A schematic diagram of an interface for quality warning prompts of a data model in an embodiment;

[0025] Figure 8 A schematic diagram of a pre-initialization processing flow of a data management method for a data model in a specific embodiment;

[0026] Fig. 9 A schematic diagram of a data model development process flow of a data model management method in a specific embodiment;

[0027] Fig.10 is a structural block diagram of a data management device for a data model in an embodiment;

[0028] Fig.11 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0029] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0030] The present application provides a data management method for a data model, which is applied to Figure 1 In the application environment shown. Figure 1 As shown, the data integration platform 100 is used to implement a data management method of a data model of the present application. Specifically, the terminal 300 initiates a data modeling construction request to the data integration platform 100. When constructing the data model, the data integration platform 100 obtains data standard information from the database 200, generates data type mapping relationships and data quality rules based on the data standard information, and binds the data type mapping relationships and data quality rules to the data model. When the data integration platform 100 publishes the data model to the terminal 300, it prompts the terminal 300 with the number of data construction times based on the number of times the data standard information is referenced. When the data model is published, data is read from the target database 400 and the read data is converted according to the data type mapping relationship and stored in the database 200 to provide data support for the operation of the data model. After the data model is published, the data integration platform 100 performs a data quality check on the data model according to the data quality rules.

[0031] In one embodiment, Figure 2 As shown, a data management method for a data model is provided, and the method is applied to Figure 1 The data integration platform 100 in FIG. 1 is used as an example to illustrate the method, which includes the following steps:

[0032] S202: When constructing a data model, determine data standard information.

[0033] In this embodiment, the data standard information can be determined by investigating and sorting out business knowledge, laws and regulations, and company specifications. The data standard information can include relevant descriptive and binding information for defining data, such as Chinese name, English name, business definition, source, basis, business rules, data category, code table information, data length, precision, unit, management department, and other information.

[0034] In one embodiment, before S202, the process further includes: if no data standard information is detected, prompting the user to manually enter the data standard information and displaying an entry interface for the data standard information, and obtaining the manually entered data standard information through the entry interface.

[0035] Specifically, if temporary data without reference standards is used, the data standard information needs to be manually entered. In addition, since temporary data is used, it is necessary to regularly check whether the manually entered data standard information needs to be retired. If so, the data standard information is deleted. If not, it is converted to data standard information for operation and added to the data standard information library.

[0036] S204, generating a data type mapping relationship and a data quality rule according to the data standard information, and binding the data type mapping relationship and the data quality rule to the data model.

[0037] In this embodiment, the data standard information includes the relevant descriptive information of the defined data, and the standard data type can be extracted from the relevant descriptive information of the defined data, and the data type mapping relationship can be generated based on the standard data type. Specifically, the data specification of the database is used as a reference to generate the data type mapping relationship of each data type and the related type parameter. For example, if the data type is a text class, its data type mapping relationship can select string type, varchar type or char type; if the data type is a measurement class, its data type mapping relationship can select bigint type or decimal type; if the data type is an enumeration class, its data type mapping relationship can select string type or boolean type; if the data type is a flag class, its data type mapping relationship can select string type or boolean type. The data type mapping relationship is bound to the data model, so that in the case where the data source on which the dimensional modeling depends may have quality problems, the data from different data sources can be converted into a standardized and normalized unified format type based on the bound data type mapping relationship, making the dimensional modeling more unified and efficient.

[0038] In one example, generating a data type mapping relationship according to data standard information includes: identifying parameter types and data types of a target database; and generating a data type mapping relationship according to the parameter types and data types of the target database and the data standard information.

[0039] Specifically, the parameter types and data types of different database configurations may be different, and it is necessary to generate corresponding data type mapping relationships based on the parameter types and data types of each database corresponding to the data model. Therefore, it is possible to maintain the corresponding data types and related type parameters in different databases. For example, the databases corresponding to the data model include the mysql database and the maxcomputer database. Both databases provide user-coded parameters and gender parameters. The data types corresponding to the two parameters in the mysql database are vatchar types, and the data types corresponding to the maxcomputer database are string types. Therefore, the data type mapping relationship set for the user-coded parameters and gender parameters in the mysql database and the data types of the two is: the user-coded data of the vatchar type and the gender data of the vatchar type are mapped to the char type data uniformly set by the data model. Therefore, the data type mapping relationship set for the user-coded parameters and gender parameters in the maxcomputer database and the data types of the two is: the string type user-coded data and the vatchar type gender data are mapped to the char type data uniformly set by the data model.

[0040] In this embodiment, the data standard information includes constraint information, and data quality rules can be generated based on the constraint information. That is, when the constraint information is not satisfied, it can be determined that the data has quality problems. For example, the constraint information includes that the mobile phone number is 11 digits. If the multiple data of the mobile phone number include non-digital data, it is determined that the data of the mobile phone number has quality problems.

[0041] S206, when the data model is published, a data construction count prompt is provided according to the number of citations of the data standard information.

[0042] In this embodiment, during the data model publishing stage, corresponding prompts are given according to the situation in which the data standard is referenced to inform the number of times the same data is constructed.

[0043] In one example, the above-mentioned prompting of the number of data construction times based on the number of times the data standard information is cited includes: when the number of times the data standard information is cited is less than a first threshold number, displaying a prompt message indicating whether reconstruction is required; when the number of times the data standard information is cited is greater than a second threshold number, providing a manual intervention prompt; wherein the second threshold is greater than or equal to the first threshold.

[0044] Among them, the second threshold value may be equal to the first threshold value, or the second threshold value may be greater than the first threshold value. For example, the first threshold value is 1, and the second threshold value is 5. When the data standard information has been cited once, a prompt message "Please confirm whether to repeat the construction" is displayed on the terminal to prompt the user of the terminal that the data standard information has been cited, that is, the same data has been built based on it. Such a prompt can reduce repeated development and directly reuse the constructed data. When the data standard information has been cited 5 times or more, it is prompted that it needs to be confirmed by relevant personnel before it can be cited, that is, it needs to be confirmed by users with certain permissions before the data standard information can be cited. Among them, if the data standard information is cited too many times, it means that there is a lot of data duplication, and manual intervention is required to check whether it is stopped. Therefore, the number of citations of the data standard information serves as a handle for data duplication, reduces data redundancy, ensures data consistency, and thus improves the efficiency and reliability of the data warehouse.

[0045] S208, when the data model is published, data is read from the target database and the read data is converted and stored according to the data type mapping relationship to provide data support for the operation of the data model.

[0046] In this embodiment, the data model construction phase configures the data type mapping relationship. Therefore, when the data in the database needs to be read to support the operation of the data model during the data model publishing phase, the data is read from the target database and converted according to the data type mapping relationship. The converted data meets the format requirements of the data model operation, and can provide data support for the data operation. Therefore, the accuracy, integrity and consistency of the data can be ensured, solving the problem of uncontrollable data quality in traditional technologies.

[0047] In one embodiment, before the above step S208, the steps are also included: generating a controlled vocabulary according to the keywords, keyword relationships and regular relationships of the controlled data; configuring a security level for the controlled vocabulary according to the data security standard information; configuring a data processing method for the controlled vocabulary according to the data security management system information and based on the security level. The above reading data from the target database and converting the read data according to the data type mapping relationship and then storing it includes: if the data read from the target database is a controlled vocabulary, converting the read data according to the data type mapping relationship, and then processing the converted controlled vocabulary based on the data processing method of the controlled vocabulary and then storing it.

[0048] Among them, before the above-mentioned step of generating a controlled vocabulary based on the keywords, keyword relationships and regular relationships of the controlled data, it also includes the steps of: collecting system specification information, business standard information and data control information; generating data standard information and controlled data based on the system specification information, business standard information and data control information.

[0049] In this embodiment, specifically, the system specification information is the specification information set to meet the unified requirements of the system. For example, the system specification information specifies the user's Chinese name and English name, ID number, user code, data category, code table information, data length, precision, etc. For example, the system specification requires that the user code is a 32-bit string, and the ID number stipulates 18 digits and the first 6 digits are for the division. Business standard information is standard information set based on the business. For example, business standard information includes information such as business definition, business rules, units and departments involved in the business. For example, the profit value is set to retain two decimal places in the business rules. Data control information can refer to relevant national laws and regulations to determine the type of data to be managed and refer to the management needs of the enterprise to determine the type of data to be managed. For example, data control information includes personal sensitive information, such as name, birthday, residential address, etc. Among them, data standard information can be generated based on system specification information and business standard information, and controlled data can be generated based on data control information.

[0050] The controlled data includes keywords, relationships between keywords, and regular relationships. For example, the relationships between keywords include synonyms, near-synonyms, and antonyms. Regular relationships are determined by regular expressions. For example, regular relationships include logical relationships of AND, OR, and NOT. A controlled vocabulary is generated based on the keywords, relationships between keywords, and regular relationships of the controlled data. For example, the keyword is academic qualification, and the annotation content contained in the controlled data contains academic qualification, highest degree, education level, and other words, or the English name hits education_level. The highest degree and education level are determined through regular relationships and / or keyword relationships, and the controlled vocabulary includes academic qualification, highest academic qualification, and education level. Furthermore, after the controlled vocabulary is determined, the controlled vocabulary can also be marked to identify the controlled vocabulary through marking.

[0051] Data security standard information can be extracted from relevant documents such as the data security standard system stored in the system. Different security levels are assigned to different controlled vocabularies in the data security standard information, and different security levels represent the degree of control of the controlled vocabulary. Therefore, after determining the controlled vocabulary, the corresponding security level can be configured for the controlled vocabulary based on the data security standard information.

[0052] In order to protect the personal privacy of users, consider the security of user information and the security of corporate data, enterprises formulate data security management systems. The data security management system specifies the relevant regulations that users need to comply with when using network data and the relevant security operations that need to be performed. Therefore, the data processing methods of controlled vocabularies of various security levels can be extracted from the information of the data security management system. For example, the security levels are divided into low, medium and high. Low-level controlled vocabularies need to be encrypted. Medium-level controlled vocabularies can only be read by users with high permissions. High-level controlled vocabularies require access control.

[0053] If the data read from the target database is a controlled vocabulary, it will be converted according to the data type mapping relationship, and then further processed according to its corresponding data processing method before being stored, so that the controlled vocabulary read during the construction of the data model or the operation of the data model can meet its security management and control requirements. Specifically, each system needs to be processed separately according to different controlled vocabularies. For example, when the data model is published, if the security level of the controlled vocabulary is greater than or equal to 3, it needs to be automatically encrypted. Or, if the security level of the controlled vocabulary is greater than or equal to 5, the security level is high, and it is set to be used only by specific people.

[0054] In a specific implementation scenario, when adding field definitions to the data model construction, you can directly select the established data standard information to automatically generate the corresponding naming, data type mapping relationship and corresponding data quality rules. Figure 3 As shown, in the field definition part, data standard information, data type, security level, non-empty, unique, check constraint and other fields are introduced, so that when building the data model, the information in the field definition part is bound to the data model. During the data model publishing stage, prompts are given based on the references of the data standard information. For example, if the data standard information has been referenced once, please confirm whether it needs to be repeated, or if it has been referenced more than 5 times, please initiate the process to contact the data manager for confirmation. After the data model is released, data is checked regularly according to data quality rules to ensure data quality.

[0055] S210: After the data model is published, a data quality check is performed on the data model according to data quality rules.

[0056] In this embodiment, after the data model is published, the data model is used by the terminal. During the use of the data model, the data in the target database is continuously called for operation. In this process, the data called by the data model can be quality checked by combining the operation of the data model with the data quality rules to ensure the quality of the data used by the data model. For example, the data quality rules include quality rules for data consistency verification, quality rules for data accuracy verification, etc. For example: For example, the quality rules for data consistency verification are used to detect whether the names of the same user in different systems are consistent. If not, a data quality problem occurs. The quality rules for data accuracy verification are used to detect whether the user's age is correct. The maximum value of the user's age is set to 60 and the minimum value is set to 18 in the quality rules for data accuracy verification, so as to detect whether the user's age data is abnormal.

[0057] The data management method of the above-mentioned data model determines the data standard information when constructing the data model; generates the data type mapping relationship and the data quality rules according to the data standard information, and binds the data type mapping relationship and the data quality rules to the data model; when the data model is released, the data construction number is prompted according to the number of times the data standard information is cited; when the data model is released, the data is read from the target database and the read data is converted and stored according to the data type mapping relationship to provide data support for the operation of the data model; after the data model is released, the data quality of the data model is checked according to the data quality rules. Therefore, when the data model refers to the data of any database, the uniformity and integrity of the referenced data format can be guaranteed. In addition, the data construction number is prompted by the number of times the data standard information is cited, so as to reduce the repeated construction of data and reduce data redundancy. In addition, after the data model is released, the data quality of the data model is checked according to the data quality rules to ensure the accuracy of the data of the data model.

[0058] In one embodiment, the data quality rules include data stability rules, data integrity rules and data uniqueness rules, the data stability rules are configured as strong rules or weak rules, the data integrity rules are configured as strong rules or weak rules, and the data uniqueness rules are configured as strong rules or weak rules; in the above step S210, the data quality check of the data model is performed according to the data quality rules, including: when the inspection result of the data quality check of the data model according to the data stability rules is that the data is unstable, if the data stability rule is configured as a strong rule, a first alarm processing is performed, and if the data stability rule is configured as a weak rule, a second alarm processing is performed. Second alarm processing; when the inspection result of the data quality check on the data model according to the data integrity rule is that the data is incomplete, if the data integrity rule is configured as a strong rule, the first alarm processing is performed, and if the data integrity rule is configured as a weak rule, the second alarm processing is performed; when the inspection result of the data quality check on the data model according to the data uniqueness rule is that the data is not unique, if the data uniqueness rule is configured as a strong rule, the first alarm processing is performed, and if the data uniqueness rule is configured as a weak rule, the second alarm processing is performed; wherein, the alarm urgency of the first alarm processing is greater than the alarm urgency of the second alarm processing.

[0059] In one example, the data standard information includes data-related description information and data constraint information. The above-mentioned generating data quality rules based on the data standard information includes: generating data stability rules, data integrity rules and data uniqueness rules based on the data-related description information and data constraint information.

[0060] Specifically, the data-related descriptive information is the descriptive information that defines the data. For example, the descriptive information that defines the data includes the data length and precision and its corresponding data structure. The data-related descriptive information can be represented in the form of a data table structure. The data table structure defines whether the relevant data is non-empty and whether it is unique, thereby generating data integrity rules and data uniqueness rules accordingly. Data constraint information can be a constraint condition for manually configured data. For example, if the data constraint information sets the sales volatility of product a to 20%, the corresponding data stability rule generated is to verify the data volatility of product a sales. If the volatility exceeds 20%, it means that the sales of product a are unstable, and if the volatility does not exceed 20%, it means that the sales of product a are stable.

[0061] Data stability rules, data integrity rules, and data uniqueness rules can all be manually configured with their strength or weakness attributes. If configured as a strong rule, the first alarm processing is performed when the corresponding problem occurs in the inspection results of the data quality check. If configured as a weak rule, the second alarm processing is performed when the corresponding problem occurs in the inspection results of the data quality check. The processing method of the first alarm processing is more urgent than the processing method of the second alarm processing. For example, the processing method of the first alarm processing is to notify the user by phone, that is, to call the user to notify him to quickly deal with the problem. The processing method of the second alarm processing is to notify the user by email or text message, and the user can learn about the specific problem when opening the email or text message.

[0062] Specifically, the data quality checks of the data stability rules are as follows: Figure 4 As shown in Figure 2, the data quality checks for data integrity rules are as follows: Figure 5 As shown in the figure, the data quality check of the data uniqueness rule is as follows Figure 6 As shown, the alarm processing is as follows Figure 7 Therefore, not only can the data uniformity and security of the data model be ensured when the data model is constructed and published, but the data quality of the data model can also be regularly checked in subsequent applications of the data model, and alarm processing can be performed when quality problems of varying degrees occur, thus ensuring the high-quality operation of the data model.

[0063] The above-mentioned data management method for a data model can ensure that the data quality of the data model is controllable, enhance the data security of the data model, reduce data redundancy in the system, improve data consistency, and improve data processing and analysis efficiency.

[0064] For the data management method of the above data model, a specific implementation process is given below:

[0065] The overall process can be divided into two parts:

[0066] The first part Figure 8 Shown: Pre-initialization processing flow.

[0067] Step 1: Research and sort out business knowledge, laws and regulations, company regulations, etc.;

[0068] Step 2: Generate data standards and controlled vocabulary based on the collected content. This step will produce two branches:

[0069] Branch 1: Data standards define relevant descriptive and constraining information of data, such as Chinese name, English name, business definition, source, basis, business rules, data category, code table information, data length, precision, unit, management department and other information; Generate corresponding data type mapping relationships based on data standard information, which refers to maintaining corresponding data types and related type parameters in different databases; Generate data quality rules, which refers to generating rules for relevant data constraint verification, such as mobile phone numbers, and perform numerical verification according to the rules for generating mobile phone numbers;

[0070] Branch 2: Controlled vocabulary mainly includes keywords extracted from controlled data, keyword relationships (such as synonyms, near synonyms, etc.), and related matching regular relationships, etc.; according to the controlled vocabulary and combined with data security standards, different security levels are assigned to different controlled words; according to the national and company data security management systems, different executable security level rules are set, such as sensitive field encryption, access control, etc.

[0071] The second part Fig. 9 Shown: Data model development process flow.

[0072] When adding fields to the data model, you can directly select the established data standard to automatically generate the corresponding naming, data type and data quality rules. When publishing the data model, you will be prompted based on the citation of the data standard. For example, if this data standard information has been cited once, please confirm whether it needs to be repeated. If it has been cited more than 5 times, please initiate the process and contact the data manager for confirmation. After the data model is published, the data is regularly checked according to the data quality rules to ensure data quality.

[0073] If temporary data does not reference data standard information, it can be entered manually. At this time, a regular inspection reminder task will be adjusted to either retire the data or convert it to data standard information in subsequent operations to supplement the data standard information. If data standard information is found to be missing, new data standard information can be created. After supplementing the information according to the requirements of the data standard information, the repeatability check can be performed. It can be checked according to the description and data constraints. After passing the check, the data standard information will be automatically released. Data standard information will be regularly checked by the data governance team as the last step to check for deficiencies and make up for gaps.

[0074] The information entered in the above steps is unified in a standard system, which is more efficient and flexible, making the data modeling process more automated, standardized and unified.

[0075] The data management method of the above data model can be summarized as follows:

[0076] 1. Data governance mechanism: An advanced data governance mechanism is introduced, which is lacking in traditional technologies. This mechanism can ensure the accuracy, integrity and consistency of data, thus solving the problem of uncontrollable data quality in traditional technologies.

[0077] 2. Security measures: Emphasis is placed on data security. By implementing strict security measures and protocols, the risk of data leakage and illegal access is reduced, which is an important improvement over the insufficient security of traditional technologies.

[0078] 3. Data standard control: By optimizing data standard control, data redundancy is reduced, data consistency is ensured, and the efficiency and reliability of the data warehouse are improved. This is an effective response to the data redundancy and inconsistency problems in traditional technologies.

[0079] 4. Data standardization and normalization: Promote the standardization of data and improve the consistency and efficiency of data processing and analysis by formulating and implementing unified data standards and specifications. This is a positive response to the lack of standardization and normalization of traditional technologies.

[0080] It should be understood that, although the steps in the flowchart are displayed in sequence as indicated by the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and these steps can be executed in other orders. Moreover, at least a portion of the steps in the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a portion of the sub-steps or stages of other steps.

[0081] The present application also provides a data management device for a data model. Fig.10As shown, a data management device for a data model includes a determination module 1002, a generation module 1004, a prompt module 1006, a storage module 1008, and a quality inspection module 1010. The determination module 1002 is used to determine data standard information when constructing a data model; the generation module 1004 is used to generate a data type mapping relationship and a data quality rule according to the data standard information, and bind the data type mapping relationship and the data quality rule to the data model; the prompt module 1006 is used to provide a data construction number prompt according to the number of references of the data standard information when the data model is released; the storage module 1008 is used to read data from a target database and convert the read data according to the data type mapping relationship and store it when the data model is released, so as to provide data support for the operation of the data model; the quality inspection module 1010 is used to perform a data quality inspection on the data model according to the data quality rules after the data model is released.

[0082] In one embodiment, a data management device for a data model also includes a security level configuration module, which is used to generate a controlled vocabulary based on keywords, keyword relationships, and regular relationships of controlled data; configure a security level for the controlled vocabulary based on data security standard information; configure a data processing method for the controlled vocabulary based on data security management system information and based on the security level; read data from a target database and convert the read data according to a data type mapping relationship and then store it, including: if the data read from the target database is a controlled vocabulary, convert the read data according to the data type mapping relationship, and then process the converted controlled vocabulary based on the data processing method of the controlled vocabulary and then store it.

[0083] In one of the embodiments, a data management device for a data model also includes a collection module for collecting system specification information, business standard information and data control information; and generating data standard information and controlled data based on the system specification information, business standard information and data control information.

[0084] In one of the embodiments, a prompt for the number of data construction times is provided based on the number of times the data standard information is cited, including: when the number of times the data standard information is cited is less than a first threshold number, displaying a prompt message indicating whether reconstruction is required; when the number of times the data standard information is cited is greater than a second threshold number, providing a prompt for manual intervention; wherein the second threshold is greater than or equal to the first threshold.

[0085] In one embodiment, generating a data type mapping relationship according to data standard information includes: identifying parameter types and data types of a target database; and generating a data type mapping relationship according to the parameter types and data types of the target database and the data standard information.

[0086] In one of the embodiments, the data quality rules include a data stability rule, a data integrity rule, and a data uniqueness rule, the data stability rule is configured as a strong rule or a weak rule, the data integrity rule is configured as a strong rule or a weak rule, and the data uniqueness rule is configured as a strong rule or a weak rule; performing a data quality check on the data model according to the data quality rules, including: when the check result of the data quality check on the data model according to the data stability rule is that the data is unstable, if the data stability rule is configured as a strong rule, a first alarm processing is performed, and if the data stability rule is configured as a weak rule, a second alarm processing is performed. processing; when the result of the data quality check on the data model according to the data integrity rule is that the data is incomplete, if the data integrity rule is configured as a strong rule, a first alarm processing is performed, and if the data integrity rule is configured as a weak rule, a second alarm processing is performed; when the result of the data quality check on the data model according to the data uniqueness rule is that the data is not unique, if the data uniqueness rule is configured as a strong rule, a first alarm processing is performed, and if the data uniqueness rule is configured as a weak rule, a second alarm processing is performed; wherein the alarm urgency of the first alarm processing is greater than the alarm urgency of the second alarm processing.

[0087] In one of the embodiments, the data standard information includes data-related description information and data constraint information, and data quality rules are generated based on the data standard information, including: generating data stability rules, data integrity rules and data uniqueness rules based on the data-related description information and data constraint information.

[0088] For the specific definition of a data management device for a data model, please refer to the definition of a data management method for a data model mentioned above, which will not be repeated here. Each module in the data management device for the above-mentioned data model can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0089] In one embodiment, a computer device is provided. The computer device may be a server supporting the operation of the data integration platform 100. The internal structure diagram thereof may be as follows: Fig.11As shown. The computer device includes a processor, a memory, a network interface and a database connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to interact with the terminal for data, etc. When the computer program is executed by the processor, the data management method of the above data model is implemented.

[0090] Those skilled in the art will understand that Fig.11 The structure shown in the figure is only a block diagram of a part of the structure related to the scheme of the present application, and does not constitute a limitation on the computer device to which the scheme of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different arrangement of components.

[0091] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the following steps are implemented: when constructing a data model, data standard information is determined; data type mapping relationships and data quality rules are generated according to the data standard information, and the data type mapping relationships and data quality rules are bound to the data model; when the data model is published, a data construction number prompt is provided according to the number of times the data standard information is cited; when the data model is published, data is read from a target database and the read data is converted and stored according to the data type mapping relationship to provide data support for the operation of the data model; after the data model is published, a data quality check is performed on the data model according to the data quality rules.

[0092] In one of the embodiments, when the processor executes the computer program, the following steps are implemented: generating a controlled vocabulary based on keywords, keyword relationships, and regular relationships of controlled data; configuring a security level for the controlled vocabulary based on data security standard information; configuring a data processing method for the controlled vocabulary based on the data security management system information and based on the security level; when the processor executes the computer program to implement the above-mentioned steps of reading data from the target database and converting the read data according to the data type mapping relationship and then storing it, the following steps are specifically implemented: if the data read from the target database is a controlled vocabulary, the read data is converted according to the data type mapping relationship, and then the converted controlled vocabulary is processed based on the data processing method of the controlled vocabulary and then stored.

[0093] In one embodiment, when the processor executes the computer program, the following steps are implemented: collecting system specification information, business standard information and data control information; generating data standard information and controlled data based on the system specification information, business standard information and data control information.

[0094] In one of the embodiments, when the processor executes a computer program to implement the above-mentioned step of prompting the number of data construction times according to the number of times the data standard information is cited, the following steps are specifically implemented: when the number of times the data standard information is cited is less than a first threshold number, a prompt message indicating whether reconstruction is required is displayed; when the number of times the data standard information is cited is greater than a second threshold number, a manual intervention prompt is performed; wherein the second threshold is greater than or equal to the first threshold.

[0095] In one of the embodiments, when the processor executes a computer program to implement the above-mentioned step of generating a data type mapping relationship based on data standard information, the following steps are specifically implemented: identifying the parameter type and data type of the target database; generating a data type mapping relationship based on the parameter type and data type of the target database and the data standard information.

[0096] In one embodiment, the data quality rules include data stability rules, data integrity rules and data uniqueness rules, the data stability rules are configured as strong rules or weak rules, the data integrity rules are configured as strong rules or weak rules, and the data uniqueness rules are configured as strong rules or weak rules; when the processor executes the computer program to implement the above-mentioned step of performing a data quality check on the data model according to the data quality rules, the following steps are specifically implemented: when the inspection result of the data quality check on the data model according to the data stability rule is that the data is unstable, if the data stability rule is configured as a strong rule, a first alarm processing is performed, and if the data stability rule is configured as a When the data quality check on the data model according to the data integrity rule shows that the data is incomplete, if the data integrity rule is configured as a strong rule, the first alarm processing is performed, and if the data integrity rule is configured as a weak rule, the second alarm processing is performed; when the data quality check on the data model according to the data uniqueness rule shows that the data is not unique, if the data uniqueness rule is configured as a strong rule, the first alarm processing is performed, and if the data uniqueness rule is configured as a weak rule, the second alarm processing is performed; wherein the alarm urgency of the first alarm processing is greater than the alarm urgency of the second alarm processing.

[0097] In one of the embodiments, the data standard information includes data-related descriptive information and data constraint information. When the processor executes a computer program to implement the above-mentioned step of generating data quality rules based on the data standard information, the following steps are specifically implemented: generating data stability rules, data integrity rules and data uniqueness rules based on the data-related descriptive information and the data constraint information.

[0098] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the following steps are implemented: when building a data model, data standard information is determined; data type mapping relationships and data quality rules are generated according to the data standard information, and the data type mapping relationships and data quality rules are bound to the data model; when the data model is published, a data construction number prompt is provided according to the number of times the data standard information is cited; when the data model is published, data is read from a target database and the read data is converted and stored according to the data type mapping relationship to provide data support for the operation of the data model; after the data model is published, a data quality check is performed on the data model according to the data quality rules.

[0099] In one embodiment, when the computer program is executed by a processor, the following steps are implemented: generating a controlled vocabulary based on keywords, keyword relationships, and regular relationships of controlled data; configuring a security level for the controlled vocabulary based on data security standard information; configuring a data processing method for the controlled vocabulary based on the data security management system information and based on the security level; when the computer program is executed by the processor to implement the above-mentioned steps of reading data from a target database and converting the read data according to a data type mapping relationship and then storing it, the following steps are specifically implemented: if the data read from the target database is a controlled vocabulary, the read data is converted according to the data type mapping relationship, and then the converted controlled vocabulary is processed based on the data processing method of the controlled vocabulary and then stored.

[0100] In one embodiment, when the computer program is executed by a processor, the following steps are implemented: collecting system specification information, business standard information and data control information; generating data standard information and controlled data based on the system specification information, business standard information and data control information.

[0101] In one of the embodiments, when the computer program is executed by the processor to implement the above-mentioned step of prompting the number of data construction times based on the number of times the data standard information is cited, the following steps are specifically implemented: when the number of times the data standard information is cited is less than a first threshold number, a prompt message indicating whether reconstruction is required is displayed; when the number of times the data standard information is cited is greater than a second threshold number, a manual intervention prompt is performed; wherein the second threshold is greater than or equal to the first threshold.

[0102] In one of the embodiments, when the computer program is executed by a processor to implement the above-mentioned step of generating a data type mapping relationship based on data standard information, the following steps are specifically implemented: identifying the parameter type and data type of the target database; generating a data type mapping relationship based on the parameter type and data type of the target database and the data standard information.

[0103] In one embodiment, the data quality rules include data stability rules, data integrity rules and data uniqueness rules, the data stability rules are configured as strong rules or weak rules, the data integrity rules are configured as strong rules or weak rules, and the data uniqueness rules are configured as strong rules or weak rules; when the computer program is executed by the processor to implement the above-mentioned steps of performing data quality check on the data model according to the data quality rules, the following steps are specifically implemented: when the inspection result of performing data quality check on the data model according to the data stability rules is that the data is unstable, if the data stability rule is configured as a strong rule, a first alarm processing is performed, and if the data stability rule is configured as a When the data quality check on the data model according to the data integrity rule shows that the data is incomplete, if the data integrity rule is configured as a strong rule, the first alarm processing is performed, and if the data integrity rule is configured as a weak rule, the second alarm processing is performed; when the data quality check on the data model according to the data uniqueness rule shows that the data is not unique, if the data uniqueness rule is configured as a strong rule, the first alarm processing is performed, and if the data uniqueness rule is configured as a weak rule, the second alarm processing is performed; wherein the alarm urgency of the first alarm processing is greater than the alarm urgency of the second alarm processing.

[0104] In one of the embodiments, the data standard information includes data-related descriptive information and data constraint information. When the computer program is executed by the processor to implement the above-mentioned step of generating data quality rules based on the data standard information, the following steps are specifically implemented: generating data stability rules, data integrity rules and data uniqueness rules based on the data-related descriptive information and the data constraint information.

[0105] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0106] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0107] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A data management method for a data model, characterized in that: The method comprises: When building a data model, determine data standard information; Generate a data type mapping relationship and a data quality rule according to the data standard information, and bind the data type mapping relationship and the data quality rule to the data model; When the data model is published, a reminder of the number of data construction times is provided according to the number of citations of the data standard information; When the data model is published, data is read from the target database and the read data is converted and stored according to the data type mapping relationship to provide data support for the operation of the data model; After the data model is published, performing a data quality check on the data model according to the data quality rules; The prompting of the number of data construction times according to the number of times the data standard information is cited includes: when the number of times the data standard information is cited is less than a first threshold number, displaying prompting information on whether reconstruction is required; when the number of times the data standard information is cited is greater than a second threshold number, performing manual intervention prompting; wherein the second threshold is greater than or equal to the first threshold; Wherein, the data quality rules include data stability rules, data integrity rules and data uniqueness rules, the data stability rules are configured as strong rules or weak rules, the data integrity rules are configured as strong rules or weak rules, and the data uniqueness rules are configured as strong rules or weak rules; the data quality check on the data model according to the data quality rules includes: when the inspection result of the data quality check on the data model according to the data stability rules is that the data is unstable, if the data stability rule is configured as a strong rule, a first alarm processing is performed, and if the data stability rule is configured as a weak rule, a second alarm processing is performed. ; When the result of the data quality check performed on the data model according to the data integrity rule is that the data is incomplete, if the data integrity rule is configured as a strong rule, a first alarm processing is performed, and if the data integrity rule is configured as a weak rule, a second alarm processing is performed; when the result of the data quality check performed on the data model according to the data uniqueness rule is that the data is not unique, if the data uniqueness rule is configured as a strong rule, a first alarm processing is performed, and if the data uniqueness rule is configured as a weak rule, a second alarm processing is performed; wherein the alarm urgency of the first alarm processing is greater than the alarm urgency of the second alarm processing.

2. The method according to claim 1, characterized in that The method further comprises: Generate controlled vocabulary based on keywords, keyword relationships, and canonical relationships of controlled data; configuring a security level for the controlled vocabulary according to data security standard information; configuring a data processing method for the controlled vocabulary according to the data security management system information and based on the security level; Reading data from the target database and converting and storing the read data according to the data type mapping relationship includes: If the data read from the target database is the controlled vocabulary, the read data is converted according to the data type mapping relationship, and then the converted controlled vocabulary is processed and stored based on the data processing method of the controlled vocabulary.

3. The method according to claim 2, characterized in that The method further comprises: Collect system specification information, business standard information and data control information; The data standard information and the controlled data are generated according to the system specification information, the business standard information and the data control information.

4. The method according to claim 1, characterized in that The generating a data type mapping relationship according to the data standard information includes: Identify parameter types and data types of the target database; A data type mapping relationship is generated according to the parameter type and data type of the target database and the data standard information.

5. The method according to claim 1, characterized in that The data standard information includes data-related description information and data constraint information, and generating data quality rules according to the data standard information includes: The data stability rule, the data integrity rule and the data uniqueness rule are generated according to the data-related description information and the data constraint information.

6. A data management device for a data model, characterized in that: The device comprises: A determination module is used to determine data standard information when building a data model; A generating module, configured to generate a data type mapping relationship and a data quality rule according to the data standard information, and bind the data type mapping relationship and the data quality rule to the data model; A prompt module, used for prompting the number of data construction times according to the number of citations of the data standard information when the data model is published; A storage module, used for reading data from a target database and converting the read data according to the data type mapping relationship and storing the converted data when the data model is published, so as to provide data support for the operation of the data model; A quality check module, used for performing a data quality check on the data model according to the data quality rules after the data model is published; The prompting of the number of data construction times according to the number of times the data standard information is cited includes: when the number of times the data standard information is cited is less than a first threshold number, displaying prompting information on whether reconstruction is required; when the number of times the data standard information is cited is greater than a second threshold number, performing manual intervention prompting; wherein the second threshold is greater than or equal to the first threshold; Wherein, the data quality rules include data stability rules, data integrity rules and data uniqueness rules, the data stability rules are configured as strong rules or weak rules, the data integrity rules are configured as strong rules or weak rules, and the data uniqueness rules are configured as strong rules or weak rules; the data quality check on the data model according to the data quality rules includes: when the inspection result of the data quality check on the data model according to the data stability rules is that the data is unstable, if the data stability rule is configured as a strong rule, a first alarm processing is performed, and if the data stability rule is configured as a weak rule, a second alarm processing is performed. ; When the result of the data quality check performed on the data model according to the data integrity rule is that the data is incomplete, if the data integrity rule is configured as a strong rule, a first alarm processing is performed, and if the data integrity rule is configured as a weak rule, a second alarm processing is performed; when the result of the data quality check performed on the data model according to the data uniqueness rule is that the data is not unique, if the data uniqueness rule is configured as a strong rule, a first alarm processing is performed, and if the data uniqueness rule is configured as a weak rule, a second alarm processing is performed; wherein the alarm urgency of the first alarm processing is greater than the alarm urgency of the second alarm processing.

7. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 5 are implemented.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.

Images