Core data identification method and system
By building a core data identification model and using judgment factors and algorithm models to automatically identify core data, the inefficiency problem of enterprises in core data identification is solved and efficient and accurate data management is achieved.
Patent Information
- Application Number
- CN202110837745.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Priority Date
- 2021-05-19
- Filing Date
- 2021-07-23
- Publication Date
- 2024-12-31
- Estimated Expiration
- 2041-07-23
AI Technical Summary
In the prior art, enterprises rely on manual methods in core data identification, resulting in inefficiency and lack of technical means to identify core data.
Build a core data recognition model, and automatically identify core data by determining the evaluation factors and judgment factors, combining semantic similarity, grammatical similarity and corpus vector similarity algorithm models.
It realizes the rapid and accurate identification of core data, reduces data management costs, improves management efficiency, and clarifies the work objects and scope of data management.
Smart Images
Figure CN113505605B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of natural language processing, and in particular to a core data recognition method and system. Background Art
[0002] The identification of core data is the focus of data management in many enterprises. Currently, most enterprises rely on manual methods to identify core data and generally lack the methods to identify core data through technical means. The current manual identification method takes up a lot of working time of business personnel and is relatively inefficient.
[0003] The information disclosed in this background technology section is only intended to enhance the understanding of the overall background of the invention and should not be regarded as an acknowledgment or any form of suggestion that the information constitutes the prior art already known to a person skilled in the art. Summary of the invention
[0004] The object of the present invention is to provide a core data identification method and system, which construct a core data identification model, based on which core data can be quickly and automatically identified.
[0005] To achieve the above-mentioned purpose, the present invention provides a core data identification method, which includes: determining the evaluation factors of core data; dividing the core data into various judgment factors and the weights of the various judgment factors according to the evaluation factors of the core data to construct a core data identification model; determining the algorithm model of the various judgment factors; importing data resources of the core data to be identified; and identifying the core data of the imported data resources according to the core data identification model and the algorithm model.
[0006] In one embodiment of the present invention, the evaluation factors include: the system scope of data coverage, the business relevance of data, the degree of influence of data on decision-making, the correlation between data and enterprise-level data models, and the security sensitivity of data; the judgment factors include: data coverage factor, business relevance factor, indicator relevance factor, data sharing factor and management relevance factor.
[0007] In one embodiment of the present invention, the core data identification model is:
[0008]
[0009] Among them, M is the value of the core data coefficient, A is the value of the data coverage factor, f1 is the weight value of the data coverage factor, B is the value of the business relevance factor, f2 is the weight value of the business relevance factor, C is the value of the indicator relevance factor, f3 is the weight value of the indicator relevance factor, D is the value of the data sharing factor, f4 is the weight value of the data sharing factor, E is the value of the management relevance factor, and f5 is the weight value of the management relevance factor.
[0010] In one embodiment of the present invention, the algorithm models of the various judgment factors include: a semantic similarity algorithm model, a grammatical similarity algorithm model, and a corpus vector similarity algorithm model.
[0011] In one embodiment of the present invention, importing data resources of core data to be identified includes: directly importing data resources of core data to be identified from a powerdesigner data import interface, an erwin data import interface, an xml data import interface, a word data import interface, or an excel data import interface, and directly obtaining metadata information from a database through an oracle data interface, a mysql data interface, or a db2 data interface, and storing the metadata information as data resources of the core data to be identified in a centralized warehouse; establishing a data resource directory to classify the various types of imported data resources.
[0012] In one embodiment of the present invention, identifying core data of imported data resources based on the core data identification model and the algorithm model includes: testing the constructed core data identification model, and correcting the coefficients of the core data identification model based on the test results; and identifying core data of imported data resources based on the corrected core data identification model.
[0013] In one embodiment of the present invention, the testing of the constructed core data identification model includes: selecting a data sample set, judging the core data therein through business, as a data sample; using the core data identification model to calculate the selected data sample set, comparing and analyzing the calculation results with the data sample, and testing the accuracy of core data identification.
[0014] Based on the same inventive concept, the present invention also provides a core data identification system, which includes: a judgment factor determination module for determining the judgment factors of core data; an identification model construction module coupled with the judgment factor determination module, for dividing the core data into various judgment factors and the weights of the various judgment factors according to the judgment factors of the core data, thereby constructing a core data identification model; an algorithm model determination module for determining the algorithm model of the various judgment factors; an import module for importing data resources of the core data to be identified; an identification module coupled with the identification model construction module, the algorithm model determination module and the import module, for identifying the core data of the imported data resources according to the core data identification model and the algorithm model.
[0015] Based on the same inventive concept, the present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of the core data identification method described in any of the above-mentioned embodiments are implemented.
[0016] Based on the same inventive concept, the present invention also provides a non-transitory computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the core data identification method described in any of the above embodiments are implemented.
[0017] Compared with the prior art, the core data identification method and system according to the present invention innovatively proposes a core data identification model in the power grid industry, clarifies the key factors and weight settings for core data identification, and can quickly and automatically identify core data based on the core data identification model. It achieves efficient and accurate identification of core data of various professions and systems, accurately defines the work objects and scope of data management, and helps enterprises invest limited resources in the management of core data, greatly reducing data management costs and improving management effectiveness. Moreover, feasibility verification can be carried out through the data of power grid companies, which is of great significance for promoting the development of industry data management. BRIEF DESCRIPTION OF THE DRAWINGS
[0018] Figure 1 The steps of the core data identification method according to one embodiment of the present invention are as follows;
[0019] Figure 2 It is a module composition of a core data identification system according to an embodiment of the present invention. DETAILED DESCRIPTION
[0020] The specific implementation modes of the present invention are described in detail below in conjunction with the accompanying drawings, but it should be understood that the protection scope of the present invention is not limited by the specific implementation modes.
[0021] Unless explicitly stated otherwise, throughout the specification and claims, the term “comprise” or variations such as “include” or “comprising”, etc., will be understood to include the stated elements or components but not to exclude other elements or components.
[0022] In order to overcome the problems of the prior art, the present invention provides a core data identification method and system, which constructs a core data identification model, based on which core data can be quickly and automatically identified.
[0023] Figure 1 The steps of the core data identification method according to an embodiment of the present invention are as follows: Step S1 to Step S5.
[0024] In step S1, the evaluation factors of core data are determined. Core data is a small part of data that is crucial to the success of the company's continued operation. The purpose of core data identification is to extract effective and valuable data from a large data group, carry out data management, and improve data management effectiveness. Core data can be judged by the following standards: 1. Data coverage, which refers to the system scope covered by the data. The quality problems of such data may affect the development of multiple systems or businesses; 2. Business relevance of data. The quality problems of such data will lead to the inability to carry out business normally, such as equipment location information, personnel age information, etc.; 3. The degree of influence of data on decision-making, such as asset type, power consumption, user arrears information, etc., are all data used to calculate important indicators; 4. The degree of data sharing between systems, that is, the correlation between data and enterprise-level data models. 5. The security sensitivity of data, such as consistency with the list of highly sensitive data that the company focuses on supervising. Therefore, the evaluation factors include: the system scope covered by data, the business relevance of data, the degree of influence of data on decision-making, the correlation between data and enterprise-level data models, and the security sensitivity of data.
[0025] In step S2, various judgment factors of the core data and weights of the various judgment factors are divided according to the judgment factors of the core data to construct a core data recognition model.
[0026] Specifically, the evaluation factors of core data are parsed into specific judgment factors, including five factors: data coverage, business relevance, indicator relevance, data sharing and management relevance. The weight of each factor is determined to construct a core data identification model.
[0027] Among them, the core data recognition model is:
[0028]
[0029] Among them, M is the value of the core data coefficient, A is the value of the data coverage factor, f1 is the weight value of the data coverage factor, B is the value of the business relevance factor, f2 is the weight value of the business relevance factor, C is the value of the indicator relevance factor, f3 is the weight value of the indicator relevance factor, D is the value of the data sharing factor, f4 is the weight value of the data sharing factor, E is the value of the management relevance factor, and f5 is the weight value of the management relevance factor.
[0030] In step S3, the algorithm model of each judgment factor is determined. The algorithm model of each judgment factor includes: a semantic similarity algorithm model, a grammatical similarity algorithm model, and a corpus vector similarity algorithm model. Optionally, this step also includes managing the algorithm model, such as managing the semantic similarity algorithm model online, establishing a general interface for data model calls, and managing the input factors, weights, and outputs of each model according to the requirements of different algorithm models.
[0031] Specifically, the algorithm model is used to analyze the correlation between a data item and the keywords of each system, and the value of the data coverage factor of a data item is determined. For example, if a customer information table exists in multiple systems, it means that the coverage of this information is relatively high and needs to be paid special attention.
[0032] Based on the algorithm model, the similarity between a data item and a key business term is analyzed to determine the relevance of a data item and the key business, that is, the value of the business relevance factor.
[0033] Based on the algorithm model, the correlation between a data item and the key analysis indicator is analyzed to determine the degree of support of a data item for the key indicator, that is, the value of the indicator correlation factor.
[0034] Based on the algorithm model, the correlation between a data item and the SG-CIM model is analyzed to determine the sharing of the data item, that is, the value of the data correlation factor.
[0035] Based on the algorithm model, the association between a data item and the data in the negative list is analyzed to determine the security sensitivity of the data item, that is, the value of the management correlation factor.
[0036] In step S4, the data resource of the core data to be identified is imported.
[0037] Importing data resources of the core data to be identified includes: directly importing data resources of the core data to be identified from the powerdesigner data import interface, the erwin data import interface, the xml data import interface, the word data import interface, or the excel data import interface, and directly obtaining metadata information from the database through the oracle data interface, the mysql data interface, or the db2 data interface, and storing the metadata information as the data resources of the core data to be identified in a centralized warehouse; establishing a data resource directory, classifying various types of imported data resources, and facilitating users to query the data in a visual manner.
[0038] In step S5, core data of the imported data resources are identified according to the core data identification model and the algorithm model.
[0039] Specifically, identifying the core data of the imported data resources according to the core data identification model and the algorithm model includes: testing the constructed core data identification model, and correcting the coefficients of the core data identification model according to the test results; and identifying the core data of the imported data resources based on the corrected core data identification model.
[0040] Among them, the testing of the constructed core data recognition model includes: selecting a data sample set, judging the core data therein through business, as a data sample; using the core data recognition model to calculate the selected data sample set, comparing and analyzing the calculation results with the data sample, and testing the accuracy of core data recognition.
[0041] Optionally, in one embodiment, after the core data is identified, it also includes: establishing a classification label system for core data according to management requirements, such as dividing it into core data, important data, general data, etc., formulating classification standards and management requirements for each type of data, and establishing the relationship between the core data label and the core data identification algorithm model to facilitate the upgrade and adjustment of the algorithm. If it is divided into core data, important data, and general data, the data with a score of 0.7 or above calculated by the model is core data, which should be the data with the first priority of data management, and should be focused on and managed first; the data with a score between 0.4 and 0.7 calculated by the model is important data, which should be the data with the second priority of data management; the data with a score below 0.4 calculated by the model is general data, which should be the data with the third priority of data management. In addition, the data range that needs to be identified and managed is selected in the enterprise data resource directory, and the core data identification is started. The system automatically starts multi-threaded concurrent processing, identifies the relationship between related data, core data labels, etc., clarifies which data in each part of the data belongs to core data and which data belongs to important data, etc., and establishes clear labels to facilitate the company's subsequent data management work.
[0042] Based on the same inventive concept, Figure 2 The illustrated embodiment also provides a core data identification system, which includes: an evaluation factor determination module 10 , an identification model construction module 11 , an algorithm model determination module 12 , an import module 13 , and an identification module 14 .
[0043] The evaluation factor determination module 10 is used to determine the evaluation factors of the core data, wherein the evaluation factors include: the system scope covered by the data, the business relevance of the data, the impact of the data on the decision, the relevance of the data to the enterprise-level data model, and the security sensitivity of the data.
[0044] The identification model building module 11 is coupled with the evaluation factor determination module 10, and is used to divide the core data into various judgment factors and the weights of the various judgment factors according to the evaluation factors of the core data, so as to build a core data identification model. The judgment factors include: data coverage factor, business relevance factor, indicator relevance factor, data sharing factor and management relevance factor.
[0045] The core data identification model is:
[0046]
[0047] Among them, M is the value of the core data coefficient, A is the value of the data coverage factor, f1 is the weight value of the data coverage factor, B is the value of the business relevance factor, f2 is the weight value of the business relevance factor, C is the value of the indicator relevance factor, f3 is the weight value of the indicator relevance factor, D is the value of the data sharing factor, f4 is the weight value of the data sharing factor, E is the value of the management relevance factor, and f5 is the weight value of the management relevance factor.
[0048] The algorithm model determination module 12 is used to determine the algorithm model of each judgment factor. The algorithm model of each judgment factor includes: a semantic similarity algorithm model, a grammatical similarity algorithm model, and a corpus vector similarity algorithm model.
[0049] The import module 13 is used to import the data resources of the core data to be identified. Specifically, the import module 13 is used to directly import the data resources of the core data to be identified from the powerdesigner data import interface, the erwin data import interface, the xml data import interface, the word data import interface, or the excel data import interface, and directly obtain metadata information from the database through the oracle data interface, the mysql data interface, or the db2 data interface, and store the metadata information as the data resources of the core data to be identified in the centralized storage library; establish a data resource directory to classify the various types of imported data resources.
[0050] The identification module 14 is coupled with the identification model construction module 11, the algorithm model determination module 12, and the import module 13, and is used to identify the core data of the imported data resources according to the core data identification model and the algorithm model. Specifically, the identification module 14 includes: testing the constructed core data identification model, and correcting the coefficients of the core data identification model according to the test results; and identifying the core data of the imported data resources based on the corrected core data identification model.
[0051] Optionally, in one embodiment, the system is also used to: establish a classification label system for core data according to management requirements, such as core data, important data, general data, etc., formulate classification standards and management requirements for each type of data, and establish the relationship between core data labels and core data identification algorithm models to facilitate the upgrade and adjustment of the algorithm. If divided into core data, important data, and general data, data with a score of 0.7 or above calculated by the model is core data, which should be the first priority data of data management, and should be focused on and managed first; data with a score between 0.4 and 0.7 calculated by the model is important data, which should be the second priority data of data management; data with a score below 0.4 calculated by the model is general data, which should be the third priority data of data management. And select the data range that needs to be identified and managed in the enterprise data resource directory, start core data identification, and the system automatically starts multi-threaded concurrent processing, identifies the relationship between related data, core data labels, etc., clarifies which data in each part of the data belongs to core data and which data belongs to important data, etc., and establishes clear labels to facilitate the company's subsequent data management work.
[0052] Based on the same inventive concept, an electronic device is also provided in one embodiment, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of the core data identification method described in any of the above embodiments are implemented.
[0053] Based on the same inventive concept, an embodiment further provides a non-transitory computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the core data identification method described in any of the above embodiments are implemented.
[0054] In summary, according to the core data identification method and system of this embodiment, a core data identification model is innovatively proposed in the power grid industry, and the key factors and weight settings of core data identification are clarified. Based on the core data identification model, the core data can be quickly and automatically identified. It realizes efficient and accurate identification of core data of various professions and systems, accurately defines the work objects and scope of data management, and helps enterprises invest limited resources in the management of core data, greatly reducing data management costs and improving management effectiveness. Moreover, feasibility verification can be carried out through the data of power grid companies, which is of great significance to promoting the development of industry data management.
[0055] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.
[0056] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0057] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0058] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.
[0059] The foregoing description of specific exemplary embodiments of the present invention is for the purpose of illustration and demonstration. These descriptions are not intended to limit the present invention to the precise form disclosed, and it is clear that many changes and variations can be made based on the above teachings. The purpose of selecting and describing the exemplary embodiments is to explain the specific principles of the present invention and its practical application, so that those skilled in the art can realize and utilize various different exemplary embodiments of the present invention and various different selections and changes. The scope of the present invention is intended to be limited by the claims and their equivalents.
Claims
1. A core data identification method, characterized in that: include: Determine the evaluation factors of core data; According to the evaluation factors of the core data, the core data is divided into various judgment factors and the weights of the various judgment factors, so as to construct a core data identification model; Determine the algorithm model of each judgment factor; Import data resources of core data to be identified; Identify the core data of the imported data resources according to the core data identification model and the algorithm model; The evaluation factors include: the system scope covered by the data, the business relevance of the data, the impact of the data on decision-making, the relevance of the data to the enterprise-level data model, and the security sensitivity of the data; The judgment factors include: data coverage factor, business relevance factor, indicator relevance factor, data sharing factor and management relevance factor; Among them, identifying the core data of the imported data resources according to the core data identification model and the algorithm model includes: Testing the constructed core data recognition model, and correcting the coefficients of the core data recognition model according to the test results; Identifying core data of the imported data resources based on the modified core data identification model; Among them, the core data recognition model is: Wherein, M is the value of the core data coefficient, A is the value of the data coverage factor, f1 is the weight value of the data coverage factor, B is the value of the business relevance factor, f2 is the weight value of the business relevance factor, C is the value of the indicator relevance factor, f3 is the weight value of the indicator relevance factor, D is the value of the data sharing factor, f4 is the weight value of the data sharing factor, E is the value of the management relevance factor, and f5 is the weight value of the management relevance factor; The algorithm models of the various judgment factors include: semantic similarity algorithm model, grammatical similarity algorithm model, and corpus vector similarity algorithm model; Among them, the data resources for importing the core data to be identified include: Directly import the data resources of the core data to be identified from the powerdesigner data import interface, erwin data import interface, xml data import interface, word data import interface, or excel data import interface, and directly obtain metadata information from the database through the oracle data interface, mysql data interface, or db2 data interface, and store the metadata information as the data resources of the core data to be identified in the centralized warehouse; Establish a data resource directory and classify the various types of imported data resources; The testing of the constructed core data recognition model includes: Select a data sample set and use the business to determine the core data in it as a data sample; The core data recognition model is used to calculate the selected data sample set, and the calculation results are compared and analyzed with the data samples to test the accuracy of core data recognition; Wherein, determining the algorithm model of each judgment factor also includes online management of the semantic similarity algorithm model, establishing a general interface for data model calling, and managing the input factors, weights and outputs of each model according to the requirements of different algorithm models; Among them, after the core data is identified, it also includes: establishing a classification label system for core data according to management requirements, formulating classification standards and management requirements for each type of data, and establishing the association between core data labels and core data identification algorithm models to facilitate algorithm upgrades and adjustments.
2. A core data identification system, characterized in that: include: The judging factor determination module is used to determine the judging factors of the core data; An identification model building module, coupled with the evaluation factor determination module, is used to divide the core data into various judgment factors and the weights of the various judgment factors according to the evaluation factors of the core data, thereby building a core data identification model; An algorithm model determination module, used to determine the algorithm model of each judgment factor; Import module, used to import data resources of core data to be identified; An identification module, coupled with the identification model building module, the algorithm model determination module and the import module, for identifying core data of imported data resources according to the core data identification model and the algorithm model; The evaluation factors include: the system scope covered by the data, the business relevance of the data, the impact of the data on decision-making, the relevance of the data to the enterprise-level data model, and the security sensitivity of the data; The judgment factors include: data coverage factor, business relevance factor, indicator relevance factor, data sharing factor and management relevance factor; Among them, identifying the core data of the imported data resources according to the core data identification model and the algorithm model includes: Testing the constructed core data recognition model, and correcting the coefficients of the core data recognition model according to the test results; Identifying core data of the imported data resources based on the modified core data identification model; Among them, the core data recognition model is: Wherein, M is the value of the core data coefficient, A is the value of the data coverage factor, f1 is the weight value of the data coverage factor, B is the value of the business relevance factor, f2 is the weight value of the business relevance factor, C is the value of the indicator relevance factor, f3 is the weight value of the indicator relevance factor, D is the value of the data sharing factor, f4 is the weight value of the data sharing factor, E is the value of the management relevance factor, and f5 is the weight value of the management relevance factor; The algorithm models of the various judgment factors include: semantic similarity algorithm model, grammatical similarity algorithm model, and corpus vector similarity algorithm model; Among them, the data resources for importing the core data to be identified include: Directly import the data resources of the core data to be identified from the powerdesigner data import interface, erwin data import interface, xml data import interface, word data import interface, or excel data import interface, and directly obtain metadata information from the database through the oracle data interface, mysql data interface, or db2 data interface, and store the metadata information as the data resources of the core data to be identified in the centralized warehouse; Establish a data resource directory and classify the various types of imported data resources; The testing of the constructed core data recognition model includes: Select a data sample set and use the business to determine the core data in it as a data sample; The core data recognition model is used to calculate the selected data sample set, and the calculation results are compared and analyzed with the data samples to test the accuracy of core data recognition; Wherein, determining the algorithm model of each judgment factor also includes online management of the semantic similarity algorithm model, establishing a general interface for data model calling, and managing the input factors, weights and outputs of each model according to the requirements of different algorithm models; Among them, after the core data is identified, it also includes: establishing a classification label system for core data according to management requirements, formulating classification standards and management requirements for each type of data, and establishing the association between core data labels and core data identification algorithm models to facilitate algorithm upgrades and adjustments.
3. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps of the core data identification method as claimed in claim 1 are implemented.
4. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the core data identification method as claimed in claim 1 are implemented.