Data model construction method based on data value evaluation
Through the method based on data value evaluation, dynamic update of the data model is solved, and the problem that data models in the existing technology cannot dynamically adapt to business changes is improved, and the model update efficiency and intelligence are improved.
Patent Information
- Application Number
- CN202411492008.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-24
- Publication Date
- 2025-02-14
Smart Images
Figure CN119441182A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to data model construction technology in the field of big data, and in particular to a data model construction method based on data value assessment. Background Art
[0002] With the advancement of digitalization, traditional subject-oriented data model construction methods are increasingly unable to meet the requirements of efficiency, ease of use and responding to changes in business needs. Therefore, dynamic data model construction technology oriented to data value has become a hot topic.
[0003] Existing data model construction methods: 1) Aggregate multi-source heterogeneous data models, domain data models, coding tables and model data within the enterprise to build an original data model warehouse; 2) Build a basic data model warehouse after cleaning, extracting, classifying, reorganizing and numerically correcting the original data model; 3) Based on the basic data model warehouse, cut, link and calculate the model to build a thematic data model warehouse for different businesses; 4) Provide data services to meet the data needs of different business domains through the data in the thematic data model warehouse.
[0004] The problems and shortcomings of the existing technology are: Step 3 relies on the developer's understanding of business requirements and designs the data model in advance. After the designed data model is released and used, it cannot be changed dynamically. If the business requirements change, a new data model needs to be designed, developed and released.
[0005] Improvement of the existing technology: To solve this problem, the industry uses an MPP architecture database. This architecture database skips step 3), so that the business data requirements directly use the original data. This method reduces step 3), but also brings new problems. The database based on the MPP architecture has a large difference in data models and a large number of data elements in the data model, which makes it very difficult for the business side to use it. Summary of the invention
[0006] The purpose of this application is to solve the problems existing in existing data model construction technology and MPP architecture database. This application discloses a data model construction method based on data value assessment, which adjusts the data model design and change work that requires the participation of R&D personnel in the existing solution to a model construction technology based on data value; so that the subject data model used in the business domain can dynamically adapt to business changes based on the value of the data without being manually designed and set.
[0007] The purpose of this application is achieved through the following technical solutions:
[0008] A data model construction method based on data value assessment, the data model construction method comprising:
[0009] S1: Build a basic data model based on the MPP architecture database;
[0010] S2: Based on the basic data model, construct the initial topic data model;
[0011] S3: The business domain calls data and records data value assessment factors;
[0012] S4: Obtain observation data of data value assessment elements;
[0013] S5: training data value assessment model;
[0014] S6: Calculate the observation data obtained in S4 based on the data value evaluation model obtained in S5 to obtain the data value evaluation value of the corresponding model operation state;
[0015] S7: Based on the data value assessment, dynamically reconstruct or update the subject data model in the running state.
[0016] According to a preferred embodiment, step S1 includes: building a basic data model on an MPP architecture database. When the data required by the business cannot be served by the subject data model, the data request can be transmitted to the MPP architecture database, and the MPP architecture database provides data services.
[0017] According to a preferred implementation, step S2 includes: based on the basic data model warehouse, cutting, linking and calculating the model, and building an initial data model warehouse for different business themes.
[0018] According to a preferred implementation, step S3 includes: the business party uses query, statistics, and screening methods to obtain data required by the business based on the initial subject data model, and records element information of data value assessment.
[0019] According to a preferred embodiment, the element information of data value assessment in step S3 includes but is not limited to the following elements:
[0020] Time element, service element, construction element, storage element.
[0021] According to a preferred embodiment, the time element includes: several time points and time ranges; the service elements include: caller, number of calls, number of updates, number of hits and average response time; the construction elements include: construction time, number of freezes, number of retries, number of failures and bloodline path; the storage elements include: storage capacity and number of fields.
[0022] According to a preferred embodiment, step S4 includes: by feature engineering, using a feature signature hash function H, calculating the data item i of the data model m as a feature vector V, where V=H (data model identifier, data item identifier).
[0023] According to a preferred embodiment, in step S5, the data value assessment model K(V) is obtained by assessment based on the value assessment elements of each data model.
[0024] According to a preferred embodiment, in step S5, the process of training the data value assessment model includes:
[0025] Through historical data, the value of the K(V) model is: the more important the fitting time period, the more callers, the more calls, the more updates, the more hits, the shorter the average response time, the shorter the build time, the fewer freezes, the fewer retries, and the higher the value assessment.
[0026] The corresponding mathematical expectation expression is:
[0027]
[0028] Among them, P ij is the preset value of the j-th dimension variable of the i-th value factor, δ ij is the value parameter of the j-th dimension variable of the i-th value element, n represents the total number of data value element categories, k represents the total number of sub-items of the corresponding data value element category, and through model training, v of the f(V) formula of the fitting mathematical expectation is obtained. ij Default value.
[0029] According to a preferred embodiment, step S7 includes: when the data value assessment score is greater than a preset threshold value, the data item of the corresponding data model is retained or added; when the data value assessment score is less than the preset threshold value, the data item of the corresponding data model is discarded, thereby realizing that in the running state, the subject data model dynamically updates the data value assessment based on the data value assessment model K (V).
[0030] The aforementioned main scheme of the present application and its further options can be freely combined to form multiple schemes, all of which are schemes that can be adopted and claimed for protection in the present application. After understanding the scheme of the present application, those skilled in the art can understand that there are multiple combinations based on the prior art and common knowledge, all of which are technical schemes to be protected by the present application, and they are not exhaustively listed here.
[0031] Beneficial effects of this application:
[0032] In the field of industrial manufacturing, the data generated in the core stages of design, manufacturing, operation and maintenance, after being aggregated into the system, requires the intervention of R&D personnel (IT personnel) to design the model according to the data requirements of the business side, if there is a change in data requirements. IT personnel also need to redesign the data model and publish it. There is no solution that can be dynamically adjusted according to business needs. In the production and manufacturing scenario, the design data model generated in the design phase contains more than 200 data items. More than 300 data items are generated in the production and manufacturing phase, and many data items are also included in the operation and maintenance phase. How these data models can be dynamically and flexibly changed according to the actual business needs has become a very hot core topic at present.
[0033] This application adjusts the data model design and change work that requires the participation of R&D personnel in the existing solution to a model building technology based on data value; so that the subject data model used in the business domain can dynamically adapt to business changes based on the value of the data without manual design and setting.
[0034] This application supports dynamic expansion and contraction. The basis for contraction and expansion is to obtain the value of data through machine learning using the data value assessment model K(V), and then judge whether it is needed based on this value; thus, in the running state, the subject data model is dynamically updated based on the assessment of data value by the data value assessment model K(V). This improves the efficiency and intelligence of model updates and reduces the human involvement process. BRIEF DESCRIPTION OF THE DRAWINGS
[0035] Figure 1 This is a schematic diagram of the application process;
[0036] Figure 2 This is a schematic diagram of the data value assessment factors for this application. DETAILED DESCRIPTION
[0037] The following describes the embodiments of the present application through specific examples, and those skilled in the art can easily understand other advantages and effects of the present application from the contents disclosed in this specification. The present application can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present application. It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict.
[0038] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.
[0039] The present application discloses a data model construction method based on data value assessment, and the data model construction method includes the following steps.
[0040] Step S1: Build a basic data model based on the MPP architecture database.
[0041] Specifically, step S1 includes: building a basic data model on an MPP architecture database. When the data required by the business cannot be served by the subject data model, the data request can be transmitted to the MPP architecture database, and the MPP architecture database provides data services.
[0042] Step S2: Based on the basic data model, construct an initial topic data model.
[0043] Specifically, step S2 includes: data developers cut, link and calculate the model based on the basic data model warehouse, and build an initial data model warehouse for different business themes. The initial model design depends on the current business needs and the experience of developers.
[0044] Step S3: The business domain calls the data and records the data value assessment elements.
[0045] Specifically, step S3 includes: the business party uses query, statistics, and screening methods to obtain data required by the business based on the initial subject data model, and records element information of data value assessment.
[0046] Preferably, the element information of data value assessment in step S3 includes but is not limited to the following elements: time element, service element, construction element, and storage element.
[0047] Furthermore, the time elements include: several time points and time ranges; the service elements include: caller, number of calls, number of updates, number of hits and average response time; the construction elements include: construction time, number of freezes, number of retries, number of failures and bloodline path; the storage elements include: storage capacity and number of fields.
[0048] Step S4: Obtain observation data of data value assessment elements.
[0049] Step S4 includes: by feature engineering, using a feature signature hash function H, calculating the data item i of the data model m as a feature vector V, where V=H (data model identifier, data item identifier).
[0050] For example:
[0051] The data model identifier is: fact_design_data_doc_d;
[0052] The data item identification is: plating_coating;
[0053] Calculate the eigenvector:
[0054] V=H(fact_design_data_doc_d, plating_coating)
[0055] =06e65ef1ff434a2676d697421bad0a19.
[0056] Step S5: training data value assessment model.
[0057] Preferably, in step S5, the data value assessment model K(V) is obtained by assessment based on the value assessment elements of each data model.
[0058] In step S5, the process of training the data value assessment model includes: using a large amount of historical data to make the value of the K(V) model be: the more important the time period to be fitted, the more callers, the more calls, the more updates, the more hits, the shorter the average response time, the shorter the construction time, the fewer freezes, the fewer retries, and the higher the mathematical expectation of the value assessment.
[0059] The corresponding mathematical expectation expression is:
[0060]
[0061] Among them, P ij is the preset value of the j-th dimension variable of the i-th value factor, δ ij is the value parameter of the j-dimensional variable of the i-th value element, n represents the total number of data value element categories, and k represents the total number of sub-items of the corresponding data value element category. Through model training, the δ of the fitting mathematical expectation f(V) formula is obtained. ij Default value.
[0062] Step S6: Calculate the observed data obtained in S4 based on the data value evaluation model obtained in S5 to obtain the data value evaluation value of the corresponding model operation state. For each subject data model MODEL, calculate the data item value evaluation value of the model.
[0063] Step S7: Based on the data value assessment value, dynamically reconstruct or update the subject data model in the running state.
[0064] Specifically, if the data value evaluation score is greater than the preset threshold value, the data item of the data model is retained or added. If the data value evaluation score is less than the preset threshold value, the data item of the data model is discarded.
[0065] At this point, in the system operation state, the subject data model is dynamically updated based on the evaluation of data value by the data value evaluation model K(V).
[0066] In the field of industrial manufacturing, the data generated in the core stages of design, manufacturing, operation and maintenance, after being aggregated into the system, requires the intervention of R&D personnel (IT personnel) to design the model according to the data requirements of the business side, if there is a change in data requirements. IT personnel also need to redesign the data model and publish it. There is no solution that can be dynamically adjusted according to business needs. In the production and manufacturing scenario, the design data model generated in the design phase contains more than 200 data items. More than 300 data items are generated in the production and manufacturing phase, and many data items are also included in the operation and maintenance phase. How these data models can be dynamically and flexibly changed according to the actual business needs has become a very hot core topic at present.
[0067] This application adjusts the data model design and change work that requires the participation of R&D personnel in the existing solution to a model building technology based on data value; so that the subject data model used in the business domain can dynamically adapt to business changes based on the value of the data without manual design and setting.
[0068] This application proposes for the first time the data value assessment model K(V) and the assessment factors that the data value model relies on. It supports dynamic expansion and dynamic contraction. The basis for contraction and expansion is to obtain the value of the data through machine learning through the data value assessment model K(V), and then judge whether it is needed based on this value; thus, in the running state, the subject data model is dynamically updated based on the assessment of the data value by the data value assessment model K(V). It improves the efficiency and intelligence of model updates and reduces the human participation process.
[0069] The above description is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present application should be included in the protection scope of the present application.
Claims
1. A data model construction method based on data value assessment, characterized in that: The data model construction method comprises: S1: Build a basic data model based on the MPP architecture database; S2: Based on the basic data model, construct the initial topic data model; S3: The business domain calls data and records data value assessment factors; S4: Obtain observation data of data value assessment elements; S5: training data value assessment model; S6: Calculate the observation data obtained in S4 based on the data value evaluation model obtained in S5 to obtain the data value evaluation value of the corresponding model operation state; S7: Based on the data value assessment, dynamically reconstruct or update the subject data model in the running state.
2. The data model construction method according to claim 1, characterized in that: Step S1 includes: building a basic data model on an MPP architecture database. When the data required by the business cannot be served by the subject data model, the data request can be transmitted to the MPP architecture database, and the MPP architecture database provides data services.
3. The data model construction method according to claim 1, characterized in that: Step S2 includes: based on the basic data model warehouse, cutting, linking and calculating the model, and building an initial data model warehouse for different business themes.
4. The data model construction method according to claim 1, characterized in that: Step S3 includes: the business party uses query, statistics, and screening methods to obtain the data required by the business based on the initial subject data model, and records the element information of data value assessment.
5. The data model construction method according to claim 4, characterized in that: The element information of data value assessment in step S3 includes but is not limited to the following elements: time element, service element, construction element, and storage element.
6. The data model construction method according to claim 5, characterized in that: The time element includes: a number of time points and time ranges; The service elements include: caller, number of calls, number of updates, number of hits and average response time; Build factors include: build time, number of freezes, number of retries, number of failures, and lineage path; The storage elements include: storage capacity and number of fields.
7. The data model construction method according to claim 1, characterized in that: Step S4 includes: by feature engineering, using a feature signature hash function H, calculating the data item i of the data model m as a feature vector V, where V=H (data model identifier, data item identifier).
8. The data model construction method according to claim 1, characterized in that: In step S5, the data value assessment model K(V) is obtained by evaluating the value assessment elements of each data model.
9. The data model construction method according to claim 8, characterized in that: In step S5, the process of training the data value assessment model includes: Through historical data, the value of the K(V) model is: the more important the fitting time period, the more callers, the more calls, the more updates, the more hits, the shorter the average response time, the shorter the build time, the fewer freezes, the fewer retries, and the higher the value assessment. The corresponding mathematical expectation expression is: Among them, P ij is the preset value of the j-th dimension variable of the i-th value factor, δ ij is the value parameter of the j-dimensional variable of the i-th value element, n represents the total number of data value element categories, and k represents the total number of sub-items of the corresponding data value element category. Through model training, the δ of the f(V) formula fitting the mathematical expectation is obtained. ij Default value.
10. The data model construction method according to claim 1, characterized in that: Step S7 includes: when the data value assessment score is greater than the preset threshold value, the data item of the corresponding data model is retained or added; when the data value assessment score is less than the preset threshold value, the data item of the corresponding data model is discarded, thereby realizing that in the running state, the subject data model dynamically updates the data value assessment based on the data value assessment model K (V).