Concrete industry multi-bin construction method

Abstract

The invention discloses a concrete industry multi-warehouse construction method, which belongs to the technical field of data management, and specifically comprises the following steps: S1, establishing a three-layer architecture; s2, extracting metadata from a plurality of business systems of concrete; s3, standardizing the metadata; s4, extracting entities, entity attributes and entity relationships from the metadata, and building an entity model through class diagram modeling combined with the extracted data; s5, establishing a number bin model based on the entity model; s6, processing the data of the data warehouse model; s7, a data interface is established on the intelligent data service layer, and concrete industry counting warehouse construction is completed. Business data are combed by using a field modeling means, and an entity model is established through entities, entity attributes and entity relationships, so that the content of each process is standardized, business requirements can be accurately reflected, and the accuracy and practicability are improved; therefore, the warehouse counting can better meet the unique business requirements of the concrete industry, and surpasses the surface model construction of the traditional method.

Description

Technical Field

[0001] The present invention belongs to the technical field of data management, and in particular relates to a data warehouse construction method for the concrete industry. Background Art

[0002] In the context of the rapid evolution of information technology and the era of big data, enterprises in the concrete industry are increasingly aware of the urgency and necessity of data governance. At present, the data ecology of concrete enterprises covers multiple business nodes such as procurement, inventory, production scheduling, logistics distribution, marketing, and leadership decision-making. Data fragmentation is scattered in various independent systems, lacking coordination and integration with each other, resulting in the phenomenon of data islands. What is more serious is that non-programmed data modification behaviors occur frequently, and data deviations and tampering caused by human factors seriously restrict the full release and efficient use of data potential, weaken the ability to make scientific decisions, and become a key bottleneck restricting the digital transformation of enterprises.

[0003] The core challenges of data governance are composed of multi-dimensional data quality issues, which are as follows:

[0004] 1. Lack of data standardization: Due to the lack of unified and rigorous data definitions and standards, different understandings of the same data or one interpretation corresponds to different data, which makes it easy for inconsistencies and confusion to occur during data collection, storage and processing, which is the root cause of data quality degradation;

[0005] 2. The phenomenon of non-programmed data modification is serious: data deviation or even data tampering caused by human input and modification has damaged the accuracy and integrity of data, exacerbating the uncertainty of data quality. Summary of the invention

[0006] In order to solve the problems raised in the above background technology, the present invention provides a data warehouse construction method for the concrete industry to solve the serious problems of lack of data standardization and non-programmed data modification in data governance.

[0007] To achieve the above object, the present invention provides the following technical solutions:

[0008] A method for building a data warehouse in the concrete industry, comprising the following steps:

[0009] S1: Establish a three-layer architecture including the original data layer, business classification layer and intelligent data service layer;

[0010] S2: Data collection: extracting source data from multiple concrete business systems and importing them into the original data layer;

[0011] S3: Data standardization: clean and standardize the imported source data;

[0012] S4: Domain modeling: Extract entities, entity attributes and entity relationships from the structured data and unstructured data of the source data after data standardization through data definition language and natural language processing methods, combine the extracted data with class diagram modeling, and build an entity model;

[0013] S5: Data warehouse model creation: Based on the entity model, a conceptual model of the neuron framework for data analysis is established, the conceptual model is converted into a logical model, business fact tables and dimension tables are defined in the business classification layer, and mapping rules between the original data layer and the business classification layer are formulated. The logical model extracts, converts, and loads the data of the entity model into the business fact table and dimension table through the mapping rules;

[0014] S6: Data processing: In-depth processing of data at the business classification layer based on business needs and data asset management strategies;

[0015] S7: Intelligent data analysis and services; import deeply processed data into the intelligent data service layer and establish a data interface in the intelligent data service layer to complete the construction of the data warehouse for the concrete industry.

[0016] Preferably, S3 specifically includes the following steps:

[0017] S3.1: Conduct a comprehensive assessment of the data source, data format, and data quality of the source data. Based on the assessment results, identify and record defects in the data.

[0018] S3.2: Based on the evaluation results, develop data standards and write data standard documents;

[0019] S3.3: Clean, convert and normalize the data in the raw data layer according to the established data standards.

[0020] Preferably, S4 specifically comprises the following steps:

[0021] S4.1: Collect and analyze the specific needs of concrete industry companies’ businesses, and design the principles of data definition language to extract entities, attributes, and relationships of structured data in source data;

[0022] S4.2: Introduce natural language processing methods, use named entity recognition methods and relationship extraction methods in natural language processing methods to perform in-depth automatic analysis of unstructured data in source data, and extract entities, attributes and relationships;

[0023] S4.3: Conduct cluster analysis of business processes and divide and refine data into key business links;

[0024] S4.4: Identify the entities required in all business processes and build a modular data model;

[0025] S4.5: Use class diagram modeling combined with modular data modeling to complete the construction of the entity model of the data system.

[0026] Preferably, S5 specifically includes the following steps:

[0027] S5.1: Based on the entity model, design the conceptual model of the neuron framework for data analysis, determine the entity categories and dimensions that need to be stored in the neuron architecture, transform the conceptual model into a logical model, and formulate the mapping rules between the original data layer and the business classification layer;

[0028] S5.2: Refine the logical model into a physical framework and define business fact tables and dimension tables in the business classification layer;

[0029] S5.3: The logical model extracts, transforms, and loads data from the entity model into business fact tables and dimension tables through mapping rules.

[0030] Preferably, the deep processing methods in S6 include data aggregation, indicator calculation, data source association, one-hot encoding and machine learning algorithms.

[0031] Preferably, S7 specifically includes the following steps:

[0032] S7.1: Define the data mapping rules between the business classification layer and the intelligent data service layer, and convert the data of the business classification layer to the intelligent data service layer as needed according to the data mapping rules;

[0033] S7.2: Set up data interfaces in the intelligent data service layer, define access parameters, output data, and query logic of the data interfaces, and perform testing and optimization.

[0034] Compared with the prior art, the present invention has the following beneficial effects:

[0035] Traditional data warehouse model design often focuses on technical considerations and may ignore comprehensive considerations of business logic, resulting in a disconnect between the model and actual business needs. This application uses domain modeling to sort out business data, establishes entity models through entities, entity attributes, and entity relationships, and standardizes the content of each process to accurately reflect business needs, reduce manual input and modification processes, improve accuracy and practicality, and make the data warehouse more in line with the unique business needs of the concrete industry, surpassing the superficial model construction of traditional methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 This is a schematic diagram of the architecture of this application;

[0037] Figure 2 This is a schematic diagram of the data standardization process;

[0038] Figure 3Process diagrams created for domain modeling and data warehouse models;

[0039] Figure 4 Model a schematic diagram for the production area. DETAILED DESCRIPTION

[0040] In order to facilitate those skilled in the art to understand the technical content of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and specific examples. It should be understood that the specific examples described herein are only used to explain the present invention and are not used to limit the present invention.

[0041] Example 1

[0042] like Figure 1 As shown, a method for constructing and managing a data warehouse system based on the concrete industry is characterized by comprising:

[0043] A1: Establish a three-layer architecture including the original data layer, business classification layer and intelligent data service layer.

[0044] A2: Extract source data from various business systems in the concrete industry and import them into the original data layer to provide original data for subsequent processing, as follows:

[0045] Set up a data center, use the MySQL database and Flink processing framework, and use the distributed cluster composed of Broker nodes to connect to the data acquisition gateway deployed in the concrete plant to achieve data collection and synchronization.

[0046] A3: If Figure 2 As shown, evaluate the existing data status and industry data characteristics, formulate unified data standards, and perform data cleaning and data standardization, as follows:

[0047] Data features for assessing the concrete industry:

[0048] ① Data sources are diverse, including procurement, inventory, production scheduling, logistics distribution, marketing, quality control, supply chain, finance, equipment status, environmental monitoring and other fields;

[0049] ② Huge amount of data: The concrete production scheduling process involves a large number of data points, such as the material ratio, mixing time, temperature, humidity, etc. of each batch;

[0050] ③ Geographic dispersion: Concrete industry companies usually have multiple production plants or service areas, and data is collected from different locations, which is geographically dispersed;

[0051] ④ Cross-system compatibility: data may need to be exchanged between different information systems, such as ERP system, CRM system, SCM system, etc.;

[0052] Conduct a comprehensive audit of the data source, data format, data quality, etc. of the existing data. Based on the audit results, identify the main problems in the data, such as data duplication, missing values, inconsistency, etc., and record the differences between different data sources and different system data, such as field naming rules, encoding methods, etc.;

[0053] Based on the evaluation results and in combination with the specific needs of the concrete industry, formulate data standards, data definitions, data formats, coding rules, data quality rules, etc., and write detailed data standard documents to clarify the meaning, data type, and allowable value range of each data element, so as to ensure that the data is interpreted and used in a consistent manner throughout the enterprise;

[0054] According to the defined data standards, the data in the raw data layer is cleaned, converted and normalized, including:

[0055] Data cleaning: remove invalid data, duplicate records, outliers, and correct erroneous data;

[0056] Format unification: adjust all data formats to standard formats, such as unified date and time formats, unified numerical precision, etc.;

[0057] Naming standardization: Unify field naming according to the data dictionary and eliminate synonyms and heteronyms.

[0058] A4: Figure 3 As shown in the figure, the idea of ​​domain modeling is combined with the natural language processing (NLP) method to sort out the business process, optimize the neural network nodes of the data, and build the entity model of the data system. The details are as follows:

[0059] Collect and analyze the specific business needs of concrete industry companies, define core business areas based on domain-driven design (DDL) principles, and extract key business entities, attributes, and relationships;

[0060] According to DDL, core business areas include procurement, logistics, weighing, technology, production, sales, and settlement. Taking the production area as an example:

[0061] The production domain includes business entities such as production tasks, production technicians, and production equipment. The production task business entity includes the attributes of production task number, dispatcher, concrete quantity, mortar quantity, vehicle number, plan nature, production mode, mix ratio, grade, order, and creation time. The production technician business entity includes the attributes of personnel number, name, position, and working hours. The production equipment business entity includes mixer number, plant station, unit, volume per plate, production technician, and creation time. The relationship between production tasks and production equipment is that one or more production equipment is used for each production plan, and the relationship between production tasks and production technicians is that each production plan is executed by one or more production technicians.

[0062] Introducing natural language processing (NLP) technology, using named entity recognition (NER) and relation extraction (RE) to perform in-depth automated analysis of unstructured data such as business documents and user interaction records, and assisting in the refinement and optimization of domain models through semantic network analysis;

[0063] Among them, NER input is business documents and user interaction records, and NER output is the identified key entities, such as:

[0064] A sentence in the business document "Production task 18363-AFVD has been assigned to mixer 3-2-3232 and mixer 2-493 for operation, and technicians Zhang San and Li Si are designated to be jointly responsible for and supervise the completion of this task", sample output: production task number: 18363-AFVD, equipment number: 3-2-3232, equipment number: 2-493, technician: Zhang San, technician: Li Si; RE input is the entities and business documents identified in NER, and RE output is the relationship between entities, sample input: entity list and business document output by NER, sample output: production task number 18363-AFVD <association> equipment number 3-2-3232 <association> equipment number 2-493 <association> technician Zhang San <association> technician Li Si;

[0065] Based on the results of business process analysis, data is intelligently divided into key business links such as procurement, logistics, weighing, technology, production, sales, and settlement through cluster analysis;

[0066] Identify the data entities required in all business links, including contracts, orders, IoT devices, personnel, goods, materials, transportation tools, etc., and build a modular data model;

[0067] Use class diagrams to display each data entity and its attributes and relationships, including but not limited to association, aggregation, inheritance, etc., to complete the construction of the entity model of the data system. The modeling results in the production field are as follows: Figure 4 shown.

[0068] A5: Figure 3 As shown in the figure, based on the modeling results, a neuron framework for data analysis is built to extract, transform, and load standardized data into the business classification layer according to the model, as follows:

[0069] Based on the entity model, design the conceptual model of the neuron framework for data analysis, determine the data entities and dimensions that need to be stored in the neuron framework for data analysis, transform the conceptual model into a logical model, formulate the mapping rules from the original data layer to the business classification layer, and design data tables, index optimization and storage strategies;

[0070] Based on the logical model, it is further refined into a physical framework to define the business fact table;

[0071] For example, in the production field, a production-production schedule and a production-production process table are established. The fields defined in the production-production schedule are: site-factory station (varchar(255)), cement_agreement_of_construction_id-contract project ID (varchar(255)), produce_task_num-production task number (varchar(255)), a_duty_officer-Party A person in charge (varchar(255)), a_duty_officer_mobile-Party A person in charge contact number (varchar(255)), dispatcher_id-dispatcher ID (int4), concrete_volume-concrete volume (numeric(10,2)), mortar_volume-mortar volume (numeric(10,2)), sum_volume-cumulative volume (numeric(10,2)), sum_deliv _count total number of trips (varchar(255)), produce_plan_property_id-plan property (int4), production_model_type_id-production model (int4), produce_formula_id-mixing ratio ID (int4), create_time-creation time (timestamp(0)), update_time-update time (timestamp(0)), create_user_id-creator ID (varchar(255)), update_user_id-updater ID (varchar(255)), del_flag-logical deletion (int2), CPFZT-full name of label (varchar(255)), order_num-order number (varchar(255)), order_channel_code-order channel (varchar(255));

[0072] Create dimension tables and implement dimension update strategies around dimensions such as employees, equipment, plants, commodities, transportation tools, and contracts to ensure the timeliness and consistency of dimension information;

[0073] For example, the employee dimension table defines the following fields: site-factory site (varchar(255)), name-employee name (varchar(255)), phone-contact number (varchar(255), belong_ent_id-organization ID (varchar(255), employee_type-employee type (varchar(255), create_user_id-creator ID (varchar(255), update_user_id-modifier ID (varchar(255), create_time-creation time (timestamp(6)), update_time-modification time (timestamp(6));

[0074] The logical model extracts, transforms, and loads data from the entity model into business fact tables and dimension tables through mapping rules.

[0075] A6: According to business needs and data asset management strategies, the data at the business classification layer is deeply processed, and methods such as data aggregation, indicator calculation, data source association, data preprocessing, one-hot encoding, and machine learning algorithms are fully utilized to enhance the value of enterprise data. The details are as follows:

[0076] Clarify indicator requirements, including the types of indicators required, calculation methods, and business scenarios;

[0077] Design data aggregation logic based on demand, determine aggregation dimensions (such as daily, monthly, and plant-based) and metrics (such as output, sales, etc.), for example, aggregate output by day;

[0078] Determine the indicators that need to be calculated and write the calculation logic;

[0079] Associating data sources, such as associating logistics data with sales data to analyze the impact of logistics factors on sales;

[0080] Data cleaning, deleting duplicate records, processing outliers and missing values, and removing noise and outliers in the data;

[0081] Min-Max data normalization scales the data proportionally so that data of different magnitudes and units can be compared and processed on the same scale. The mathematical formula is as follows:

[0082] ;

[0083] Where: X is the original data value, is the minimum value in the data set, is the maximum value in the data set, is the normalized data value;

[0084] Use One-Hot Encoding to process categorical variables, identify categorical variables in the data (such as product category, region, etc.), and convert categorical variables into One-Hot Encoding form to facilitate subsequent data analysis and machine learning model processing;

[0085] Select appropriate machine learning algorithms (such as linear regression, decision tree, random forest, neural network, etc.) according to business needs and data characteristics, use preprocessed data to train machine learning models, adjust model parameters to optimize performance, and use cross-validation, accuracy, recall and other indicators to evaluate model performance after training to improve the intelligence level of business decision-making;

[0086] Take the demand in the production field as an example:

[0087] Demand: Use historical data to predict the concrete production volume for the next week so as to arrange production plans, allocate raw materials and human resources in advance;

[0088] Clarify indicator requirements;

[0089] Indicator type: concrete volume, Business scenario: predict future production plans based on historical production data;

[0090] Data aggregation logic design;

[0091] Aggregation dimension: Time dimension - by week, Measure: Output - amount of concrete;

[0092] Data cleaning;

[0093] Delete duplicate records and fill missing values ​​with the mean;

[0094] Data normalization;

[0095] Use Min-Max normalization to scale the data to the [0,1] interval;

[0096] In this demand, the normalized concrete production data for the first eight weeks of Plant A, Plant B, and Plant C are:

[0097] Plant A: [0.7407, 0.7778, 0.5926, 0.8889, 0.9630, 1.0000, 0.8148, 0.6667];

[0098] Plant B: [0.6667, 0.7037, 0.5926, 0.7778, 0.8333, 0.8519, 0.7778, 0.5556];

[0099] Plant C: [0.5926, 0.6296, 0.5185, 0.6667, 0.7037, 0.7407, 0.6296, 0.4815];

[0100] One-hot encoding;

[0101] Encode the plant name into a one-hot vector, [1,0,0] represents plant A, [0,1,0] represents plant B, and [0,0,1] represents plant C;

[0102] The linear regression model is used to predict demand. All data are used to train the model. The data from the first 7 weeks are used as the training set, and the data from the 8th week are used as the test set to calculate the mean square error (MSE) to evaluate the model performance. The prediction results are shown in Table 1. The coefficients and intercepts of the fitting result function are shown in Table 2. According to the prediction results, production tasks and raw material procurement are arranged in advance, and human resources are deployed together to improve operational efficiency and economic benefits.

[0103] Table 1 Prediction results of concrete production in the next week

[0104] Mean Squared Error 0.01129803106666667 Predicted production for FactoryA(Week9) 0.8055625 Predicted production for FactoryB(Week9) 0.719925 Predicted production for FactoryC(Week9) 0.62222

[0105] Table 2 Coefficients and intercepts of fitting results function

[0106] Coefficients Intercept FactoryA 0.08966 0.7159025 FactoryB 0.0040225 0.7159025 FactoryC -0.0936825 0.7159025

[0107] A7: Import the processed data into the intelligent data service layer, establish a data interface, and provide flexible data services and decision support, as follows:

[0108] Analyze data demands according to business needs, determine business usage scenarios and business goals, identify key business indicators and data dimensions, plan data logical architecture, and design intelligent data service layer models;

[0109] Define the data mapping rules between the business classification layer and the intelligent data service layer, and convert the data of the business classification layer to the intelligent data service layer as needed;

[0110] Define the access parameters, output data and query logic of the data interface, implement the intelligent data service interface, and perform testing and optimization;

[0111] In summary, the present invention proposes a data warehouse system construction and management method based on the concrete industry, which mainly solves the following problems:

[0112] Data dispersion and lack of standardization: Through data collection, standardized processing, domain modeling, and data warehouse model creation, data scattered in different systems are integrated, unified data standards are established, and data silos and consistency issues are resolved;

[0113] Data distortion and accuracy issues: Implementing data quality management and full-process quality control improves data accuracy and reliability, and solves the problem of misleading decisions caused by inaccurate and inconsistent data;

[0114] Insufficient data analysis capabilities: A data warehouse model and data service adapted to the characteristics of the concrete industry have been established to support complex data analysis and intelligent decision-making, making up for the limitations of simple report statistical analysis;

[0115] This application comprehensively improves the level of data governance, optimizes the entire chain from data collection to analysis and application, forms an efficient data governance system, and enhances the value of data assets of concrete enterprises.

Claims

1. A method for building a digital warehouse in the concrete industry, characterized in that: The following steps are involved: S1: Establish a three-layer architecture including the original data layer, business classification layer and intelligent data service layer; S2: Data collection: extracting source data from multiple concrete business systems and importing them into the original data layer; S3: Data standardization: clean and standardize the imported source data; S4: Domain modeling: Extract entities, entity attributes and entity relationships from the structured data and unstructured data of the source data after data standardization through data definition language and natural language processing methods, combine the extracted data with class diagram modeling, and build an entity model; S5: Data warehouse model creation: Based on the entity model, a conceptual model of the neuron framework for data analysis is established, the conceptual model is converted into a logical model, business fact tables and dimension tables are defined in the business classification layer, and mapping rules between the original data layer and the business classification layer are formulated. The logical model extracts, converts, and loads data from the entity model into the business fact tables and dimension tables through mapping rules; S6: Data processing: In-depth processing of data at the business classification layer based on business needs and data asset management strategies; S7: Intelligent data analysis and services; import deeply processed data into the intelligent data service layer and establish a data interface in the intelligent data service layer to complete the construction of the data warehouse for the concrete industry.

2. A method for building a data warehouse in the concrete industry according to claim 1, characterized in that: S3 specifically includes the following steps: S3.1: Conduct a comprehensive assessment of the data source, data format, and data quality of the source data. Based on the assessment results, identify and record defects in the data. S3.2: Based on the evaluation results, develop data standards and write data standard documents; S3.3: Clean, convert and normalize the data in the raw data layer according to the established data standards.

3. A method for building a data warehouse in the concrete industry according to claim 1, characterized in that: S4 specifically includes the following steps: S4.1: Collect and analyze the specific needs of concrete industry companies’ businesses, and design the principles of data definition language to extract entities, attributes, and relationships of structured data in source data; S4.2: Introduce natural language processing methods, use named entity recognition methods and relationship extraction methods in natural language processing methods to perform in-depth automatic analysis of unstructured data in source data, and extract entities, attributes and relationships; S4.3: Conduct cluster analysis of business processes and divide and refine data into key business links; S4.4: Identify the entities required in all business processes and build a modular data model; S4.5: Use class diagram modeling combined with modular data modeling to complete the construction of the entity model of the data system.

4. A method for building a data warehouse in the concrete industry according to claim 3, characterized in that: S5 specifically includes the following steps: S5.1: Based on the entity model, design the conceptual model of the neuron framework for data analysis, determine the entity categories and dimensions that need to be stored in the neuron architecture, transform the conceptual model into a logical model, and formulate the mapping rules between the original data layer and the business classification layer; S5.2: Refine the logical model into a physical framework and define business fact tables and dimension tables in the business classification layer; S5.3: The logical model extracts, transforms, and loads data from the entity model into business fact tables and dimension tables through mapping rules.

5. A method for building a data warehouse in the concrete industry according to claim 1, characterized in that: The deep processing methods in S6 include data aggregation, indicator calculation, data source association, one-hot encoding, and machine learning algorithms.

6. A method for building a data warehouse in the concrete industry according to claim 1, characterized in that: S7 specifically includes the following steps: S7.1: Define the data mapping rules between the business classification layer and the intelligent data service layer, and convert the data of the business classification layer to the intelligent data service layer as needed according to the data mapping rules; S7.2: Set up data interfaces in the intelligent data service layer, define access parameters, output data, and query logic of the data interfaces, and perform testing and optimization.

Images