A college relationship data processing method and system based on lake warehouse integration

Abstract

The application relates to a college relationship data processing method and system based on a lake warehouse, which comprises the following steps: extracting original data of various business systems of colleges into a lake warehouse and dividing initial metadata into the lake warehouse, and storing labeled metadata into the lake warehouse; constructing a first edition college data standard mapping dictionary, and assimilating Chinese abbreviations in the data standard into model semantic category labels; performing semantic recognition on metadata without labels extracted into the lake warehouse, correcting metadata labeling based on the model semantic category labels, and updating and backfilling the metadata labeling into the metadata in the lake warehouse; based on the college metadata standard mapping dictionary in the data warehouse, mapping, integrating and constructing a hierarchical classified college data asset directory; and realizing data asset directory publishing and query searching. The application constructs a unified college metadata standard mapping data dictionary, realizes the association mapping between business systems and the unified metadata standard of different business systems and colleges, and constructs a unified data asset directory.

Description

Technical Field

[0001] This invention relates to the field of data governance technology, and in particular to a method and system for processing university relationship data based on lake-warehouse integration. Background Technology

[0002] In today's data-driven era, universities have built various business information systems in their informatization construction, such as student management systems, academic affairs management systems, research management systems, and asset management systems, accumulating a large amount of data. Due to the diverse characteristics of data in these systems in terms of type, structure, and source, data users find it difficult to cross departments and find the data they need in a one-stop manner. Furthermore, university leadership cannot have a comprehensive grasp of the university's data assets, resulting in serious "information silos" between university departments. This hinders the full realization of data value and impedes the coordinated development of university informatization. In response to the national education digitalization strategy and the requirements for intelligent and informatized university construction, effectively organizing university data and constructing a clear and orderly unified data asset catalog can help build a comprehensive digital architecture for universities. This facilitates a comprehensive understanding of the university's data assets for university leadership and management personnel, while also helping data users in relevant departments quickly and centrally locate the data they need, enabling more effective data insight and analysis. This provides strong support for data governance, offers efficient and convenient retrieval conditions for data sharing, and provides data support for precise management and decision-making.

[0003] However, current research and applications in building data catalogs are mostly concentrated in government, healthcare, and corporate sectors, and the construction methods are often simply summarizing and compiling metadata from existing business systems. Research on data catalog construction techniques applicable to universities is lacking. Furthermore, in metadata aggregation, most researchers have not yet addressed the practical problems that universities urgently need to solve: First, traditional data integration ETL tools can interfere with the connection to the original relational database of the business system during data extraction. Business system administrators are generally unwilling to directly open full data access to ETL, usually opening partial data interfaces through data views. This masks certain relationships between data, thus affecting the quality of the data asset catalog and data resource platform. Second, data synchronization from business systems to the data resource center is generally done using timed or real-time methods, but due to system upgrades, data updates, and changes in the requirements of the original business system, these methods are subject to change. Changes in metadata caused by this cannot be detected and reflected in the data asset catalog in a timely manner, usually requiring the business system software vendor to upgrade the data view or interface later. Secondly, due to the technical protection of business system software vendors, insufficient metadata information is not injected into the database table structure, resulting in the lack of semantic information in the metadata information obtained by the data resource center through traditional ETL methods, which increases the difficulty of building the data asset catalog. In addition, due to the different data resource standards and specifications in different systems of universities, semantic conflicts such as homonyms and synonyms occur, which cannot provide support for consistent data asset classification and consistent data metadata description across multiple business systems in universities.

[0004] Lake-warehouse integration technology fully combines the advantages of data lakes and data warehouses. It is built on the low-cost data storage architecture of data lakes and inherits the data processing and management functions of data warehouses. It can aggregate and store multiple data types from multiple business systems in a unified manner, realize seamless scheduling and management of data between the lake and the warehouse, and access, query and analyze data through a unified interface at the upper layer. It can effectively solve problems such as complex ETL logic, difficulty in changing metadata, and inconsistent metadata. Summary of the Invention

[0005] To address the problems existing in the aforementioned technologies, this invention provides a lake-warehouse integrated data processing method for universities. It extracts raw data from various university business systems into a lake warehouse and stores standardized metadata therein. Based on relationships, tagged metadata is stored in a lake warehouse association table. Metadata without tags is semantically intelligently identified through the data records contained in its column attributes. A university data standard mapping data dictionary is established, thereby constructing a unified university data asset catalog and enabling visualization and querying of data assets. This effectively overcomes the shortcomings of existing technologies, such as connection interference, inability to promptly detect changes in the original system data structure, inability to identify the meaning of metadata, and inconsistent data standards, thus constructing a clear and orderly unified data asset catalog for universities.

[0006] To address the issues arising during the metadata aggregation process, this invention utilizes a lake-warehouse integrated technology to extract raw structured data into the lake warehouse using initial stock extraction, continuous batch incremental extraction, and real-time methods. Historical data is loaded into the data lake all at once, establishing a complete foundation of university data and its relationships. Simultaneously, it maintains synchronization between the data in the data lake and the data source, ensuring data real-time performance and accuracy. A semantic recognition model is used to obtain detailed semantic information of metadata from data records contained in column attributes within the business system, achieving metadata consistency. A unified standard university data asset catalog is constructed to manage university data resources as data assets, facilitating maintenance, management, analysis, and mining. This releases data value, promotes data sharing, and achieves the goal of university data governance.

[0007] Terminology Explanation:

[0008] 1. CDM: Cloud Data Migration (CDM) is an efficient and easy-to-use batch data migration service. CDM focuses on big data migration to the cloud and intelligent data lake solutions, providing simple and easy-to-use migration capabilities and integration capabilities from multiple data sources to the data lake. This reduces the complexity of data source migration and integration for customers and effectively improves the efficiency of data migration and integration.

[0009] 2. CDL: CDL (Change Data Capture) is a simple, convenient, real-time and efficient data synchronization and integration service that can capture data from various OLTP data sources and push it to the Kafka message queue. Finally, the SinkConnector consumes the data in the Topic and transmits it to big data ecosystem software applications, thereby realizing real-time data entry into the Kafka ecosystem.

[0010] 3. DataHub: DataHub is a scalable metadata management platform open-sourced by the LinkedIn team. It provides an integrated data catalog and metadata management system designed to enable end-to-end data discovery, data observability, and data governance. This scalable metadata platform is built for developers to accommodate the complexity of their rapidly evolving data ecosystem and for data practitioners to leverage the total value of data within their organizations.

[0011] 4. Hudi: Hudi (Hadoop Upserts Delete and Incremental) is a data lake framework that supports insert, update, delete, and incremental processing. Apache Hudi directly integrates core repository and database functionalities into the data lake. Hudi provides tables, transactions, efficient upserts / deletes, advanced indexes, streaming ingestion services, data clustering / compression optimizations, and concurrency, while maintaining the open-source file format of the data.

[0012] The technical solution of this invention is as follows:

[0013] This invention provides a university data processing method based on lake-warehouse integration, comprising:

[0014] The raw data from various business systems of universities are extracted into the lake warehouse and the initial metadata is segmented and stored into the warehouse. Based on the relationship, the tagged metadata is stored into the lake warehouse.

[0015] Based on the GB / T 29808 national standard, an initial version of the university data standard mapping dictionary was constructed, and the Chinese abbreviations in its data standard were assimilated into model semantic category labels;

[0016] Based on the semantic recognition model, semantic recognition is performed on the unlabeled metadata extracted into the data warehouse. Based on the semantic category labels of the model, the metadata labels are corrected and updated back into the metadata labels of the data warehouse. The existing labels backfilled by the data warehouse association and the metadata labels after error correction are added to the standardized metadata descriptions that have been entered into the data warehouse and the university data standard mapping dictionary of the data warehouse.

[0017] Based on the standard mapping dictionary of university metadata in the data warehouse, a hierarchical and classified catalog of university data assets is mapped, integrated, and constructed.

[0018] Enables the publishing and searching of data asset catalogs.

[0019] Furthermore, in this method, the original data from various university business systems is extracted into a lake warehouse, and the initial metadata is segmented and stored in the warehouse. Tag-based metadata is then stored in the lake warehouse based on relationships; this includes:

[0020] First, the raw data from various university business systems is migrated to the data lake using the batch data migration tool CDM and the real-time data lake access tool CDL. Specifically, CDM is used to migrate structured data from the relational databases of the business systems to the data lake through initial stock extraction and batch incremental extraction. CDL is used to capture and parse the relational database change logs in real time, generating data records for insert, delete, and modify commands that operate on the data lake in real time, enabling real-time incremental migration of relational database data into the data lake. The migrated raw data is stored in MOR (Merge On Read) mode, using columnar format for immutable basic data files and row-format for newly added or modified incremental data files.

[0021] Secondly, the metadata of the databases of various business systems of universities is extracted using the metadata management tool DataHub as the initial metadata. The underlying database of the metadata management tool DataHub is then used to convert the initial metadata into a dictionary format, segment and standardize it, and connect it to Hudi. The metadata is stored in COW (Copy On Write) mode to realize real-time metadata loading.

[0022] Finally, based on the association relationship, the metadata with tags is stored in the Lake Warehouse Association Table Tag; the metadata with existing annotation information in some tables or views is backfilled into the metadata without annotation information in the related Lake Warehouse table fields using SQL statements through primary and foreign key association relationships and related view relationships, forming metadata annotation A.

[0023] Furthermore, in this method, an initial version of the university data standard mapping dictionary is constructed, and the Chinese abbreviations in the data standards are assimilated into model semantic category labels; including:

[0024] The initial version of the university data standard mapping dictionary is constructed based on the GB / T 29808 national standard. The university data standard mapping dictionary includes raw metadata and standardized metadata. The raw metadata is based on the initial metadata of the data entering the warehouse and is constructed in the form of raw field name, raw field description, raw field data type, raw character encoding, and raw sorting rule. The standardized metadata is based on GB / T 29808 and adds the corresponding standard definitions: the standardized metadata includes standard field number, standard field data item name, standard field Chinese abbreviation, standard field length, standard field type, standard field constraints, standard field values, and standard field examples.

[0025] The Chinese abbreviations in the data standard mapping dictionary are used as semantic category labels for the model. The semantic category labels are stored as strings in a list, and each semantic label is an independent string. The original data field names and standard definition names are associated with the university data standard mapping dictionary to realize the association mapping between the databases of various business systems of the university and the university metadata standards.

[0026] Furthermore, in this method, metadata semantic recognition and error correction include:

[0027] The unlabeled metadata extracted from the lake warehouse is represented as column data in the university business database that does not identify column descriptions, i.e., relational column data of the university business database. The relational column data records of the university business database are linearized and encoded, and input into the CSR (Column-Semantic-Recognition) column semantic recognition model proposed in this invention to learn relevant features and perform metadata semantic recognition, resulting in metadata label B. The recognized metadata label B and the original data label metadata label A are used to perform similarity analysis through text similarity calculation to correct metadata errors. The corrected metadata label is then updated and backfilled into the metadata label of the lake warehouse. Based on the association relationship, the backfilled metadata label stored in the lake warehouse is added to the description field of the initial segmented metadata that has been entered into the warehouse and the blank field of metadata label A in the constructed university data standard mapping dictionary. The corrected metadata label is then added to the description field of the initial segmented metadata that has been entered into the warehouse and the blank field of metadata label A in the constructed university data standard mapping dictionary.

[0028] Furthermore, metadata semantic recognition includes:

[0029] The relational column data of the university business database is concatenated row by row, and special identifiers are added for text marking and attribute encoding, outputting a preliminary column vector; the formula for the preliminary column vector matrix is ​​shown in Equation (I):

[0030]

[0031] In formula (Ⅰ), C represents the initial column vector for each column, n represents the total number of relational data columns, and C_Embed represents the n-dimensional initial column vector matrix encoded by attributes;

[0032] Contextual features are extracted from the sample vector, i.e., the initial column vector, using a multi-head self-attention mechanism.

[0033] The vector formula for the output incorporating learned features is shown in Equation (II):

[0034]

[0035] In formula (II), This represents the output column vector that has learned its own and contextual features, where n represents the total number of relational data columns, C_Embed represents the n-dimensional initial column vector matrix, Mu_Se_Atten represents the n-dimensional column matrix after learning contextual features, and C_Atten represents the n-dimensional column vector matrix after learning its own and contextual semantic features.

[0036] The output vector is passed through a fully connected MLP layer and normalized using the Softmax function to finally output the semantic category probability of each sample for prediction and classification.

[0037] The formula for the Softmax function is shown in equation (Ⅲ):

[0038]

[0039] In equation (Ⅲ), c i It is the i-th output column vector, and j represents the number of output column vectors. Softmax(c i () represents the probability distribution corresponding to the output of the normalization operation;

[0040] The formula for outputting semantic categories is shown in equation (IV):

[0041] Logit=Softmax(w2*tanh(w1*C_Atten+b1)+b2)(IV)

[0042] In Equation (IV), w1 and w2 represent the weight matrices used for linear combination, C_Atten represents the column vector matrix after learning semantics, b1 and b2 represent the bias parameters, tanh represents the hyperbolic tangent activation function used to learn nonlinear features, and Logit represents the final semantic category output of the CSR column semantic recognition model.

[0043] The objective of the CSR column semantic recognition model is shown in equation (V):

[0044]

[0045] In equation (V), y i c i Categorical variables, S j Represents the observed class truth value;

[0046] The cross-entropy loss function used by the CSR column semantic recognition model is shown in equation (VI):

[0047]

[0048] In equation (VI), n represents the number of semantic categories, and y iLet P(y) represent the predicted semantic category label variable for the i-th class. i |c1,...,c n ) represents the predicted probability of the i-th type of label, and Loss represents the difference between the predicted probability and the actual label.

[0049] Furthermore, metadata correction includes:

[0050] The cosine similarity algorithm is used to analyze the similarity between metadata label B and metadata label A. Labels with a similarity threshold of 0.9 are backfilled into the blank fields of metadata label A in the initial metadata label of the data warehouse segmentation and the university data standard mapping dictionary of the data warehouse. The cosine similarity formula is shown in Equation (VII).

[0051]

[0052] In equation (VII), x and y represent two vectors whose similarity is to be compared. i and y i Let x and y represent the components of vectors x and y, and let cosθ represent the similarity of the vectors after the dot product of the two vectors.

[0053] Furthermore, in this method, the steps for constructing the university asset catalog include:

[0054] First, define the subject areas and related attributes; based on the well-established university metadata standard mapping dictionary and the standardized metadata stored in the data warehouse, determine the subject areas, such as organization management, personnel management, teaching management, financial management, scientific research management, asset management, service management, etc.

[0055] Then, construct the data model; construct a corresponding data model for each subject area, which includes a fact table and dimension tables;

[0056] Subsequently, data tables are created in the data warehouse to store data for each subject area, and appropriate storage formats (such as COW or MOR) and configuration options are selected for each table;

[0057] Finally, load the relevant data from the fact table and dimension table into the corresponding data tables.

[0058] Before publication, the directory must be reviewed by relevant business and technical experts, and will be published only after approval. The directory search methods include simple search and advanced search. Simple search only requires entering keywords for the university data being searched, while advanced search requires setting multiple conditions for combined searches.

[0059] A computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the steps of a high-level data processing method based on lake-warehouse integration.

[0060] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of a lake-warehouse integrated high-level data processing method.

[0061] This invention provides a university data processing system based on lake-warehouse integration, used to implement the above-mentioned university data processing method based on lake-warehouse integration, including a data extraction module, a university data standard mapping dictionary construction module, a metadata semantic recognition and error correction module, a directory construction module, and a directory publishing module;

[0062] The data extraction module is used to extract the original data and initial metadata of various business systems of universities into the data warehouse.

[0063] The university data standard mapping dictionary construction module is used to construct a university data mapping standard dictionary based on the GB / T 29808 national standard and the original data entered into the lake by the data extraction module.

[0064] The metadata semantic recognition and error correction module is used to perform semantic recognition and error correction on the unlabeled metadata extracted into the lake warehouse by the data extraction module, and write the corrected metadata labels into the segmentation initial metadata and university data standard mapping dictionary.

[0065] The directory building module is used to build a directory from the error-corrected university data standard mapping dictionary stored in the university data standard mapping dictionary;

[0066] The directory publishing module is used to review and publish the data directories generated by the directory building module, and to provide data querying.

[0067] Furthermore, the data extraction module includes a raw data import module, a metadata extraction module, and a metadata storage module;

[0068] The raw data import module is used to migrate raw data from various university business systems, such as the student management system, academic affairs management system, scientific research management system, and asset management system, into the lake; it includes a batch data import migration module and a real-time data import access module.

[0069] The batch data migration module is used to migrate the original data from various business systems of universities into the lake through a full data migration once and a full data migration plus batch incremental migration.

[0070] The real-time data ingestion module is used to capture and parse the binlog logs of the original business system in real time to generate add, delete, and modify operation commands for real-time changed data, and to perform modification operations on the data records in the data lake, so as to realize the real-time ingestion of relational data into the data lake.

[0071] The metadata extraction module is used to extract metadata from the business system database using the metadata management tool DataHub, and connects to the underlying database of DataHub to convert the initial metadata into a dictionary format for segmentation and standardization, so as to realize the real-time entry of metadata into the lake.

[0072] The metadata storage module is used to store tagged metadata from the metadata extraction module into the lake warehouse based on the association relationship and store the metadata in the lake warehouse association table tag.

[0073] Furthermore, the university data standard mapping dictionary construction module includes a university data standard mapping dictionary management module and a university data standard mapping dictionary storage module;

[0074] The university data standard mapping dictionary management module is used to configure the original business system name, database name, table name, column name, column description, data type, character encoding, sorting rules, and string length information to manage the mapping dictionary; the university data standard mapping dictionary storage module is used to store model semantic category labels in the form of strings.

[0075] Furthermore, the metadata semantic recognition and error correction module includes a metadata semantic recognition module, a metadata error correction module, and a metadata correction module;

[0076] The metadata semantic recognition module is used to perform semantic recognition on the unlabeled metadata extracted into the lake warehouse by the metadata extraction module based on the CSR column semantic recognition model; the metadata error correction module is used to perform similarity analysis between the recognition results of the metadata semantic recognition module and the original data labels of the university data standard mapping dictionary construction module to correct the metadata errors; the metadata correction module is used to write the metadata annotations corrected by the metadata error correction module.

[0077] Furthermore, the directory construction module includes a subject domain definition module, a data model design module, a data directory storage module, and a data loading module;

[0078] The subject domain definition module is used to determine the subject domains of universities based on metadata and the corrected university data standard mapping dictionary; the data model design module is used to manage the data model designed for each subject domain; the data catalog storage module is used to create data tables to store the data of each subject domain based on the data model formed by the data model design module; and the data loading module is used to retrieve relevant data from the lake warehouse and load it into the corresponding data tables.

[0079] Furthermore, the directory publishing module includes a directory review module and a directory query module;

[0080] The catalog review module is used to enable relevant business and technical experts to review and manage the catalog, and publish it after approval; the catalog query module is used to enable data users to query the data.

[0081] The beneficial effects of this invention are as follows:

[0082] 1. By extracting raw business system data into a lake using batch and real-time methods, the meaning of unlabeled metadata in the university's global business system database within the integrated lake-warehouse system is intelligently identified through the data records contained in their column attributes. Traditional models only learn semantic features of column-level data and lack feature learning of contextual relationships at the relational table level. Furthermore, the column-level data features of relational tables are order-independent. If the model is highly sensitive to column order, the prediction accuracy decreases, leading to incorrect semantic category labels. Therefore, the CSR model proposed in this invention leverages the order-independent nature of multi-head self-attention mechanisms to effectively learn the interaction of similar attributes of relational column data and the contextual semantic feature information of relational table-level data. This method overcomes the problems of connection interference, inability to promptly detect changes in the original system's data structure, and inability to determine the meaning of metadata present in existing technologies.

[0083] 2. This invention constructs a unified metadata standard mapping data dictionary for universities, realizing the association mapping between different business systems and the unified metadata standard of universities, and constructing a unified data asset catalog. This invention forms a complete data asset catalog for universities, assisting university leaders and managers in understanding the unified data asset indicators across the campus, and helping data users in different business departments within the university to quickly locate the data they need, interpret the data, and realize the business value of data used collaboratively by multiple departments across universities. University management and decision-makers can effectively conduct data insights and analysis to unlock data value, promote data sharing, and achieve the goals of university data governance. Attached Figure Description

[0084] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0085] Figure 1 This is a flowchart illustrating the method for constructing a unified university relational data asset catalog according to the present invention.

[0086] Figure 2 This is a schematic diagram of the modules of the lake-warehouse integrated university relational data asset catalog construction system of the present invention;

[0087] Figure 3 This is a schematic diagram of the CSR column semantic recognition model framework proposed in this invention. Detailed Implementation

[0088] Obviously, the examples listed in the specific embodiments are only a part of the examples of this invention, and not all of them. All other examples obtained by those skilled in the art based on the examples of this invention without inventive effort should fall within the protection scope of this invention. The invention is further defined below with reference to the accompanying drawings and embodiments, but is not limited thereto.

[0089] Example 1

[0090] This invention provides a university data processing method based on lake-warehouse integration, such as... Figure 1 As shown, it includes:

[0091] The raw data from various business systems of universities are extracted into the lake warehouse and the initial metadata is segmented and stored into the warehouse. Based on the relationship, the tagged metadata is stored into the lake warehouse.

[0092] Based on the GB / T 29808 national standard, an initial version of the university data standard mapping dictionary was constructed, and the Chinese abbreviations in its data standard were assimilated into model semantic category labels;

[0093] Based on the semantic recognition model, semantic recognition is performed on the unlabeled metadata extracted into the data warehouse. Based on the semantic category labels of the model, the metadata labels are corrected and updated back into the metadata labels of the data warehouse. The existing labels backfilled by the data warehouse association and the metadata labels after error correction are added to the standardized metadata descriptions that have been entered into the data warehouse and the university data standard mapping dictionary of the data warehouse.

[0094] Based on the standard mapping dictionary of university metadata in the data warehouse, a hierarchical and classified catalog of university data assets is mapped, integrated, and constructed.

[0095] Enables the publishing and searching of data asset catalogs.

[0096] Example 2

[0097] The difference between the university data processing method based on lake-warehouse integration described in Example 1 and the method described in Example 1 is as follows:

[0098] The system extracts raw data from various university business systems into a data repository and segments initial metadata for storage. Based on relationships, it then stores tagged metadata into the data repository. This includes:

[0099] This embodiment analyzes nearly 300,000 pieces of actual data from the business systems of university departments and annotates some metadata. The analysis covers data resources from the information systems of 16 departments, including the Technology Transfer Office, Innovation and Entrepreneurship Center, Development and Planning Office, International Cooperation and Exchange Office, Finance Office, Academic Affairs Office, Graduate School, Research Department, Social Sciences Office, Human Resources Department, Talent Office, Youth League Committee, Alumni Office, Student Affairs Office, Organization Department, and Faculty Affairs Department. It includes 52 basic tables such as lists of invention patents and intellectual property rights, technology transfer, student entrepreneurship information, student competition information, major information, teaching and research projects, party branches, student party members, student honors, and student information.

[0100] First, raw data is extracted and imported into the data lake: Raw data from various university business systems is imported into the data lake using the batch data import migration tool CDM and the real-time data import access tool CDL. Specifically, CDM is used to migrate structured data from the relational databases of the business systems to the data lake through initial stock extraction and batch incremental extraction. CDL is used to capture and parse real-time change logs from the relational databases, generating data records for insert, delete, and modify commands that operate on the data lake in real time, achieving real-time incremental migration of relational database data into the data lake. The migrated raw data is stored in MOR (Merge On Read) mode, using columnar format for immutable basic data files and row format for newly added or modified incremental data files. This includes:

[0101] The Hudi data migration tool uses the Batch Data Migration (CDM) and Real-Time Data Access (CDL) tools to migrate data, converting the migrated data to the Parquet format supported by Hudi for storage. The data migration process is as follows:

[0102] S111: Use the Batch Data Migration (CDM) tool to configure the inbound connection information, configure the business system relational database as the source, and the HDFS / Hive directory as the destination;

[0103] S112: Use the Batch Data Migration (CDM) tool to perform the first stock extraction, start the task of importing the stock data of the relational database of the university business system into the lake, and use the Where command to filter and restrict the conditions for importing into the lake, with the time limit being from the initial migration time to the current time;

[0104] S113: Use the Batch Data Migration (CDM) tool to perform batch incremental data extraction, where the time in the WHERE command filter condition is adjusted to the migration time of the previous batch task up to the current time. Once all data before Ts has been migrated and extracted to the HDFS directory by the batch incremental task, stop the batch incremental extraction and migration task.

[0105] S114: Use the real-time data access tool (CDL) to configure the lake connection information, configure the business system relational database as the source end, and configure the Hudi file directory as the destination end;

[0106] S115: Run and start the real-time data access tool to achieve real-time incremental migration of relational database data into the lake. Capture and parse the binlog change logs of the relational database in real time, and call the Hudi interface to insert, delete, and modify relevant data records to the Hudi file directory in real time.

[0107] The raw data migrated into the lake is converted to Hudi-supported Merge On Read (MOR) format, using a hybrid approach of columnar file format (Parquet) and row file format (Avro) for data storage. Merge On Read uses columnar Parquet format to store immutable base files, while using row-based Avro format to store newly added or modified delta files. The delta files are associated with the base files, and a COMPACTION operation is performed in real time to merge the delta data into the columnar files according to a configurable strategy.

[0108] Secondly, the metadata management tool DataHub is used to extract metadata from the databases of various business systems in universities as initial metadata. The underlying database of DataHub then converts the initial metadata into a dictionary format, segments and standardizes it, and connects it to Hudi for storage in COW (Copy On Write) mode, enabling real-time metadata ingestion. Specifically, this includes:

[0109] The Datahub metadata management tool is used to extract metadata from the university's business system based on the connection information and related extraction configuration information. Database connection information includes: data source type (Kafka, Redshift, MySQL, etc.) and data source configuration (IP port number, database name, username, and password). Database extraction execution scheduling includes: minutes, hours, days, months, and grades, set according to the dynamic changes required by the business system. The name of the newly extracted metadata is set, and the Datahub metadata management tool is run to automatically extract metadata from the business system database. A Python program is used to convert the initial metadata to a dictionary format for standardization, removing extraneous characters, and connecting to Hudi to implement batch real-time data entry into the Datahub database. The extracted initial metadata is stored in a columnar file format using a COW table type. During the data writing process, each write operation reads the most recent complete copy, generates a new data file, and does not modify existing data files.

[0110] Finally, based on the association relationships, the metadata with tags is stored in the Lakewaregs associated table `Label`. Using primary and foreign key relationships, metadata with existing tags in some tables is used to populate the metadata without tags in the associated Hudi tables using SQL statements, forming tag A. Similarly, using the associated view relationships, metadata with existing tags in some views is used to populate the metadata without tags in the associated Hudi tables using SQL statements, forming metadata tag A.

[0111] In this method, an initial version of the university data standard mapping dictionary is constructed, and the Chinese abbreviations in the data standards are assimilated into model semantic category labels; including:

[0112] The initial version of the university data standard mapping dictionary is constructed based on the GB / T 29808 national standard. The university data standard mapping dictionary includes raw metadata and standardized metadata. The raw metadata is based on the initial metadata of the data entering the warehouse and is constructed in the form of raw field name, raw field description, raw field data type, raw character encoding, and raw sorting rule. The standardized metadata is based on GB / T 29808 and adds the corresponding standard definitions: the standardized metadata includes standard field number, standard field data item name, standard field Chinese abbreviation, standard field length, standard field type, standard field constraints, standard field values, and standard field examples.

[0113] In this embodiment, a preliminary version of the university data standard mapping dictionary is constructed based on the GB / T 29808 national standard, and its file format is stored in the data warehouse as a Hudi table. The university data standard mapping dictionary is designed according to actual business departments, such as constructing a university data standard mapping dictionary that includes data from 16 business departments, including the University Technology Transfer Office, Innovation and Entrepreneurship Center, Development and Planning Office, International Cooperation and Exchange Office, Finance Office, Academic Affairs Office, Graduate School, Scientific Research Department, Social Sciences Office, Human Resources Department, Talent Office, Youth League Committee, Alumni Office, Student Affairs Office, Organization Department, and Faculty Affairs Department.

[0114] Define standardized names for data items in the data tables and data fields of the business system. These standardized names will serve as the corresponding standard names for the metadata in the business system. The standard definition includes:

[0115] (1) Numbering: A globally unified four-segment code will be used as the unique identifier for college data items.

[0116] (2) Data item name: The first letter of the pinyin of the Chinese data item abbreviation is capitalized and used as the standard for university data.

[0117] (3) Chinese abbreviation: The name of the data element with semantic information in the user-oriented business system in universities, and its Chinese abbreviation is used as the semantic type label of the model.

[0118] (4) Length: The maximum number of characters that a data item can contain.

[0119] (5) Type: The data type contained in the data item

[0120] (6) Constraints: Description of the constraint status of a data item

[0121] (7) Value range: The range of values ​​for a data item

[0122] (8) Examples: Examples of data item attributes

[0123] The construction format of the university data standard mapping dictionary is as follows: original business system name, database name, table name, field name, field type, original field annotation A information, and the addition of the above standard definitions (number, data item name, Chinese abbreviation, length, type, constraint, value, example definition).

[0124] The Chinese abbreviations in the data standard mapping dictionary are used as semantic category labels for the model. These semantic category labels are stored as strings in a list, with each semantic label being an independent string. The original data field names and standard definition names are simultaneously associated with each other in the university's data standard mapping dictionary, achieving a mapping between the databases of various university business systems and the university's metadata standards. The mapping process includes:

[0125] S21: Use Hudi to load the standard mapping dictionary of college data. The standard mapping dictionary of college data is stored in a CSV file. The dictionary contains raw metadata and corresponding standard definition mapping value fields. The mapping dictionary is loaded into memory using a Map data structure.

[0126] S22: Read the initial metadata in Hudi. Read the extracted initial metadata from the Hudi dataset and use the API provided by Hudi to read the data.

[0127] S23: Association Mapping. For each extracted metadata, obtain the corresponding standard field value from the data standard mapping dictionary based on the original field value, and update the Hudi initial metadata based on the mapped standard field value.

[0128] In this method, metadata semantic recognition and error correction include:

[0129] The representation of the unlabeled metadata extracted into the lake warehouse is the column data described by the unlabeled columns in the data tables of the university business database, that is, the relational column data of the university business database; linearize and encode the records of the relational column data of the university business database, and input it into the CSR (Column-Semantic-Recognition) column semantic recognition model proposed by this invention to learn relevant features for metadata semantic recognition, and obtain metadata annotation B; use the identified metadata annotation B and the metadata annotation A of the original data label to perform similarity analysis through the text similarity calculation method to achieve metadata error correction, and update and fill back the corrected and recognized metadata annotation into the metadata annotation of the lake warehouse; supplement the filled-back metadata annotation stored in the lake warehouse to the description field of the segmented initial metadata already stored in the warehouse and the blank field of the metadata annotation A in the constructed university data standard mapping dictionary; supplement the corrected and recognized metadata annotation to the description field of the segmented initial metadata already stored in the warehouse and the blank field of the metadata annotation A in the constructed university data standard mapping dictionary.

[0130] Metadata semantic recognition includes:

[0131] Concatenate the attribute values of the columns to be predicted in the relational table row by row, regard it as a piece of text, and convert the sequence into tokens through the word-piece tokenization method. For example, for the relational column of "student name", the tokenized input text is "[CLS]Wu Yi, Gao Chengyan, Guo Liang,..., Zhang Hui[SEP]".

[0132] Select the open-source RoBERTa pre-trained model containing a Transformer encoder with 12 layers and 768 hidden layers as the basic framework for semantic recognition of this invention. Segment the original text into words through tokenization, and use words as the smallest processing unit. Use the tokenized text as the input of the RoBERTa model for vector representation, and for the output vector c corresponding to the identifier [CLS] i (1) As the preliminary column vector of each column in the relational table, the final output is an n-dimensional vector matrix.

[0133] The relational column data of the university business database is concatenated row by row, added with special identifiers for text marking and attribute encoding, and the preliminary column vector is output; the definition formula of the preliminary column vector matrix is shown in formula (Ⅰ):

[0134]

[0135] In formula (Ⅰ), This represents the initial column vector for each column, where n represents the total number of columns in the relational data, and C_Embed represents the n-dimensional initial column vector matrix encoded by attributes. Since RoBERTa has a limitation on input length, typically limited to 512 tags, a fixed input sequence length is chosen. For each column in the relational table with a different number of cells, its sequence needs to be truncated to the maximum sequence length set by RoBERTa. If the truncated length does not meet the model's limitation, zero-padding will be used to reduce the maximum sequence length.

[0136] Contextual features are extracted from the initial column vector obtained from the sample vector using a multi-head self-attention mechanism; including:

[0137] Using linearized data as input, and through a multi-head self-attention mechanism, the initial column vectors learn features at the column attribute level and contextual features at the relation table level.

[0138] The common features of the entire column of relational attribute values ​​and all attribute values ​​under the same topic in each table are used as the column attribute hierarchical context and relational table hierarchical context for feature extraction, respectively. For cases where features are extremely similar and semantic recognition cannot be performed using a single column attribute value, the common features of the table hierarchical context help determine the semantic information. For example, if two tables share the attribute feature of time, the semantic information in one table is "textbook publication date," while the semantic information in the other table is "scientific paper publication date," the actual semantic information can be distinguished by the common attributes of the relational table hierarchical context.

[0139] A multi-head attention mechanism is adopted, in which the input sequence is input as a query, key, and value into multiple attention heads. By calculating the attention scores between the query and the key, the values ​​are weighted and summed to obtain the contextual representation of each tag, which helps the model capture key information and semantic associations in the sequence, thereby extracting useful features.

[0140] The vector formula for the output incorporating learned features is shown in Equation (II):

[0141]

[0142] In formula (II), This represents the output column vector that has learned its own and contextual features, where n represents the total number of relational data columns, C_Embed represents the n-dimensional initial column vector matrix, Mu_Se_Atten represents the n-dimensional column matrix after learning contextual features, and C_Atten represents the n-dimensional column vector matrix after learning its own and contextual semantic features.

[0143] The output vector is passed through a fully connected MLP layer and normalized using the Softmax function to finally output the semantic category probability of each sample for prediction and classification.

[0144] A feed-forward neural network is employed, using a multilayer perceptron (MLP) network to introduce nonlinear transformations to assist in extracting more complex features and better capturing the semantic information of the text. A fully connected layer is used, taking the output of the previous layer as input, and finally, a tanh activation function is added to map it to a new feature space to learn nonlinear feature information. The resulting vector is then applied to a softmax normalization function for semantic label classification. The softmax function formula is shown in equation (Ⅲ).

[0145]

[0146] In equation (Ⅲ), c i It is the i-th output column vector, and j represents the number of output column vectors. Softmax(c i ) represents the probability distribution corresponding to the output of the normalization operation, indicating the probability that the sample belongs to the semantic category.

[0147] The formula for outputting semantic categories is shown in equation (IV):

[0148] Logit=Softmax(w2*tanh(w1*C_Atten+b1)+b2)(IV)

[0149] In Equation (IV), w1 and w2 represent the weight matrices used for linear combination, C_Atten represents the column vector matrix after learning semantics, b1 and b2 represent the bias parameters, tanh represents the hyperbolic tangent activation function used to learn nonlinear features, and Logit represents the final semantic category output of the CSR column semantic recognition model.

[0150] The objective of the CSR column semantic recognition model is shown in equation (V):

[0151]

[0152] In equation (V), y i c i The categorical variable, S j Represents the observed class truth value;

[0153] The cross-entropy loss function used by the CSR column semantic recognition model is shown in equation (VI):

[0154]

[0155] In equation (VI), n represents the number of semantic categories, and y i Let P(y) represent the predicted semantic category label variable for the i-th class. i |c1,...,c n ) represents the predicted probability of the i-th type of label, and Loss represents the difference between the predicted probability and the actual label.

[0156] like Figure 3 As shown, the network architecture of the CSR column semantic recognition model consists of three parts;

[0157] The first part is the text tagging and linearization encoding of relational data attribute columns. The relational columns of the university business database are used as input to the CSR column semantic recognition model. A 12-layer Roberta model based on the Transformer encoder architecture is used to encode each attribute column. Each relational attribute column is tokenized by splitting with "." and the relational attribute columns are concatenated into a text by row. The BERT model generates a vector representation of each token, extracts features from it, and selects the output vector corresponding to the special identifier [CLS] as the initial column vector.

[0158] The second part is context attribute feature extraction. The initial column vectors obtained in the first part are used to capture and learn the features of the relation attribute columns themselves and the context interaction features of the relation table in a loop using the 12 column attention heads of the Self-Attention mechanism with three column attention layers in the Transformer model. This further updates the input embedding vectors and parameters of the three-layer interaction.

[0159] The third part is the prediction and classification. First, the output vector, already incorporating semantic features, is passed through a fully connected layer in a multilayer perceptron network (MLP) and a tanh activation function is used to learn non-linear features. Second, a 1D convolutional layer is introduced to convolve the output of the previous layer to capture higher-level local features. Next, to extract the most important features from the convolutional layer output, global max pooling is performed to retain the most salient information in each feature channel. Finally, the pooled features are passed through a fully connected layer to the Softmax activation function to obtain the probability of each semantic category label corresponding to each relation attribute column vector for prediction and classification.

[0160] Metadata correction, including:

[0161] The prediction results of the CSR column semantic recognition model are used as the metadata label B and compared with the existing metadata label A through similarity algorithm to correct the metadata. The text similarity calculation adopts the cosine similarity algorithm. First, the text labels A and B are segmented, merged and deduplicated. The word frequency of each short text is calculated. Finally, the samples are vectorized and the cosine similarity is calculated. The two vectors are projected into a multi-dimensional space and the similarity difference between the two samples is plotted. The closer the cosine similarity value of the two samples is to 1, the more related the two vectors are.

[0162] The cosine similarity algorithm is used to analyze the similarity between metadata label B and metadata label A. Labels with a similarity threshold of 0.9 are backfilled into the blank fields of metadata label A in the initial metadata label of the data warehouse segmentation and the university data standard mapping dictionary of the data warehouse. The cosine similarity formula is shown in Equation (VII).

[0163]

[0164] In equation (VII), x and y represent two vectors whose similarity is to be compared. i and y i Let x and y represent the components of vectors x and y, and let cosθ represent the similarity of the vectors after the dot product of the two vectors.

[0165] Metadata correction: For each text annotation, the metadata annotation B that reaches the similarity threshold of 0.9 is updated and backfilled into the original metadata annotation of the data warehouse, and supplemented into the blank field of metadata annotation A in the constructed university metadata standard mapping dictionary.

[0166] In this method, based on the extracted metadata and the revised university metadata standard mapping dictionary, the standardized definitions of each table and field in the original business system are classified and integrated according to the university standard mapping dictionary based on the standard feature data of the subject domain, and a hierarchical unified data asset catalog of universities is constructed. The steps for constructing the university asset catalog include:

[0167] First, define the subject areas and related attributes. Based on the established university metadata standard mapping dictionary and the standardized metadata stored in the data warehouse, determine the subject areas, such as organization management, personnel management, teaching management, financial management, scientific research management, asset management, and service management. In this embodiment, based on the revised university metadata standard mapping dictionary and the initial metadata stored in the data warehouse, determine the university subject areas. This integrates 16 business systems and 55 relationship tables, and designs and divides the university data assets into seven major subject areas: organization, personnel, teaching, scientific research, finance, assets, and services. The subject area division includes:

[0168] (1) Organizational Management Domain: This includes information on secondary colleges, classes, research groups, and student organizations.

[0169] (2) Personnel Management Domain: including student information, teacher information, faculty and staff information, etc.

[0170] (3) Teaching management domain: including teaching plan information, student course selection information, student exam results information, textbook information, etc.

[0171] (4) Financial Management Domain: This includes accounting information, school operating fund income information, scientific research award information, etc.

[0172] (5) Research Management Domain: This includes information on research projects, research papers, and research teams.

[0173] (6) Asset Management Domain: This includes assets (teaching and research instruments, teaching platforms, etc.).

[0174] (7) Service Management Domain: including library information, dormitory information, enrollment information, employment information, etc.

[0175] Next, construct the data model; construct a corresponding data model for each subject area, the data model includes fact tables and dimension tables; for example, in the personnel management subject area, you can design a data model that includes fact tables and dimension tables, such as student fact tables, teacher fact tables, college dimension tables, and major dimension tables.

[0176] The student fact table contains basic student information and related metrics, such as student ID, name, gender, age, enrollment date, and academic performance.

[0177] Teacher Fact Table: Contains basic information about teachers and related indicator data, such as teacher ID, name, gender, professional title, and affiliated college.

[0178] College Dimension Table: Contains basic information about the college and its related attributes, such as college ID, college name, dean's name, establishment date, etc.

[0179] Major Dimension Table: Contains basic information about the major and its related attributes, such as major ID, major name, and affiliated college.

[0180] Then, use Hudi's API or command-line tools to create data tables in the Hudi data warehouse to store data for each subject area, and select the appropriate storage format (such as COW or MOR) and configuration options for each table;

[0181] Finally, load the relevant data from the fact table and dimension table into the corresponding data tables.

[0182] The specific steps to implement data asset catalog publishing and querying are as follows:

[0183] The data asset catalog must be reviewed by relevant business and technical experts before it can be released.

[0184] The data asset catalog offers two search methods: simple search and advanced search. A simple search requires entering keywords related to the university data being searched, such as "XS," to retrieve information related to XS, including basic information like the Chinese abbreviation "Student" and relevant data attributes. A complex search requires setting multiple conditions for combined queries, such as "XS" and "XM," to retrieve data in the asset catalog containing both "XS" and "XM." This data is presented as data asset cards, displaying the relevant information for the searched university.

[0185] Example 3

[0186] A computer device includes a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of a university data processing method based on lake-warehouse integration as described in Embodiment 1 or 2.

[0187] Example 4

[0188] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of a university data processing method based on lake-warehouse integration as described in Embodiment 1 or 2.

[0189] Example 5

[0190] This invention provides a university data processing system based on lake-warehouse integration, such as... Figure 2 As shown, the method for implementing the integrated lake and warehouse data processing method for universities in Embodiment 1 or 2 includes a data extraction module, a university data standard mapping dictionary construction module, a metadata semantic recognition and error correction module, a directory construction module, and a directory publishing module.

[0191] The data extraction module is used to extract the original data and initial metadata from various business systems of universities into the data repository.

[0192] The data extraction module includes a raw data import module, a metadata extraction module, and a metadata storage module;

[0193] The raw data import module is used to migrate raw data from various university business systems, such as the student management system, academic affairs management system, scientific research management system, and asset management system, into the lake; it includes a batch data import migration module and a real-time data import access module.

[0194] The batch data migration module is used to migrate raw data from various business systems of universities into the lake through a full data migration once and a full data migration plus batch incremental migration once.

[0195] The real-time data ingestion module is used to capture and parse the binlog logs of the original business system in real time to generate add, delete, and modify operation commands for real-time changed data, and to perform modification operations on the data records in the data lake, so as to realize the real-time ingestion of relational data into the data lake.

[0196] The metadata extraction module is used to extract metadata from the business system database using the metadata management tool DataHub, and connects to the underlying database of DataHub to convert the initial metadata into a dictionary format for segmentation and standardization, so as to realize the real-time entry of metadata into the lake.

[0197] The metadata storage module is used to store tagged metadata from the metadata extraction module into the lake warehouse based on the association relationship and store the metadata in the lake warehouse association table tag.

[0198] The module for constructing a standard mapping dictionary for university data is used to construct a standard mapping dictionary for university data based on the original data entered into the lake using the GB / T 29808 national standard and the data extraction module.

[0199] The module for constructing a university data standard mapping dictionary includes a university data standard mapping dictionary management module and a university data standard mapping dictionary storage module.

[0200] The university data standard mapping dictionary management module is used to configure the original business system name, database name, table name, column name, column description, data type, character encoding, sorting rules, and string length information to manage the mapping dictionary; the university data standard mapping dictionary storage module is used to store model semantic category labels in the form of strings.

[0201] The metadata semantic recognition and error correction module is used to perform semantic recognition and error correction on the unlabeled metadata extracted by the data extraction module into the lake warehouse, and write the corrected metadata labels into the initial metadata segmentation and university data standard mapping dictionary.

[0202] The metadata semantic recognition and error correction module includes a metadata semantic recognition module, a metadata error correction module, and a metadata correction module.

[0203] The metadata semantic recognition module is used to perform semantic recognition on the unlabeled metadata extracted into the lake warehouse by the metadata extraction module based on the CSR column semantic recognition model; the metadata error correction module is used to perform similarity analysis between the recognition results of the metadata semantic recognition module and the original data labels of the university data standard mapping dictionary construction module to correct the metadata; the metadata correction module is used to write the metadata annotations corrected by the metadata error correction module.

[0204] The directory building module is used to build a directory from the error-corrected university data standard mapping dictionary stored in the university data standard mapping dictionary;

[0205] The directory construction module includes a subject area definition module, a data model design module, a data directory storage module, and a data loading module;

[0206] The subject domain definition module is used to determine the subject domains of universities based on metadata and the corrected university data standard mapping dictionary; the data model design module is used to manage the data model designed for each subject domain; the data catalog storage module is used to create data tables to store the data of each subject domain based on the data model formed by the data model design module; and the data loading module is used to retrieve relevant data from the lake warehouse and load it into the corresponding data tables.

[0207] The directory publishing module is used to review and publish the data directories generated by the directory building module, and to provide data querying.

[0208] The directory publishing module includes a directory review module and a directory query module;

[0209] The catalog review module is used to enable relevant business and technical experts to review and manage the catalog, and publish it after approval; the catalog query module is used to enable data users to query the data.

Claims

1. A data processing method for universities based on lake-warehouse integration, characterized in that, include: The raw data from various business systems of universities are extracted into the lake warehouse and the initial metadata is segmented and stored into the warehouse. Based on the relationship, the tagged metadata is stored into the lake warehouse. Construct an initial version of the university data standard mapping dictionary, and assimilate the Chinese abbreviations in the data standards into model semantic category labels; Based on the semantic recognition model, semantic recognition is performed on the unlabeled metadata extracted into the data warehouse. Based on the semantic category labels of the model, the metadata labels are corrected and updated back into the metadata labels of the data warehouse. The existing labels backfilled by the data warehouse association and the metadata labels after error correction are added to the standardized metadata descriptions that have been entered into the data warehouse and the university data standard mapping dictionary of the data warehouse. Based on the standard mapping dictionary of university metadata in the data warehouse, a hierarchical and classified catalog of university data assets is mapped, integrated, and constructed. Enables the publishing and searching of data asset catalogs; In this method, metadata semantic recognition and error correction include: The unlabeled metadata extracted into the data repository is represented as column data from the university business database that does not have labeled column descriptions, i.e., relational column data from the university business database. This relational column data is linearly encoded and input into a CSR column semantic recognition model to learn relevant features for metadata semantic recognition, resulting in metadata label B. The identified metadata label B and the original data label's metadata label A are compared using text similarity calculation to correct metadata errors. The corrected metadata labels are then updated and backfilled into the data repository's metadata labels. Based on the association relationships, the backfilled metadata labels stored in the data repository are added to the description fields of the initial segmented metadata already in the repository and to the blank fields of metadata label A in the constructed university data standard mapping dictionary. The corrected metadata labels are then added to the description fields of the initial segmented metadata already in the repository and to the blank fields of metadata label A in the constructed university data standard mapping dictionary. Metadata semantic recognition, including: The relational column data of the university business database is concatenated row by row, and special identifiers are added for text marking and attribute encoding, outputting a preliminary column vector; the formula for defining the preliminary column vector matrix is ​​shown in equation (I): In formula (Ⅰ), This represents the initial column vector for each column, where n represents the total number of relational data columns, and C_Embed represents the n-dimensional initial column vector matrix encoded by attributes; Contextual features are extracted from the sample vector, i.e., the initial column vector, using a multi-head self-attention mechanism. The vector formula for the output incorporating learned features is shown in equation (II): In formula (II), This represents the output column vector that has learned its own and contextual features, where n represents the total number of relational data columns, C_Embed represents the n-dimensional initial column vector matrix, Mu_Se_Atten represents the n-dimensional column matrix after learning contextual features, and C_Atten represents the n-dimensional column vector matrix after learning its own and contextual semantic features. The output vector is passed through a fully connected MLP layer and normalized using the Softmax function to finally output the semantic category probability of each sample for prediction and classification. The formula for the Softmax function is shown in equation (Ⅲ): In formula (Ⅲ), Let be the i-th output column vector, and j represent the number of output column vectors. This represents the corresponding probability distribution of the output of the normalization operation; The formula for outputting semantic categories is shown in equation (Ⅳ): In formula (Ⅳ), , C_Atten represents the weight matrix used for linear combination, and C_Atten represents the column vector matrix after semantic learning. and represents the bias parameter, tanh represents the hyperbolic tangent activation function used to learn nonlinear features, and Logit represents the final semantic category output of the CSR column semantic recognition model; The objective of the CSR column semantic recognition model is shown in equation (V): In formula (V), express Category variables, Represents the observed class truth value; The cross-entropy loss function used by the CSR column semantic recognition model is shown in equation (VI): In equation (VI), n represents the number of semantic categories. This represents the predicted semantic category label variable for the i-th class. This represents the predicted probability of the i-th type of label, and Loss represents the difference between the predicted probability and the actual label. Metadata correction, including: The cosine similarity algorithm is used to analyze the similarity between metadata annotation B and metadata annotation A. Annotations with a similarity threshold of 0.9 are backfilled into the blank fields of metadata annotation A in the initial metadata annotation of the data warehouse segmentation and the university data standard mapping dictionary of the data warehouse. The cosine similarity formula is shown in equation (VII): In equation (VII), x and y represent two vectors whose similarity is to be compared. and Represents the components of vectors x and y. This represents the similarity of two vectors as a result of their dot product.

2. The method for university data processing based on lake-warehouse integration according to claim 1, characterized in that, In this method, raw data from various university business systems is extracted into a data repository and initial metadata is segmented and stored in the repository. Tag-based metadata is then stored in the data repository based on relationships. This includes: First, the raw data from various business systems of the university are migrated into the lake using the batch data migration tool CDM and the real-time data access tool CDL. Second, the metadata of the databases of various business systems of the university is extracted using the metadata management tool DataHub as the initial metadata. The underlying database of the metadata management tool DataHub is then used to convert the initial metadata into a dictionary format, segment and standardize it, and connect it to Hudi for storage in COW mode, so as to realize the real-time entry of metadata into the warehouse. Finally, based on the association relationship, the metadata with tags is stored in the Lake Warehouse Association Table Tag; the metadata with existing annotation information in some tables or views is backfilled into the metadata without annotation information in the related Lake Warehouse table fields using SQL statements through primary and foreign key association relationships and related view relationships, forming metadata annotation A.

3. The method for university data processing based on lake-warehouse integration according to claim 1, characterized in that, In this method, an initial version of the university data standard mapping dictionary is constructed, and the Chinese abbreviations in the data standards are assimilated into model semantic category labels; including: Construct an initial version of the university data standard mapping dictionary; the university data standard mapping dictionary includes raw metadata and standardized metadata; the raw metadata is based on the initial metadata of the data entering the warehouse, and its construction format includes raw field name, raw field description, raw field data type, raw character encoding, and raw sorting rule; the standardized metadata includes standard field number, standard field data item name, standard field Chinese abbreviation, standard field length, standard field type, standard field constraints, standard field values, and examples of standard fields; The Chinese abbreviations in the data standard mapping dictionary are used as semantic category labels for the model. The semantic category labels are stored as strings in a list, and each semantic label is an independent string. The original data field names and standard definition names are associated with the university data standard mapping dictionary to realize the association mapping between the databases of various business systems of the university and the university metadata standards.

4. The university data processing method based on lake-warehouse integration according to claim 1, characterized in that, In this method, the steps for constructing the university asset catalog include: First, define the subject areas and related attributes; determine the subject areas based on the well-established university metadata standard mapping dictionary and the standardized metadata stored in the data warehouse. Then, construct the data model; construct a corresponding data model for each subject area, which includes a fact table and dimension tables; Subsequently, data tables are created in the data warehouse to store data for each subject area, and appropriate storage formats and configuration options are selected for each table; Finally, load the relevant data from the fact table and dimension table into the corresponding data tables.

5. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the university data processing method based on lake-warehouse integration as described in any one of claims 1-4.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the university data processing method based on lake-warehouse integration as described in any one of claims 1-4.

7. A university data processing system based on lake-warehouse integration, used to implement the university data processing method based on lake-warehouse integration as described in any one of claims 1-4, characterized in that, It includes a data extraction module, a university data standard mapping dictionary construction module, a metadata semantic recognition and error correction module, a directory construction module, and a directory publishing module; The data extraction module is used to extract the original data and initial metadata of various business systems of universities into the data warehouse. The university data standard mapping dictionary construction module is used to construct a university data mapping standard dictionary based on the GB / T 29808 national standard and the original data entered into the lake by the data extraction module. The metadata semantic recognition and error correction module is used to perform semantic recognition and error correction on the unlabeled metadata extracted into the lake warehouse by the data extraction module, and write the corrected metadata labels into the segmentation initial metadata and university data standard mapping dictionary. The directory building module is used to build a directory from the error-corrected university data standard mapping dictionary stored in the university data standard mapping dictionary. The directory publishing module is used to review and publish the data directories generated by the directory building module, and to provide data querying.

8. A university data processing system based on lake-warehouse integration according to claim 7, characterized in that, The data extraction module includes a raw data import module, a metadata extraction module, and a metadata storage module; The raw data import module is used to migrate raw data from various business systems of universities into the lake; it includes a batch data import migration module and a real-time data import access module. The batch data migration module is used to migrate the raw data from various business systems of universities into the lake. The real-time data ingestion module is used to capture and parse the binlog logs of the original business system in real time to generate add, delete, and modify operation commands for real-time changed data, and to perform modification operations on the data records in the data lake, so as to realize the real-time ingestion of relational data into the data lake. The metadata extraction module is used to extract metadata from the business system database using the metadata management tool DataHub, and connects to the underlying database of DataHub to convert the initial metadata into a dictionary format for segmentation and standardization, so as to realize the real-time entry of metadata into the lake. The metadata storage module is used to store tagged metadata from the metadata extraction module into the lake warehouse based on the association relationship and store the metadata in the lake warehouse association table tag.

9. A university data processing system based on lake-warehouse integration as described in claim 7 or 8, characterized in that, The university data standard mapping dictionary construction module includes a university data standard mapping dictionary management module and a university data standard mapping dictionary storage module; The university data standard mapping dictionary management module is used to configure the original business system name, database name, table name, column name, column description, data type, character encoding, sorting rules, and string length information to manage the mapping dictionary; the university data standard mapping dictionary storage module is used to store model semantic category labels in the form of strings. The metadata semantic recognition and error correction module includes a metadata semantic recognition module, a metadata error correction module, and a metadata correction module; The metadata semantic recognition module is used to perform semantic recognition on the unlabeled metadata extracted into the lake warehouse by the metadata extraction module based on the CSR column semantic recognition model. The metadata error correction module is used to perform similarity analysis between the recognition results of the metadata semantic recognition module and the original data tags of the university data standard mapping dictionary construction module to achieve metadata error correction; The metadata correction module is used to write the metadata annotations corrected by the metadata error correction module. The directory construction module includes a subject domain definition module, a data model design module, a data directory storage module, and a data loading module; The subject domain definition module is used to determine the subject domains of universities based on metadata and the corrected university data standard mapping dictionary; The data model design module is used to manage the data models designed for each subject area; The data catalog storage module is used to create data tables to store data for each subject area based on the data model formed by the data model design module; the data loading module is used to retrieve relevant data from the lake warehouse and load it into the corresponding data tables. The directory publishing module includes a directory review module and a directory query module; The catalog review module is used to enable relevant business and technical experts to review and manage the catalog, and publish it after approval; the catalog query module is used to enable data users to query the data.

Images