A campus data governance management method and system based on machine learning

Abstract

The present invention provides a campus data governance management method and system based on machine learning, the method comprising responding to a data acquisition request from a data terminal and sending the request to a data middle station; triggering a data processing flow according to the data identification field and target interface configuration returned by the data middle station; performing structured processing on the original campus data through machine learning to generate governance target data; generating a governance strategy based on the governance target data and updating data metadata; if the terminal interface configuration is not deployed, associating the updated metadata with a preset governance field to generate a metadata set; updating the data middle station interface deployment parameters based on the metadata set; returning the data identification field to the terminal; responding to the terminal assignment operation, executing an interface call through the data middle station and returning the governance target data. The present invention can realize the automated governance and efficient circulation of campus data, and improve data quality, interface compatibility, and system response efficiency.

Description

Technical Field

[0001] The present invention relates to the field of data governance and machine learning technology. More specifically, the present invention relates to a campus data governance management method and system based on machine learning. Background Art

[0002] In today's digital age, campus informatization is rapidly advancing, accumulating vast amounts of data across various areas, including teaching, research, administration, and student life. This data is not only massive in volume but also comes from a wide range of sources and formats, including structured, semi-structured, and unstructured data. For example, teaching management systems store structured data such as student course grades and course selection records, semi-structured data such as library borrowing records, and unstructured data such as photos and videos of campus activities. Effective management of this data is crucial for improving campus management efficiency, optimizing teaching resource allocation, promoting scientific research innovation, and enhancing the student learning experience.

[0003] Traditional campus data governance methods primarily rely on manual operations or simple data processing tools. Manual governance is not only inefficient but also prone to errors, making it difficult to cope with the complex processing requirements of large-scale data. For example, when manually analyzing student performance data, data entry errors may occur due to negligence, which in turn affects subsequent decision-making. While simple data processing tools can perform some basic data cleaning and organization functions, they generally lack intelligent and automated capabilities, making it impossible to conduct in-depth data mining and analysis, making it difficult to meet the increasingly complex needs of campus data governance. For example, these tools cannot automatically identify outliers, duplicate records, and other issues in the data, nor can they automatically adjust data governance policies based on data characteristics.

[0004] With the continuous development of information technology, machine learning technology has been widely used in the field of data processing. Machine learning algorithms can automatically discover patterns and features in large amounts of data by learning and analyzing them, thereby enabling intelligent data processing and analysis. For example, in the financial sector, machine learning algorithms can be used for risk assessment and fraud detection. By learning from massive amounts of transaction data, they can automatically identify potentially risky transaction patterns. In the medical field, machine learning algorithms can be used for disease diagnosis and treatment recommendation. By learning from patient medical records and examination data, they can provide doctors with auxiliary diagnostic recommendations. However, the application of machine learning technology in campus data governance is relatively limited, and its potential to improve the efficiency and quality of campus data governance has not yet been fully realized.

[0005] In the process of implementing the embodiments of the present invention, the inventors found that there are at least the following problems or defects in the existing technology: traditional campus data governance methods cannot effectively respond to the complex governance needs of large-scale, multi-source heterogeneous data, resulting in low data governance efficiency and difficulty in ensuring data quality; existing methods lack intelligence and automation capabilities, cannot automatically adjust governance strategies according to data characteristics, and it is difficult to achieve deep mining and analysis of data; existing technologies have not fully utilized the advantages of machine learning technology and have failed to fully tap its potential in improving the efficiency and quality of campus data governance. Summary of the Invention

[0006] The present invention provides a campus data governance management method and system based on machine learning.

[0007] In a first aspect of the present invention, a campus data governance management method based on machine learning is provided, comprising:

[0008] In response to detecting data acquisition request information of the target data set from the data terminal, sending the acquisition request information of the target data set to the data middle station;

[0009] In response to receiving the data identification field information of the corresponding target data set sent by the data middle station, triggering the data processing flow of the data middle station according to the data identification field information and the target interface configuration information of the data middle station;

[0010] The original campus data corresponding to the data identification field information is structured through the data center. The structured processing includes data feature extraction based on machine learning algorithms to obtain governance target data.

[0011] Generate data governance policy information corresponding to the governance target data based on the governance target data;

[0012] According to the data governance policy information, the model parameters of each data metadata corresponding to the target data set are updated, wherein each data metadata includes metadata representing the data identifier;

[0013] In response to determining that the interface configuration information of the corresponding data terminal is not deployed, each updated data meta information is associated with the preset governance field configuration of the data middle platform to generate a meta information set;

[0014] Update the interface deployment parameters of the data center based on the preset data interface configuration information corresponding to the metadata set and the governance target data;

[0015] Sending the data identification field information of the associated target data set to the data terminal;

[0016] In response to detecting the data terminal's assignment operation on the preset governance field configuration of the preset data interface configuration information, the assigned preset governance field configuration is converted into an interface call parameter through the data middle platform, and the data interface call operation is executed to return the governance target data to the data terminal.

[0017] Furthermore, the data processing flow of the data center is triggered, including:

[0018] Extract the target field mapping rules corresponding to the target interface configuration information from the interface configuration library of the data center;

[0019] Inject the data identification field information into the parameter container corresponding to the target field mapping rule;

[0020] Execute parameterized interface calls through the interface call engine of the data center to obtain the original campus data associated with the data identification field information.

[0021] Furthermore, the structured processing includes:

[0022] Perform data quality verification on the original campus data to generate basic data that passes the verification;

[0023] Through the machine learning algorithm integrated in the data middle platform, the basic data is transformed into feature space to generate governance target data with standardized data structure characteristics.

[0024] Furthermore, the model parameter update process includes:

[0025] Generate a data governance feature vector through data governance policy information;

[0026] Input the data governance feature vector into the metadata update module of the data center to calculate the weight adjustment parameters of each data metadata;

[0027] The metadata storage structure of each data element information is updated according to the weight adjustment parameters.

[0028] Furthermore, the association process includes:

[0029] Align the updated data metadata with the preset governance field configuration for feature dimensions;

[0030] The semantic similarity parameters between data metadata and preset governance field configurations are calculated through the association relationship learning module of the data center.

[0031] A meta-information set with a hierarchical structure is constructed based on semantic similarity parameters.

[0032] Furthermore, the interface deployment parameter update includes:

[0033] Convert the metadata collection into the interface description file of the data center;

[0034] Compile the interface description file into an executable data interface configuration through a model-driven interface generator;

[0035] Deploy the data interface configuration to the interface service gateway of the data center.

[0036] Furthermore, the data quality verification process includes:

[0037] Obtain the field integrity parameters of the original campus data through the data verification engine of the data center;

[0038] When the field integrity parameter reaches the preset verification threshold, the data completion module of the data center is activated;

[0039] The data completion module performs interpolation calculations on missing fields to generate basic data that passes verification.

[0040] Furthermore, the feature space conversion process includes:

[0041] Call the dimensionality reduction algorithm library of the data center to perform principal component analysis on the basic data;

[0042] Through the standardization module of the data center, Z-score standardization calculation is performed on the data processed by principal component analysis;

[0043] Write the standardized calculation results into the feature repository of the data center to generate governance target data with standardized data structure characteristics.

[0044] Furthermore, the semantic similarity parameter construction includes:

[0045] The word vector generator of the data center converts the data metadata and the preset governance field configuration into a word vector set;

[0046] Based on the attention mechanism module of the data center, the cosine similarity matrix of the word vector set is calculated;

[0047] Through the graph neural network of the data center, a metadata set with a hierarchical structure is generated based on the cosine similarity matrix.

[0048] In a second aspect of the present invention, a campus data governance and management system based on machine learning is provided, comprising:

[0049] a detection module configured to, in response to detecting data acquisition request information of a data terminal for a target data set, send the acquisition request information of the target data set to the data middle station;

[0050] The interface trigger module is configured to trigger the data processing flow of the data middle station in response to receiving the data identification field information of the corresponding target data set sent by the data middle station, according to the data identification field information and the target interface configuration information of the data middle station;

[0051] The data processing engine is integrated into the data center and is configured to perform structured processing on the original campus data corresponding to the data identification field information, wherein the structured processing includes data feature extraction processing based on machine learning algorithms to obtain governance target data;

[0052] A policy generation module is configured to generate data governance policy information corresponding to the governance target data based on the governance target data;

[0053] A parameter updating module is configured to update model parameters for each piece of data metadata corresponding to the target data set according to the data governance policy information, wherein each piece of data metadata includes metadata representing the data identifier;

[0054] an association processing module configured to, in response to determining that the interface configuration information of the corresponding data terminal is not deployed, associate each updated data meta information with the preset governance field configuration of the data middle platform to generate a meta information set;

[0055] The interface deployment module is configured to update the interface deployment parameters of the data center based on the preset data interface configuration information corresponding to the metadata set and the governance target data;

[0056] A field transmission module configured to send data identification field information associated with a target data set to a data terminal;

[0057] The interface execution engine is integrated in the data center and is configured to respond to the detection of the data terminal's assignment operation on the preset governance field configuration of the preset data interface configuration information, convert the assigned preset governance field configuration into an interface call parameter, and execute the data interface call operation to return the governance target data to the data terminal.

[0058] The above embodiments of the present invention have at least the following beneficial effects:

[0059] 1. By using machine learning algorithms to automatically verify the data quality, transform the feature space, and perform structured processing on the original campus data, it can effectively solve the problems of low processing efficiency, strong subjectivity, and high error rate in traditional manual data governance methods, ensuring that the output data has standardized data structure characteristics and high-quality data content, providing a reliable foundation for subsequent data analysis and application.

[0060] 2. By establishing a dynamic association mechanism between data identification fields and preset governance fields, combined with model-driven interface automatic generation technology, it can effectively solve the problems of inconsistent multi-source heterogeneous data interface standards and poor compatibility that are common in campus information systems, realize automatic adaptation and efficient calling of data interfaces between different systems, and greatly improve the accuracy and real-time performance of cross-system data interaction.

[0061] 3. Through the intelligent calculation of data governance feature vectors and dynamic adjustment of metadata weights, combined with semantic similarity analysis and hierarchical structure construction technology, it can effectively solve the problem that traditional static data governance strategies are difficult to adapt to campus business changes and data feature evolution, so that data governance strategies can be automatically optimized and adjusted according to actual business needs and data feature changes, providing flexible and reliable data governance support for the continuous expansion and upgrading of campus informatization construction. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] The above and other objects, features and advantages of the exemplary embodiments of the present invention will become readily apparent by reading the following detailed description with reference to the accompanying drawings, in which several embodiments of the present invention are shown by way of example and not limitation, in which:

[0063] Figure 1 A schematic diagram of a process flow of a campus data governance management method based on machine learning provided by one embodiment of the present invention;

[0064] Figure 2 A schematic diagram of the structure of a campus data governance and management system based on machine learning provided by one embodiment of the present invention;

[0065] Figure 3 The figure schematically shows the structure of an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0066] The principles and spirit of the present invention will be described below with reference to several exemplary embodiments. It should be understood that these embodiments are provided solely to enable those skilled in the art to better understand and implement the present invention, and are not intended to limit the scope of the present invention in any way. Rather, these embodiments are provided to make the present invention more thorough and complete, and to fully convey the scope of the present invention to those skilled in the art.

[0067] Those skilled in the art will appreciate that the embodiments of the present invention may be implemented as a system, apparatus, device, method, or computer program product. Therefore, the present invention may be implemented in the following forms: entirely in hardware, entirely in software (including firmware, resident software, microcode, etc.), or in a combination of hardware and software.

[0068] It should be noted that any number of elements in the drawings is for illustration only and not for limitation, and any naming is only for distinction and does not have any limiting meaning.

[0069] Reference below Figure 1 , Figure 1 The flowchart of the campus data governance management method based on machine learning provided by one embodiment of the present invention is as follows. Figure 1 As shown, a campus data governance management method based on machine learning includes:

[0070] S1. In response to detecting a data acquisition request information of a target data set from a data terminal, sending the acquisition request information of the target data set to a data middle station;

[0071] S2. In response to receiving the data identification field information of the corresponding target data set sent by the data middle station, trigger the data processing flow of the data middle station according to the data identification field information and the target interface configuration information of the data middle station;

[0072] S3. Use the data center to perform structured processing on the original campus data corresponding to the data identification field information. The structured processing includes data feature extraction based on machine learning algorithms to obtain governance target data.

[0073] S4. Generate data governance policy information corresponding to the governance target data based on the governance target data;

[0074] S5. Based on the data governance policy information, update the model parameters of each data meta-information corresponding to the target data set, wherein each data meta-information includes meta-information representing the data identifier;

[0075] S6. In response to determining that the interface configuration information of the corresponding data terminal is not deployed, the updated data metadata is associated with the preset governance field configuration of the data center to generate a metadata set;

[0076] S7. Update the interface deployment parameters of the data center based on the preset data interface configuration information corresponding to the metadata set and the governance target data;

[0077] S8. Sending the data identification field information of the associated target data set to the data terminal;

[0078] S9. In response to detecting that the data terminal has performed an assignment operation on the preset governance field configuration of the preset data interface configuration information, the assigned preset governance field configuration is converted into an interface call parameter through the data middle platform, and the data interface call operation is executed to return the governance target data to the data terminal.

[0079] It should be noted that when a data terminal detects a data acquisition request for a target data set, the system will send the request information to the data middle station. The data terminal here refers to various devices or systems used to obtain data on campus, such as the academic affairs management system, library management system, etc., which can initiate requests for specific data sets. The target data set refers to the specific data set requested by the terminal, such as a student information data set or a course grade data set. The data middle station is the core platform for campus data governance, responsible for operations such as receiving requests, processing data, and returning results. This process realizes the initial transmission of data acquisition requests and lays the foundation for subsequent data processing.

[0080] Specifically, the data acquisition request information includes information such as the terminal's identity, the name of the required data set, and the specific parameters of the request. For example, when the educational management system, as a data acquisition terminal, requests a student information data set, it will carry its own system identification, data set name student information, and request parameters such as querying student information of a specific major. After receiving this information, the data middle station will parse and process it according to the preset interface configuration. Data identification field information refers to field information that can uniquely identify the target data set. For example, in a student information data set, the student ID field can be used as a data identification field. Target interface configuration information refers to the interface configuration related to the target data set in the data middle station, including the interface address, parameter format, etc. Through this information, the data middle station can trigger the corresponding data processing process to ensure the correct acquisition and processing of data.

[0081] Preferably, when the data processing flow of the data center is triggered, the target field mapping rules corresponding to the target interface configuration information can be extracted from the interface configuration library of the data center. For example, for requests for student information data sets, the interface configuration library will store the mapping relationship between fields such as student ID, name, major, and interface parameters. After injecting the data identification field information into the parameter container corresponding to the target field mapping rule, the interface call engine of the data center executes a parameterized interface call to obtain the original campus data associated with the data identification field information. This process ensures the accuracy and efficiency of data acquisition and provides a reliable data foundation for subsequent data governance.

[0082] In some embodiments, the data processing flow of triggering the data middle station includes:

[0083] Extract the target field mapping rules corresponding to the target interface configuration information from the interface configuration library of the data center;

[0084] Inject the data identification field information into the parameter container corresponding to the target field mapping rule;

[0085] Execute parameterized interface calls through the interface call engine of the data center to obtain the original campus data associated with the data identification field information.

[0086] It should be noted that when triggering the data processing flow of the data middle station, it is first necessary to extract the target field mapping rules corresponding to the target interface configuration information from the interface configuration library of the data middle station. The interface configuration library here is an important component of the data middle station. It stores all configuration information related to the data interface, including field mapping rules, interface addresses, parameter formats, etc. The target field mapping rules refer to the specific rules for mapping data identification field information with interface parameters. These rules define how to convert the field information in the request into a parameter form that the interface can recognize. Next, the data identification field information is injected into the parameter container corresponding to the target field mapping rule. This process ensures that the data identification field information can be correctly passed to the interface call engine. Finally, the interface call engine of the data middle station executes a parameterized interface call to obtain the original campus data associated with the data identification field information, thereby completing the triggering of the data processing process and providing basic data support for subsequent data governance.

[0087] Specifically, the data center is a platform that integrates data storage, processing, and management. It manages the configuration information of various data interfaces through the interface configuration library. The target field mapping rules in the interface configuration library are pre-set according to different data interface requirements. For example, when processing the student information data set, the target field mapping rules include mapping the student ID field to the interface parameter student_id, mapping the name field to the interface parameter name, etc. These mapping rules ensure the consistency and accuracy of the data format during transmission. The parameter container is a temporary storage area for storing and passing parameters. It can receive data identification field information and convert it into the parameter format defined by the target field mapping rule. The interface call engine is a module in the data center used to execute interface calls. It executes the corresponding interface call operation based on the parameters in the parameter container to obtain the required original campus data. These components work together to ensure the efficient and accurate triggering of the data processing process.

[0088] Preferably, when executing a parameterized interface call, the interface call engine will send a request to the data storage layer according to the parameter values in the parameter container, in accordance with the preset interface protocol and parameter format. For example, if the interface protocol is a RESTful API, the interface call engine will construct an HTTP request and send the parameters in the parameter container as query parameters or request body of the request. After obtaining the original campus data, the interface call engine will perform preliminary parsing and verification of the data to ensure the integrity and accuracy of the data. If data is found to be missing or in the wrong format, the interface call engine will trigger the data completion module, perform interpolation calculations on the missing fields, and generate basic data that has passed verification. This process not only improves the reliability of data acquisition, but also provides a high-quality data foundation for subsequent data governance.

[0089] In some embodiments, the structuring process includes:

[0090] Perform data quality verification on the original campus data to generate basic data that passes the verification;

[0091] Through the machine learning algorithm integrated in the data middle platform, the basic data is transformed into feature space to generate governance target data with standardized data structure characteristics.

[0092] It should be noted that structured processing is a series of operations performed on raw campus data, aiming to convert unstructured or semi-structured data into governance target data with standardized data structure characteristics. This process consists of two main steps: data quality verification and feature space conversion. Data quality verification checks the raw data to ensure its integrity and accuracy, generating verified basic data. Feature space conversion utilizes machine learning algorithms to process the basic data, extract data features, and generate standardized governance target data. This series of processes effectively improves data quality and usability, providing support for subsequent data analysis and applications.

[0093] Specifically, data quality verification processing refers to a series of inspection operations performed on the original campus data to ensure the integrity and accuracy of the data. This process usually includes checking the integrity of the data fields, whether the data type is correct, whether there are duplicate records, etc. For example, for the student information data set, check whether the student ID field is missing, whether the name field is empty, etc. Basic data refers to data that meets the basic quality requirements after data quality verification processing. These data are the basis for subsequent feature space conversion processing. Feature space conversion processing refers to the use of machine learning algorithms to process basic data, extract the features of the data, and convert it into governance target data with standardized data structure characteristics. This process usually includes operations such as data dimensionality reduction and standardization. Machine learning algorithms refer to a series of algorithms used for data processing that can automatically learn the characteristics and laws of data, such as the principal component analysis PCA algorithm for dimensionality reduction and the Z-score normalization algorithm for data standardization.

[0094] Preferably, during the data quality verification process, the field integrity parameters of the original campus data can be obtained through the data verification engine of the data center. If the field integrity parameters do not reach the preset verification threshold, the data completion module will be activated, and the interpolation calculation will be performed on the missing fields to generate basic data that has passed the verification. For example, for the missing grade field in the student information data set, the mean value interpolation method can be used to complete it. During the feature space conversion process, the dimensionality reduction algorithm library of the data center can be called to perform principal component analysis on the basic data to extract the main features of the data. Then, through the standardization module of the data center, the Z-score standardization calculation is performed on the data after principal component analysis, and the data is converted into standard normal distribution data with a mean of 0 and a standard deviation of 1. Finally, the standardized calculation results are written into the feature repository of the data center to generate governance target data with standardized data structure characteristics. This process not only improves the quality of the data, but also provides a standardized data foundation for subsequent data governance and analysis.

[0095] In some embodiments, the model parameter update process includes:

[0096] Generate a data governance feature vector through data governance policy information;

[0097] Input the data governance feature vector into the metadata update module of the data center to calculate the weight adjustment parameters of each data metadata;

[0098] The metadata storage structure of each data element information is updated according to the weight adjustment parameters.

[0099] It should be noted that the model parameter update process is the process of adjusting the various data metadata corresponding to the target data set based on the data governance policy information. The purpose of this process is to dynamically optimize the storage structure and weights of data metadata according to the governance policy to better reflect the importance and governance needs of the data. Among them, data governance policy information refers to a series of guiding information generated based on the governance target data, which is used to guide the specific operations of data governance. Data metadata refers to the basic unit information that constitutes the data set, including data identification, data type, data source, etc. The model parameter update process updates the metadata storage structure of the data metadata by calculating the weight adjustment parameters, thereby optimizing the effect of data governance.

[0100] Specifically, data governance policy information is generated through analysis of governance target data, and it contains a series of feature vectors and rules used to guide data governance. These feature vectors can represent the importance and relevance of data. The data governance feature vector is a vector generated based on the governance target data, which reflects the characteristics of the data and governance requirements. For example, for a student information dataset, the data governance feature vector includes the importance weights of fields such as student ID, name, and grades. The metadata update module is a module in the data center, responsible for calculating the weight adjustment parameters of each data metadata based on the data governance feature vector. These weight adjustment parameters are used to update the metadata storage structure of the data metadata, such as adjusting the storage location of the data metadata, optimizing the data access path, etc.

[0101] Preferably, in the process of updating model parameters, a data governance feature vector is first generated through data governance policy information. This process can be based on statistical analysis of the data, such as calculating the variance, mean and other statistics of each field to determine the importance of the field. Then, the data governance feature vector is input into the metadata update module of the data center, which calculates the weight adjustment parameters of each data metadata based on the feature vector. For example, for the grade field, if its variance is large, it means that the distribution of the grade data is more dispersed and requires a higher weight. Finally, the metadata storage structure of each data metadata is updated according to the weight adjustment parameters, such as adjusting the index structure of the data element to optimize data query efficiency. This process not only improves the flexibility of data governance, but also enhances the adaptability of data governance, and can dynamically adjust the storage and management methods of data elements according to different governance needs.

[0102] In some embodiments, the association process includes:

[0103] Align the updated data metadata with the preset governance field configuration for feature dimensions;

[0104] The semantic similarity parameters between data metadata and preset governance field configurations are calculated through the association relationship learning module of the data center.

[0105] A meta-information set with a hierarchical structure is constructed based on semantic similarity parameters.

[0106] It should be noted that association processing is the process of integrating updated data metadata with preset governance field configurations, with the goal of constructing a metadata set with a hierarchical structure. This process is achieved through feature dimension alignment and semantic similarity calculation to ensure semantic consistency and logical association between data metadata and governance field configurations. Feature dimension alignment refers to adjusting the feature dimensions of data metadata and governance field configurations to enable effective semantic similarity calculations. The semantic similarity parameter is a parameter that measures the degree of semantic similarity between data metadata and governance field configurations. The calculated similarity parameter can be used to construct a metadata set with a hierarchical structure.

[0107] Specifically, the updated data metadata refers to the data metadata after the model parameter update processing, which contains the latest weight adjustment parameters and the optimized storage structure. The preset governance field configuration refers to the field configuration pre-defined by the data center, which is used to guide the specific operations of data governance. These configurations include field name, type, weight and other information. Feature dimension alignment refers to adjusting the feature dimensions of the data metadata and the governance field configuration so that they can perform effective semantic similarity calculations. For example, if the feature dimensions of the data metadata are student number, name, and grades, and the feature dimensions of the governance field configuration are student ID, student name, and test scores, then the student number needs to be aligned with the student ID, the name with the student name, and the grades with the test scores. The semantic similarity parameter is obtained through calculation and is used to measure the degree of semantic similarity between the data metadata and the governance field configuration. For example, cosine similarity can be used to calculate the similarity between two vectors.

[0108] Preferably, during the association processing, the updated data metadata is first aligned with the preset governance field configuration for feature dimension. This process can be achieved through the association relationship learning module of the data middle station, which automatically identifies and aligns the feature dimensions of the data metadata and the governance field configuration according to preset rules and algorithms. Then, the data metadata and the governance field configuration are converted into a word vector set through the word vector generator of the data middle station. Next, based on the attention mechanism module of the data middle station, the cosine similarity matrix of the word vector set is calculated. Finally, through the graph neural network of the data middle station, a metadata set with a hierarchical structure is generated based on the cosine similarity matrix. This process not only improves the correlation between the data metadata and the governance field configuration, but also enhances the intelligence level of data governance. For example, for a student information dataset, by calculating the cosine similarity between the student number and the student ID, the name and the student name, and the grades and the test scores, a metadata set with a hierarchical structure can be constructed, thereby achieving efficient management and governance of the data.

[0109] In some embodiments, updating interface deployment parameters includes:

[0110] Convert the metadata collection into the interface description file of the data center;

[0111] Compile the interface description file into an executable data interface configuration through a model-driven interface generator;

[0112] Deploy the data interface configuration to the interface service gateway of the data center.

[0113] It should be noted that the interface deployment parameter update is the process of converting the metadata set into the interface description file of the data middle platform, compiling it into an executable data interface configuration through a model-driven interface generator, and finally deploying these configurations to the interface service gateway of the data middle platform. The purpose of this process is to ensure that the data middle platform can dynamically adjust the interface configuration according to the updated metadata set, so as to better support data governance and data access requirements. Among them, the interface description file is a file that defines information such as interface functions, parameters, and return values. It is the basis for interface generation. The model-driven interface generator is a tool that can generate specific interface code based on the interface description file. The interface service gateway is a component of the data middle platform, responsible for managing and routing interface requests.

[0114] Specifically, the metadata set refers to a set of data metadata and governance field configurations generated after association processing. It has a hierarchical structure and can reflect the logical relationship between data elements. The interface description file is a standardized file that describes in detail the interface name, input parameters, output parameters, return value type and other information. For example, for a student information query interface, the interface description file will define the interface name as GetStudentInfo, the input parameter as student_id, and the output parameters as name, age, score, etc. The model-driven interface generator is a tool based on the model-driven architecture MDA. It can automatically generate interface code according to the interface description file, reducing the workload and errors of manual coding. The interface service gateway is a middleware that is responsible for receiving external requests, routing the requests to the corresponding interface processing module according to the requested interface name and parameters, and returning the processing results.

[0115] Preferably, during the interface deployment parameter update process, the metadata set is first converted into an interface description file. This process can be implemented through the conversion module of the data middle station, which will generate an interface description file that conforms to the predefined format based on the hierarchical structure and content of the metadata set. Then, the interface description file is compiled into an executable data interface configuration through a model-driven interface generator. The interface generator will generate specific interface code based on the definition in the interface description file, such as generating RESTful API interface code. Finally, the generated interface configuration is deployed to the interface service gateway of the data middle station. During the deployment process, the interface service gateway will load the interface configuration to ensure that the interface can operate normally and provide services to the outside world. For example, for the student information query interface, the interface service gateway will receive the query request based on the interface configuration, call the back-end data processing module to obtain student information, and return the result to the requester. This process not only improves the efficiency of interface deployment, but also enhances the maintainability and scalability of the interface.

[0116] In some embodiments, the data quality verification process includes:

[0117] Obtain the field integrity parameters of the original campus data through the data verification engine of the data center;

[0118] When the field integrity parameter reaches the preset verification threshold, the data completion module of the data center is activated;

[0119] The data completion module performs interpolation calculations on missing fields to generate basic data that passes verification.

[0120] It should be noted that data quality verification involves checking the field integrity of raw campus data and, when necessary, activating the data completion module to address missing fields. This process aims to ensure data integrity and accuracy, providing high-quality foundational data for subsequent data governance. The field integrity parameter is a parameter used to measure whether a data field is missing, reflecting the degree of data integrity. The data completion module is a tool for handling missing data, completing missing fields through methods such as interpolation.

[0121] Specifically, data quality verification processing refers to a series of inspection operations performed on the original campus data to ensure the integrity and accuracy of the data. This process usually includes checking the field integrity of the data, whether the data type is correct, whether there are duplicate records, etc. The field integrity parameter is a quantitative indicator used to measure the integrity of the data field. For example, for a student information data set containing fields such as student number, name, and grades, the field integrity parameter can be obtained by calculating the ratio of the number of missing fields to the total number of fields. If the field integrity parameter does not reach the preset verification threshold, for example, the number of missing fields exceeds 10% of the total number of fields, the data completion module is activated. The data completion module will perform interpolation calculations on the missing fields according to the preset completion strategy. For example, for the missing grade field, the average interpolation method can be used, that is, the average value of the student's grades in other courses is used to complete the missing grade field.

[0122] Preferably, during the data quality verification process, the field integrity parameters of the original campus data are first obtained through the data verification engine of the data center. The data verification engine scans each field in the data set, counts the number of missing fields, and calculates the field integrity parameters. If the field integrity parameters do not reach the preset verification threshold, for example, the number of missing fields exceeds 10% of the total number of fields, the data completion module is activated. The data completion module selects an appropriate interpolation method based on the type and distribution of the data. For example, for numerical data such as grades, the mean interpolation method can be used; for categorical data such as gender, the mode interpolation method can be used. The completed data will be checked again by the data verification engine to ensure that the completed data meets the quality requirements. This process not only improves the integrity of the data, but also provides a reliable data foundation for subsequent data governance and analysis.

[0123] In some embodiments, the feature space conversion process includes:

[0124] Call the dimensionality reduction algorithm library of the data center to perform principal component analysis on the basic data;

[0125] Through the standardization module of the data center, Z-score standardization calculation is performed on the data processed by principal component analysis;

[0126] Write the standardized calculation results into the feature repository of the data center to generate governance target data with standardized data structure characteristics.

[0127] It should be noted that feature space conversion processing is a series of processing operations performed on basic data, the purpose of which is to convert the data into governance target data with standardized data structure characteristics. This process includes calling the dimensionality reduction algorithm library to perform principal component analysis on the basic data, and performing Z-score standardization calculation on the processed data through the standardization module. Through these steps, the dimension of the data can be effectively reduced while ensuring the standardization of the data, thereby improving the availability and processing efficiency of the data. Among them, the dimensionality reduction algorithm library refers to a collection of multiple dimensionality reduction algorithms used to reduce the dimension of the data; the Z-score standardization calculation is a data standardization method used to convert the data into standard normal distribution data with a mean of 0 and a standard deviation of 1.

[0128] Specifically, feature space transformation refers to processing underlying data using dimensionality reduction algorithms and normalization methods to extract key features and reduce the data's dimensionality. This process typically involves two main steps: dimensionality reduction and normalization. A dimensionality reduction algorithm library is a collection of various dimensionality reduction algorithms, such as principal component analysis (PCA) and linear discriminant analysis (LDA). These algorithms improve data processing efficiency by reducing the data's dimensionality while preserving its key features. Principal component analysis (PCA) is a commonly used dimensionality reduction method that reduces the data's dimensionality by projecting it onto the principal components. For example, for a dataset of student information containing multiple features, PCA can project these features onto a few principal components, thereby reducing the data's dimensionality. Z-score normalization is a data normalization method that converts the data into a standard normal distribution with a mean of 0 and a standard deviation of 1 by calculating its mean and standard deviation. This process eliminates the effects of dimensionality and improves data comparability.

[0129] Preferably, during the feature space conversion process, the dimensionality reduction algorithm library of the data middle station is first called to perform principal component analysis on the basic data. This process can be achieved by calculating the covariance matrix of the data and finding the eigenvalues and eigenvectors of the covariance matrix. Then, several eigenvectors with the largest eigenvalues are selected as principal components, and the data are projected onto the directions of these principal components, thereby reducing the dimension of the data. Next, the Z-score standardization calculation is performed on the data processed by the principal component analysis through the standardization module of the data middle station. This process includes calculating the mean and standard deviation of the data, and then subtracting the mean from each data point and dividing it by the standard deviation to obtain the standardized data. Finally, the standardized calculation results are written into the feature repository of the data middle station to generate governance target data with standardized data structure characteristics. This process not only improves the data processing efficiency, but also provides a high-quality data foundation for subsequent data analysis and application.

[0130] In some embodiments, constructing the semantic similarity parameter includes:

[0131] The word vector generator of the data center converts the data metadata and the preset governance field configuration into a word vector set;

[0132] Based on the attention mechanism module of the data center, the cosine similarity matrix of the word vector set is calculated;

[0133] Through the graph neural network of the data center, a metadata set with a hierarchical structure is generated based on the cosine similarity matrix.

[0134] It should be noted that the construction of semantic similarity parameters is a process of converting data metadata and preset governance field configurations into word vectors and calculating the cosine similarity matrix, ultimately generating a metadata set with a hierarchical structure. The purpose of this process is to ensure semantic consistency between data metadata and governance field configurations through semantic analysis, thereby improving the accuracy and efficiency of data governance. Among them, the word vector generator is a tool used to convert text data into word vectors to facilitate subsequent semantic similarity calculations; the attention mechanism module is an algorithm module used to calculate the similarity between word vectors; and the graph neural network is a neural network used to process graph-structured data and is used to construct a metadata set with a hierarchical structure.

[0135] Specifically, the word vector generator is a tool that converts text data into word vectors for semantic similarity calculation. For example, for a student ID in data metadata and a student ID in the preset governance field configuration, the word vector generator converts these two texts into corresponding word vectors. These word vectors are vectors in a high-dimensional space that can represent the semantic information of the text. The cosine similarity matrix is a matrix obtained by calculating the cosine similarity between word vectors. It reflects the degree of similarity between word vectors. For example, a cosine similarity value close to 1 indicates that two word vectors are very similar, while a value close to 0 indicates that they are not similar. The attention mechanism module is an algorithm module that calculates the similarity between word vectors to generate a cosine similarity matrix. The attention mechanism helps the model better focus on important word vectors, improving the accuracy of similarity calculations. Graph neural networks are neural networks designed to process graph-structured data. They can generate a hierarchical set of metadata based on the cosine similarity matrix. For example, graph neural networks can cluster word vectors with high similarity to form a hierarchical structure, facilitating data management.

[0136] Preferably, during the semantic similarity parameter construction process, the data meta-information and the preset governance field configuration are first converted into a set of word vectors using the word vector generator in the data center. This process can be implemented using a pre-trained word embedding model, such as Word2Vec or BERT, to map text data into word vectors in a high-dimensional space. Then, the cosine similarity matrix of the word vector set is calculated using the attention mechanism module in the data center. Based on the similarity between the word vectors, the attention mechanism generates a matrix, where each element represents the similarity between two word vectors. Finally, a hierarchical meta-information set is generated based on the cosine similarity matrix using the graph neural network in the data center. Based on the similarity matrix, the graph neural network clusters similar word vectors together to form a hierarchical structure, thereby constructing the meta-information set. This process not only improves the semantic consistency between the data meta-information and the governance field configuration but also enhances the intelligent level of data governance. For example, for a student information dataset, by calculating the cosine similarity between student ID and student ID, name and student name, and grades and test scores, a hierarchical meta-information set can be constructed, thereby achieving efficient data management and governance.

[0137] The above embodiments of the present invention have the following beneficial effects:

[0138] 1. It can significantly improve the automation and intelligence level of campus data governance: By using machine learning algorithms to automatically verify data quality, transform feature spaces, and perform structured processing on raw campus data, it can effectively solve the problems of low processing efficiency, strong subjectivity, and high error rate in traditional manual data governance methods, ensuring that the output data has standardized data structure characteristics and high-quality data content, providing a reliable foundation for subsequent data analysis and application.

[0139] 2. It can realize the intelligent adaptation and efficient circulation of campus data interfaces: By establishing a dynamic association mechanism between data identification fields and preset governance fields, combined with model-driven interface automatic generation technology, it can effectively solve the problems of inconsistent standards and poor compatibility of multi-source heterogeneous data interfaces that are prevalent in campus information systems, realize automatic adaptation and efficient calling of data interfaces between different systems, and greatly improve the accuracy and real-time performance of cross-system data interaction.

[0140] 3. An adaptive campus data governance strategy system can be built: Through the intelligent calculation of data governance feature vectors and dynamic adjustment of metadata weights, combined with semantic similarity analysis and hierarchical structure construction technology, it can effectively solve the problem that traditional static data governance strategies are difficult to adapt to campus business changes and data feature evolution, so that data governance strategies can be automatically optimized and adjusted according to actual business needs and data feature changes, providing flexible and reliable data governance support for the continuous expansion and upgrading of campus informatization construction.

[0141] like Figure 2 As shown, some embodiments provide a campus data governance and management system based on machine learning, the system comprising:

[0142] The detection module 201 is configured to, in response to detecting a data acquisition request information of a target data set from a data terminal, send the acquisition request information of the target data set to the data middle station;

[0143] The interface trigger module 202 is configured to trigger the data processing flow of the data center station in response to receiving the data identification field information of the corresponding target data set sent by the data center station, according to the data identification field information and the target interface configuration information of the data center station;

[0144] The data processing engine 203 is integrated into the data center and configured to perform structured processing on the original campus data corresponding to the data identification field information, wherein the structured processing includes data feature extraction processing based on a machine learning algorithm to obtain governance target data;

[0145] A policy generation module 204 is configured to generate data governance policy information corresponding to the governance target data based on the governance target data;

[0146] The parameter updating module 205 is configured to update the model parameters of each data metadata corresponding to the target data set according to the data governance policy information, wherein each data metadata includes metadata representing the data identifier;

[0147] The association processing module 206 is configured to, in response to determining that the interface configuration information of the corresponding data terminal is not deployed, associate the updated data metadata with the preset governance field configuration of the data middle platform to generate a metadata set;

[0148] The interface deployment module 207 is configured to update the interface deployment parameters of the data center based on the preset data interface configuration information corresponding to the metadata set and the governance target data;

[0149] The field transmission module 208 is configured to send the data identification field information associated with the target data set to the data terminal;

[0150] The interface execution engine 209 is integrated in the data center and is configured to respond to the detection of the data terminal's assignment operation on the preset governance field configuration of the preset data interface configuration information, convert the assigned preset governance field configuration into an interface call parameter, and execute the data interface call operation to return the governance target data to the data terminal.

[0151] It is understandable that the modules and references recorded in the campus data governance management system based on machine learning Figure 1 The steps in the campus data governance management method based on machine learning correspond to each other. Therefore, the operations, features, and beneficial effects described above for the campus data governance management method based on machine learning are also applicable to the campus data governance management system based on machine learning and the modules contained therein, and will not be repeated here.

[0152] Reference below Figure 3 , which shows a schematic structural diagram of an electronic device 300 suitable for implementing some embodiments of the present invention. The electronic devices in some embodiments of the present invention may include, but are not limited to, mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 3 The terminal device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0153] like Figure 3As shown, electronic device 300 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 301, which can perform various appropriate actions and processes based on programs stored in a read-only memory (ROM) 302 or programs loaded from a storage device 308 into a random access memory (RAM) 303. RAM 303 also stores various programs and data required for the operation of electronic device 300. Processing device 301, ROM 302, and RAM 303 are interconnected via a bus 304. An input / output (I / O) interface 305 is also connected to bus 304.

[0154] Typically, the following devices may be connected to the I / O interface 305: an input device 306 including, for example, a touch screen, a touchpad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 307 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 308 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 309. The communication device 309 may allow the electronic device 300 to communicate with other devices wirelessly or by wire to exchange data. Figure 3 The electronic device 300 is shown with various devices, but it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed instead. Figure 3 Each block shown in the figure may represent one device, or may represent multiple devices as needed.

[0155] Furthermore, the storage medium of the embodiment of the present application stores program instructions that can implement all the above methods, wherein the program instructions can be stored in the above storage medium in the form of a software product, including a number of instructions for causing a computer device (which can be a personal computer, server, or network device, etc.) or a processor to execute all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, or a terminal device such as a computer, server, mobile phone, or tablet.

[0156] The above descriptions merely illustrate some preferred embodiments of the present invention and the underlying technical principles. Those skilled in the art should understand that the scope of the invention encompassed by the embodiments of the present invention is not limited to technical solutions formed by specific combinations of the aforementioned technical features. It also encompasses other technical solutions formed by any combination of the aforementioned technical features or their equivalents, without departing from the aforementioned inventive concept. For example, a technical solution formed by replacing the aforementioned features with (but not limited to) technical features with similar functions disclosed in the embodiments of the present invention.

Claims

1. A campus data governance management method based on machine learning, comprising: In response to detecting data acquisition request information of the target data set from the data terminal, sending the acquisition request information of the target data set to the data middle station; In response to receiving the data identification field information of the corresponding target data set sent by the data middle station, triggering the data processing flow of the data middle station according to the data identification field information and the target interface configuration information of the data middle station; Through the data center, the original campus data corresponding to the data identification field information is structured to obtain the governance target data; Generate data governance policy information based on governance target data; Perform model parameter update processing on each data metadata corresponding to the target data set, including data identifiers, according to the data governance policy information; In response to determining that the interface configuration information of the corresponding data terminal is not deployed, the updated data metadata is associated with the preset governance field configuration of the data middle platform to generate a metadata set; Update the interface deployment parameters of the data center based on the preset data interface configuration information corresponding to the metadata set and the governance target data; Sending the data identification field information of the associated target data set to the data terminal; In response to detecting that the data terminal has assigned a value to a preset governance field configuration of preset data interface configuration information, the data middle platform converts the assigned preset governance field configuration into an interface call parameter, executes the data interface call operation, and returns the governance target data to the data terminal; wherein the model parameter update process includes: Generate a data governance feature vector through data governance policy information; Input the data governance feature vector into the metadata update module of the data center to calculate the weight adjustment parameters of each data metadata; Update the metadata storage structure of each data metadata information according to the weight adjustment parameters; Association processing includes: Align the updated data metadata with the preset governance field configuration for feature dimensions; The semantic similarity parameters between data metadata and preset governance field configurations are calculated through the association relationship learning module of the data center. Constructing a meta-information set with a hierarchical structure based on semantic similarity parameters; The construction of semantic similarity parameters includes: The word vector generator of the data center converts the data metadata and the preset governance field configuration into a word vector set; Based on the attention mechanism module of the data center, the cosine similarity matrix of the word vector set is calculated; Through the graph neural network of the data center, a metadata set with a hierarchical structure is generated based on the cosine similarity matrix.

2. The method according to claim 1, wherein Triggering the data processing flow of the data center includes: Extract the target field mapping rules corresponding to the target interface configuration information from the interface configuration library of the data center; Inject the data identification field information into the parameter container corresponding to the target field mapping rule; Execute parameterized interface calls through the interface call engine of the data center to obtain the original campus data associated with the data identification field information.

3. The method according to claim 1, wherein Structural processing includes: Perform data quality verification on the original campus data to generate basic data that passes the verification; Through the machine learning algorithm integrated in the data middle platform, the basic data is transformed into feature space to generate governance target data with standardized data structure characteristics.

4. The method according to claim 1, wherein Interface deployment parameter updates include: Convert the metadata collection into the interface description file of the data center; Compile the interface description file into an executable data interface configuration through a model-driven interface generator; Deploy the data interface configuration to the interface service gateway of the data center.

5. The method according to claim 3, wherein Data quality verification processing includes: Obtain the field integrity parameters of the original campus data through the data verification engine of the data center; When the field integrity parameter reaches the preset verification threshold, the data completion module of the data center is activated; The data completion module performs interpolation calculations on missing fields to generate basic data that passes verification.

6. The method according to claim 3, wherein Feature space transformation processing includes: Call the dimensionality reduction algorithm library of the data center to perform principal component analysis on the basic data; Through the standardization module of the data center, Z-score standardization calculation is performed on the data processed by principal component analysis; Write the standardized calculation results into the feature repository of the data center to generate governance target data with standardized data structure characteristics.

7. A campus data governance and management system based on machine learning, comprising: a detection module configured to, in response to detecting data acquisition request information of a data terminal for a target data set, send the acquisition request information of the target data set to the data middle station; The interface trigger module is configured to trigger the data processing flow of the data middle station in response to receiving the data identification field information of the corresponding target data set sent by the data middle station, according to the data identification field information and the target interface configuration information of the data middle station; The data processing engine is integrated into the data center and is configured to perform structured processing on the original campus data corresponding to the data identification field information, wherein the structured processing includes data feature extraction processing based on machine learning algorithms to obtain governance target data; A policy generation module is configured to generate data governance policy information corresponding to the governance target data based on the governance target data; A parameter updating module is configured to update model parameters for each piece of data metadata corresponding to the target data set according to the data governance policy information, wherein each piece of data metadata includes metadata representing the data identifier; an association processing module configured to, in response to determining that the interface configuration information of the corresponding data terminal is not deployed, associate each updated data meta information with the preset governance field configuration of the data middle platform to generate a meta information set; The interface deployment module is configured to update the interface deployment parameters of the data center based on the preset data interface configuration information corresponding to the metadata set and the governance target data; A field transmission module configured to send data identification field information associated with a target data set to a data terminal; An interface execution engine, integrated into the data center, is configured to, in response to detecting an assignment operation by a data terminal to a preset governance field configuration of preset data interface configuration information, convert the assigned preset governance field configuration into an interface call parameter, execute the data interface call operation, and return the governance target data to the data terminal; The model parameter update process includes: Generate a data governance feature vector through data governance policy information; Input the data governance feature vector into the metadata update module of the data center to calculate the weight adjustment parameters of each data metadata; Update the metadata storage structure of each data metadata information according to the weight adjustment parameters; Association processing includes: Align the updated data metadata with the preset governance field configuration for feature dimensions; The semantic similarity parameters between data metadata and preset governance field configurations are calculated through the association relationship learning module of the data center. Constructing a meta-information set with a hierarchical structure based on semantic similarity parameters; The construction of semantic similarity parameters includes: The word vector generator of the data center converts the data metadata and the preset governance field configuration into a word vector set; Based on the attention mechanism module of the data center, the cosine similarity matrix of the word vector set is calculated; Through the graph neural network of the data center, a metadata set with a hierarchical structure is generated based on the cosine similarity matrix.

Images