An AI cluster governance intelligent agent driven big data storage governance system and method

Abstract

This application provides a big data storage governance system and method driven by an AI cluster governance agent, relating to the field of data processing technology. The system includes: identifying governance objects within the cluster and collecting their information to generate a candidate list; extracting decision features from the candidate list; analyzing these features using an AI decision model that integrates rules and machine learning to predict the storage governance actions of the governance objects; analyzing the basic information of the governance objects using natural language processing technology driven by a large language model, and generating auxiliary decision information based on their correlation with a pre-defined relational database; integrating storage governance actions, auxiliary decision information, and confidence levels to generate a storage governance strategy; generating comprehensive governance information based on the storage governance strategy using an AI cluster governance agent and creating work orders to obtain a work order task list; and executing the corresponding storage governance strategy based on the confirmation instructions in the work order task list, thereby improving maintenance efficiency and reducing the risk of accidental deletion.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a big data storage and governance system and method driven by an AI cluster governance agent. Background Technology

[0002] With the continuous expansion of enterprise business and the rapid accumulation of data assets, the amount of data stored has exploded. Hundreds of thousands of business tables are stored in distributed storage layers such as HDFS and S3, forming hundreds of petabytes of existing data and several petabytes of incremental data every day. Storage costs continue to soar, and the need for storage governance is becoming increasingly urgent.

[0003] Traditional storage governance relies on manual maintenance and periodic manual inspections. Operations personnel need to manually determine which tables require cold backups, which tables can be deleted, whether they are within the whitelist protection range, and perform periodic backup / deletion operations. Its main problems include: (1) Low storage governance efficiency: Existing methods rely on manual analysis and offline communication to carry out storage governance, which is time-consuming and difficult to adapt to the rapidly growing data scale.

[0004] (2) Insufficient intelligence: In the full table cold backup and periodic cold backup process, the current methods mostly rely on fixed planned tasks or manual triggering by operation and maintenance personnel. Traditional scripts cannot dynamically adjust the governance strategy, resulting in resource waste (such as long-term occupation of high-cost storage for low-value data) or risk of misoperation.

[0005] (3) Insufficient security: Traditional governance methods for full table deletion and periodic deletion typically rely on manual confirmation or pre-set rule scripts to determine whether to execute. Due to the lack of intelligent identification, there is a risk of accidentally deleting critical business tables or omitting redundant data. For example, if the importance of a table and its downstream dependencies are not accurately assessed, core business tables may be accidentally deleted, causing business interruption.

[0006] Regarding whitelist maintenance, manual maintenance is prone to omissions or inconsistencies. Different departments may maintain different whitelists, causing the protection mechanism to fail. For example, the same critical table may be whitelisted in one system but not protected in another script, which may lead to the critical table being accidentally deleted or not properly protected.

[0007] (4) Poor scalability and consistency: The rule scripts need to be continuously updated and maintained manually, making it difficult to adapt to changes in business. Furthermore, the execution standards of different operation and maintenance personnel are not uniform, which increases the management difficulty and operation and maintenance risks.

[0008] This paper proposes a big data storage governance system and method driven by AI cluster governance agents. Summary of the Invention

[0009] This specification provides a big data storage governance system and method driven by an AI cluster governance agent, in order to address the limitations and shortcomings of existing technologies that rely on manual inspection and fixed scripts for storage management.

[0010] This application provides an AI-driven big data storage governance method for cluster governance. This method is applied to an operating environment deploying a multi-node distributed big data cluster, interfacing with the cluster source system, cluster metadata repository, and distributed storage layer. It identifies massive amounts of data tables across the cluster as governance objects in batches, including: Identify governance objects within the cluster and collect their information to generate a candidate list; The AI ​​cluster governance agent predicts storage governance strategies based on the candidate list. Specifically, it extracts decision features of governance objects from the candidate list; analyzes these decision features using an AI decision model that integrates rules and machine learning to predict the storage governance actions of the governance objects; analyzes the basic information of the governance objects using natural language processing technology driven by a large language model; calculates the correlation between the basic information and a preset relational database to generate a table importance score; determines a semantic importance score based on the table importance score; generates auxiliary decision information based on the semantic importance score; verifies the confidence level of the storage governance actions; and integrates the storage governance actions, the auxiliary decision information, and the confidence level to generate the storage governance strategy. Utilizing the analytical capabilities of the AI ​​cluster governance agent, comprehensive governance information is generated based on the storage governance strategy; Based on the comprehensive governance information, a batch production work order task list is generated. Execute the corresponding storage governance policy according to the work order confirmation instruction.

[0011] Optionally, the step of utilizing the analytical capabilities of the AI ​​cluster governance agent to generate comprehensive governance information based on the storage governance strategy includes: Based on the storage governance strategy, the data of the governance objects is integrated to generate an intermediate table; Statistical information for predicting the storage governance actions based on the intermediate table; Dashboard information is constructed based on the statistical information and the storage governance strategy; An approval package is constructed by combining the governance object information and the dashboard information, serving as the comprehensive governance information; wherein, the governance object is aggregated by combining the stored governance actions, risk levels, and responsible person information to generate an execution batch; and an approval package is assembled for each execution batch.

[0012] Optionally, the step of identifying governance objects within the cluster and collecting governance object information to generate a candidate list includes: Identify governance objects within the big data cluster from the source systems, metadata repositories, and distributed storage layers that are connected to the big data cluster; Extract information for each of the governance objects; The candidate list is constructed by summarizing the information of the governance objects and their corresponding governance objects.

[0013] Optionally, the statistical information includes: spatial statistical information, cost statistical information, and risk level; The statistical information used to predict the storage governance actions based on the intermediate table includes: Determine the spatial statistics; The cost statistics are predicted by combining price parameters with the spatial statistics. The risk level of the governance object is classified based on the first basic indicator in the intermediate table.

[0014] Optionally, the dashboard information includes: action distribution information, cost saving information, risk distribution information, and a TopN list; The construction of dashboard information based on the statistical information and the storage governance strategy includes: Identify the governance object corresponding to each of the storage governance actions to obtain the action distribution information; The cost statistics are aggregated and processed according to the first basic indicator to obtain the cost saving information; By cross-statistically analyzing the risk level and the storage governance actions, the risk distribution information is obtained. The TopN list is constructed based on the governance object information; For each of the TopN lists, mark the storage governance actions and auxiliary decision-making information.

[0015] This application provides an AI-driven big data storage governance system for cluster governance, deployed in a multi-node distributed big data cluster. It interfaces with the cluster source system, cluster metadata repository, and distributed storage layer, and identifies massive amounts of data tables across the cluster as governance objects. The system employs the following technical solutions: The data collection module is used to identify governance objects within the cluster, collect information about these objects, and generate a candidate list. The AI-driven strategy prediction module is used to predict storage governance strategies based on the candidate list through an AI cluster governance agent. The integrated management module is used to generate comprehensive governance information based on the storage governance strategy by utilizing the analytical capabilities of the AI ​​cluster governance agent. The work order creation module is used to generate a batch work order task list based on the comprehensive governance information. The work order execution module is used to execute the corresponding storage governance strategy according to the work order confirmation instruction; The AI-driven strategy prediction module integrates the AI ​​cluster governance agent, which includes at least: The feature extraction submodule is used to extract the decision features of the governance objects from the candidate list; The intelligent agent decision-making submodule is used to analyze the decision features through an AI decision-making model that integrates rules and machine learning, and predict the storage governance actions of the governance object; The importance scoring submodule is used to analyze the basic information of the governance object based on natural language processing technology driven by a large language model; calculate the correlation between the basic information and a preset relational database to generate a table importance score; and determine the semantic importance score based on the table importance score. The auxiliary decision-making information generation submodule is used to generate auxiliary decision-making information based on the semantic importance score; The confidence verification submodule is used to verify the confidence level of the storage governance action; The strategy encapsulation submodule is used to integrate the storage governance actions, the auxiliary decision information, and the confidence level to generate the storage governance strategy.

[0016] Optionally, the integrated management module includes: The intermediate table construction submodule is used to integrate the data of the governance objects based on the storage governance strategy and generate an intermediate table. The statistical information prediction submodule is used to predict the statistical information of the storage governance actions based on the intermediate table; The dashboard information construction submodule is used to construct dashboard information based on the statistical information and the storage governance strategy. The approval package construction submodule is used to construct an approval package by combining the governance object information and the dashboard information, which serves as the comprehensive governance information; wherein, the governance object is aggregated by combining the stored governance actions, risk levels, and responsible person information to generate execution batches; and an approval package is assembled for each execution batch.

[0017] Optionally, the acquisition module includes: The object identification submodule is used to identify governance objects within the big data cluster from the source systems, metadata repositories, and distributed storage layers that are connected to the big data cluster. The information extraction submodule is used to extract information about each of the governance objects. The information aggregation submodule is used to aggregate the governance objects and their corresponding governance object information to construct the candidate list.

[0018] Optionally, the statistical information includes: spatial statistical information, cost statistical information, and risk level; The statistical information prediction submodule includes: Spatial prediction unit, used to determine the spatial statistical information; A cost prediction unit is used to predict the cost statistics by combining price parameters with the spatial statistics. The risk prediction unit is used to classify the risk level of the governance object based on the first basic indicator in the intermediate table.

[0019] Optionally, the dashboard information includes: action distribution information, cost saving information, risk distribution information, and a TopN list; The dashboard information construction submodule includes: An action recognition unit is used to identify the governance object corresponding to each of the stored governance actions and obtain the action distribution information; An aggregation unit is used to aggregate the cost statistics information according to the first basic indicator to obtain the cost saving information. A cross-validation unit is used to cross-validate the risk level and the storage governance action to obtain the risk distribution information; A list construction unit is used to construct a TopN list based on the governance object information; The list optimization unit is used to mark the storage governance actions and auxiliary decision-making information for each of the TopN lists.

[0020] Optionally, the AI ​​cluster governance agent further includes: a model self-iteration unit; The model self-iteration unit is used to collect data from manual review and work order execution logs and feed it back to train the AI ​​decision-making model, thereby continuously optimizing the decision-making accuracy of the AI ​​cluster governance agent.

[0021] This specification also provides a computer device, wherein the computer device includes: Processor; and, A memory that stores computer-executable instructions, which, when executed, cause the processor to perform any of the methods described above.

[0022] This specification also provides a computer-readable storage medium that stores one or more programs / instructions that, when executed by a processor, implement any of the methods described above.

[0023] This specification also provides a computer program product, wherein the computer program product includes: a computer program / instruction, which, when executed by a processor, implements any of the methods described above.

[0024] This invention identifies governance objects within a cluster and collects their information to generate a candidate list. An AI cluster governance agent then predicts storage governance strategies based on this candidate list. Specifically, the invention extracts decision features from the candidate list; analyzes these features using an AI decision model that integrates rules and machine learning to predict the storage governance actions of the governance objects; analyzes the basic information of the governance objects using natural language processing technology driven by a large language model; calculates the correlation between the basic information and a preset relational database to generate a table importance score; determines a semantic importance score based on the table importance score; generates auxiliary decision information based on the semantic importance score; verifies the confidence level of the storage governance actions; and integrates the storage governance actions, the auxiliary decision information, and the confidence level to generate the storage governance strategy, thereby improving maintenance efficiency and reducing the risk of accidental deletion. Attached Figure Description

[0025] Figure 1 A schematic diagram illustrating the principle of a big data storage governance method driven by an AI cluster governance agent, as provided in the embodiments of this specification. Figure 2 Timing of a Big Data Storage Governance Method Driven by an AI Cluster Governance Agent Provided in the Embodiments of this Specification Figure 1 ; Figure 3 Timing of a Big Data Storage Governance Method Driven by an AI Cluster Governance Agent Provided in the Embodiments of this Specification Figure 2 ; Figure 4 This specification provides a schematic diagram of the structure of a big data storage governance system driven by an AI cluster governance agent, as illustrated in an embodiment of the present specification. Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this specification; Figure 6 This is a schematic diagram of a computer-readable storage medium provided for an embodiment of this specification. Detailed Implementation

[0026] The following description is intended to disclose the present invention and enable those skilled in the art to implement it. The preferred embodiments described below are merely examples, and other obvious variations will occur to those skilled in the art. The basic principles of the invention defined in the following description can be applied to other embodiments, modifications, improvements, equivalents, and other technical solutions that do not depart from the spirit and scope of the invention.

[0027] Exemplary embodiments of the invention will now be described more fully with reference to the accompanying drawings. While conforming to the inventive concept, the features, structures, characteristics, or other details described in a particular embodiment may be combined in one or more other embodiments in a suitable manner.

[0028] The terms “and / or” or “and / or” include all combinations of any one or more of the listed items.

[0029] If the technical solution of this application involves personal information, the product using this technical solution has clearly informed the user of the personal information processing rules and obtained the user's voluntary consent before processing the personal information. If the technical solution of this application involves sensitive personal information, the product using this technical solution has obtained the user's separate consent before processing the sensitive personal information, and also meets the requirement of "express consent". For example, at personal information collection devices such as cameras, clear and prominent signs are set up to inform users that they have entered the scope of personal information collection and that personal information will be collected. If an individual voluntarily enters the collection scope, it is deemed that they have agreed to the collection of their personal information; or on the personal information processing device, with clear signs / information informing users of the personal information processing rules, authorization is obtained from the individual through pop-up information or by asking the individual to upload their personal information; wherein, the personal information processing rules may include information such as the personal information processor, the purpose of personal information processing, the processing method, and the types of personal information processed.

[0030] First, a brief explanation of the terms used in the embodiments of this application will be provided: AI, short for Artificial Intelligence, includes algorithms such as machine learning (ML), natural language processing (NLP), and deep learning, used to automate decision-making and process complex data tasks.

[0031] Storage governance involves systematically analyzing the access behavior, data characteristics, and storage status of stored files in a big data cluster. This process identifies inefficient, redundant, or long-unused data objects and, in conjunction with business value and data lifecycle, formulates and implements differentiated storage governance strategies. These strategies include measures such as hot / cold tiering, compression optimization, partition adjustment, archiving, or cleanup. The aim is to reduce overall storage usage and cluster operation and maintenance costs, while simultaneously improving resource utilization and the standardization of data management.

[0032] AI Cluster Governance Agent: An AI / rule-based automated execution and decision-making module that can dynamically determine whether to perform cold backup, deletion, or protection operations based on the importance of the table, its dependencies, storage costs, and business characteristics.

[0033] Full table cold backup: This refers to performing a complete backup of all data in a single table and storing it in a relatively low-cost, infrequently accessed cold storage. It is typically used for disaster recovery protection of critical tables to ensure that the table can be restored from the backup if it is accidentally deleted, damaged, or needs to be rolled back.

[0034] Full table deletion: This refers to directly clearing all data records from a table (usually preserving the table structure). It is often used for test data cleanup, removal of expired data, or reconstruction of erroneous data. This operation carries significant risk and generally requires approval and protection mechanisms (such as cold backups or whitelisting mechanisms).

[0035] Periodic cold backup: refers to performing cold backups on historical partitions of a data table periodically according to a preset time period (such as daily, weekly, or monthly). This mechanism ensures that historical partition data can be used for data tracing, historical analysis, or recovery after accidental operations.

[0036] Periodic deletion refers to the automatic cleanup of expired, redundant, or low-value data (such as log data older than 30 days) according to business or storage governance strategies and at preset intervals. This mechanism can free up storage space, reduce storage costs, and maintain the efficient operation of the data warehouse.

[0037] Whitelist: A security mechanism used in storage governance or operations to protect specified tables, databases, or fields from accidental operations (such as deletion, overwriting, or data masking). Only objects not included in the whitelist are allowed to perform high-risk operations. The whitelist mechanism can reduce the risk of data loss due to human error.

[0038] Figure 1 This is a schematic diagram illustrating the principle of a big data storage governance method driven by an AI cluster governance agent, as provided in the embodiments of this specification. The corresponding system is deployed in a big data cluster environment, connecting to the business source system, HDFS / S3 distributed storage layer, and Hive / Atlas-like metadata warehouse. It can batch identify massive data tables within the cluster as objects to be governed, and completes full-link automated storage governance based on the AI ​​cluster governance agent, specifically including: S1 identifies governance objects within the cluster and collects information about them, generating a candidate list. S2 uses an AI cluster governance agent to predict storage governance strategies based on the candidate list. Specifically, it extracts decision features of governance objects from the candidate list; analyzes these decision features using an AI decision model that integrates rules and machine learning to predict the storage governance actions of the governance objects; analyzes the basic information of the governance objects using natural language processing technology driven by a large language model; calculates the correlation between the basic information and a preset relational database to generate a table importance score; determines a semantic importance score based on the table importance score; generates auxiliary decision information based on the semantic importance score; verifies the confidence level of the storage governance actions; and integrates the storage governance actions, the auxiliary decision information, and the confidence level to generate the storage governance strategy. S3 utilizes the analytical capabilities of the AI ​​cluster governance agent to generate comprehensive governance information based on the storage governance strategy; S4 generates a batch work order task list based on the comprehensive governance information; S5 executes the corresponding storage governance policy based on the work order confirmation instruction.

[0039] Storage governance is a crucial aspect of ensuring enterprise data asset security and optimizing costs. Currently, enterprises typically rely on manual inspections and basic scripts for data storage governance; however, this approach is inefficient and lacks intelligent support.

[0040] Based on this, this invention introduces AI-driven automatic identification, classification, and lifecycle management of data assets, achieving intelligent and automated operations such as full table cold backup, full table deletion, whitelist protection, periodic cold backup, and periodic deletion. This improves the efficiency, security, and reliability of storage governance while reducing manual intervention and operational risks, and optimizes storage resource utilization while ensuring data security. Specifically, this method includes: S1 identifies governance objects within the cluster and collects information about them, generating a candidate list. This step is performed within a big data cluster environment. It identifies all data tables within the cluster as governance objects from the source system, distributed storage, and metadata repository connected to the cluster, and extracts various indicators to generate a candidate list.

[0041] This invention systematically acquires multi-dimensional basic information about the objects to be governed, providing data support for the generation of subsequent storage governance strategies.

[0042] S11 identifies governance objects within the cluster from the source systems, metadata repositories, and distributed storage layers connected to the big data cluster; The governance targets tables in big data clusters that may have storage redundancy, inefficiency, or expiration risks and require storage governance.

[0043] The source system is used to store dynamic behavior / context information. The source system includes, but is not limited to: logs (access / query / scan), storage layer (HDFS / S3) metadata, data directory, and operation and maintenance whitelist.

[0044] Metadata repositories are used to store structured metadata. Metadata repositories include, but are not limited to, Hive, Glue, InfoSchema, and Atlas.

[0045] The identification of governance objects in this invention covers both the metadata repository and the source system, ensuring cross-validation of multi-dimensional data.

[0046] S12 Extract information for each of the governance objects; The governance object information includes several basic indicators. In one embodiment of this specification, the types of governance object information include, but are not limited to: table attribute information, access behavior information, dependency relationship information, and business attribute information.

[0047] S111 retrieves table-level statistics of the governance object and obtains table attribute information; Table attribute information refers to the basic attributes of a table. Specifically, the basic metrics corresponding to table attribute information include, but are not limited to: number of partitions (partition_count), data size (total_bytes), number of files (file_count), last modified time (last_modified), and last read time (last_read_ts).

[0048] The most recent modification time includes: the most recent write time (last_write_ts) and the most recent update time.

[0049] In one embodiment of this specification, partition count, data size, and last modification time are directly read as table attribute information from the table statistics interface of a metadata repository (e.g., Hive, HDFS, S3) or file system metadata. If no row count is available, sampling or direct counting is used, for example, SELECT COUNT( For large tables (such as those in the TB range), prioritize using computational statistics or approximate methods to avoid full table scans.

[0050] S112 retrieves access behavior information of the governance object; Access behavior information refers to the historical behavior data of the governance object being queried and scanned.

[0051] Specifically, access behavior information includes, but is not limited to: access frequency, last access time (last_access_ts), and downstream consumption.

[0052] The access frequency includes, but is not limited to: the most recent read time (last_read_ts), the number of reads in the last 30 days (read_count_30d), and the number of unique users querying the service in the last 30 days (distinct_users_30d).

[0053] S113 collects dependency information of the managed objects; Dependency information represents the upstream and downstream data link relationships of the governance object, reflecting the scope of the impact of deletion / modification operations on the business.

[0054] The basic metrics corresponding to dependency information include, but are not limited to: downstream impact and downstream dependency count. The downstream dependency count represents the number of downstream tasks / tables that directly or indirectly depend on the governance object.

[0055] In one embodiment of this specification, the "upstream → table → downstream" link relationship is analyzed and extracted from a scheduling system, data directory, or static code to form a graph structure; wherein, the scheduling system includes, but is not limited to: Airflow, Dag / Spark jobs.

[0056] Based on the location of the governance object, the downstream influence is calculated by the number of downstream tables affected, the business scope and importance of the business to which the downstream tables belong, and the proportion of all tables in the downstream table site, which serves as the dependency information of the governance object.

[0057] S114 Collects business attribute information of the governance objects; Business attribute information is the business metadata used to assess the business risks of governance operations.

[0058] Specifically, the basic indicators corresponding to business attribute information include, but are not limited to: owner / contact information, business domain, sensitivity level, and whitelist status.

[0059] The responsible person information refers to the team / contact person corresponding to the governance object. The business domain refers to the business area to which the governance object belongs, such as transactions, user behavior, settlement, etc. Sensitivity level represents the data's sensitivity level, such as public, internal, confidential, highly confidential. Whitelist status indicates whether the governance object is included in the whitelist (protected list), marking tables that are prohibited from deletion / cold backup.

[0060] In one embodiment of this specification, the responsible person information for the governance object is associated with the Configuration Management Database (CMDB); the business domain and sensitivity level are directly retrieved from the business metadata module of the data catalog; and the whitelist status of the governance object is synchronized from the operations and maintenance whitelist system. Of course, business attribute information can also be collected through automated questionnaires / email confirmations, etc.

[0061] S13 summarizes the governance objects and their corresponding governance object information to construct the candidate list.

[0062] The candidate list is in CSV or table format, where each row contains basic metadata and statistical metrics.

[0063] S2 uses an AI cluster governance agent to predict storage governance strategies based on the candidate list; This invention is based on multi-source data and decision-making models to generate executable stored governance actions for each governance object and output auxiliary decision-making information and confidence level to support subsequent automated execution.

[0064] Storage governance policies include, but are not limited to: storage governance actions and explanatory information; Explanatory information includes, but is not limited to: decision support information and confidence levels.

[0065] S21 Constructing a decision-making model; The decision model is the core of storage governance action prediction, supporting both rule engines and machine learning models, balancing business interpretability and model generalization ability. As the preferred choice, the decision model is a classification model.

[0066] S211 constructs decision features; In one embodiment of this specification, the decision features include, but are not limited to: the number of reads in the last 30 days (read_count_30d), the last access time (last_access_ts), the last modification time, the data size (total_bytes), the growth rate (last 30 / 90 days), the number of downstream dependencies, the whitelist status, the sensitivity level, the owner confirmation flag, and the number of times the data has been restored in the past.

[0067] The growth rate refers to the rate of increase in table storage size. This growth rate can be calculated using the daily data size (total_bytes) metric (over the last 30 / 90 days). The growth rate is used to detect table anomalies: generally, table growth rates are stable; a very rapid growth rate is considered abnormal.

[0068] S212 determines the preset weight for each decision feature; The weights determine the influence of each decision feature on governance decisions, supporting both manual rule setting and automatic model learning, balancing business flexibility and model objectivity.

[0069] Specifically, fixed weights can be assigned to decision-making features based on business experience and cost considerations.

[0070] Alternatively, historical governance records can be obtained and used to train a decision-making model. By introducing AI / ML to enhance the judgment, the importance weights of decision features can be automatically derived. These historical governance records include: the input decision features and the actual stored governance actions output.

[0071] At this point, the preferred decision model is a labeled historical decision-trained binary / multi-class classification model, such as XGBoost or LightGBM. The decision features are the fields mentioned above, and the output stores the type and probability of the governance action.

[0072] S22 standardizes the information of the governance targets; In one embodiment of this specification, the information of the governance object is standardized to eliminate the dimensional differences between features. Specifically, numerical features are standardized, and categorical features are one-hot encoded.

[0073] S23 predicts the stored governance actions and auxiliary decision-making information of the governance object based on the decision-making model; S231 Extracts the decision characteristics of the governance objects from the candidate list; S232 analyzes the decision characteristics using an AI decision model that integrates rules and machine learning to predict the storage governance actions of the governance object; Specifically, based on standardized decision characteristics and their preset weights, the probability distribution of each storage governance action is predicted; Storage governance actions include, but are not limited to: cold backup, deletion, protection, and retention.

[0074] Based on the probability distribution of the storage governance actions, the storage governance action with the highest probability is found and used as the final storage governance action.

[0075] This invention is based on multi-dimensional judgment and automatically generates storage governance actions such as "(required) cold backup / (required) deletion / (required) protection / no operation required" by integrating metadata, data dependencies, storage costs and business continuity requirements.

[0076] S233 uses natural language processing technology driven by a large language model to analyze the basic information of the governance object; calculates the correlation between the basic information and the preset relational database, and generates a table importance score; and determines a semantic importance score based on the table importance score. In one embodiment of this specification, the basic information of the governance object is converted into a first vector representation; the basic information includes, but is not limited to, the table name, column name, and description information of the governance object.

[0077] The first vector is correlated with a preset relational database, and the correlation is used as the table importance score. The preset relational database includes, but is not limited to, a core business keyword database and a domain knowledge base. Specifically, through semantic understanding, access frequency analysis, and downstream dependency resolution, different importance levels such as core business tables, secondary business tables, and redundant tables are automatically identified for table importance assessment. In one embodiment of this specification, the correlation between the table name, column names, description information, and core business keywords and the domain knowledge base is determined; the higher the correlation, the higher the table importance score.

[0078] A semantic importance score is determined based on the table importance score; wherein, the semantic importance score is positively correlated with the table importance score. That is, the higher the table importance score, the more important the governance object, and the higher its semantic importance score.

[0079] S234 generates auxiliary decision-making information based on the semantic importance score; In one embodiment of this specification, for scenarios where the storage governance action is (required) cold backup / (required) deletion / (required) protection / no operation required: for each specific storage governance action prediction sample, the model provides a prediction f(x). The contribution of each decision feature to the storage governance action is calculated using an interpretable method (SHAP, SHapley Additive ex Planations), yielding the contribution value. If the contribution value is greater than 0, it is considered that the feature makes the model more inclined to take that storage governance action. If the contribution value is less than 0, it means that the feature lowers the model's inclination to take that storage governance action.

[0080] Based on contribution values, a natural language summary is generated using an LLM (Limited Language Management) system as auxiliary decision-making information. This auxiliary decision-making information includes decision features that play a crucial role in storage governance actions. In one embodiment of this specification, the decision features are used as input, and the LLM performs a natural language summary: features corresponding to contribution values ​​with larger absolute values ​​are identified as important decision features. The LLM then organizes the specific decision features, positive and negative contributions, and their magnitudes into a readable explanation.

[0081] S24 generates a prompt message based on storage management actions; If the storage governance action is recommended deletion, the system automatically checks if a snapshot / cold backup of the object being governed exists. If neither a snapshot nor a cold backup exists, the object is marked with a notification message. Marked tables indicate that governance has been performed and no further governance is needed within a preset timeframe. The notification message is used to annotate periodic deletions or periodic cold backups, allowing users to quickly identify the cause when querying for missing table data. The notification message includes: "Perform a cold backup first."

[0082] S25 verifies the confidence level of the storage management action; By using confidence scoring and manual sampling, the accuracy and reliability of the strategy are ensured, and feedback is used to optimize the model.

[0083] S251 performs a confidence prediction for each storage governance action to obtain the corresponding confidence level.

[0084] In one embodiment of this specification, the confidence level is the probability of the decision model or the strength of rule matching.

[0085] S252 automatically generates manual review tasks for low-confidence or high-risk items.

[0086] Among them, low-confidence items are storage governance actions with a confidence level below a preset threshold. High-risk items are deletion operations in table storage governance.

[0087] In one embodiment of this specification, during the table storage governance process, a confidence assessment and risk identification are performed on each governance action to be executed. For items with low confidence, they will be automatically marked as "requiring manual review," and the automatic execution process will be paused.

[0088] Based on the above judgment results, the corresponding manual review task is automatically generated, and the task details are pushed to the manual review end to ensure that manual confirmation is obtained before execution, thereby effectively avoiding the risk of data loss due to misoperation.

[0089] S253 optimizes the AI ​​cluster governance agent based on the review results of the manual review task; In one embodiment of this specification, the AI ​​cluster governance agent further includes a model self-iteration unit.

[0090] After manual review is completed, the manual review results and work order execution log data are collected through the model's self-iterative unit and fed back to train the AI ​​decision-making model to continuously optimize the decision-making accuracy of the AI ​​cluster governance agent.

[0091] S254 samples storage governance actions; In one embodiment of this specification, the top N% of COLD_BACKUP / DELETE recommendations are sampled each period to verify accuracy and are used for model retraining.

[0092] This invention optimizes the model based on a sample of high-confidence automated execution data and low-confidence / high-risk manually reviewed data, thereby comprehensively improving the decision-making robustness and accuracy of the AI ​​cluster governance agent.

[0093] S26 summarizes the storage governance actions, auxiliary decision-making information, and confidence levels to construct a storage governance strategy.

[0094] In one embodiment of this specification, the storage governance policy (governance_decision) includes: periodic deletion is recommended because the storage has not been accessed in the last 365 days, has no downstream dependencies, and is not on the whitelist; confidence level 92%; key evidence: last_read=450days, lineage_down_count=0, size=120GB.

[0095] This invention generates table-level storage governance actions by fusing five-dimensional features, including access frequency, table size, downstream dependency, sensitivity, and white status, as well as by combining rule-based and AI-based judgment, dynamic priority sorting and confidence calculation, and using reinforcement learning to optimize weights based on historical governance data.

[0096] S3 utilizes the analytical capabilities of the AI ​​cluster governance agent to generate comprehensive governance information based on the storage governance strategy; Comprehensive governance information includes: the approval package for each batch of execution; S31 integrates the data of the managed objects based on the storage governance strategy and generates an intermediate table; S311 uses key information as a unique identifier to associate storage governance actions with target basic indicators; The key information (table_id) includes: environment (env), database (db), and table name (table).

[0097] The target basic metrics include, but are not limited to: metadata, whitelist status, sensitivity level, and dependency information.

[0098] S312 standardizes the field definitions of the target's basic indicators; In one embodiment of this specification, field standardization includes, but is not limited to: Convert bytes to GB, i.e., size_gb=total_bytes / 10243; Define the number of days since the last access, i.e., last_read_days = now - last_read_ts (days).

[0099] The downstream impact is quantified, i.e., downstream_degree = lineage_down_count; is_core = (domain in core business domain or semantic importance ≥ threshold).

[0100] S313 defines the scope of storage governance actions; Storage governance actions include: COLD_BACKUP, DELETE, PROTECT, ARCHIVE, and KEEP_ACTIVE. That is, the storage governance actions are defined as: action∈{COLD_BACKUP, DELETE, PROTECT, ARCHIVE, KEEP_ACTIVE}.

[0101] S314 summarizes key information and basic target indicators to obtain an intermediate table; The intermediate table (governance_fact) is used for subsequent statistics and visualization.

[0102] S32 predicts statistical information about the storage governance actions based on the intermediate table; The statistical information includes: spatial statistical information, cost statistical information, and risk level; S321 Determines the spatial statistics; Spatial statistical information includes: space release information and space migration information; The space release information is the space released by the deletion operation: space_release_gb(DELETE). In one embodiment of this specification, space_release_gb(DELETE) = size_gb, that is, the space released by the deletion operation is the size of the managed object.

[0103] The space migration information is the space space for cold backup / archive operations: space_tier_shift_gb(COLD_BACKUP / ARCHIVE). In one embodiment of this specification, space_tier_shift_gb(COLD_BACKUP / ARCHIVE) = size_gb, that is, the space for cold backup / archive operations is the size of the governance object.

[0104] S322 combines price parameters with spatial statistical information to predict cost statistical information; The pricing parameters include: hot storage unit price (price_hot), cold storage unit price (price_cold), and backup cost (backup_cost_once).

[0105] Cost statistics include: average monthly cost savings (cost_saving_monthly) and tiered storage savings.

[0106] In one embodiment of this specification, the monthly cost saving parameter cost_saving_monthly = space release information space_release_gb × hot storage unit price - (backup cost / amort_months); Tiered storage saving parameter tiering_saving_monthly = space migration information space_tier_shift_gb × (hot storage unit price price_hot - cold storage unit price price_cold).

[0107] S323 classifies the risk level of the governance object based on the first basic indicator in the intermediate table; The primary indicators include, but are not limited to: whitelist status, sensitivity level, and access behavior information.

[0108] If the object under governance is in the first state, then the risk level of the object under governance is determined to be the first risk level; wherein, the first state includes: the object under governance is in the whitelist state, and / or the sensitivity of the object under governance is high.

[0109] If the object under governance is in the second state, its risk level is determined to be the second risk level. The second state includes: the object being in a long-term inaccessible state, and the object's size exceeding a preset memory threshold. A long-term inaccessible state means that the number of days since the object's last access is within the preset inaccessible threshold. Preferably, the preset inaccessible threshold is 90-180 days.

[0110] If the object of governance is neither in the first state nor the second state, then the risk level of the object of governance is determined to be the third risk level.

[0111] The risk levels are ranked as follows: First risk level > Second risk level > Third risk level.

[0112] Specifically, the first risk level is high risk, the second risk level is medium risk, and the third risk level is low risk.

[0113] In one embodiment of this specification, the risk level is risk=HIGH; if is_whitelisted=1 or sensitivity=HIGH, but the downstream impact rate is greater than or equal to the preset dependency threshold; the risk level is risk=MED, if last_read_days∈[90,180) and ize_gb>preset memory threshold; otherwise LOW.

[0114] S325 verifies the confidence level.

[0115] Objects with low confidence levels require manual verification for re-marking or reuse of the original mark. In one embodiment of this specification, objects with low confidence levels are marked and notified for manual review.

[0116] S33 constructs dashboard information based on the statistical information and the storage governance strategy; The dashboard information includes: action distribution information, cost saving information, risk distribution information, and a Top N list.

[0117] S331 identifies the governance object corresponding to each of the storage governance actions and obtains action distribution information; Action distribution information includes: quantity distribution information and capacity distribution information; The quantity distribution information represents the number of governance objects corresponding to each storage governance action; The capacity distribution information represents the percentage of the total capacity of the managed objects corresponding to each storage governance action.

[0118] S332 aggregates and processes cost statistics information according to the first basic indicator to obtain cost saving information; The first basic indicator is at least one of the following: storage governance actions, responsible person information, and business domain.

[0119] Specifically, the monthly average cost saving parameters are aggregated based on the first basic indicator to obtain the first saving parameter; the tiered storage saving parameters are aggregated based on the first basic indicator to obtain the second saving parameter; the sum of the first saving parameter and the second saving parameter is used as the cost saving information.

[0120] S333 cross-statistically analyzes the risk level and the storage governance action to obtain risk distribution information; S334 Constructs a TopN list based on the governance object information; The TopN list is a list consisting of the top N tables selected from the candidate tables after sorting them according to a specific metric.

[0121] S334-1 Sort the governance objects in descending order according to the total capacity percentage, extract the governance objects with the target number in turn, and construct the percentage list; The objects being managed consume the most storage costs; deleting or cold-backing them can free up a significant amount of space. S334-2 sorts the governance objects in descending order according to the last read time, extracts the target number of governance objects in sequence, and constructs an activity list; Tables that have not been accessed for a long time have a high risk of redundancy and may be "zombie tables". S334-3 Sort the governance objects in descending order according to downstream dependency, extract the governance objects of the target number in turn, and construct a dependency list; Deleting highly dependent tables carries significant risks and requires careful evaluation or priority protection. S335 marks each of the TopN lists with the stored governance actions and auxiliary decision-making information.

[0122] S34 builds visual charts based on dashboard information; S341 generates pie / bar charts based on motion distribution information; S342 generates bar charts based on cost-saving information; for example, it can rank governance targets based on cost-saving information according to business domain or responsible person information. S343 generates a heatmap / matrix based on risk distribution information; S344 generates table styles based on TopN lists.

[0123] S35 combines the governance object information and the dashboard information to construct an approval package, which serves as the comprehensive governance information; S351 combines the storage governance actions, risk levels, and responsible person information to aggregate the governance objects and generate an execution batch; By aggregating storage governance actions, risk levels, and responsible person information, governance objects with the same storage governance actions, risk levels, and responsible persons are grouped into a single execution batch. This involves batch processing of governance objects based on action consistency to reduce redundant operations, standardizing approval criteria for execution batches based on risk homogeneity, and centralizing the delivery of relevant information to the same responsible parties based on clear accountability, thus avoiding inefficiencies caused by fragmented communication.

[0124] As a preferred option, execute batch governance_batch associated statistics; specifically, execute batch associated release space information and cost statistics.

[0125] S352 is the assembly approval package for each of the aforementioned execution batches; For each execution batch: Identify the number of governance objects, total capacity, cost statistics, and risk level in the current execution batch, and use these as a batch summary; Extract the most recent access time, dependency information, whitelist status, sensitive status, and decision support information of each governance object in each current execution batch as a chain of evidence.

[0126] Find the backup ID / snapshot policy and recovery SLA of the currently executed batch, and determine the prerequisites and rollback plan; among them, set a minimum buffer period, such as recommending that at least 7 days of temporary cold backup be retained for rollback when deleting.

[0127] An approval process is built based on the information of the person in charge to improve the automation, security, and compliance of subsequent storage instruction approvals.

[0128] This invention provides a standardized approval package for subsequent approval and execution stages by outputting lists, charts, summaries, and traceable chains of evidence to both business and technical audiences.

[0129] S36 verifies the execution batch; S361 performs consistency verification on the governance objects; Specifically, the statistical criteria for each governance target must be verified; the actions taken for the same governance target must be consistent across different statistical criteria; and no deletion actions are allowed in the whitelist.

[0130] S362 performs a completeness check on the governance targets; Specifically, the non-empty rate of key fields for each governance object is verified; if the non-empty rate of key fields is ≥99%, the completeness verification is considered passed. The low-confidence list has been marked for manual review.

[0131] S363 performs benefit verification and validation on the governance targets; Specifically, randomly sample the cost statistics of the entities being addressed to check whether the price parameters match the actual price parameters.

[0132] S364 performs a reproducibility verification of the governance targets; Specifically, determine whether the governance object retains the SQL / script and parameter versions, and record timestamps and data snapshots.

[0133] S37 generates comprehensive governance information; This invention summarizes and quantifies the conclusions of the aforementioned table-level governance to form comprehensive governance information that enables decision-making. Specifically, it aggregates dashboard information, visual charts, and approval packages to construct comprehensive governance information.

[0134] In one embodiment of this specification, the layout includes: cover and summary, current status overview, key objects, governance recommendations and batches, methods and definitions, and appendices. The cover and summary include: scope, key conclusions, cost savings, and risk warnings. The current status overview includes: action / risk / domain distribution diagrams. Key objects include: a TopN list and its basis. Governance recommendations and batches include: execution priorities and windows. Methods and definitions include: fields, thresholds, and calculation formulas. The appendix includes: a table-level detailed list.

[0135] After generating comprehensive governance information, export the comprehensive governance information and Excel details in PDF / Word format; include calculation methods and SQL / Notebook links (or numbers); and store intermediate tables, execution batches, and report_meta data in the database for easy traceability and recalculation. The comprehensive governance information is preferably a governance analysis report.

[0136] S38 pushes comprehensive governance information; Specifically, the approval package and batch ID are pushed to the approval system / work order system.

[0137] Based on the aforementioned list output form, historical governance records, business priority tags, and cost parameters (storage unit price, backup fee), this invention evaluates the governance strategy of each governance object through multi-dimensional quantitative scoring and AI / rule-based judgment.

[0138] S4 generates a batch work order task list based on the comprehensive governance information; The storage governance actions in the comprehensive governance information are interpreted as actionable tasks. Based on the storage governance actions, standardized work orders are automatically or semi-automatically generated and distributed to the corresponding responsible persons or teams, ensuring that the storage governance actions smoothly transition from "analysis results" to "execution and implementation," forming a closed loop.

[0139] S41 reads table-level records based on the table-level detail list; The governance details list (Excel / CSV) includes: storing governance actions, confidence levels, risk labels, and benefit calculation results.

[0140] This involves calculating revenue based on the storage unit price and backup costs in the cost parameters, resulting in a revenue calculation outcome that achieves space savings, cost estimation, and risk classification. Revenue Calculation Result = Storage Unit Price Table size before remediation - storage unit price The size of the table after treatment.

[0141] In one embodiment of this specification, the table-level record includes: key information, responsible person information, storage governance actions, risk level, confidence level, and cost saving.

[0142] S42 uses whitelist status, risk level, and low confidence level to specially mark governance targets; If the object to be governed is in a whitelist state, the storage governance action of the object will be marked as protected, and the generation of deletion work orders will be prohibited.

[0143] If the target of governance is at a high risk level, a multi-level approval process will be automatically required.

[0144] If the confidence level of the target being addressed is low, then the labeling requires manual confirmation.

[0145] S43 combines execution batches and special markers to generate work orders; S421 identifies the person in charge of each execution batch; The storage governance strategy of this invention requires the responsible person to agree before the specific storage governance actions can be confirmed and executed.

[0146] Specifically, based on the responsible party mapping table, the governance objects and / or execution batches are mapped to the responsible party accounts.

[0147] The responsible person mapping table includes: governance object → responsible person → corresponding work order system account.

[0148] S422 prioritizes work orders based on storage governance actions and / or risk levels: Based on comprehensive governance information, retrieve and store governance actions and / or risk levels.

[0149] Preferably, when the storage management action is deletion, the work order priority is high; when the storage management action is cold backup, the work order priority is medium; and when the storage management action is protection, the work order priority is no priority.

[0150] When the risk level is high, the work order priority is high; when the risk level is medium, the work order priority is medium; when the risk level is low, the work order priority is low.

[0151] High-priority orders require manual approval; medium-priority orders are automatically approved; and orders with no priority are information-based work orders that only require notification.

[0152] The priority order is: high priority > medium priority > no priority.

[0153] S423 creates work orders based on approval packages; Call the work order system API to create work orders in batches and write them to the work order distribution record table.

[0154] The work order system configuration includes: API / template / field specifications. Specifically, the work order template includes: work order title and work order content.

[0155] In one embodiment of this specification, the work order title rule is: [Storage Governance][Action] Table / Batch Governance Task - {db}.{table} or {batch_id}.

[0156] The fields in the work order content include: table / batch information, storage governance actions, basis summary, risk level, confidence level, preconditions, rollback plan, approval chain, and execution window.

[0157] The table / batch information includes: database name, table name, data size, and last access time. The storage governance action is one of the following: cold backup, deletion, archiving, or protection. The summary is dashboard information, which can be formatted as: no access in the last 180 days, capacity 500GB, estimated monthly savings of xxx yuan. Risk level and confidence level are also included. Preconditions can be formatted as: a full table cold backup (snapshot_xxx) is required. The rollback plan is one of the following: recovery method or SLA. The execution window can be formatted as: off-peak hours, weekend early morning.

[0158] The work order issuance record table includes: work order ID, responsible person, issuance time, and work order status (pending / confirmed / in progress).

[0159] S424 writes back the created work order to obtain the work order task list.

[0160] Write the work order ID back to the work order task list in the governance platform database to maintain traceability.

[0161] The work order task list includes several work order information entries. Work order information includes: storage governance actions, decision support information, execution conditions, and rollback plans. Each governance object / execution batch corresponds to one work order entry.

[0162] S5 executes the corresponding storage management policy based on the work order confirmation instruction; This invention constructs a closed-loop process for work order generation, approval, circulation, execution, and feedback, enabling automatic parsing and analysis of governance objects, generation of batch work orders, and scheduling of execution tasks, thus ensuring the efficient implementation of storage governance actions.

[0163] S51 obtains the responsible person's confirmation log; The responsible person's confirmation log includes: the responsible person's confirmation / rejection / supplementary explanation of the work order.

[0164] In one embodiment of this specification, after a work order is issued, the responsible person receives a notification from email / IM / work order platform. They then process the order within a specified time: if execution is confirmed, the work order enters the approval stage; if the work order is rejected, the reason for rejection is stated, such as business dependency or being on a whitelist. If supplementary explanations are requested, modifications such as adjusting the execution window or adding precautions can be made. The responsible person's actions are recorded and synchronized to the governance platform, generating a responsible person confirmation log.

[0165] S52 approves the work order after the person in charge confirms its execution; S521 automatically determines the approval process based on the risk level of the work order; The approval process involves several approval nodes. These nodes include at least one of the following: the responsible person, the superior responsible person, the security and compliance department, and the database administrator (DBA).

[0166] In one embodiment of this specification, the approval chain for high-risk levels is: responsible person → superior responsible person → security and compliance department → DBA. The approval chain for medium-risk levels is: responsible person → superior responsible person → DBA. The approval chain for low-risk levels is: responsible person → DBA.

[0167] S522 performs approval verification for each approval node; In one embodiment of this specification, the approval verification includes: The review process includes verifying whether the work order content complies with governance guidelines; verifying the reasonableness of the execution plan and rollback scheme; and verifying whether there are any approvals, and whether the current approvals are consistent with the storage governance actions. Approvals include: approval, rejection, and modification.

[0168] During the approval and verification process, approval comments and timestamps are automatically recorded to ensure audit traceability.

[0169] After approval, S53 schedules the work order. In one embodiment of this specification, after approval, the work order enters the "pending execution" state; The executor formulates an execution schedule based on the work order content; determines the execution time window and execution order, where the execution time window is usually during off-peak business periods; Prepare rollback resources for the work order, including but not limited to: snapshots and backup locations; Once the execution plan is confirmed, it is sent to the responsible person and relevant parties. The executor includes at least one of the following: DBA and storage administrator.

[0170] This invention ensures that issued governance work orders flow smoothly among responsible persons, approvers, and executors according to established procedures. A systematic work order approval and execution mechanism reduces the risk of accidental deletion or misoperation. A transparent and traceable task processing chain is established, facilitating auditing and accountability.

[0171] S54 executes work orders; S541 prepares for the execution of the current work order; The executor confirms whether the execution time window for the current work order has arrived; Check if there are any whitelist conflicts involving the governance objects in the current work order: if the governance object is in the whitelist, the execution will be automatically skipped and the reason will be recorded; if the governance object is not in the whitelist, the execution process will proceed.

[0172] Pre-configure rollback resources for the current work order: If the storage management action is a cold backup, then reserve storage space and verify read and write permissions; if the storage management action is deletion, then automatically trigger a snapshot or incremental backup.

[0173] S542 performs storage management actions on the current work order; S542-1 If the storage management action is a cold backup, execute the storage management action and verify the cold backup result after the cold backup; Cold backup includes: full table cold backup and periodic cold backup.

[0174] A full table cold backup refers to performing a complete backup of all data in a single table and storing it in a relatively low-cost, infrequently accessed cold storage. It is typically used for disaster recovery protection of critical tables to ensure that the table can be restored from the backup if it is accidentally deleted, damaged, or requires rollback.

[0175] Periodic cold backup refers to performing cold backups on historical partitions of a data table periodically according to a preset time period (such as daily, weekly, or monthly). This mechanism ensures that historical partition data can be used for data tracing, historical analysis, or recovery after accidental operations.

[0176] Verify the cold backup results, including: comparing data volume (source table vs. backup); verifying hash value consistency; and recording the backup storage path.

[0177] S542-2 If the storage management action is deletion, the deletion operation will be performed after the security check; The security check is a pre-execution security check, which includes: determining whether the cold backup is complete; if the cold backup is not complete, blocking the deletion operation; and determining whether there are downstream dependencies; if there are downstream dependencies, manual confirmation or blocking the deletion operation is required.

[0178] Deletion includes full table deletion and periodic deletion. Full table deletion refers to directly clearing all data records of a table (usually preserving the table structure), and is often used for test data cleaning, removal of expired data, or reconstruction of erroneous data. This operation is relatively risky and generally requires approval and protection mechanisms (such as cold backup or whitelist mechanisms).

[0179] Periodic deletion refers to the automatic cleanup of expired, redundant, or low-value data (such as log data older than 30 days) according to business or storage governance strategies and at preset intervals. This mechanism can free up storage space, reduce storage costs, and maintain the efficient operation of the data warehouse.

[0180] After performing the delete operation, check if the number of rows / partitions in the managed object's table matches expectations. Record the execution log of the delete action.

[0181] During the work order execution phase, this invention utilizes a whitelist protection mechanism to automatically verify whether the execution target is within the whitelist, preventing accidental deletion. A secondary verification is performed immediately after execution to ensure no impact on critical business tables. The protection results are recorded and written to the audit log for easy traceability.

[0182] S543 determines the execution status of storage management actions; If the execution status of the stored governance action is successful, the work order status is updated to "completed"; the result is written to the governance database for subsequent analysis and report generation.

[0183] If a storage governance action fails, a rollback mechanism is automatically triggered, such as restoring the data table using a cold backup. A rollback log is also recorded, and an anomaly alert is pushed to the responsible person and the governance administrator.

[0184] S543 updates the status of work orders based on the work order processing flow. S543-1 Automatic Maintenance Work Order Status Flow: In one embodiment of this specification, the execution sequence of the work order status includes: pending confirmation → pending approval → approved → pending execution → in execution → completed / abnormal.

[0185] S543-2 records each state change; Specifically, each status change is written to ticket_flow_log, including the operator, time, and operation result.

[0186] This invention supports real-time query of work order status, allowing responsible persons and management administrators to track it at any time.

[0187] S55 handles abnormal work orders; Abnormal work orders include: timeout abnormal work orders, transfer abnormal work orders, and execution abnormal work orders.

[0188] If a work order is not confirmed within the specified time, S551 will treat it as an abnormal work order and automatically remind or escalate it to the superior.

[0189] S552 If the approval of a work order is rejected, the work order will be treated as an abnormal work order and returned to the responsible person for modification and resubmission.

[0190] If the execution of a work order fails, the work order will be treated as an execution exception work order. For execution exception work orders, a rollback plan will be triggered and the work order will be marked as a failure pending review.

[0191] This invention originates from the actual needs of enterprises in the implementation and maintenance of data warehouses and big data platforms, especially against the backdrop of continuously rising storage costs and frequent operational risks. It leverages AI to drive storage governance. Based on the generated comprehensive governance information, combined with AI's ability to summarize and categorize text, it automatically parses and generates batch work orders and assigns them to responsible personnel. The work order assignment process includes a summary of the basis, preconditions, and a rollback plan. During the work order flow, multi-level approval links, work order status tracking, and anomaly handling are used to monitor the work order execution process in real time, dynamically capturing abnormal events and automatically triggering alarms or rollbacks. Detailed operation logs and audit records are generated to ensure the transparency and compliance of governance operations. After the work order is completed, the corresponding storage strategy is automatically implemented, thus completing an automated closed-loop process from analysis and interpretation, work order generation, approval flow to strategy execution and feedback. This protects the data flow connection throughout the entire process, the work order batching strategy, risk verification, and rollback control technology.

[0192] Following approved work orders and governance strategies, storage governance operations are completed securely and efficiently, ensuring that storage governance actions (full table cold backup, periodic cold backup, full table deletion, periodic deletion, whitelist status) strictly adhere to execution specifications. The entire operation process is monitored and traceable, and rapid rollback is guaranteed. This invention ensures idempotency and traceability of operations through a policy execution and rollback mechanism (including a complete execution method and technology for anomaly-triggered rollback and recovery). It also enables real-time monitoring and status logging of execution batches (anomaly alarms, approval records, execution logs, and rollback logs) and rapid data recovery.

[0193] The core innovation of this invention lies in AI-driven big data cluster storage governance. It employs a multi-dimensional table-level governance strategy algorithm, comprehensively considering table access frequency, size, dependencies, sensitivity tags, whitelist status, and historical governance records to quantify table importance and governance priority from multiple dimensions. It supports a hybrid judgment system combining rules and AI, dynamically outputting cold backup, deletion, archiving, or protection policies, thus solving the problems of low efficiency and error-proneness in traditional manual static judgment.

[0194] This invention, through the deep integration of AI, achieves intelligent storage governance for ultra-large-scale big data clusters, comprehensively addressing the pain points of inefficiency, inaccuracy, and high risk inherent in traditional manual governance. Specifically: the data analysis efficiency for storage governance is reduced from hours to minutes using traditional manual processing, an efficiency improvement of approximately 90%. The entire storage governance process (analysis, communication, and governance) is shortened from days offline to ten minutes online, an overall efficiency improvement of approximately 99%. By integrating multi-dimensional data, expert experience, and AI reasoning capabilities, this invention generates storage governance strategies with an accuracy rate of up to 85%. The entire process of this invention is automated online, from data analysis and strategy confirmation to strategy execution, reducing the risks of manual operation, ensuring traceability and auditability, and significantly improving governance security and compliance.

[0195] The following combination Figure 2 Briefly describe the specific interactive process for generating governance analysis reports (S1-S3).

[0196] Figure 2 The roles involved in the process include: Operations personnel: the party initiating the "storage analysis" request (usually a system operations engineer); Big data experts: the intermediary responsible for "intent understanding" and "report distribution"; Cluster governance agent: the core execution component that performs "data acquisition," "analysis," and "report generation"; MCP: the data source that provides "stored data" and "lineage data." Among these, the cluster governance agent marked in the flowchart of this invention, i.e., the AI ​​cluster governance intelligent agent described in this invention, is the core module deployed within the big data cluster for intelligent decision-making regarding data table storage governance.

[0197] The interaction process mainly includes: operations and maintenance personnel sending a storage analysis request to a big data expert → indicating that the operations and maintenance personnel want to analyze and optimize "storage issues"; the big data expert internally processes the intent parsing and routing; specifically, the big data expert first understands the operations and maintenance personnel's "storage analysis" needs through LLM intent analysis (LLM large language model, responsible for parsing natural language requirements); then, through routing distribution, the requirements are forwarded to the cluster governance agent for LLM intent analysis; the cluster governance agent initiates two data requests to the MCP: the MCP returns storage data and lineage data (dependency information) respectively; the cluster governance agent performs intelligent analysis on the data to generate a governance analysis report, and feeds it back to the big data expert and operations and maintenance personnel in sequence.

[0198] The following combination Figure 3 Briefly describe the interaction process of work order processing (S4-S5).

[0199] Figure 3 The roles involved in the process include: Operations personnel: initiators of requests and reviewers / confirmers of the work order process. Governance Agent: the "transfer station" and initial processor of requests (integrating requests and generating work orders). Work Order System: the management center for the entire lifecycle of work orders (creation, notification, status synchronization, and end notification). DDS / DAM: the business-side system (presumably a "data / policy management system," responsible for interfacing with users and executing policies). Users: confirmers of work order content and recipients of policy execution results.

[0200] The interaction process mainly includes: operations personnel initiating a request to the governance agent; the governance agent internally processing the request (summarizing LLM content + batch creating work orders), and then creating a work order in the work order system; based on the user's modification / confirmation, the work order system returns the result to the governance agent; the governance agent provides feedback to the operations personnel; based on the operations personnel's review and confirmation, the work order system confirms "executable"; the work order system sends a callback to DDS / DAM to execute the strategy, triggering DDS / DAM to execute the preset strategy; after the strategy is implemented, the work order status is marked as "completed" and DDS / DAM notifies the user through Teams.

[0201] Figure 4 This is a schematic diagram of the structure of a big data storage governance system driven by an AI cluster governance agent, provided in an embodiment of this specification. The system is deployed in a multi-node distributed big data cluster, interfacing with the cluster source system, the cluster metadata repository, and the distributed storage layer. It identifies massive amounts of data tables across the cluster as governance objects in batches, specifically including: The data acquisition module 410 is used to identify governance objects within the cluster, collect information about these objects, and generate a candidate list. AI-driven strategy prediction module 420 is used to predict storage governance strategies based on the candidate list by an AI cluster governance agent. The comprehensive management module 430 is used to generate comprehensive governance information based on the storage governance strategy by utilizing the analytical capabilities of the AI ​​cluster governance agent. Work order creation module 440 is used to generate a batch work order task list based on the comprehensive governance information; The work order execution module 450 is used to execute the corresponding storage governance strategy according to the work order confirmation instruction; The AI-driven strategy prediction module 420 includes: The AI-driven strategy prediction module 420 integrates the AI ​​cluster governance agent, which includes at least: The feature extraction submodule is used to extract the decision features of the governance objects from the candidate list; The intelligent agent decision-making submodule is used to analyze the decision features through an AI decision-making model that integrates rules and machine learning, and predict the storage governance actions of the governance object; The importance scoring submodule is used to analyze the basic information of the governance object based on natural language processing technology driven by a large language model; calculate the correlation between the basic information and a preset relational database to generate a table importance score; and determine the semantic importance score based on the table importance score. The auxiliary decision-making information generation submodule is used to generate auxiliary decision-making information based on the semantic importance score; The confidence verification submodule is used to verify the confidence level of the storage governance action; The strategy encapsulation submodule is used to integrate the storage governance actions, the auxiliary decision information, and the confidence level to generate the storage governance strategy.

[0202] Optionally, the integrated management module 430 includes: The intermediate table construction submodule is used to integrate the data of the governance objects based on the storage governance strategy and generate an intermediate table. The statistical information prediction submodule is used to predict the statistical information of the storage governance actions based on the intermediate table; The dashboard information construction submodule is used to construct dashboard information based on the statistical information and the storage governance strategy. The approval package construction submodule is used to construct an approval package by combining the governance object information and the dashboard information, which serves as the comprehensive governance information; wherein, the governance object is aggregated by combining the stored governance actions, risk levels, and responsible person information to generate execution batches; and an approval package is assembled for each execution batch.

[0203] Optionally, the acquisition module 410 includes: The object identification submodule is used to identify governance objects within the big data cluster from the source systems, metadata repositories, and distributed storage layers that are connected to the big data cluster. The information extraction submodule is used to extract information about each of the governance objects. The information aggregation submodule is used to aggregate the governance objects and their corresponding governance object information to construct the candidate list.

[0204] Optionally, the statistical information includes: spatial statistical information, cost statistical information, and risk level; The statistical information prediction submodule includes: Spatial prediction unit, used to determine the spatial statistical information; A cost prediction unit is used to predict the cost statistics by combining price parameters with the spatial statistics. The risk prediction unit is used to classify the risk level of the governance object based on the first basic indicator in the intermediate table.

[0205] Optionally, the dashboard information includes: action distribution information, cost saving information, risk distribution information, and a TopN list; The dashboard information construction submodule includes: An action recognition unit is used to identify the governance object corresponding to each of the stored governance actions and obtain the action distribution information; An aggregation unit is used to aggregate the cost statistics information according to the first basic indicator to obtain the cost saving information. A cross-validation unit is used to cross-validate the risk level and the storage governance action to obtain the risk distribution information; A list construction unit is used to construct the TopN list based on the governance object information; The list optimization unit is used to mark the storage governance actions and auxiliary decision-making information for each of the TopN lists.

[0206] Optionally, the AI ​​cluster governance agent further includes: a model self-iteration unit; The model self-iteration unit is used to collect data from manual review and work order execution logs and feed it back to train the AI ​​decision-making model, thereby continuously optimizing the decision-making accuracy of the AI ​​cluster governance agent.

[0207] The functions of the system in this embodiment have been described in the above method embodiments. Therefore, for any parts not detailed in this embodiment, please refer to the relevant descriptions in the foregoing embodiments, which will not be repeated here.

[0208] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this specification. The electronic device includes a memory 501 and a processor 502. The memory 501 is used to store computer-executable instructions. When the computer-executable instructions are executed by the processor 502, they can implement the steps of the above-described method embodiments.

[0209] Figure 6 This is a schematic diagram of the structure of a computer-readable storage medium provided in an embodiment of this specification. The computer-readable storage medium 600 stores one or more computer programs, which, when executed by a processor, can implement the steps of the above-described method embodiments.

[0210] This specification also provides a computer program product, including a computer program / computer executable instructions, which, when executed by a processor, can implement the steps of the above-described method embodiments.

[0211] Those skilled in the art will understand that all or part of the processes in the above method embodiments can be implemented by a computer program instructing related hardware. When the computer program is executed, it may include the processes of the embodiments of the above methods.

[0212] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. An AI cluster governance intelligent agent driven big data storage governance system, characterized in that, The system is deployed on a multi-node distributed big data cluster, connecting to the cluster source system, cluster metadata repository, and distributed storage layer. It identifies massive amounts of data tables across the cluster as governance objects, specifically including: The data collection module is used to identify governance objects within the cluster, collect information about these objects, and generate a candidate list. The AI-driven strategy prediction module is used to predict storage governance strategies based on the candidate list through an AI cluster governance agent. The integrated management module is used to generate comprehensive governance information based on the storage governance strategy by utilizing the analytical capabilities of the AI ​​cluster governance agent. The work order creation module is used to generate a batch work order task list based on the comprehensive governance information. The work order execution module is used to execute the corresponding storage governance strategy according to the work order confirmation instruction; The AI-driven strategy prediction module integrates the AI ​​cluster governance agent, which includes at least: The feature extraction submodule is used to extract the decision features of the governance objects from the candidate list; The intelligent agent decision-making submodule is used to analyze the decision features through an AI decision-making model that integrates rules and machine learning, and predict the storage governance actions of the governance object; The importance scoring submodule is used to analyze the basic information of the governance object based on natural language processing technology driven by a large language model; calculate the correlation between the basic information and a preset relational database to generate a table importance score; and determine the semantic importance score based on the table importance score. The auxiliary decision-making information generation submodule is used to generate auxiliary decision-making information based on the semantic importance score; The confidence verification submodule is used to verify the confidence level of the storage governance action; The strategy encapsulation submodule is used to integrate the storage governance actions, the auxiliary decision information, and the confidence level to generate the storage governance strategy.

2. The AI swarm governance intelligent agent driven big data storage governance system of claim 1, wherein, The integrated management module includes: The intermediate table construction submodule is used to integrate the data of the governance objects based on the storage governance strategy and generate an intermediate table. The statistical information prediction submodule is used to predict the statistical information of the storage governance actions based on the intermediate table; The dashboard information construction submodule is used to construct dashboard information based on the statistical information and the storage governance strategy. The approval package construction submodule is used to construct an approval package by combining the governance object information and the dashboard information as comprehensive governance information; wherein, the governance object is aggregated by combining the stored governance actions, risk level, and responsible person information to generate execution batches; and an approval package is assembled for each execution batch. 3.The AI cluster governance intelligent agent driven big data storage governance system of claim 1, wherein, The acquisition module includes: The object identification submodule is used to identify governance objects within the big data cluster from the source systems, metadata repositories, and distributed storage layers that are connected to the big data cluster. The information extraction submodule is used to extract information about each of the governance objects. The information aggregation submodule is used to aggregate the governance objects and their corresponding governance object information to construct the candidate list. 4.The AI cluster governance intelligent agent driven big data storage governance system of claim 2, wherein, The statistical information includes: spatial statistical information, cost statistical information, and risk level; The statistical information prediction submodule includes: Spatial prediction unit, used to determine the spatial statistical information; A cost prediction unit is used to predict the cost statistics by combining price parameters with the spatial statistics. The risk prediction unit is used to classify the risk level of the governance object based on the first basic indicator in the intermediate table. 5.The AI cluster governance intelligent agent driven big data storage governance system of claim 4, wherein, The dashboard information includes: action distribution information, cost saving information, risk distribution information, and a TopN list; The dashboard information construction submodule includes: An action recognition unit is used to identify the governance object corresponding to each of the stored governance actions and obtain the action distribution information; An aggregation unit is used to aggregate the cost statistics information according to the first basic indicator to obtain the cost saving information. A cross-validation unit is used to cross-validate the risk level and the storage governance action to obtain the risk distribution information; A list construction unit is used to construct the TopN list based on the governance object information; The list optimization unit is used to mark the storage governance action and the auxiliary decision information for each of the TopN lists.

6. An AI cluster governance intelligent agent driven big data storage governance method, characterized in that, The method is applied to a runtime environment deploying a multi-node distributed big data cluster, interfacing with the cluster source system, cluster metadata repository, and distributed storage layer. It identifies massive amounts of data tables across clusters as governance objects, specifically including: Identify governance objects within the cluster and collect their information to generate a candidate list; The storage governance strategy is predicted based on the candidate list by an AI cluster governance agent. This includes: extracting decision features of governance objects from the candidate list; analyzing the decision features using an AI decision model that integrates rules and machine learning to predict the storage governance actions of the governance objects; analyzing the basic information of the governance objects using natural language processing technology driven by a large language model; calculating the correlation between the basic information and a preset relational database to generate a table importance score; determining a semantic importance score based on the table importance score; generating auxiliary decision information based on the semantic importance score; verifying the confidence level of the storage governance actions; and integrating the storage governance actions, the auxiliary decision information, and the confidence level to generate the storage governance strategy. Utilizing the analytical capabilities of the AI ​​cluster governance agent, comprehensive governance information is generated based on the storage governance strategy; Based on the comprehensive governance information, a batch production work order task list is generated. Execute the corresponding storage governance policy according to the work order confirmation instruction.

7. The big data storage governance method driven by AI cluster governance intelligent agent as described in claim 6, characterized in that, The step of utilizing the analytical capabilities of the AI ​​cluster governance agent to generate comprehensive governance information based on the storage governance strategy includes: Based on the storage governance strategy, the data of the governance objects is integrated to generate an intermediate table; Statistical information for predicting the storage governance actions based on the intermediate table; Dashboard information is constructed based on the statistical information and the storage governance strategy; An approval package is constructed by combining the governance object information and the dashboard information as comprehensive governance information; wherein, the governance object is aggregated by combining the stored governance actions, risk levels, and responsible person information to generate execution batches; and an approval package is assembled for each execution batch.

8. The big data storage governance method driven by AI cluster governance intelligent agent as described in claim 6, characterized in that, The process of identifying governance objects within the cluster, collecting governance object information, and generating a candidate list includes: Identify governance objects within the big data cluster from the source systems, metadata repositories, and distributed storage layers that are connected to the big data cluster; Extract information for each of the governance objects; The candidate list is constructed by summarizing the information of the governance objects and their corresponding governance objects.

9. The big data storage governance method driven by AI cluster governance intelligent agent as described in claim 7, characterized in that, The statistical information includes: spatial statistical information, cost statistical information, and risk level; The statistical information used to predict the storage governance actions based on the intermediate table includes: Determine the spatial statistics; The cost statistics are predicted by combining price parameters with the spatial statistics. The risk level of the governance object is classified based on the first basic indicator in the intermediate table.

10. The AI ​​cluster governance agent-driven big data storage governance method as described in claim 9, characterized in that, The dashboard information includes: action distribution information, cost saving information, risk distribution information, and a TopN list; The construction of dashboard information based on the statistical information and the storage governance strategy includes: Identify the governance object corresponding to each of the storage governance actions to obtain the action distribution information; The cost statistics are aggregated and processed according to the first basic indicator to obtain the cost saving information; By cross-statistically analyzing the risk level and the storage governance actions, the risk distribution information is obtained. The TopN list is constructed based on the governance object information; For each of the TopN lists, mark the storage governance action and the auxiliary decision information.

11. A computer device, characterized in that, The computer device includes: Processor; and, A memory storing computer-executable instructions, which, when executed, cause the processor to perform the method of any one of claims 6-10.

12. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores one or more programs / instructions, which, when executed by a processor, implement the method as described in any one of claims 6-10.

13. A computer program product, characterized in that, Includes a computer program / instruction, which, when executed by a processor, implements the method as described in any one of claims 6-10.

Images