Automatic offline method and device for large database table, electronic equipment and storage medium

Abstract

The invention relates to the technical field of big data, can be applied to the field of science and technology finance / digital medical treatment, and discloses an automatic offline method and device for a big database table, electronic equipment and a storage medium. The method comprises the following steps: setting a permission approval person of a virtual user in a target distributed computing framework as an owner of the virtual user, and synchronizing owner information and permission configuration change information; automatically synchronizing dependency, authorization records and auditing records at the downstream of the library table to a data middle table; performing automatic cleaning and conversion on downstream dependency, authorization records and auditing records of the library table; generating a unified wide table according to downstream dependency, authorization records and auditing records of the library table; receiving an input library table to be evaluated offline, generating evaluation information and virtual user owner information of the library table to be evaluated offline, and sending an offline notification to an associated party for confirmation; and receiving feedback confirmation information returned by the related party, and performing offline processing on the to-be-offline evaluation library table. According to the method, the automation degree and efficiency of large database table offline are improved.

Description

Technical Field

[0001] The present application relates to the field of big data technology and can be applied to the fields of science and technology finance / digital medicine, and in particular to a method, device, electronic device and storage medium for automatically shutting down a large database table. Background Art

[0002] In the full life cycle management of databases and tables in the field of big data management based on Hadoop, when a database or table is about to enter the offline stage, it is necessary to conduct pre-offline analysis and evaluation, and it is necessary to integrate the data of various related platforms. However, the existing technology has deficiencies in information fusion, transparency, reusability and automation, and it is difficult to form an automated and rapid offline analysis closed loop. The specific problems are as follows: First, the Hadoop cluster is large in scale, with many virtual users hduser, and its owner information is not updated in a timely manner, is not transparent, is not easy to use and is difficult to reuse. For example, in the financial field, a bank's Hadoop cluster may have a large number of hdusers corresponding to financial data managers in different business departments. If the owner information is updated lagging behind, it may cause personnel in other departments to be unable to Accurately knowing the ownership of data makes it difficult to quickly determine the owner responsible for approval when conducting cross-departmental database and table offline assessments, affecting the assessment progress; second, database and table task scheduling information, authorization information, actual access information, etc. are scattered across independent platforms, lacking correlation analysis and use, and lacking automated integration and evaluation analysis. For example, in the medical field, in the hospital's Hadoop platform, the task scheduling information of patient data-related databases and tables may be on the hospital's internal business system platform, authorization information on the authority management platform, and actual access information on the audit platform. The independence of each platform leads to medical staff manually collecting and analyzing information from multiple platforms when assessing whether patient data-related databases and tables are offline. This is not only time-consuming and labor-intensive, but also prone to information omissions or errors. Summary of the Invention

[0003] The main technical problem solved by the embodiments of the present application is the inefficiency of information integration and automated processing in the existing Hadoop database and table offline evaluation.

[0004] In order to solve the above technical problems, the first technical solution adopted by the implementation method of the present application is: to provide a method for automatically taking a large database table offline, including: in a pre-deployed unified management platform, setting the permission approver of the virtual user in the target distributed computing framework as the owner of the virtual user, and synchronizing the owner information and the change information of the permission configuration corresponding to the owner through the unified management platform; automatically synchronizing the downstream dependencies, authorization records, and audit records of the library table from the task scheduling platform, authorization management platform, and audit management platform to the data middle platform, and obtaining downstream tables, queryable users, and historical access records; based on the downstream dependencies of the library table, the authorization records, and the audit records synchronized to the data middle platform, according to the preset offline evaluation rules Perform automated cleaning and conversion; generate a unified wide table based on the results of cleaning, conversion and integration of the downstream dependencies of the library table, the authorization records and the audit records, and store the unified wide table in the data middle platform; after receiving the input of the evaluation library table to be taken offline, automatically associate the unified wide table with the owner information synchronized with the unified management platform through the data middle platform, generate the evaluation information and virtual user owner information of the evaluation library table to be taken offline, and send an offline notification to the related party for confirmation; after receiving the feedback confirmation information returned by the related party, automatically record the feedback confirmation information in the data middle platform, and perform offline processing on the evaluation library table to be taken offline based on the evaluation information in the unified wide table and the preset offline rules.

[0005] Optionally, the step of automatically synchronizing the downstream dependencies, authorization records, and audit records of the library table from the task scheduling platform, authorization management platform, and audit management platform to the data middle platform includes: obtaining the stored task configuration file by calling the task scheduling query application program interface, parsing the input table and output table fields in the task configuration file, and building a library table dependency chain based on the input table and output table fields; based on the owner information synchronized by the unified management platform, associating each downstream table with the corresponding virtual user information, and synchronizing the dependency chain and the associated virtual user information to the data middle platform; obtaining the authorization record of the library table by calling the permission query application program interface; matching the authorization record with the owner information in the unified management platform, generating an authorization list containing the owner and the responsible person, and synchronizing the authorization list to the permission management table of the data middle platform; obtaining all access records of the library table within a preset historical period by calling the log analysis application program interface; associating the access record with the owner information in the unified management platform, generating historical access heat distribution data of preset categories, and synchronizing the historical access heat distribution data to the data middle platform.

[0006] Optionally, the step of automatically cleaning and converting according to preset offline evaluation rules includes: screening the task scheduling records in the downstream dependencies of the library table according to the task execution frequency and the last execution time, and removing the task dependencies that have no execution records in the last preset first number of consecutive days; classifying and counting the virtual user access rights in the authorization records, generating an authorized user list for each library table, and marking the permissions that have not been used for a preset second number of consecutive days; based on the audit records, calculating the library table heat value according to the average daily visits, data change frequency and business value score of each library table, wherein the business value score is dynamically generated according to the preset field importance weight and access heat; based on the authorized user list and the library table heat value, performing correlation analysis on the library table downstream dependencies, the authorization records and the audit records, and identifying library tables that have no downstream dependencies, no valid authorization and have not been accessed within a preset time range.

[0007] Optionally, after the step of identifying a library table that has no downstream dependency, no valid authorization, and has not been accessed within a preset time range, the step further includes: if the number of downstream task executions of the library table is 0 within the last preset third number of days, and the library table has no valid authorized user access record, then setting the corresponding library table as a high-priority offline library table; if the average daily access volume of the library table is lower than the preset fourth number, and the business value score is within the last preset first percentage of all library tables, then setting the corresponding library table as a medium-priority offline library table; if the library table has a valid authorized user access record, but the data change frequency of the library table is 0 within the preset fifth number of days, then setting the corresponding library table as a library table in a state to be observed; and storing the information data of the high-priority offline library table, the medium-priority offline library table, and the library table in a state to be observed in the unified wide table.

[0008] Optionally, after the step of executing offline processing on the library table to be evaluated for offline processing based on the evaluation information in the unified wide table and the preset offline rules, the step further includes: constructing a library table offline impact prediction model through a preset machine learning algorithm, and using historical library table offline records, historical library table authorization change records, and historical library table access records to train the library table offline impact prediction model to obtain the trained library table offline impact prediction model; after receiving a newly input library table to be evaluated for offline processing, inputting the library table to be evaluated for offline processing into the library table offline impact prediction model to obtain offline risk assessment data after the offline operation is predicted for the library table to be evaluated for offline processing; after the offline operation is executed on the library table to be evaluated for offline processing, obtaining the business system response time, throughput, and dependency change data of the associated library tables as first training data; combining the information data of the library table to be evaluated for offline processing, the offline risk assessment data, and the first training data as optimized training data; performing incremental training on the library table offline impact prediction model through the optimized training data, and optimizing the parameters of the library table offline impact prediction model according to the training results.

[0009] Optionally, after the step of automatically synchronizing the downstream dependencies, authorization records, and audit records of the library tables from the task scheduling platform, the authorization management platform, and the audit management platform to the data middle platform, it also includes: using the historical synchronized data in the data middle platform to train an artificial intelligence-based library table anomaly detection model, the library table anomaly detection model uses an isolation forest algorithm or a long short-term memory network to identify abnormal data in task scheduling frequency, authorization change mode, and audit access behavior; using the library table anomaly detection model to perform real-time analysis on the downstream dependencies of the library tables, the authorization records, and the audit records synchronized to the data middle platform, and generate an anomaly detection model for each library table. Regular scoring; if the anomaly score exceeds the preset first threshold, the first level warning is automatically triggered, prompting the administrator to manually review the offline database table; if the anomaly score exceeds the preset second threshold and occurs for the first preset number of times in succession, the second level warning is automatically triggered, and the offline evaluation process of the corresponding database table is suspended; the abnormal database table with anomaly scores exceeding the first threshold and the second threshold, as well as the context information of the abnormal database table, are stored in the anomaly record table of the data center; the data of the anomaly record table is used to continuously optimize and train the database table anomaly detection model, and the currently running database table anomaly detection model is updated according to the results of the optimization training.

[0010] Optionally, after the step of performing offline processing on the evaluation library table to be offline based on the evaluation information and preset offline rules in the unified wide table, it also includes: deploying a workflow engine in the data middle platform to encapsulate the processing operations of data synchronization, cleaning and conversion, evaluation grading, and approval notification into configurable atomic tasks; through the workflow engine interface, sending instructions to the task scheduling platform to suspend scheduling tasks that have dependencies on the library table to be offline, sending instructions to the authorization management platform to freeze new authorizations for the library table to be offline, and sending instructions to the audit management platform to start a full audit of change operations on the library table to be offline; automatically executing the atomic task according to the preset process through the workflow engine, if the number of task failures is less than or equal to the second preset number, it is re-executed, if the number of failures is greater than the second preset number, it is switched to the backup data source or backup processing node for re-execution; generating a library table offline processing report in the management interface of the data middle platform, wherein the library table offline processing exposure includes data processing time, risk assessment results and approval chain records.

[0011] In order to solve the above technical problems, the second technical solution adopted in the implementation method of the present application is: to provide a large database table automatic offline device, including: a unified management platform module, which is used to set the permission approver of the virtual user in the target distributed computing framework as the owner of the virtual user in the pre-deployed unified management platform, and synchronize the owner information and the change information of the permission configuration corresponding to the owner through the unified management platform; a data automatic synchronization module, which is used to automatically synchronize the downstream dependencies, authorization records, and audit records of the library table from the task scheduling platform, authorization management platform and audit management platform to the data middle platform, and obtain downstream tables, queryable users and historical access records; a data cleaning and conversion module, which is used to follow the preset offline evaluation rules based on the downstream dependencies of the library table, the authorization records and the audit records synchronized to the data middle platform Perform automated cleaning and conversion; a unified wide table management module is used to generate a unified wide table after the downstream dependencies of the library table, the authorization records and the audit records are cleaned, converted and integrated, and store the unified wide table in the data middle platform; an offline notification management module is used to receive the input of the library table to be offline for evaluation, automatically associate the unified wide table with the owner information synchronized with the unified management platform through the data middle platform, generate the evaluation information and virtual user owner information of the library table to be offline for evaluation, and send an offline notification to the related party for confirmation; a library table offline execution module is used to automatically record the feedback confirmation information returned by the related party to the data middle platform, and perform offline processing on the library table to be offline for evaluation based on the evaluation information in the unified wide table and the preset offline rules.

[0012] In order to solve the above technical problems, the third technical solution adopted in the implementation of the present application is: to provide an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the large database table automatic offline method as described above.

[0013] In order to solve the above technical problems, the fourth technical solution adopted in the embodiment of the present application is: to provide a non-volatile computer-readable storage medium, characterized in that the non-volatile computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are executed by an electronic device, the electronic device executes the large database table automatic offline method as described above.

[0014] Different from the related technologies, this application realizes the centralized management and change synchronization of owner information and permission configuration through a unified management platform to ensure clear data permission responsibilities; automatically synchronizes multi-platform data to the data middle platform and cleans and converts it to improve data quality; generates a unified wide table to realize multi-source data integration and storage, providing standardized support for evaluation; automatically associates data to generate evaluation information and triggers notification confirmation to reduce manual intervention; performs offline processing based on feedback to improve the efficiency and accuracy of database and table offline, and enhance data security and resource utilization. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] One or more embodiments are exemplarily illustrated by corresponding drawings, which do not constitute limitations on the embodiments. Elements with the same reference numerals in the drawings are represented as similar elements, and unless otherwise stated, the figures in the drawings do not constitute proportional limitations.

[0016] Figure 1 This is a schematic diagram of the operating environment of the large database table automatic offline method provided in the embodiment of the present application.

[0017] Figure 2 It is a schematic diagram of the execution flow of the large database table automatic offline method provided in the embodiment of the present application.

[0018] Figure 3 This is a schematic diagram of the execution flow of data synchronization in the automatic offline method for a large database table provided in an embodiment of the present application.

[0019] Figure 4 This is a schematic diagram of the execution flow of data synchronization in the automatic offline method for a large database table provided in an embodiment of the present application.

[0020] Figure 5 It is a system structure diagram of the large database table automatic offline device provided in an embodiment of the present application.

[0021] Figure 6 This is a hardware structure diagram of an electronic device for executing the automatic offline method for a large database table provided in an embodiment of the present application. DETAILED DESCRIPTION

[0022] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0023] It should be noted that, if there is no conflict, the various features in the embodiments of the present application can be combined with each other and are all within the scope of protection of the present application. In addition, although the functional modules are divided in the device schematics and the logical order is shown in the flow charts, in some cases, the steps shown or described can be performed in a different order than the module division in the device schematics or the order in the flow charts.

[0024] Unless otherwise defined, all technical and scientific terms used in this specification have the same meanings as those commonly understood by those skilled in the art to which this application belongs. The terms used in this specification are intended only to describe specific embodiments and are not intended to limit this application. The term "and / or" as used in this specification includes any and all combinations of one or more of the relevant listed items.

[0025] To facilitate understanding of this embodiment, firstly, a large database table automatic offline method disclosed in the embodiment of this application is introduced in detail. Figure 1 , Figure 1 This is a schematic diagram of the operating environment of the large database table automatic offline method provided in the embodiment of the present application, such as Figure 1 As shown, the execution subject of the large database table automatic offline method provided in the embodiment of the present application is generally an electronic device with certain computing capabilities, such as a computer device. In some possible implementations, the large database table automatic offline method can be implemented by a processor calling computer-readable instructions stored in a memory. Figure 1 The computer device in the above can be a server. The server can be an independent server or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. It can be understood that Figure 1 The number of computer devices in the figure is only for reference and can be expanded to any number according to actual needs.

[0026] Please continue reading Figure 2 , Figure 2 This is a schematic diagram of the execution flow of the large database table automatic offline method provided in the embodiment of the present application, such as Figure 2 As shown, the process includes the following steps S1 to S6.

[0027] S1. In the pre-deployed unified management platform, the permission approver of the virtual user in the target distributed computing framework is set as the owner of the virtual user, and the owner information and the change information of the permission configuration corresponding to the owner are synchronized through the unified management platform.

[0028] Among them, the unified management platform is the core hub for implementing permission and owner information management. It is usually built on a distributed system architecture and uses a metadata storage module to centrally store and manage virtual user owner information, including the department to which they belong, the person in charge, and contact information, as well as the corresponding permission configuration, such as data query, write, delete, etc., in a structured manner. For virtual users in the target distributed computing framework, they are themselves stateless system-level accounts used to perform various data operations. The unified management platform establishes a logical mapping relationship to associate virtual users with actual owners (business departments or responsible persons) and clarify the responsible parties for permission approval. When it comes to changes in virtual user permissions, the owner can approve requests for permission addition, modification, revocation, etc. through the workflow engine or approval interface within the platform. The approval process is executed according to pre-set rules, and multi-level approval can also be implemented for changes that exceed a certain permission range.

[0029] Regarding information synchronization, the unified management platform utilizes change data capture technology. Once changes occur in owner information or permission configuration, the platform detects these changes in real time and, through an event-driven architecture and communication mechanisms like message queues, automatically synchronizes these changes to related systems or modules. This ensures consistency of owner information and permission configuration across all platforms, providing an accurate data foundation for subsequent operations such as data processing, permission control, and offline evaluation of databases and tables based on this information.

[0030] As an example, in the financial sector, a large bank built a unified management platform to manage data permissions. Multiple virtual users exist within the bank's internal distributed computing framework, processing core business data such as credit approval and transaction settlement. Through the unified management platform, the virtual user's permission approver, responsible for processing credit approval data, is assigned to the head of the credit risk control department. This head can manage the virtual user's permission configuration, such as limiting it to reading and processing credit application data within a specific time period. When risk control strategies are adjusted and the virtual user's access rights to high-risk customer data need to be changed, the unified management platform automatically synchronizes the permission configuration change information with relevant systems to ensure the compliance and security of data access.

[0031] As another example, in the healthcare industry, a tertiary hospital utilizes a unified management platform to manage medical data permissions. Within the hospital's distributed computing framework, virtual users are responsible for data processing tasks such as electronic medical record storage and medical image analysis. For electronic medical record data, the unified management platform sets the virtual user permission approver responsible for electronic medical record storage as the director of the hospital's information management department. This director can configure virtual user permissions, stipulating that only authorized doctor and nurse accounts can read the medical records of patients in the corresponding department through virtual users. When a patient transfers to another department or is discharged, and medical record access rights need to be adjusted, the unified management platform synchronizes changes to the owner information and permission configuration in real time to ensure that medical data access complies with patient privacy protection regulations and hospital management standards, preventing improper access and leakage of medical data.

[0032] S2. Automatically synchronize the downstream dependencies, authorization records, and audit records of the database tables from the task scheduling platform, authorization management platform, and audit management platform to the data center to obtain downstream tables, queryable users, and historical access records.

[0033] Among them, step S2 mainly relies on cross-platform data interaction and integration technology to achieve efficient aggregation of multi-source data. The task scheduling platform, authorization management platform and audit management platform assume different functions, and each stores key information related to the library table. The task scheduling platform stores task configuration metadata. By parsing the input and output table fields in the task process, it can sort out the downstream dependency chain between the library tables. The authorization management platform maintains complete permission control data and can obtain the authorized user list, permission level and change history of the library table. The audit management platform records the full life cycle operation log of the library table, including historical access records such as user access time, operation type, and data reading volume.

[0034] In addition, to achieve data synchronization, application program interface (API) calls or data middleware technology can be used to build standardized data extraction channels. For example, by writing specific data extraction scripts and periodically calling the query APIs of each platform, raw data such as dependencies, authorization records, and audit logs can be converted into a unified format. Then, using message queues or ETL (extraction, transformation, loading) tools, the data is pushed to the data middle platform in a streaming or batch transmission manner. Within the data middle platform, the metadata management module is used to clean, transform, and structure the received data, ultimately forming a complete data set of downstream tables, queryable users, and historical access records. This data provides comprehensive data support for subsequent offline evaluation of database tables, anomaly detection, and other operations. The entire process uses an automated scheduling mechanism to ensure the real-time and accuracy of the data, avoiding efficiency losses and data errors caused by manual intervention.

[0035] As an optional implementation, after the above step S2, a library table anomaly detection model can be constructed through artificial intelligence, and the model can be used to assist in library table offline analysis, which can specifically include the following steps S21 to S26.

[0036] S21. Use the historical synchronized data from the data center to train an AI-based database table anomaly detection model. The database table anomaly detection model uses the isolation forest algorithm or long short-term memory network to identify abnormal data in task scheduling frequency, authorization change patterns, and audit access behavior.

[0037] Based on the multi-dimensional data historically synchronized with the data center, we use the Isolation Forest algorithm (suitable for outlier detection) or the Long Short-Term Memory (LSTM) network (excels at time series analysis) to build a database and table anomaly detection model. The Isolation Forest algorithm rapidly identifies outliers that deviate from the normal distribution by randomly partitioning the data space, while the LSTM can capture unusual fluctuations in task scheduling frequency and authorization change patterns over time, providing a technical foundation for subsequent anomaly detection.

[0038] S22. Use the library table anomaly detection model to perform real-time analysis on the downstream dependencies, authorization records, and audit records of the libraries and tables synchronized to the data center, and generate an anomaly score for each library table.

[0039] The trained model performs in-depth analysis of real-time synchronized database and table downstream dependencies, authorization records, and audit logs. The model first extracts data features and converts them into vectors. Then, based on pre-trained rules, it matches these features with normal patterns and calculates an anomaly score that reflects the degree of deviation from the database and table data. If unexpected changes in dependencies or frequent modifications to authorization records are detected, the anomaly score is increased accordingly.

[0040] S23. If the abnormality score exceeds the preset first threshold, a first-level warning is automatically triggered, prompting the administrator to manually review the database table offline.

[0041] A pre-set threshold serves as the initial anomaly determination standard. When the database or table anomaly score exceeds this threshold, a Level 1 alert is automatically triggered. This alert is sent to administrators via various channels, prompting them to review detailed database and table data and comprehensively determine whether further action is necessary, such as investigating the anomaly, adjusting permissions, or initiating a shutdown process.

[0042] S24. If the abnormality score exceeds the preset second threshold and occurs for the first preset number of times consecutively, a second-level warning is automatically triggered, and the offline evaluation process of the corresponding database table is suspended.

[0043] A stricter second threshold is set as the standard for severe anomalies. If the database or table anomaly score exceeds the second threshold and occurs repeatedly for a preset number of times, a second-level warning is automatically triggered and the offline evaluation process is suspended. This operation is intended to prevent offline operations when the database or table is in an abnormal state, avoiding serious consequences such as data loss and business interruption caused by anomalies, and ensuring safe and stable database and table processing.

[0044] S25. Store the abnormality library table whose abnormality scores exceed the first threshold and the second threshold, and the context information of the abnormality library table into the abnormality record table of the data center.

[0045] For database tables with anomaly scores exceeding the threshold, in addition to recording basic information and scores, we also collect contextual information such as the time of the anomaly, the users involved, and the associated tasks to construct a detailed profile of the anomaly scenario. This information is uniformly stored in the anomaly record table on the data center, forming a structured anomaly database that provides data support for subsequent problem tracing and model optimization.

[0046] S26. Use the data in the abnormal record table to continuously optimize and train the library table abnormality detection model, and update the currently running library table abnormality detection model based on the results of the optimization training.

[0047] Regularly optimizing and training the model using real data from the anomaly record table improves its detection capabilities by integrating new anomaly features and adjusting parameter rules. The optimized model is then updated to the production environment, replacing the original version. This forms a closed-loop process of continuous improvement, ensuring the effectiveness and reliability of database and table anomaly detection and the stable operation of the data center.

[0048] As an example, in the financial sector, a bank's data center uses the isolation forest algorithm to train a table anomaly detection model based on historically accumulated data such as transaction task scheduling frequency, user permission change records, and audit logs. For example, by analyzing the scheduling frequency of credit card transaction data, the model can identify abnormally high-frequency transaction task scheduling requests that suddenly appear within a certain period of time. This outlier may indicate a system attack or internal data abuse risk. The model analyzes the downstream dependencies and authorization records of the real-time synchronized credit card table. If it detects an abnormal increase in the list of authorized users for a credit card table within a short period of time, accompanied by high-frequency access records, it generates a corresponding anomaly score, providing a basis for the bank to promptly identify potential data security risks.

[0049] As another example, in the healthcare industry, a hospital leverages historical data from its data center and employs a long short-term memory (LSTM) network to train a database and table anomaly detection model. This model learns the time series patterns of access frequency and authorization change patterns for electronic medical record data. For example, the model analyzes the frequency of access to patient medical records by different departments to establish access patterns under normal business scenarios. If a department's electronic medical record database shows abnormally high access rates for multiple consecutive days during non-business hours, and authorization changes indicate the addition of multiple user permissions to non-related departments, the model generates an anomaly score based on pre-set rules. Hospital management can use this score to promptly intervene and investigate any illegal access to medical record data or abuse of permissions, thereby safeguarding patient privacy and ensuring the compliant use of medical data.

[0050] Through the above steps S21 to S26, a closed-loop database and table anomaly detection and optimization system is constructed to achieve intelligent identification and dynamic prevention and control of data risks. With the help of artificial intelligence algorithms such as isolation forests and long short-term memory networks, it is possible to capture data spatial distribution anomalies and time series anomalies at the same time, and accurately identify risks such as sudden changes in task scheduling frequency, abnormal authorization modes, and deviations in audit behavior. Through threshold grading, an automated step-by-step response from mild anomalies to severe anomalies is achieved, and early warnings are triggered in time to suspend the offline evaluation process of abnormal databases and tables, avoiding the lag and risk of misoperation caused by manual investigations, and ensuring the security of the entire data processing chain. The case data accumulated in the abnormal record table forms a knowledge base for continuously optimizing the model, promoting the continuous evolution of detection capabilities, while automated detection and intelligent resource scheduling mechanisms effectively reduce the workload and resource consumption of manual inspections, improve operation and maintenance efficiency, and reduce costs.

[0051] S3. Based on the downstream dependencies, authorization records, and audit records of the database tables synchronized to the data center, automatic cleaning and conversion are performed according to the preset offline evaluation rules.

[0052] Among them, step S3 builds a data preprocessing system before the offline evaluation of the library and table, focusing on the core capabilities of the data center. The data center receives the downstream dependencies, authorization records, and audit records of the library and table from the task scheduling platform, authorization management platform, and audit management platform. This raw data often has problems such as inconsistent format, data redundancy, and noise interference, and cannot be directly used for offline evaluation of the library and table. Therefore, through the preset offline evaluation rules, ETL (extraction, transformation, loading) technology and data cleaning algorithms are used to deeply process the data.

[0053] From a data cleansing perspective, duplicate data, such as duplicate authorization change records or redundant audit logs, is removed based on rules. Missing values ​​are also filled, for example, by inferring missing fields in downstream dependencies based on historical data patterns. During data conversion, unstructured or semi-structured data is converted into a unified structured format to facilitate subsequent analysis. For example, authorization time formats recorded across different platforms are standardized to timestamps, and descriptive text of downstream dependencies is parsed into relational data.

[0054] Furthermore, data is standardized using pre-set rules, such as categorizing and coding operational behaviors in audit records and normalizing numerical data such as access frequency. This automated cleaning and conversion not only improves data quality, ensuring the accuracy and reliability of subsequent database and table offline assessment results, but also reduces assessment errors caused by data inconsistencies, providing a solid foundation for data-driven decision-making and ultimately ensuring the stability of the database and table offline process.

[0055] As an optional implementation, please continue to refer to Figure 3 , Figure 3 This is a schematic diagram of the execution flow of data synchronization in the large database table automatic offline method provided in the embodiment of the present application, such as Figure 3 As shown, the process includes the following steps S301 to S306.

[0056] S301. Obtain a stored task configuration file by calling a task scheduling query application program interface, parse the input table and output table fields in the task configuration file, and build a library table dependency chain based on the input table and output table fields.

[0057] The task scheduling platform's query API is used to retrieve the task configuration file, which records the input and output table information required for the task. Parsing techniques are used to extract table fields from the file. By analyzing the input and output relationships between tables, a complete library-table dependency chain is constructed, clarifying the upstream and downstream positions of each library table in the data processing flow.

[0058] S302. Based on the owner information synchronized from the unified management platform, each downstream table is associated with the corresponding virtual user information, and the dependency chain and associated virtual user information are synchronized to the data center.

[0059] Based on owner information synchronized from the unified management platform, downstream tables are associated with virtual users, clarifying the management responsibilities of each table. The complete dependency chain and associated virtual user information are then synchronized to the data center via a data transfer protocol, providing foundational data for subsequent permission management and library and table evaluation.

[0060] S303: Obtain the authorization record of the library table by calling the permission query application program interface.

[0061] S304. Match the authorization record with the owner information in the unified management platform, generate an authorization list containing the owner and the responsible person, and synchronize the authorization list to the permission management table of the data center.

[0062] Among them, the obtained authorization records are matched with the owner information in the unified management platform, and each authorized user is mapped to the actual responsible department or person in charge, and an authorization list containing the owner and person in charge is generated to ensure that the responsibility for permission management is assigned to the person, and the list is synchronized to the permission management table of the data center.

[0063] S305: Obtain all access records of the database table within a preset historical period by calling the application program interface of the log analysis.

[0064] By calling the log analysis API of the audit management platform, we can obtain all access records of database tables within a preset historical period, including access time, access user, operation type, etc. This record is an important basis for analyzing database and table usage.

[0065] S306. Associate the access records with the owner information in the unified management platform, generate historical access heat distribution data of preset categories, and synchronize the historical access heat distribution data to the data center.

[0066] Among them, the access records are associated with the owner information of the unified management platform, and the access data is classified and counted by dimensions such as department and user to generate historical access popularity distribution data, intuitively displaying the usage frequency of database tables in different time periods and under different responsibilities, and synchronized with the data middle platform for subsequent database table evaluation and optimization.

[0067] As another alternative implementation, please refer to Figure 4 , Figure 4 This is a schematic diagram of the execution flow of data synchronization in the large database table automatic offline method provided in the embodiment of the present application, such as Figure 4 As shown, the process includes the following steps S307 to S310.

[0068] S307: Filter the task scheduling records in the downstream dependencies of the library table according to the task execution frequency and the last execution time, and remove the task dependencies that have no execution records within the last preset first number of consecutive days.

[0069] For downstream task scheduling records, a filtering strategy is implemented based on two key dimensions: task execution frequency and last execution time. By setting a threshold of "the last preset number of consecutive days" as the judgment threshold, task dependencies that have never been executed within that time period are automatically identified and removed. This operation effectively cleans up redundant data, making library and table dependencies more consistent with actual business operations, improving data processing efficiency and the accuracy of downstream assessments.

[0070] S308: Classify and count the virtual user access rights in the authorization records, generate a list of authorized users for each library table, and mark the rights that have not been used for a preset second number of consecutive days.

[0071] The system comprehensively categorizes and compiles the access rights of virtual users in the authorization records, combing through the authorized user information corresponding to each database table to generate a detailed list of authorized users. Furthermore, the system scans the list for permission usage based on a "preset second consecutive number of days," marking permissions that have not been used for a long time. This facilitates subsequent evaluation of the effectiveness of permission configurations and promptly identifies unused permissions to optimize resource allocation.

[0072] S309. Based on the audit records, calculate the table popularity value according to the average daily access volume, data change frequency and business value score of each table. The business value score is dynamically generated based on the preset field importance weight and access popularity.

[0073] We leverage audit records and comprehensively consider multiple indicators to quantify table popularity. Daily average visits reflect table usage activity, data change frequency reflects updates to table data, and business value scores are dynamically calculated using preset field importance weights combined with visit popularity to comprehensively assess the core importance of a table in the business. Ultimately, this data is integrated to generate a straightforward table popularity value, providing a quantitative basis for table evaluation.

[0074] S310: Based on the authorized user list and the library table heat value, correlation analysis is performed on the library table downstream dependencies, authorization records, and audit records to identify libraries and tables that have no downstream dependencies, no valid authorization, and have not been accessed within a preset time range.

[0075] The analysis focuses on authorized user lists and database table popularity values, cross-correlating the database table's downstream dependencies, authorization records, and audit records. By integrating and comparing multi-dimensional data, we accurately identify database tables that have no downstream task dependencies, no valid authorized users, and have never been accessed within a preset timeframe. This allows us to identify high-potential candidates for offline removal, facilitating the rational optimization and management of resources.

[0076] As another optional implementation, after executing the above step S310, the library table may be further analyzed to obtain different offline priorities, which may specifically include the following steps S311 to S314.

[0077] S311: If the number of downstream task executions for a library table is 0 within the latest consecutive preset third number of days, and there is no valid authorized user access record for the library table, then set the corresponding library table as a high-priority offline library table.

[0078] By analyzing the execution records of downstream tasks and authorized user access logs for database tables, a dual judgment condition is established: if a database table has no downstream tasks executed and no valid authorized user access records for a preset third number of consecutive days, it will be marked as a high-priority offline database table. This automated identification mechanism, based on the principle of "business dormancy," quickly locates redundant database tables that have been unused for a long time, providing precise targets for resource recovery.

[0079] S312: If the average daily visits to the table are lower than the fourth preset number and the business value score is within the first preset percentage of all tables, set the corresponding table as a medium-priority offline table.

[0080] A differentiated assessment is implemented based on average daily visits and business value scores. When average daily visits fall below a preset fourth threshold and the business value score falls within a preset first percentile, the database or table is considered a medium-priority decommissioning candidate. This strategy balances data usage frequency with business contribution, avoiding the sole criterion of visit volume and ensuring that less important databases and tables are prioritized for decommissioning.

[0081] S313: If the database table has a valid authorized user access record, but the data change frequency of the database table is 0 for a consecutive preset fifth number of days, then the corresponding database table is set to a database table in an observation state.

[0082] A special observation mechanism is implemented for database tables with valid authorized users but stagnant data changes. When a database table's data change frequency is zero for a preset fifth consecutive day, it is placed in an observation state. This design prevents inadvertent offline access to authorized but temporarily unused database tables while also identifying potential "zombie" data through continuous monitoring, balancing data security and resource utilization.

[0083] S314: Store the information data of the high-priority offline library table, the medium-priority offline library table, and the to-be-observed status library table into a unified wide table.

[0084] Key information about high-priority and medium-priority offline tables and tables in observation status, including evaluation indicators, decision criteria, and business relevance, is uniformly stored in a wide table structure. This integration approach breaks down data silos and provides a standardized data model for subsequent batch processing, cross-dimensional analysis, and visualization, enabling management decision-makers to fully understand the full picture of offline tables and libraries.

[0085] As an example, in the financial sector, a bank parsed the task configuration files of credit card transaction data and constructed a library table dependency chain. It discovered that the downstream task of a library table recording historical transaction details had been executed 0 times for 90 consecutive days and had no valid authorized user access records. Based on the rules, it was set as a high-priority offline library table, freeing up storage resources to optimize data center performance. At the same time, customer address information libraries with an average daily visit volume of less than 50 times and a business value score in the bottom 20% were identified as medium-priority offline objects to prevent redundant data from affecting the efficiency of the core transaction system. For temporary risk indicator calculation tables that still have authorized access from the risk control department but have not had data updates for 60 consecutive days, the system marks them as under observation and continuously monitors whether they are temporarily idle due to business adjustments, rather than being directly offlined.

[0086] As another example, in the medical industry, a hospital analyzed the task scheduling records of the electronic medical record system and identified a database table storing archived medical records. The downstream medical record retrieval task had not been executed for 180 consecutive days, and there was no doctor-authorized access record. The hospital listed the table as a high-priority offline table to reduce data storage pressure. A department's consumables claim record table with an average daily access volume of less than 10 times and a business value score in the bottom 30% was determined to be a medium-priority offline table to optimize resource allocation in the data center. As for the operating room scheduling table, although medical staff authorized access, there was no data change for 30 consecutive days due to adjustments to the surgical process. The system set it to an observation state to prevent the accidental deletion of critical data due to short-term business fluctuations and ensure that the data is available when subsequent surgical schedules are resumed.

[0087] Through the above steps S301 to S314, systematic data processing and analysis have achieved refined management and efficient scheduling of library and table resources. By parsing the task configuration file to build a library and table dependency chain, the data flow can be clearly sorted out to avoid system confusion caused by redundant dependencies. The owner information is associated with the library table, permissions, and access records to clarify the responsibility for data management and strengthen data security management. The screening and analysis of task scheduling, authorization, and audit records can accurately identify low-value and idle library and table resources, automatically divide high- and medium-priority offline libraries and tables and libraries to be observed, and reduce manual evaluation costs and subjective errors. The relevant information is uniformly stored in a wide table to provide structured data support for subsequent data processing and decision-making, ultimately improving the resource utilization of the data center, reducing the complexity of operation and maintenance, and ensuring the health and efficient operation of the data ecosystem.

[0088] S4. Generate a unified wide table after cleaning, converting, and integrating the downstream dependencies of the library and table, authorization records, and audit records. Store the unified wide table in the data center.

[0089] Step S4 achieves the integrated storage and efficient query of multi-source heterogeneous data by constructing a unified wide table. The data center first aligns the schema of the cleaned and converted downstream dependencies, authorization records, and audit records of the database tables, mapping fields from different sources to a unified data model. For example, the "task_id" of the task scheduling platform and the "operation_id" of the authorization management platform are linked to the "business operation ID" field of the wide table. A distributed computing framework (such as Spark) is used to optimize the column storage of the processed data, and composite indexes are established based on dimensions such as database table ID and timestamp to improve the efficiency of subsequent filtering and aggregation.

[0090] When storing the unified wide table, the wide table data is written to a distributed storage engine that supports ACID features (for example, Hudi or Delta Lake) to ensure the atomicity and consistency of data changes. At the same time, data compression (for example, Snappy / LZ4) and tiered storage strategies are enabled to store hot data on high-performance media and archive historical data to low-cost storage to balance read and write performance and storage costs. Through the metadata registration mechanism, the wide table structure and association relationships are synchronized to the metadata management system of the data center, providing a standardized interface for subsequent data servitization. This design enables wide tables to support complex analysis in OLAP scenarios, as well as meet real-time data retrieval needs, providing a unified data base for offline evaluation of libraries and tables.

[0091] As an optional implementation, after the above step S4, the workflow engine may be used to assist the library table to go offline, which may specifically include the following steps S41 to S44.

[0092] S41. Deploy a workflow engine in the data center to encapsulate the processing operations of data synchronization, cleansing and conversion, evaluation and grading, and approval and notification into configurable atomic tasks.

[0093] A workflow engine is deployed in the data center, breaking down processing operations such as data synchronization, cleansing and conversion, evaluation and grading, and approval notifications into independent and configurable atomic tasks. These atomic tasks, like building blocks, can be flexibly combined to meet different business needs. Each task has clear input and output and execution logic, laying the technical foundation for subsequent automated processing.

[0094] S42. Through the workflow engine interface, send instructions to the task scheduling platform to suspend the scheduling tasks that have dependencies on the library table to be taken offline, send instructions to the authorization management platform to freeze the new authorization of the library table to be taken offline, and send instructions to the audit management platform to start a full audit of the change operations on the library table to be taken offline.

[0095] Through the interface provided by the workflow engine, instructions are sent to the task scheduling platform to suspend scheduled tasks that have dependencies on the database table to be taken offline, preventing data processing errors during the process. Simultaneously, instructions are sent to the authorization management platform to freeze new authorizations for the database table to be taken offline, preventing problems caused by new users accessing the database. Furthermore, instructions are sent to the audit management platform to initiate a full audit of all changes to the database table to be taken offline, thereby comprehensively monitoring and recording all changes during the process.

[0096] S43. Automatically execute the atomic task according to the preset process through the workflow engine. If the number of task failures is less than or equal to the second preset number, re-execute; if the number of failures is greater than the second preset number, switch to the backup data source or backup processing node and re-execute.

[0097] S44. Generate a database and table offline processing report in the management interface of the data middle platform, where the database and table offline processing report includes data processing time, risk assessment results and approval chain records.

[0098] The management interface of the data center automatically generates a report on the offline processing of databases and tables. This report includes the time consumption of each link in the data processing process, allowing managers to understand the efficiency of the entire process. The report also presents risk assessment results, quantifying and analyzing potential risks during the offline processing of databases and tables. Furthermore, the report records relevant information about the approval chain, including the time, approver, and approval opinions of each approval link, enabling traceability and transparent management of the entire offline processing process.

[0099] As an example, in the financial sector, a bank deployed a workflow engine in its data center, encapsulating operations such as credit card data synchronization, abnormal transaction cleaning and conversion, database and table risk assessment and grading, and approval notifications into atomic tasks. When a credit card transaction flow table enters the offline process, the workflow engine, through an interface, suspends the transaction reconciliation scheduling tasks downstream of the table to prevent data processing errors during the offline period. It also freezes new authorizations for the table to prevent new business departments from accidentally accessing it. It also sends instructions to the audit platform to conduct a full audit of transaction changes made to the table six months before the offline process to ensure compliance. If the cleaning task fails twice (within the preset number) due to network fluctuations, it will automatically retry. If it fails a third time, processing will be switched to the local disaster recovery node. A report is ultimately generated on the management interface, documenting the four-hour process from data freeze to completion of the offline process. The risk assessment indicates no risk of sensitive data leakage, and the approval chain records the step-by-step approval opinions of the risk control and compliance departments.

[0100] As another example, in the medical industry, a hospital's data center uses a workflow engine to manage the offline process of electronic medical record databases and tables, and sets operations such as medical record data archiving and synchronization, privacy information desensitization and cleaning, and database table usage frequency assessment as configurable tasks. When a department's patient follow-up record database table from three years ago needs to be offline, the workflow engine suspends the scheduling of follow-up tasks associated with the database table to prevent accidental triggering of data calls; freezes new authorizations to avoid medical staff from mistakenly applying for access; and simultaneously initiates a full audit of changes to patient contact information, diagnosis results, and other operations in the database table to ensure compliance with the Personal Information Protection Law. If the desensitization task fails once (less than the preset number of 2 times), it will automatically retry. If it fails again, it will switch to the backup data processing node. The final report shows that the entire offline process took 6 hours, and the risk assessment confirmed that the privacy data processing had been completed. The approval chain records the joint approval process of the Information Department and the Medical Affairs Department.

[0101] Through steps S41 to S44, the workflow engine automates the entire process of decommissioning databases and tables. Encapsulating atomic tasks enhances process flexibility and enables rapid adaptation to diverse scenarios. Cross-platform command interaction mitigates data manipulation risks, ensuring business and regulatory compliance. Automatic task retries and failover enhance robustness and minimize interruptions. Reports containing processing time, risk assessments, and approval records are generated, ensuring transparency and traceability, improving decommissioning efficiency and the automated operations and maintenance capabilities of the middleware platform.

[0102] S5. After receiving the input of the evaluation database table to be taken offline, the data center automatically associates the owner information synchronized with the unified wide table and the unified management platform, generates the evaluation information and virtual user owner information of the evaluation database table to be taken offline, and sends the offline notification to the related party for confirmation.

[0103] Among them, step S5 technically relies on the metadata management and integration capabilities of the data middle platform. After receiving the library table to be evaluated offline, it uses the unique identifier of the library table as an index to automatically associate the downstream dependencies, authorization records, audit records and other multi-dimensional data in the unified wide table. At the same time, the library table owner information and virtual user mapping relationships are synchronized in real time through the API interface of the unified management platform. Using the built-in rule engine, the related data is analyzed according to the preset standards, and an assessment report containing risk levels and impact ranges is automatically generated. Finally, the multi-channel notification workflow is triggered through the message middleware, and the assessment conclusions and offline requests are sent to the relevant responsible persons. The entire process ensures data access compliance through permission verification, and uses an asynchronous communication mechanism to ensure high concurrent processing capabilities, realizing a fully automated process from data association, evaluation to notification, reducing manual intervention, and improving the efficiency and accuracy of offline evaluation of libraries and tables.

[0104] S6. After receiving the feedback confirmation information returned by the related party, the feedback confirmation information is automatically recorded in the data center, and based on the evaluation information in the unified wide table and the preset offline rules, the offline evaluation library table is offline processed.

[0105] Among them, step S6 relies on the workflow engine and rule executor of the data middle platform to realize feedback confirmation and automatic offline of library tables. When the confirmation information returned by the related party through the message middleware is received, it is immediately stored in the operation log table of the data middle platform to ensure the completeness of the record. The rule executor extracts the evaluation data in the unified wide table and matches it with the preset offline rules (for example, low risk and no active dependencies are automatically offline). If the conditions are met, the offline process is triggered. The workflow engine automatically calls the task scheduling platform to stop related tasks, freezes the library table permissions through the permission management system, performs data archiving, and finally updates the metadata status. This process uses idempotent design and distributed locks to avoid repeated operations and concurrency conflicts, and completes closed-loop processing.

[0106] As an optional implementation, after the above step S6, a library table offline impact prediction model can be constructed through a machine learning algorithm, and the model can be used to assist in library table offline analysis, which can specifically include the following steps S71 to S75.

[0107] S71. Build a library table offline impact prediction model using a preset machine learning algorithm. Use historical library table offline records, historical library table authorization change records, and historical library table access records to train the library table offline impact prediction model to obtain a trained library table offline impact prediction model.

[0108] For example, the XGBoost or LSTM algorithm is used to build a library and table offline impact prediction model, and multi-source time series data such as historical library and table offline records, authorization change logs, and access frequency are converted into feature vectors. Through feature engineering, dimensions such as "number of sensitive field accesses in the past 30 days" and "authorized user attenuation slope" are extracted. Cross-validation and Bayesian optimization of hyperparameters are used to complete model training on a distributed computing framework, and finally output a baseline model that includes prediction dimensions such as risk level and business impact range.

[0109] S72. After receiving the newly input evaluation database table to be taken offline, input the evaluation database table to be taken offline into the database table offline impact prediction model to obtain offline risk assessment data after the offline operation is performed on the evaluation database table to be taken offline.

[0110] When a new table to be taken offline is received, the table metadata (e.g., number of fields, number of associated tasks) and real-time access characteristics (e.g., QPS fluctuation curve) are input into the trained model through a standardized interface. The model uses an integrated learning method based on the learned pattern recognition rules to generate prediction results, including multi-dimensional risk assessment data such as the system response time fluctuation threshold (e.g., ±20ms), throughput decrease percentage (e.g., less than or equal to 15%), and probability of associated business interruption.

[0111] S73: After the offline evaluation database table is offlined, business system response time, throughput, and dependency change data of associated database tables are obtained as first training data.

[0112] After the library table is actually offline, the APM tool (such as Skywalking) is used to collect the changes in the quantile response time of the business system in real time, and the throughput fluctuation curve is obtained from the monitoring system (such as Prometheus). At the same time, the metadata management system API is called to capture the dependency changes of related library tables (for example, a surge in the number of task scheduling failures), and the real-time observation data is stored as the first training data in the time series database.

[0113] S74: Combine the information data of the offline assessment database table, the offline risk assessment data, and the first training data as optimized training data.

[0114] Among them, the original information of the database table to be taken offline (for example, the business department to which it belongs), the risk assessment data predicted by the model (for example, the predicted interruption probability), and the first training data actually collected after the offline (for example, the actual interruption duration) are spliced ​​together for features. The most discriminative feature subset is screened out through the feature importance ranking algorithm, and an optimized training dataset is constructed, which includes dimensions such as prediction error (for example, the mean square error between the predicted value and the actual value) and business impact authenticity (for example, the actual throughput reduction).

[0115] S75. Incrementally train the library table offline impact prediction model by optimizing the training data, and optimize the parameters of the library table offline impact prediction model based on the training results.

[0116] Among them, an online learning mechanism is adopted to perform incremental training on the model, and the FTRL (Follow-the-Regularized-Leader) optimization algorithm is used to dynamically adjust the weights of each feature. The concept drift detection algorithm is introduced to monitor data distribution changes in real time. When the prediction error is detected to exceed the preset threshold, the model parameter update is triggered. At the same time, the effect of the optimized model is verified through A / B testing, and finally a closed-loop optimization is achieved in which the prediction accuracy continues to improve with the accumulation of historical data.

[0117] As an example, a bank built a model to predict the impact of database and table offline operations. Using historical credit card transaction details database offline records, user permission change logs, and nightly batch task execution records as training data, the model was trained using the XGBoost algorithm to predict risk levels and potential impact. When a new database and table with address change logs was input for a customer to be offlined, the model predicted that the offline operation would result in a delay in customer information synchronization, increasing system response time from 100ms to 150ms. After the actual offline operation, APM tools detected a peak response time of 200ms for the customer information modification interface, and monitoring also revealed a 12% decrease in customer relationship management system throughput. The model then combined the original database and table information, predicted risk data, and actual observational data to generate an optimized training set that included prediction errors. The model was then incrementally trained using the FTRL algorithm, adjusting the weight of the "inter-system data synchronization frequency" feature, improving the accuracy of subsequent predictions of similar database and table offlines to 89%.

[0118] As another example, a hospital trained a model to predict the impact of offline databases based on five years of historical data. The model used inputs such as offline cases in operating room schedule records, records of changes in medical staff privileges, and peak patient visit times. Using an LSTM algorithm to capture time series patterns, the model outputs a probability prediction, such as "causing 3% of surgeries to be delayed by more than 15 minutes." The model used a database of anesthetic inventory logs to predict the impact of offline databases on the drug traceability system, predicting a drop in the number of orders processed per hour from 500 to 420. After the offline database was shut down, the system actually processed only 400 orders per hour, with two emergency drug dispensing delays. The predicted data was fused with actual observed data to generate an optimized training set. Using online learning to adjust model parameters, the model reduced the error rate in subsequent predictions of offline databases by 15%.

[0119] Through the above steps S71 to S75, a closed-loop machine learning optimization system is constructed to significantly improve the accuracy and adaptability of the risk prediction of library and table offline. The prediction model trained with historical multi-source data can quantitatively evaluate the potential impact of library and table offline on key indicators such as system response time and throughput. In the new library and table evaluation, the model quickly outputs risk data to assist in decision-making. After the actual offline, by collecting real data such as changes in business system response and dependency changes, and integrating it with the prediction information to form an optimized training set, the model is incrementally trained and parameterized in combination with the algorithm to effectively respond to changes in the business environment, continuously improve the model prediction accuracy, realize dynamic optimization of library and table offline risk prediction and management, and reduce the risk of business interruption caused by library and table offline.

[0120] The automatic offline method for large database tables provided in the embodiment of the present application realizes the automatic cleaning, conversion and integration of multi-source heterogeneous data by building a data middle platform, forming a unified wide table to provide a standardized data base for evaluation. Relying on the workflow engine, data synchronization, assessment classification, approval notification and other links are encapsulated into configurable atomic tasks, realizing the automated orchestration and cross-platform instruction interaction of the entire process of database table offline, effectively blocking operational risks and improving processing efficiency. A machine learning model is introduced to build a closed-loop prediction system, which quantitatively evaluates offline risks based on historical data training models, and dynamically optimizes parameters based on actual business feedback, significantly improving prediction accuracy and adaptability. Through the rule engine and multi-level threshold design, high and medium priority offline database tables and objects to be observed are accurately identified, reducing manual intervention errors. Ultimately, the resource utilization rate of the data middle platform is improved and the operation and maintenance complexity is reduced, forming a full-link intelligent management from data association, risk prediction to offline execution, effectively ensuring the efficiency, security and compliance of the data ecosystem.

[0121] Please continue reading Figure 5 , Figure 5 This is a schematic diagram of the system structure of the large database table automatic offline device provided in the embodiment of the present application, such as Figure 5 As shown, the large database table automatic offline device 50 includes: a unified management platform module 51, a data automatic synchronization module 52, a data cleaning and conversion module 53, a unified wide table management module 54, an offline notification management module 55 and a database table offline execution module 56.

[0122] The unified management platform module 51 is used to set the permission approver of the virtual user in the target distributed computing framework as the owner of the virtual user in the pre-deployed unified management platform, and synchronize the owner information and the change information of the permission configuration corresponding to the owner through the unified management platform.

[0123] The data automatic synchronization module 52 is used to automatically synchronize the downstream dependencies, authorization records, and audit records of the library tables from the task scheduling platform, authorization management platform, and audit management platform to the data middle platform, and obtain downstream tables, queryable users, and historical access records.

[0124] The data cleaning and conversion module 53 is used to perform automatic cleaning and conversion according to preset offline evaluation rules based on the downstream dependencies of the library tables, the authorization records and the audit records synchronized to the data middle platform.

[0125] The unified wide table management module 54 is used to generate a unified wide table based on the results of cleaning, converting and integrating the downstream dependencies of the library table, the authorization records and the audit records, and store the unified wide table in the data middle platform.

[0126] The offline notification management module 55 is used to receive the input of the evaluation library table to be offline, automatically associate the unified wide table and the owner information synchronized with the unified management platform through the data middle platform, generate the evaluation information and virtual user owner information of the evaluation library table to be offline, and send the offline notification to the related party for confirmation.

[0127] The library table offline execution module 56 is used to automatically record the feedback confirmation information to the data middle platform after receiving the feedback confirmation information returned by the related party, and perform offline processing on the evaluation library table to be offline based on the evaluation information in the unified wide table and the preset offline rules.

[0128] As an optional implementation, the data cleaning and conversion module 53 is specifically used to obtain the stored task configuration file by calling the task scheduling query application program interface, parse the input table and output table fields in the task configuration file, and build a library table dependency chain based on the input table and output table fields; based on the owner information synchronized by the unified management platform, each downstream table is associated with the corresponding virtual user information, and the dependency chain and the associated virtual user information are synchronized to the data middle platform; by calling the permission query application program interface, the authorization record of the library table is obtained; the authorization record is matched with the owner information in the unified management platform to generate an authorization list containing the owner and the responsible person, and the authorization list is synchronized to the permission management table of the data middle platform; by calling the log analysis application program interface, all access records of the library table within a preset historical period are obtained; the access record is associated with the owner information in the unified management platform, historical access heat distribution data of a preset category is generated, and the historical access heat distribution data is synchronized to the data middle platform.

[0129] As an optional implementation, the data cleaning conversion module 53 is also specifically used to filter the task scheduling records in the downstream dependencies of the library table according to the task execution frequency and the last execution time, and remove the task dependencies that have no execution records in the last preset first number of days; classify and count the virtual user access rights in the authorization records, generate an authorized user list for each library table, and mark the permissions that have not been used for a preset second number of days; based on the audit records, calculate the library table heat value according to the average daily access volume, data change frequency and business value score of each library table, wherein the business value score is dynamically generated according to the preset field importance weight and access heat; based on the authorized user list and the library table heat value, perform correlation analysis on the downstream dependencies of the library table, the authorization records and the audit records, and identify the library tables that have no downstream dependencies, no valid authorization and have not been accessed within a preset time range.

[0130] As an optional implementation, the data cleaning conversion module 53 is also specifically used to set the corresponding library table as a high-priority offline library table if the number of downstream task executions in the library table is 0 within the last preset third number of days, and the library table has no valid authorized user access record; if the average daily access volume of the library table is lower than the preset fourth number, and the business value score is within the last preset first percentage of all library tables, then the corresponding library table is set as a medium-priority offline library table; if the library table has a valid authorized user access record, but the data change frequency of the library table is 0 within the last preset fifth number of days, then the corresponding library table is set as a library table in the state to be observed; and store the information data of the high-priority offline library table, the medium-priority offline library table and the library table in the state to be observed in the unified wide table.

[0131] As an optional embodiment, the large database table automatic offline device 50 also includes a library table offline impact prediction model module, which is specifically used to construct a library table offline impact prediction model through a preset machine learning algorithm, and use historical library table offline records, historical library table authorization change records, and historical library table access records to train the library table offline impact prediction model to obtain the trained library table offline impact prediction model; after receiving a newly input library table to be offlined, input the library table to be offlined into the library table offline impact prediction model to obtain offline risk assessment data after the offline operation is predicted for the library table to be offlined; after the offline operation is executed on the library table to be offlined, obtain the business system response time, throughput, and dependency change data of the associated library table as first training data; combine the information data of the library table to be offlined, the offline risk assessment data, and the first training data as optimized training data; incrementally train the library table offline impact prediction model through the optimized training data, and optimize the parameters of the library table offline impact prediction model according to the training results.

[0132] As an optional embodiment, the large database table automatic offline device 50 also includes a library table anomaly detection model module, which is specifically used to use the data of historical synchronization in the data center to train an artificial intelligence-based library table anomaly detection model. The library table anomaly detection model uses an isolation forest algorithm or a long short-term memory network to identify abnormal data in task scheduling frequency, authorization change mode and audit access behavior; the library table anomaly detection model is used to perform real-time analysis on the downstream dependencies of the library table, the authorization records and the audit records synchronized to the data center to generate an anomaly score for each library table. ; If the anomaly score exceeds the preset first threshold, the first level warning is automatically triggered, prompting the administrator to manually review the offline database table; if the anomaly score exceeds the preset second threshold and occurs for the first preset number of times in succession, the second level warning is automatically triggered, and the offline evaluation process of the corresponding database table is suspended; the abnormal database table whose anomaly score exceeds the first threshold and the second threshold, and the context information of the abnormal database table are stored in the abnormal record table of the data center; the data of the abnormal record table is used to continuously optimize and train the database table anomaly detection model, and the currently running database table anomaly detection model is updated according to the results of the optimization training.

[0133] As an optional implementation, the large database table automatic offline device 50 also includes a workflow engine module, which is specifically used to deploy a workflow engine in the data middle platform, and encapsulate the processing operations of data synchronization, cleaning and conversion, evaluation and grading, and approval notification into configurable atomic tasks; through the workflow engine interface, send instructions to the task scheduling platform to suspend scheduling tasks that have dependencies on the library table to be offline, send instructions to the authorization management platform to freeze new authorizations for the library table to be offline, and send instructions to the audit management platform to start a full audit of change operations on the library table to be offline; automatically execute the atomic task according to the preset process through the workflow engine, if the number of task failures is less than or equal to the second preset number, re-execute, if the number of failures is greater than the second preset number, switch to the backup data source or backup processing node and re-execute; generate a library table offline processing report in the management interface of the data middle platform, wherein the library table offline processing exposure includes data processing time, risk assessment results and approval chain records.

[0134] It should be noted that the aforementioned large database table automatic offline device can execute the large database table automatic offline method provided in the embodiments of this application, and has the corresponding functional modules and beneficial effects of the execution method. For technical details not fully described in the embodiments of the large database table automatic offline device, please refer to the large database table automatic offline method provided in the embodiments of this application.

[0135] Figure 6 Schematic diagram of the hardware structure of an electronic device for executing the automatic offline method of a large database table provided in an embodiment of the present application, such as Figure 6 As shown, the electronic device 600 includes:

[0136] One or more processors 610 and memory 620, Figure 6 A processor 610 is taken as an example.

[0137] The processor 610 and the memory 620 may be connected via a bus or other means. Figure 6 The bus connection is taken as an example.

[0138] Memory 620, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer executable programs, and modules, such as the program instructions / modules corresponding to the large database table automatic offline method in the embodiments of the present application. Processor 610 executes the non-volatile software programs, instructions, and modules stored in memory 620 to execute various server functional applications and data processing, thereby implementing the large database table automatic offline method in the aforementioned method embodiment.

[0139] The memory 620 may include a program storage area and a data storage area, wherein the program storage area may store an operating system and application programs required for at least one function; the data storage area may store data created based on the use of the large database table automatic offline device, etc. In addition, the memory 620 may include a high-speed random access memory and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory 620 may optionally include a memory remotely located relative to the processor 610, and these remote memories may be connected to the large database table automatic offline device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0140] The one or more modules are stored in the memory 620, and when executed by the one or more processors 610, the large database table automatic offline method in any of the above method embodiments is executed, for example, the above described Figure 2 Steps S1 to S6 of the method, Figure 3 Steps S301 to S306 of the method, Figure 4 Steps S307 to S310 of the method are implemented Figure 5 The functions of modules 51-56 in.

[0141] The above-mentioned product can execute the method provided in the embodiment of this application, and has the functional modules and beneficial effects corresponding to the execution method. For technical details not fully described in this embodiment, please refer to the method provided in the embodiment of this application.

[0142] An embodiment of the present application provides a non-volatile computer-readable storage medium, wherein the non-volatile computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are executed by one or more processors, for example Figure 6 A processor 610 in the embodiment may enable the one or more processors to execute the large database table automatic offline method in any of the above method embodiments, for example, executing the above described Figure 2 Steps S1 to S6 of the method, Figure 3 Steps S301 to S306 of the method, Figure 4 Steps S307 to S310 of the method are implemented Figure 5 The functions of modules 51-56 in.

[0143] The embodiment of the present application provides a computer program product, which includes a computer program stored on a non-volatile computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by the electronic device, the electronic device is enabled to perform the large database table automatic offline method in any of the above method embodiments, for example, performing the above described Figure 2 Steps S1 to S6 of the method, Figure 3 Steps S301 to S306 of the method, Figure 4 Steps S307 to S310 of the method are implemented Figure 5 The functions of modules 51-56 in.

[0144] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of this embodiment.

[0145] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a general hardware platform, or of course by hardware. Those skilled in the art can understand that all or part of the processes in the above embodiment methods can be implemented by instructing the relevant hardware through a computer program, and the program can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above methods. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM), etc.

[0146] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Based on the concept of the present application, the technical features in the above embodiments or different embodiments can also be combined, the steps can be implemented in any order, and there are many other variations in different aspects of the present application as described above. For the sake of simplicity, they are not provided in detail. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for automatically shutting down a large database table, characterized in that: include: In a pre-deployed unified management platform, the permission approver of the virtual user in the target distributed computing framework is set as the owner of the virtual user, and the owner information and the change information of the permission configuration corresponding to the owner are synchronized through the unified management platform; Automatically synchronize downstream dependencies, authorization records, and audit records of database tables from the task scheduling platform, authorization management platform, and audit management platform to the data center, obtaining downstream tables, queryable users, and historical access records. Based on the downstream dependencies of the database tables, the authorization records, and the audit records synchronized to the data center, automatic cleaning and conversion are performed according to the preset offline evaluation rules; Generate a unified wide table from the results of cleaning, converting and integrating the downstream dependencies of the library table, the authorization records and the audit records, and store the unified wide table in the data middle platform; After receiving the input of the evaluation table to be taken offline, the data center automatically associates the owner information synchronized with the unified wide table and the unified management platform, generates the evaluation information and virtual user owner information of the evaluation table to be taken offline, and sends an offline notification to the related party for confirmation; After receiving the feedback confirmation information returned by the related party, the feedback confirmation information is automatically recorded in the data middle platform, and based on the evaluation information in the unified wide table and the preset offline rules, the offline processing is performed on the evaluation library table to be offline.

2. The automatic offline method for a large database table according to claim 1, characterized in that: The steps of automatically synchronizing the downstream dependencies, authorization records, and audit records of the database tables from the task scheduling platform, authorization management platform, and audit management platform to the data center platform include: By calling the task scheduling query application program interface, a stored task configuration file is obtained, the input table and output table fields in the task configuration file are parsed, and a library table dependency chain is constructed according to the input table and output table fields; Based on the owner information synchronized by the unified management platform, each downstream table is associated with the corresponding virtual user information, and the dependency chain and the associated virtual user information are synchronized to the data center; Obtain the authorization record of the library table by calling the permission query application program interface; Match the authorization record with the owner information in the unified management platform, generate an authorization list containing the owner and the responsible person, and synchronize the authorization list to the permission management table of the data center; Obtain all access records of the database table within a preset historical period by calling the application program interface of the log analysis; The access record is associated with the owner information in the unified management platform, historical access heat distribution data of preset categories is generated, and the historical access heat distribution data is synchronized to the data middle platform.

3. The automatic offline method for a large database table according to claim 1, characterized in that: The steps of automatically cleaning and converting according to the preset offline evaluation rules include: Filter the task scheduling records in the downstream dependencies of the library table according to the task execution frequency and the last execution time, and remove the task dependencies that have no execution records within the last preset first number of consecutive days; Classify and count the virtual user access rights in the authorization record, generate a list of authorized users for each library table, and mark the rights that have not been used for a preset second number of consecutive days; Based on the audit records, the table popularity value is calculated according to the average daily access volume, data change frequency and business value score of each table. The business value score is dynamically generated based on the preset field importance weight and access popularity. Based on the authorized user list and the library table heat value, correlation analysis is performed on the library table downstream dependencies, the authorization records, and the audit records to identify libraries and tables that have no downstream dependencies, no valid authorization, and have not been accessed within a preset time range.

4. The automatic offline method for a large database table according to claim 3 is characterized in that: After the step of identifying a library table that has no downstream dependency, no valid authorization, and has not been accessed within a preset time range, the method further includes: If the number of downstream task executions for a table is zero within the last three consecutive days, and there is no valid authorized user access record for the table, then the corresponding table is set as a high-priority offline table. If the average daily visits to a table are lower than the fourth preset number, and the service value score is within the first preset percentage of all tables, then the corresponding table is set as a medium-priority offline table; If a table has valid authorized user access records, but the table's data change frequency is 0 for the fifth consecutive preset number of days, the corresponding table is set to a table in the observation state; The information data of the high-priority offline library table, the medium-priority offline library table, and the to-be-observed status library table are stored in the unified wide table.

5. The automatic offline method for a large database table according to claim 1 is characterized in that: After the step of executing offline processing on the evaluation database table to be offline based on the evaluation information in the unified wide table and the preset offline rules, the method further includes: A library table offline impact prediction model is constructed using a preset machine learning algorithm. The model is trained using historical library table offline records, historical library table authorization change records, and historical library table access records to obtain a trained library table offline impact prediction model. After receiving a newly input database table for evaluation of databases to be taken offline, the database table for evaluation of databases to be taken offline is input into the database table offline impact prediction model to obtain offline risk assessment data after the offline operation is predicted for the database table for evaluation of databases to be taken offline; After the offline evaluation database table is offlined, the business system response time, throughput, and dependency change data of the associated database table are obtained as first training data; Combining the information data of the to-be-offline evaluation database table, the offline risk assessment data, and the first training data as optimized training data; The library table offline impact prediction model is incrementally trained using the optimized training data, and parameters of the library table offline impact prediction model are optimized according to the training results.

6. The automatic offline method for a large database table according to claim 1, characterized in that: After the step of automatically synchronizing the downstream dependencies, authorization records, and audit records of the database tables from the task scheduling platform, the authorization management platform, and the audit management platform to the data center, the following steps are also included: Using the historical synchronized data from Taichung, an AI-based database and table anomaly detection model is trained. The database and table anomaly detection model uses an isolation forest algorithm or a long short-term memory network to identify abnormal data in task scheduling frequency, authorization change patterns, and audit access behavior. Use the library table anomaly detection model to perform real-time analysis on the library table downstream dependencies, the authorization records, and the audit records synchronized to the data center to generate an anomaly score for each library table; If the anomaly score exceeds the preset first threshold, a first-level warning is automatically triggered, prompting the administrator to manually review the database table offline; If the anomaly score exceeds the preset second threshold and occurs for the first preset number of times consecutively, a second-level warning is automatically triggered, and the offline evaluation process of the corresponding database table is suspended; The abnormality database table whose abnormality score exceeds the first threshold and the second threshold, and the context information of the abnormality database table are stored in the abnormality record table of the data center; The data of the abnormality record table is used to continuously optimize and train the library table abnormality detection model, and the currently running library table abnormality detection model is updated according to the results of the optimization training.

7. The automatic offline method for a large database table according to claim 1, characterized in that: After the step of performing offline processing on the evaluation database table to be offline based on the evaluation information and preset offline rules in the unified wide table, the method further includes: Deploy a workflow engine in the data center to encapsulate data synchronization, cleansing and conversion, evaluation and grading, and approval and notification processing operations into configurable atomic tasks; Through the workflow engine interface, send instructions to the task scheduling platform to suspend scheduled tasks that have dependencies on the database table to be taken offline, send instructions to the authorization management platform to freeze new authorizations for the database table to be taken offline, and send instructions to the audit management platform to start a full audit of changes to the database table to be taken offline; Automatically executing the atomic task according to a preset process by the workflow engine, re-executing the task if the number of task failures is less than or equal to a second preset number, and switching to a backup data source or backup processing node for re-execution if the number of task failures is greater than the second preset number; Generate a database and table offline processing report in the management interface of the data middle platform, where the database and table offline processing exposure includes data processing time, risk assessment results and approval chain records.

8. A large database table automatic offline device, characterized in that: include: A unified management platform module is used to set the permission approver of the virtual user in the target distributed computing framework as the owner of the virtual user in the pre-deployed unified management platform, and synchronize the owner information and the change information of the permission configuration corresponding to the owner through the unified management platform; The data automatic synchronization module is used to automatically synchronize the downstream dependencies, authorization records, and audit records of the database tables from the task scheduling platform, authorization management platform, and audit management platform to the data center, and obtain downstream tables, queryable users, and historical access records; A data cleaning and conversion module is used to automatically clean and convert the data based on the downstream dependencies of the database tables, the authorization records, and the audit records synchronized to the data center according to the preset offline evaluation rules; A unified wide table management module is used to generate a unified wide table from the results of cleaning, converting and integrating the downstream dependencies of the library table, the authorization records and the audit records, and store the unified wide table in the data center; The offline notification management module is used to receive the input of the evaluation library table to be offline, automatically associate the owner information synchronized between the unified wide table and the unified management platform through the data center, generate the evaluation information and virtual user owner information of the evaluation library table to be offline, and send the offline notification to the related party for confirmation; The library table offline execution module is used to automatically record the feedback confirmation information to the data middle platform after receiving the feedback confirmation information returned by the related party, and perform offline processing on the library table to be offlined based on the evaluation information in the unified wide table and the preset offline rules.

9. An electronic device, characterized in that: include: at least one processor; as well as, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the large database table automatic offline method according to any one of claims 1 to 7.

10. A non-volatile computer-readable storage medium, characterized in that: The non-volatile computer-readable storage medium stores computer-executable instructions. When the computer-executable instructions are executed by an electronic device, the electronic device executes the large database table automatic offline method according to any one of claims 1 to 7.

Images