Data management method, system, device, computer device and storage medium

Abstract

The application relates to a data management method, system, device, computer equipment and storage medium. The method comprises the following steps: in response to a data collection instruction, obtaining account information of each database container from a plurality of data management platforms; the account information comprises a data matching IP of each database container; according to the data matching IP of each database container, obtaining supplementary data information of each database container from a target association platform; according to the account information and the supplementary data information of each database container, generating full life cycle information of each database container; and based on the full life cycle information of each database container, performing a data processing operation on data in the target association platform. The target association platform comprises at least one platform. In this way, the account information of each database container is summarized to form the full life cycle information, comprehensive data summarization and effective management are realized, and synchronous updating of data information between the plurality of data management platforms and the target association platform is realized.

Description

Technical Field

[0001] This application relates to the field of big data technology, and in particular to a data management method, system, apparatus, computer equipment, and storage medium. Background Technology

[0002] With the explosive growth in the scale of MySQL containers within data centers, the number of MySQL operation and maintenance and emergency response platforms is also increasing.

[0003] In related technologies, when management departments apply data, staff obtain ledger information from multiple data management platforms (DMPs) of the department for each database container, and summarize and manage the ledger information obtained from multiple DMPs.

[0004] However, the related technologies suffer from low data management efficiency. Summary of the Invention

[0005] Therefore, it is necessary to provide a data management method, system, device, computer equipment, and storage medium that can improve the efficiency of data management across multiple platforms to address the aforementioned technical problems.

[0006] Firstly, this application provides a data management method, which includes:

[0007] In response to data acquisition commands, the system retrieves ledger information for each database container from multiple data management platforms; the ledger information includes the data matching IP for each database container.

[0008] Based on the data matching IP of each database container, supplementary data information of each database container is obtained from the target associated platform; the target associated platform includes at least one platform;

[0009] Based on the ledger information and supplementary data of each database container, generate the full lifecycle information of each database container;

[0010] Based on the full lifecycle information of each database container, data processing operations are performed on the data in the target associated platform.

[0011] In one embodiment, the supplementary data information includes at least one of performance monitoring information, container information, and backup information; the target associated platform includes one or more of a performance capacity management platform, a database container self-service platform, and a data backup management platform.

[0012] Correspondingly, based on the data matching IP of each database container, supplementary data information of each database container is obtained from the target associated platform, including at least one of the following:

[0013] Based on the data matching IP of each database container, obtain the performance monitoring information corresponding to each data matching IP from the performance capacity management platform;

[0014] Based on the data matching IP of each database container, obtain the container information corresponding to each data matching IP from the database container self-service platform;

[0015] Based on the data matching IP of each database container, the backup information corresponding to each matching IP is obtained from the data backup management platform.

[0016] In one embodiment, based on the ledger information and supplementary data information of each database container, full lifecycle information of each database container is generated, including:

[0017] The ledger information of each database container is aggregated and processed, and the aggregated ledger information is stored in a preset data management instance table;

[0018] Based on the supplementary data, the summarized ledger information is further processed to obtain the full lifecycle information of each database container.

[0019] In one embodiment, the method further includes:

[0020] Retrieve the database container table; the database container table stores historical supplementary data information of the database containers managed by each data management platform;

[0021] Based on the data matching IP and database container table, determine whether the running environment of each database container is a new environment, and identify the new database container in the new environment and the original database container in the non-new environment.

[0022] Write the supplementary data information of the newly added database container into the database container table;

[0023] Based on the supplementary data information of the original database container, update the historical supplementary data information of the original database container in the database container table.

[0024] In one embodiment, the target associated platform is a performance capacity management platform;

[0025] Based on the full lifecycle information of each database container, data processing operations are performed on the data in the target associated platform, including:

[0026] Based on the full lifecycle information of each database container, the emergency container information in the performance capacity management platform is updated;

[0027] Based on the full lifecycle information of each database container, newly added database containers are included in the container monitoring items of the performance and capacity management platform, and the running status of newly added database containers is monitored through the performance and capacity management platform.

[0028] In one embodiment, the target associated platform is a data backup management platform;

[0029] Based on the full lifecycle information of each database container, data processing operations are performed on the data in the target associated platform, including:

[0030] Based on the full lifecycle information of each database container, newly added environments are entered into the data backup management platform for backup.

[0031] In one embodiment, the method further includes:

[0032] Based on the full lifecycle information of each database container, check the running information of each database container;

[0033] If at least one database container has abnormal operation information, an alarm will be triggered using a preset alarm method.

[0034] In one embodiment, the method further includes:

[0035] Based on preset functional indicator monitoring items, the activation status of functional indicators in multiple data management platforms is monitored.

[0036] If at least one functional indicator is not enabled, an alarm will be triggered using a preset alarm method.

[0037] Secondly, this application also provides a data management system, which includes a data management server, multiple data management platforms, and at least one associated platform;

[0038] The data management server is used to execute the data management method in any of the embodiments of the first aspect described above;

[0039] Each data management platform is used to run and maintain multiple database containers, and records ledger information through the database containers;

[0040] The associated platform is used to record supplementary data information for the database containers in each data management platform.

[0041] Thirdly, this application also provides a data management device, which includes:

[0042] The ledger acquisition module is used to respond to data collection commands and obtain ledger information of each database container from multiple data management platforms; the ledger information includes the data matching IP of each database container;

[0043] The information acquisition module is used to obtain supplementary data information of each database container from the target associated platform based on the data matching IP of each database container; the target associated platform includes at least one platform.

[0044] The data management module is used to generate full lifecycle information for each database container based on the ledger information and supplementary data information of each database container.

[0045] The data processing module is used to perform data processing operations on the data in the target associated platform based on the full lifecycle information of each database container.

[0046] Fourthly, this application also provides a computer device including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of any of the method embodiments in the first aspect described above.

[0047] Fifthly, this application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of any of the method embodiments in the first aspect described above.

[0048] In a sixth aspect, this application also provides a computer program product comprising a computer program that, when executed by a processor, implements the steps of any of the method embodiments in the first aspect described above.

[0049] The aforementioned data management method, system, apparatus, computer equipment, and storage medium involve the computer equipment responding to a data acquisition command by obtaining ledger information for each database container from multiple data management platforms. This ledger information includes the data matching IPs of each database container. Based on these IPs, supplementary data information for each database container is obtained from a target associated platform. Based on the ledger and supplementary data information, full lifecycle information for each database container is generated. Based on this full lifecycle information, data processing operations are performed on the data in the target associated platform. The target associated platform includes at least one platform. In this method, the computer equipment can not only summarize the ledger information of database containers from multiple data management platforms but also obtain supplementary data information from the target associated platform to supplement and improve the summarized ledger information, thereby forming full lifecycle information for each database container and achieving comprehensive data summarization and effective management. Furthermore, based on the summarized full lifecycle information of each database container, the data in the target associated platform is processed. Thus, by using summarized data to inspect and update the target associated platform, synchronous data updates can be achieved between multiple data management platforms and the target associated platform. Attached Figure Description

[0050] Figure 1 This is a flowchart illustrating a data management method in one embodiment;

[0051] Figure 2 This is a schematic diagram of the process for obtaining full lifecycle information in one embodiment;

[0052] Figure 3 This is a schematic diagram of the update process of a database container table in one embodiment;

[0053] Figure 4 This is a schematic diagram of multi-platform data acquisition in one embodiment;

[0054] Figure 5 This is a schematic diagram of the monitoring process for a database container in one embodiment;

[0055] Figure 6 This is a schematic diagram of the monitoring process of a data management platform in one embodiment;

[0056] Figure 7 This is a schematic diagram illustrating multi-platform data synchronization in one embodiment;

[0057] Figure 8 This is a schematic diagram of the structure of a data management system in one embodiment;

[0058] Figure 9 This is a structural block diagram of a data management device in one embodiment;

[0059] Figure 10 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0060] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0061] With the explosive growth in the scale of MySQL containers in data centers, the number of internal MySQL operation and maintenance (O&M) and emergency response platforms within banks has also increased significantly, leading to chaotic management of ledger information generated across multiple departments. Furthermore, the ledger information between these O&M and emergency response platforms requires manual recording, aggregation, and analysis to obtain the full lifecycle information for each database container. Additionally, the lack of timely information updates across these platforms makes effective data synchronization difficult.

[0062] Based on this, the present application provides a data management method, system, device, computer device, and storage medium. By collecting and summarizing the ledger information of multiple data management platforms within a department on a computer device, and recording the full life cycle information of database containers, it solves the problems of chaotic ledger information management and platform automatic operation and maintenance. That is, the present application overcomes the deficiencies of independent management of multiple platforms. By uniformly managing the ledger information of multiple platforms, it realizes synchronous update of data information between multiple platforms. Further, through the joint management of multiple platforms, it provides convenience for operations such as operation and maintenance, monitoring, and emergency response of operation and maintenance personnel, and has good universality and scalability.

[0063] It should be noted that for the data management method provided in the embodiments of the present application, the execution subject may be a data management device, and this data management device may be implemented as part or all of a computer device through software, hardware, or a combination of software and hardware. Among them, this computer device may be a server or a terminal. The servers in the embodiments of the present application include but are not limited to at least one independent server, distributed server, cloud server, and server cluster; the terminal may include but is not limited to various personal computers, laptop computers, smartphones, tablet computers, portable wearable devices, intelligent robots, and other intelligent hardware devices. In the following method embodiments, it is described by taking the execution subject as a computer device as an example.

[0064] It should be understood that the following method embodiments may be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. Obviously, the described embodiments are part of the embodiments of the present application, rather than all of the embodiments.

[0065] In one embodiment, as Figure 1 shown, a data management method is provided. Taking the application of this method to a computer device as an example, it includes the following steps:

[0066] Step 110: In response to a data collection instruction, obtain the ledger information of each database container from multiple data management platforms; the ledger information includes the data matching IP of each database container.

[0067] Among them, the data collection instruction may be triggered manually, or a scheduled job pre-deployed on the computer device, or other triggering methods, and the embodiments of the present application do not limit this.

[0068] It should be noted that the Data Management Platform (DMP) in step 110 is a MySQL database management platform, primarily used for MySQL operation, maintenance, management, monitoring, and emergency response. As business data increases, the number of data management platforms within the bank's system also increases. Each data management platform can mount and manage multiple database containers; that is, the database containers managed by the DMP platform are MySQL containers.

[0069] Furthermore, each database container is used to store ledger information generated within the banking system. Ledger information from different database containers may be related, requiring aggregation and organization to form full lifecycle information.

[0070] In addition, the ledger information includes multiple data entries, each corresponding to a unique data matching IP. Based on this data matching IP, identical or similar data can be merged and organized.

[0071] Step 120: Based on the data matching IP of each database container, obtain supplementary data information of each database container from the target associated platform; the target associated platform includes at least one platform.

[0072] The target associated platform can be one platform or multiple platforms. It can be an external platform storing resource information of the underlying hardware of the data management platform, an external platform monitoring the operating status and device information of the database containers managed by the data management platform, or an external platform storing backup data of each data management platform / database container, etc. This embodiment does not limit the number of target associated platforms or the content they store.

[0073] In some application scenarios, the target associated platform includes one or more of the following: performance capacity management platform, database container self-service platform, and data backup management platform; supplementary data information includes at least one of the following: performance monitoring information, container information, and backup information.

[0074] Correspondingly, the implementation process of step 120 includes at least one of the following:

[0075] (1) Based on the data matching IP of each database container, obtain the performance monitoring information corresponding to each data matching IP from the performance capacity management platform.

[0076] Specifically, the Performance Capacity Management (PCM) platform is primarily used to retrieve Automatic Workload Repository (AWR) reports and perform queries, diagnostics, emergency response, and monitoring. Correspondingly, performance monitoring information may include MyAWR and monitoring IP addresses.

[0077] MyAWR is a performance monitoring tool for MySQL. MyAWR samples data at regular intervals based on the MySQL instance and then saves the data to the database for analysis.

[0078] (2) Based on the data matching IP of each database container, obtain the container information corresponding to each data matching IP from the database container self-service platform.

[0079] Among them, the Self-help Service (SSER platform) is a self-service platform for MySQL, mainly used for self-service application of MySQL containerization, execution environment setup, container rebuilding and database exit.

[0080] Correspondingly, container information includes application name, database name, SetID (a classification identifier at the object management level), deployment location, host IP, file system size, central processing unit (CPU), memory, image, and deployment time.

[0081] (3) Based on the data matching IP of each database container, obtain the backup information corresponding to each matching IP from the data backup management platform.

[0082] The database backup management platform is used to implement distributed storage and backup of data from multiple DMP platforms in the data center. In this embodiment, the database backup management platform can be implemented using the KKbackup platform, which uses distributed object storage as the backup medium to provide users with automated backup services. Correspondingly, the backup information includes the backup method and backup time.

[0083] In addition, the target associated platform may also include an Open System CentralizeMonitor (OSCM) system, a Configuration Management Database (CMDB) system, and a mail server.

[0084] The OSCM system is used to centrally process and display various monitoring events in the open platform system; the CMDB system is used to integrate the basic resource data and resource management relationship information required by various application systems and manage the relationship between various configuration items; the mail server is mainly used to send alarm emails.

[0085] It should be noted that when the target associated system includes an OSCM system, a CMDB system, and a mail server, the computer device can obtain data from them for analysis, management, and use, or it can choose not to perform the data acquisition operation. This embodiment does not impose any restrictions on this.

[0086] Step 130: Generate the full lifecycle information of each database container based on the ledger information and supplementary data information of each database container.

[0087] Among them, the full lifecycle information of a database container refers to the information, data, and processes throughout the entire lifecycle of a database container, from requirements, planning, design, operation, data use, maintenance, and recycling.

[0088] In one possible implementation, step 130 is performed as follows: based on the data matching IP, the ledger information and supplementary data information of each database container are merged and processed to generate the full lifecycle information of each database container.

[0089] Step 140: Based on the full lifecycle information of each database container, perform data processing operations on the data in the target associated platform.

[0090] The data processing operations may include at least one of the following: data update, data storage, data filtering, anomaly monitoring, etc.

[0091] In addition, computer equipment can also perform other data analysis operations based on the massive amounts of data aggregated from various database containers, such as for business planning and strategy adjustments. This embodiment does not limit the scope of data use.

[0092] The aforementioned data management method involves a computer device responding to a data acquisition command by obtaining ledger information for each database container from multiple data management platforms. This ledger information includes the data matching IPs of each database container. Based on these IPs, supplementary data information for each database container is obtained from a target associated platform. Based on the ledger and supplementary data information, full lifecycle information for each database container is generated. Data processing operations are then performed on the data in the target associated platform based on this full lifecycle information. The target associated platform includes at least one platform. In this method, the computer device can not only aggregate the ledger information of database containers from multiple data management platforms but also obtain supplementary data information from the target associated platform to supplement and improve the aggregated ledger information, thereby forming full lifecycle information for each database container and achieving comprehensive data aggregation and effective management. Furthermore, the data in the target associated platform is processed based on the aggregated full lifecycle information of each database container. Thus, by using aggregated data to inspect and update the target associated platform, synchronous data updates can be achieved between multiple data management platforms and the target associated platform.

[0093] In one embodiment, such as Figure 2 As shown, the process of generating the full lifecycle information of each database container based on the ledger information and supplementary data information of each database container in step 130 above includes the following steps:

[0094] Step 210: Perform data aggregation processing on the ledger information of each database container, and store the aggregated ledger information in the preset data management instance table.

[0095] The data aggregation and processing may include data cleaning, format conversion, data item merging, and classification and summarization, and may also include other processing operations, which are not limited in this embodiment.

[0096] Specifically, when summarizing the ledger information of each database container, the ledger information of each database container can be summarized and processed based on the data matching IP corresponding to each data entry.

[0097] In addition, the database in the computer device or the database connected to the computer device has a data management instance table, which is used to record the full lifecycle information of the ledger information of each database container.

[0098] Step 220: Based on the supplementary data, perform supplementary processing on the summarized ledger information to obtain the full lifecycle information of each database container.

[0099] In one possible implementation, step 220 is performed as follows: based on the data matching IP, the supplementary data corresponding to each data in the ledger information is obtained from the supplementary data information, and the supplementary data is filled into the corresponding position in the data management instance table to form complete lifecycle information for each data.

[0100] In this embodiment, the ledger information and supplementary data information of each database container in multiple data management platforms are summarized and managed to obtain the full lifecycle information of each database container. Thus, based on the massive amount of full lifecycle information of database containers, linkage can be achieved between the data management platform and the target associated platform, and unified data management across multiple platforms is also facilitated.

[0101] In one embodiment, such as Figure 3 As shown, taking the application of this method to a computer device as an example, the update process of a database container table includes the following steps:

[0102] Step 310: Obtain the database container table; the database container table stores historical supplementary data information of the database containers managed by each data management platform.

[0103] The database container table can be stored in the database of the computer device or in an external database of the computer device, and is used to record historical supplementary data information of each database container.

[0104] Step 320: Based on the data matching IP and database container table, determine whether the running environment of each database container is a new environment, and identify the new database container in the new environment and the original database container in the non-new environment.

[0105] In one possible implementation, step 320 is performed as follows: Based on the matching IP, the database container table is traversed by data item IP. If no matching data item IP exists, the database container corresponding to that matching IP operates in a new environment, and this database container is considered a new database container. If a matching data item IP exists, the database container corresponding to that matching IP does not operate in a new environment, and this database container is considered an original database container. In this way, database containers from multiple data management platforms are categorized into new database containers and original database containers to monitor the master / slave failover status of database containers in real time.

[0106] Step 330: Write the supplementary data information of the newly added database container into the database container table.

[0107] In one possible implementation, step 330 is performed as follows: based on the supplementary data information of the newly added database container, a data record is inserted into the database container table to record the supplementary data information of the newly added database container in the database container table.

[0108] It should be understood that if there are multiple new database containers, multiple data records will be inserted into the database container table. The number of inserted data records is the same as the number of new database containers, and there is a one-to-one correspondence between the two.

[0109] The newly inserted data records include at least the data matching IP, virtual IP (SIP), DMP address, group name, server ID (i.e., the hostname managed on the DMP), and the protocol type of the Service-Level Agreement (SLA) for the newly added database container.

[0110] Step 340: Update the historical supplementary data information of the original database container in the database container table based on the supplementary data information of the original database container.

[0111] Among them, the historical supplementary data information of the original database container is generated as a data record corresponding to the original database container in the database container table.

[0112] It should be understood that for the original database container, its historical supplementary data information (i.e., supplementary data information not obtained at the current moment) is already recorded in the database container table. Therefore, each time supplementary data information of the original database container is collected, the historical supplementary data information originally stored in the database container can be updated based on the supplementary data information to supplement and improve the corresponding data records of the original database container in the database container table.

[0113] In one possible implementation, step 340 is implemented as follows: based on the data matching IP corresponding to the original database container, a data record corresponding to the original database container is determined from the database container table, and then the data record corresponding to the original database container is updated based on the supplementary data information of the original database container.

[0114] Additionally, if there are multiple original database containers, first determine multiple data records in the database container table, with a one-to-one correspondence between the original database containers and the data records. Then, update the corresponding data records for each original database container in the database container table.

[0115] In this embodiment, the database container table is updated using supplementary data obtained from the target associated platform, so as to record the container information and master / slave switch status of each database container.

[0116] Based on the above method embodiments, see [link to relevant documentation]. Figure 4 The process of collecting and aggregating data from multiple data management platforms and target association platforms through computer equipment is as follows: After the MySQL container is put into operation, it is operated and managed by the DMP platform. The computer equipment responds to the scheduled tasks, collects new environmental data and updated existing data from multiple DMP platforms to update the primary and backup roles in real time; and puts the ledger information of the MySQL container collected from multiple DMPs into the database and stores it on the computer equipment.

[0117] Furthermore, supplementary data collected from the PCM platform is used to update MyAWR and monitoring IP; supplementary data collected from the SSER platform is used to update the physical machine CPU and memory resource information of the MySQL container managed in the DMP platform, as well as the MySQL container image; and supplementary data collected from the KKbackup platform is used to update the backup time.

[0118] Optionally, when inserting the ledger information from the MySQL container into the database, error messages can be displayed to verify and check for anomalies in the inserted ledger information.

[0119] Optionally, the computer device may also be equipped with at least one backup database to back up the data stored in the computer device, so as to ensure the security of data storage.

[0120] In this way, by using computer equipment to achieve unified collection, processing, and storage of data from multiple platforms, full coverage of open platform system data is achieved, solving the problems of multi-platform operation and maintenance, as well as chaotic monitoring and management difficulties after data center scaling. Moreover, centralizing data management on a single computer device can also reduce the manual operation and maintenance costs of managing multiple platforms individually.

[0121] Based on the above embodiments, after the computer device collects and summarizes data from multiple platforms, it can also perform inspections on the data in the target associated platform based on the summarized complete data, so as to achieve synchronous updates of data across multiple platforms.

[0122] In one embodiment, when the target associated platform is a performance capacity management platform, the implementation process of performing data processing operations on the data in the target associated platform based on the full lifecycle information of each database container in step 104 above can be as follows: update the emergency container information in the performance capacity management platform based on the full lifecycle information of each database container; add new database containers to the container monitoring items of the performance capacity management platform based on the full lifecycle information of each database container, and monitor the running status of the new database containers through the performance capacity management platform.

[0123] In other words, newly added database containers are added to the PCM platform for emergency use, and MySQL instances that are not included in the PCM platform are checked.

[0124] In another embodiment, when the target associated platform is a data backup management platform, the implementation process of performing data processing operations on the data in the target associated platform based on the full lifecycle information of each database container in step 104 above is as follows: based on the full lifecycle information of each database container, the newly added environment is entered into the data backup management platform for backup.

[0125] In other words, newly added environments are automatically entered into the KKbackup platform for remote backup, and backup libraries not included in the KKbackup platform are checked.

[0126] In this embodiment, the performance capacity management platform and data backup management platform are updated using the full lifecycle information of each database container to ensure that the data in the entire system can be synchronized.

[0127] In one embodiment, such as Figure 5 As shown, computer devices can also monitor database containers across multiple data management platforms based on the full lifecycle information of the database container. Specifically, the database container monitoring process includes the following steps:

[0128] Step 510: Based on the full lifecycle information of each database container, check the running information of each database container.

[0129] The runtime information includes at least one of the following: CPU memory, user password login, and thread pool running status.

[0130] In other words, by analyzing the full lifecycle information of each database container, we can determine the operational information of each database container and, based on the operational information of each database container, determine whether there are any anomalies.

[0131] Step 520: If there is an anomaly in the running information of at least one database container, an alarm will be triggered using a preset alarm method.

[0132] As an example, the default alarm method is to send an alarm email in case of an anomaly via the mail server. The alarm email carries the identifier of the database container that is running abnormally, its running information, and the reason for the anomaly.

[0133] As another example, the default alarm method is to issue an alarm through the OSCM system and display the identifier, running information, and cause of the abnormal database container in the OSCM system's monitoring interface.

[0134] It should be understood that when at least one of the following abnormal conditions occurs: CPU memory usage exceeds a preset threshold, user login with an incorrect password, or thread pool blockage, the computer device will issue an alarm using a preset alarm method. If the operational information of each database container is normal, no alarm is required.

[0135] In this embodiment, the operation information of each database container is checked by using the full lifecycle information of each database container, so as to promptly alert when the database container is running abnormally, thereby improving the operation and maintenance efficiency of the database container.

[0136] In one embodiment, such as Figure 6 As shown, computer equipment can also monitor multiple data management platforms based on preset functional indicators. Specifically, the monitoring process of the data management platform includes the following steps:

[0137] Step 610: Based on preset functional indicator monitoring items, monitor the activation status of functional indicators in multiple data management platforms.

[0138] The preset functional monitoring indicators include whether the database container is managed by the performance and capacity management platform, whether the high availability decision of the data management platform is enabled, and whether the high availability capability of the database container is enabled.

[0139] In one possible implementation, step 610 is performed as follows: checking if any MySQL containers in multiple data management platforms are not managed by the data management platform; monitoring the enabling status of high availability decisions on each data management platform; and monitoring the enabling status of high availability capabilities of the MySQL instances managed by multiple data management platforms.

[0140] Step 620: If at least one functional indicator is not enabled, an alarm will be triggered using a preset alarm method.

[0141] Similarly, the preset alarm methods include sending alarm emails via the mail server or issuing alarms through the OSCM system.

[0142] Specifically, if a MySQL container is not managed by the data management platform, and / or the high availability decision of the data management platform is not enabled, and / or the high availability capability of the MySQL instance is not enabled, an alarm will be triggered using a preset alarm method.

[0143] In this embodiment, by monitoring the activation status of functional indicators in multiple data management platforms, and promptly issuing alarms in case of abnormalities, the managed database containers in each data management platform are effectively maintained.

[0144] Based on the above method embodiments, see [link to relevant documentation]. Figure 7Based on the collected and aggregated full lifecycle information of each database container, the computer equipment disseminates the latest data across the entire platform. Specifically, for the PCM platform, newly added environments are automatically entered into the PCM platform for emergency backups, and MySQL instances not included in the PCM platform are checked. For the KKbackup platform, newly added environments are automatically entered into the KKbackup platform for remote backups, and backup databases not included in the KKbackup platform are checked. For multiple DMP platforms, monitoring and alerts are issued for MySQL containers not managed by the DMP platform, high availability decisions not enabled, and MySQL instance high availability capabilities not enabled.

[0145] Furthermore, by collecting comprehensive and complete lifecycle information, and employing three self-developed functional indicator monitoring items, we conduct anomaly monitoring for CPU memory usage and incorrect user password logins. Simultaneously, we invoke the PCM platform to monitor and alert on thread pool congestion. In case of anomalies, we also send alerts via the mail server or OSCM system.

[0146] Additionally, during the final shutdown phase of the database container's lifecycle, the CMDB system is first invoked to check the system's node status. Then, for database containers in the system's recycling state, the SSER platform interface is invoked to perform batch database dumping.

[0147] Optionally, computer equipment can also perform performance and capacity analysis and utilization of the entire system based on the full lifecycle information of massive database containers.

[0148] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0149] Furthermore, based on the same inventive concept, such as Figure 8 As shown in the figure, this application embodiment also provides a data management system, which includes a data management server 810, multiple data management platforms 820 and at least one associated platform 830.

[0150] The data management server is used to execute the data management method shown in any of the above method embodiments; each data management platform is used to run and maintain multiple database containers and record ledger information through the database containers; the associated platform is used to record supplementary data information of the database containers in each data management platform.

[0151] In this embodiment, the data management server in the data management system obtains ledger information of each database container from multiple data management platforms and supplementary data information of each database container from at least one associated platform. Then, based on the ledger information and supplementary data information of each database container, the full lifecycle information of each database container is generated. Furthermore, based on the full lifecycle information of each database container, the data in at least one associated platform is synchronously updated.

[0152] Furthermore, based on the same inventive concept, this application also provides a data management apparatus for implementing the data management method described above. The solution provided by this apparatus is similar to the implementation described in the above method; therefore, the specific limitations in one or more data management apparatus embodiments provided below can be found in the limitations of the data management method described above, and will not be repeated here.

[0153] In one embodiment, such as Figure 9 As shown, a data management device 900 is provided, comprising: a ledger acquisition module 910, an information acquisition module 920, a data management module 930, and a data processing module 940, wherein:

[0154] The ledger acquisition module 910 is used to respond to data collection commands and obtain ledger information of each database container from multiple data management platforms; the ledger information includes the data matching IP of each database container;

[0155] The information acquisition module 920 is used to obtain supplementary data information of each database container from the target associated platform based on the data matching IP of each database container; the target associated platform includes at least one platform.

[0156] The data management module 930 is used to generate full lifecycle information for each database container based on the ledger information and supplementary data information of each database container.

[0157] The data processing module 940 is used to perform data processing operations on the data in the target associated platform based on the full lifecycle information of each database container.

[0158] In one embodiment, the supplementary data information includes at least one of performance monitoring information, container information, and backup information; the target associated platform includes one or more of a performance capacity management platform, a database container self-service platform, and a data backup management platform.

[0159] Correspondingly, the information acquisition module 920 includes at least one of the following acquisition units:

[0160] The first acquisition unit is used to obtain the performance monitoring information corresponding to each data matching IP from the performance capacity management platform based on the data matching IP of each database container.

[0161] The second acquisition unit is used to obtain the container information corresponding to each data matching IP from the database container self-service platform based on the data matching IP of each database container.

[0162] The third acquisition unit is used to obtain the backup information corresponding to each matched IP from the data backup management platform based on the data matching IP of each database container.

[0163] In one embodiment, the data management module 930 includes:

[0164] The aggregation unit is used to aggregate the ledger information of each database container and store the aggregated ledger information in a preset data management instance table.

[0165] The supplementary unit is used to supplement the summarized ledger information based on supplementary data information to obtain the full lifecycle information of each database container.

[0166] In one embodiment, the device 900 further includes:

[0167] The container table retrieval module is used to retrieve database container tables; the database container tables store historical supplementary data information of the database containers managed by each data management platform;

[0168] The determination module is used to match IPs and database container tables based on data to determine whether the running environment of each database container is a new environment, and to determine the new database container in the new environment and the original database container in the non-new environment.

[0169] The update module is used to write supplementary data information for newly added database containers into the database container table;

[0170] The update module is also used to update the historical supplementary data information of the original database container in the database container table based on the supplementary data information of the original database container.

[0171] In one embodiment, the target associated platform is a performance capacity management platform;

[0172] Data processing module 940 includes:

[0173] The update unit is used to update the emergency container information in the performance and capacity management platform based on the full lifecycle information of each database container.

[0174] The monitoring unit is used to incorporate newly added database containers into the container monitoring items of the performance and capacity management platform based on the full lifecycle information of each database container, and to monitor the running status of the newly added database containers through the performance and capacity management platform.

[0175] In one embodiment, the target associated platform is a data backup management platform;

[0176] Data processing module 940 includes:

[0177] The backup unit is used to input new environments into the data backup management platform for backup based on the full lifecycle information of each database container.

[0178] In one embodiment, the device 900 further includes:

[0179] The runtime inspection module is used to check the runtime information of each database container based on the full lifecycle information of each database container;

[0180] The first alarm module is used to issue an alarm through a preset alarm method if there is any abnormality in the running information of at least one database container.

[0181] In one embodiment, the device 900 further includes:

[0182] The function monitoring module is used to monitor the activation status of function indicators in multiple data management platforms based on preset function indicator monitoring items.

[0183] The second alarm module is used to trigger an alarm using a preset alarm method if at least one functional indicator is not enabled.

[0184] Each module in the aforementioned data management device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the operations corresponding to each module.

[0185] In one embodiment, a computer device is provided, which may be a server, specifically... Figure 8 The internal structure diagram of the data management server 810 shown is as follows: Figure 10As shown, the computer device includes a processor, memory, and a network interface connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database, while the internal memory provides the environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The database stores data such as the full lifecycle information of database containers in multiple data management platforms. The network interface communicates with external terminals via a network connection. When the computer program is executed by the processor, it implements a data management method.

[0186] Those skilled in the art will understand that Figure 10 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0187] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform the following steps:

[0188] In response to data acquisition commands, the system retrieves ledger information for each database container from multiple data management platforms; the ledger information includes the data matching IP for each database container.

[0189] Based on the data matching IP of each database container, supplementary data information of each database container is obtained from the target associated platform; the target associated platform includes at least one platform;

[0190] Based on the ledger information and supplementary data of each database container, generate the full lifecycle information of each database container;

[0191] Based on the full lifecycle information of each database container, data processing operations are performed on the data in the target associated platform.

[0192] The computer device provided in the above embodiments has a similar implementation principle and technical effect to the above method embodiments, and will not be described again here.

[0193] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program performing the following steps when executed by a processor:

[0194] In response to data acquisition commands, the system retrieves ledger information for each database container from multiple data management platforms; the ledger information includes the data matching IP for each database container.

[0195] Based on the data matching IP of each database container, supplementary data information of each database container is obtained from the target associated platform; the target associated platform includes at least one platform;

[0196] Based on the ledger information and supplementary data of each database container, generate the full lifecycle information of each database container;

[0197] Based on the full lifecycle information of each database container, data processing operations are performed on the data in the target associated platform.

[0198] The computer-readable storage medium provided in the above embodiments has a similar implementation principle and technical effect to the above method embodiments, and will not be described again here.

[0199] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, performs the following steps:

[0200] In response to data acquisition commands, the system retrieves ledger information for each database container from multiple data management platforms; the ledger information includes the data matching IP for each database container.

[0201] Based on the data matching IP of each database container, supplementary data information of each database container is obtained from the target associated platform; the target associated platform includes at least one platform;

[0202] Based on the ledger information and supplementary data of each database container, generate the full lifecycle information of each database container;

[0203] Based on the full lifecycle information of each database container, data processing operations are performed on the data in the target associated platform.

[0204] The computer program product provided in the above embodiments has a similar implementation principle and technical effect to the above method embodiments, and will not be described again here.

[0205] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the methods described above. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, or optical storage, etc. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc.

[0206] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0207] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A data management method, characterized in that, The method includes: In response to a data acquisition command, ledger information for each database container is obtained from multiple data management platforms; the ledger information includes the data matching IP of each database container; wherein, the data matching IP of the database container is used to merge and organize identical or similar data; Based on the data matching IP of each database container, supplementary data information of each database container is obtained from the target association platform; the target association platform includes at least one platform; wherein, the supplementary data information includes at least one of performance monitoring information, container information, and backup information; the target association platform includes one or more of a performance capacity management platform, a database container self-service platform, and a data backup management platform. Based on the ledger information and supplementary data information of each database container, generate the full lifecycle information of each database container; Based on the full lifecycle information of each of the database containers, data processing operations are performed on the data in the target associated platform.

2. The method according to claim 1, characterized in that, Correspondingly, the step of obtaining supplementary data information of each database container from the target associated platform based on the data matching IP of each database container includes at least one of the following: Based on the data matching IP of each database container, obtain the performance monitoring information corresponding to each data matching IP from the performance capacity management platform; Based on the data matching IP of each database container, obtain the container information corresponding to each data matching IP from the database container self-service platform; Based on the data matching IP of each database container, the backup information corresponding to each matching IP is obtained from the data backup management platform.

3. The method according to claim 1, characterized in that, The step of generating full lifecycle information for each database container based on its ledger information and supplementary data information includes: The ledger information of each of the database containers is aggregated and processed, and the aggregated ledger information is stored in a preset data management instance table; Based on the supplementary data, the summarized ledger information is further processed to obtain the full lifecycle information of each database container.

4. The method according to claim 1, characterized in that, The method further includes: Obtain the database container table; the database container table stores historical supplementary data information of the database containers managed by each of the data management platforms; Based on the data matching IP and the database container table, determine whether the running environment of each database container is a new environment, and identify the new database container in the new environment and the original database container in the non-new environment. Write the supplementary data information of the newly added database container into the database container table; Based on the supplementary data information of the original database container, the historical supplementary data information of the original database container is updated in the database container table.

5. The method according to any one of claims 1 to 4, characterized in that, When the target associated platform is a performance and capacity management platform; The data processing operations performed on the data in the target associated platform based on the full lifecycle information of each of the database containers include: Based on the full lifecycle information of each database container, the emergency container information in the performance capacity management platform is updated; Based on the full lifecycle information of each database container, the newly added database container is included in the container monitoring item of the performance and capacity management platform, and the running status of the newly added database container is monitored through the performance and capacity management platform.

6. The method according to any one of claims 1 to 4, characterized in that, When the target associated platform is a data backup management platform; The data processing operations performed on the data in the target associated platform based on the full lifecycle information of each of the database containers include: Based on the full lifecycle information of each database container, the newly added environment is entered into the data backup management platform for backup.

7. The method according to any one of claims 1 to 4, characterized in that, The method further includes: Based on the full lifecycle information of each database container, check the running information of each database container; If at least one of the database containers has abnormal operating information, an alarm will be triggered using a preset alarm method.

8. The method according to any one of claims 1 to 4, characterized in that, The method further includes: Based on preset functional indicator monitoring items, the activation status of functional indicators in the multiple data management platforms is monitored. If at least one of the aforementioned functional indicators is not enabled, an alarm will be triggered using a preset alarm method.

9. A data management system, characterized in that, The system includes a data management server, multiple data management platforms, and at least one associated platform; The data management server is used to execute the data management method according to any one of claims 1 to 8; Each of the aforementioned data management platforms is used to run and maintain multiple database containers, and to record ledger information through the database containers; The associated platform is used to record supplementary data information of the database containers in each of the data management platforms.

10. A data management device, characterized in that, The device includes: The ledger acquisition module is used to respond to data collection instructions and obtain ledger information of each database container from multiple data management platforms; the ledger information includes the data matching IP of each database container; wherein, the data matching IP of the database container is used to merge and organize the same or similar data; The information acquisition module is used to acquire supplementary data information of each database container from a target association platform based on the data matching IP of each database container; the target association platform includes at least one platform; wherein, the supplementary data information includes at least one of performance monitoring information, container information, and backup information; the target association platform includes one or more of a performance capacity management platform, a database container self-service platform, and a data backup management platform. The data management module is used to generate full lifecycle information for each of the database containers based on the ledger information and supplementary data information of each database container. The data processing module is used to perform data processing operations on the data in the target associated platform based on the full lifecycle information of each of the database containers.

11. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the method of any one of claims 1 to 8.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 8.

Images