Database data management method, device, computer equipment and storage medium

Abstract

The present invention relates to the field of database technology, and discloses a database data management method, device, computer equipment, and storage medium, including: obtaining time information of data to be processed; determining a storage area corresponding to the data to be processed based on the time information; processing the data to be processed based on different storage areas, and determining a processing result. The method divides the data to be processed according to the time information, stores the divided data to be processed in different storage areas, and then processes the data to be processed accordingly based on the storage areas. The specific processing method can be determined according to the actual application scenario. For scenarios such as writing and reading database data, the method can avoid memory overflow caused by writing or loading a large amount of data at one time, effectively improving the stability of the database, reducing the resource usage of the database, and improving the efficiency of data management.

Description

Technical Field

[0001] The present invention relates to the technical field of databases, and in particular to a method, device, computer equipment and storage medium for managing database data. Background Art

[0002] A server is a type of computer that runs faster and handles higher loads than regular computers. It provides computing or application services to other clients on a network. Servers feature high-speed computing capabilities, long-term reliable operation, robust I / O external data throughput, and excellent scalability. Monitoring servers can help improve them and detect server failures promptly. Server performance monitoring involves monitoring system resources, such as CPU usage, memory consumption, and CPU temperature, to help identify server performance issues.

[0003] Typically, when monitoring server performance, the collected performance data is stored in a database. This data is then displayed on the front-end page, or based on user queries. Performance monitoring is typically performed on all servers in a server cluster. Due to the large number of server nodes, the database may experience memory overflows during large-scale data collection, writing, and querying. Summary of the Invention

[0004] In view of this, embodiments of the present invention provide a database data management method, apparatus, computer equipment, and storage medium to solve the problem of database memory overflow.

[0005] In a first aspect, an embodiment of the present invention provides a method for managing data in a database, the method comprising:

[0006] Get the time information of the data to be processed;

[0007] Determining a storage area corresponding to the data to be processed based on the time information;

[0008] The data to be processed is processed based on the different storage areas to determine a processing result.

[0009] The database data management method provided in this embodiment collects time information of the data to be processed, determines the storage area corresponding to the data to be processed based on the time information, and then processes the data to be processed in different storage areas. This method divides the data to be processed based on the time information, stores the divided data to be processed in different storage areas, and then processes the data accordingly based on the storage area. The specific processing method can be determined according to the actual application scenario. For scenarios such as writing and reading database data, this method can avoid memory overflow caused by writing or loading large amounts of data at once, effectively improving the stability of the database, reducing database resource usage, and improving data management efficiency.

[0010] In some optional implementations, if the data to be processed is data to be written, the time information includes the acquisition time of the data to be written, and the storage area includes a target database and a disk file, determining the storage area corresponding to the data to be processed based on the time information includes:

[0011] Calculate the time difference between the acquisition time of the data to be written and the current time;

[0012] Determine the difference between the time difference and a preset disk placement threshold, and obtain a determination result;

[0013] The data to be written is divided into first data to be written and second data to be written based on the judgment result. The storage area corresponding to the first data to be written is the target database, and the storage area corresponding to the second data to be written is the disk file.

[0014] In some optional implementations, processing the data to be processed based on the different storage areas and determining a processing result includes:

[0015] Writing the first data to be written into the target database;

[0016] The second data to be written is written into the disk file.

[0017] In some optional implementations, processing the data to be processed based on different storage areas and determining a processing result further includes:

[0018] The second data to be written is obtained from the disk file, and the second data to be written is stored in the target database.

[0019] In some optional implementations, if the data to be processed is data to be read, the time information includes the acquisition time and the query time of the data to be read, and determining the storage area corresponding to the data to be processed based on the time information includes:

[0020] storing index information corresponding to the to-be-read data whose acquisition time is less than a preset acquisition time threshold in a first reading area;

[0021] The index information corresponding to the data to be read in the first reading area whose query time is lower than the preset query time threshold is stored in the second reading area, and the index information of the data to be read in the first reading area and the index information of the data to be read in the second reading area corresponding to the data to be read are stored in the third reading area.

[0022] In some optional implementations, processing the data to be processed based on different storage areas and determining a processing result includes:

[0023] Get query request;

[0024] Determining, based on the query request, whether the first read area stores a target index corresponding to the query request;

[0025] When the target index corresponding to the query request is stored in the first read area, the target to-be-read data corresponding to the target index is acquired from the third read area.

[0026] In some optional implementations, processing the data to be processed based on different storage areas and determining a processing result further includes:

[0027] When the target index corresponding to the query request is not stored in the first read area, determining whether the target index corresponding to the query request is stored in the second read area;

[0028] When the target index corresponding to the query request is stored in the second read area, the target to-be-read data corresponding to the target index is acquired from the third read area.

[0029] In a second aspect, an embodiment of the present invention provides a data management device for a database, the device comprising:

[0030] An information acquisition module is used to obtain time information of the data to be processed;

[0031] an area determination module, configured to determine a storage area corresponding to the data to be processed based on the time information;

[0032] The data processing module is used to process the data to be processed based on different storage areas and determine the processing results.

[0033] In a third aspect, an embodiment of the present invention provides a computer device comprising: a memory and a processor, the memory and the processor being communicatively connected to each other, computer instructions being stored in the memory, and the processor executing the database data management method of the first aspect or any corresponding embodiment thereof by executing the computer instructions.

[0034] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium having computer instructions stored thereon, the computer instructions being used to enable a computer to execute the database data management method of the first aspect or any corresponding embodiment thereof. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] In order to more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the specific embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0036] Figure 1 is a flowchart of a database data management method according to some embodiments of the present invention;

[0037] Figure 2 is a flowchart of a database data management method according to some embodiments of the present invention;

[0038] Figure 3 is a schematic diagram of a cache queue according to some embodiments of the present invention;

[0039] Figure 4 is a flowchart of a database data management method according to some embodiments of the present invention;

[0040] Figure 5 is a flowchart of a database data management method according to some embodiments of the present invention;

[0041] Figure 6 is a schematic diagram of a storage area according to some embodiments of the present invention;

[0042] Figure 7 is a schematic diagram of a data query process according to some embodiments of the present invention;

[0043] Figure 8 is a schematic diagram of a data writing process according to some embodiments of the present invention;

[0044] Figure 9 is a structural block diagram of a data management device for a database according to an embodiment of the present invention;

[0045] Figure 10 Schematic diagram of the hardware structure of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0046] To make the purpose, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without making creative efforts shall fall within the scope of protection of the present invention.

[0047] Monitoring server performance helps identify server performance issues and perform timely repairs. In related technologies, server performance can be collected and displayed through monitoring software. However, since server clusters are usually monitored in large quantities, the database may face memory overflow problems in scenarios where data is written in seconds and index data changes frequently.

[0048] Based on this, an embodiment of the present invention provides a data management method for a database, thereby optimizing the memory usage of the database and improving data query performance.

[0049] According to an embodiment of the present invention, an embodiment of a method for managing data in a database is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0050] In this embodiment, a method for managing database data is provided. Figure 1 : is a flow chart of a method for managing database data according to an embodiment of the present invention. Figure 1 As shown, the process includes the following steps:

[0051] Step S11: Obtain time information of the data to be processed.

[0052] The embodiment of this method can be used for server performance monitoring software, which can be used to collect the performance data of the server, store the performance data in a database, and display the collected performance data through the front-end page. When the user needs to query the server performance, he can filter the data by entering keywords, time, etc., and display the corresponding performance data. Usually, a server cluster is monitored, and a server cluster involves multiple server devices. A collection tool is installed on each server device, and the performance data of the server is collected regularly according to a set period. The specific performance data to be collected and the period can be set according to actual needs. For example, it is set to collect every 5 seconds, and collect CPU utilization, CPU temperature, GPU thread utilization, disk file read and write performance, etc. After collecting the performance data, the collection tool sends a data write request to the database end, and the database end receives the data write request and stores the performance data.

[0053] The data to be processed refers to server performance data. The embodiments of the present invention involve scenarios such as writing and querying database data. Therefore, the data to be processed may include data that the collection tool needs to store in the database after collecting server performance data, data that users need to query and read from the database according to needs, etc. The location of the data to be processed in different scenarios is different.

[0054] Time information may include the time when server performance data was collected, the time when the data stored in the database was last queried, the frequency of queries for the data stored in the database, etc. The specific time information corresponding to the data to be processed in different scenarios varies. For example, for data that has not yet been stored in the database, its time information includes the time when the performance data was collected.

[0055] Step S12: Determine the storage area corresponding to the data to be processed based on the time information.

[0056] Before processing the data to be processed, it is necessary to store the data to be processed in different storage areas. For the data to be processed in different scenarios, the data to be processed is divided according to its time information. Specifically, a time threshold can be set, and the time information is compared with the time threshold to divide the data to be processed. The divided batches of data are stored in the corresponding storage areas.

[0057] The storage area can be in the form of a queue or a form. There is no restriction on its specific form and it can be set accordingly according to the actual scenario.

[0058] Step S13: Process the data to be processed based on different storage areas and determine the processing result.

[0059] Different processing operations are performed on the data to be processed in different storage areas. The specific processing methods for the data to be processed vary in different scenarios. When the application scenario is writing data, the final processing result is writing the processed data to the database. When the application scenario is querying data, the final processing result is reading the data from the database according to the query request.

[0060] Taking the scenario of writing data to the database as an example, the data to be processed is the data that needs to be stored in the database. Usually, it is necessary to store the collected data in the database in large quantities. In order to avoid the database memory overflow problem caused by large amounts of data storage, in this solution, the time information of the data to be processed is used to store some data that has been collected for a long time and has not yet been stored in the database in other areas, and the data with a newer collection time is stored in the database. The time information includes the collection time. In this way, the amount of data written to the database at one time is limited to avoid memory overflow caused by large amounts of data written.

[0061] The methods in this solution can be implemented through plug-ins. Different plug-ins can be used for different application scenarios, or all functions can be integrated into a single plug-in. This plug-in approach can be integrated into server performance monitoring software without intruding on the software's original business operations and without restrictions on the specific database products involved, thus reducing maintenance costs and technical risks. There are no restrictions on the database software involved, such as InfluxDB, Kdb+, and Graphite.

[0062] The database data management method provided in this embodiment collects time information of the data to be processed, determines the storage area corresponding to the data to be processed based on the time information, and then processes the data to be processed in different storage areas. In this method, the data to be processed is divided according to the time information, and the divided data to be processed is stored in different storage areas. The data to be processed is then processed accordingly based on the storage area. The specific processing method can be determined according to the actual application scenario. For scenarios such as writing and reading database data, this method can avoid memory overflow caused by writing or loading large amounts of data at one time, effectively improving the stability of the database, reducing the database's resource usage, and improving the efficiency of data management.

[0063] In this embodiment, a method for managing database data is provided. Figure 2 : is a flow chart of a method for managing database data according to an embodiment of the present invention. Figure 2 As shown, if the data to be processed is data to be written, the time information includes the acquisition time of the data to be written, and the storage area includes the target database and the disk file, the process includes the following steps:

[0064] Step S21: Obtain time information of the data to be processed.

[0065] For details, please see Figure 1 Step S11 of the illustrated embodiment will not be described in detail here.

[0066] Step S22: Calculate the time difference between the acquisition time of the data to be written and the current time.

[0067] This embodiment is applied to the scenario of writing data into a database. The data to be processed is the data to be written. The data to be written is sent to the database by the performance collection tool on the server after collecting the data. The collection tool sends a data write request to the database. The data to be written is included in the data write request. The server receives the data write request and obtains the data to be written from the data write request.

[0068] Before executing the method of step S22, the data to be written can be stored in the cache queue first. Specifically, the cache queue can be written into the cache queue in a double-hash nesting manner, such as Figure 3 As shown, the details are as follows:

[0069] 1. Hash Table 1 (HashTable1): uses an array + linked list structure, where the key is the name of the node to be collected and the value is a nested hash list. The size of HashTable1 is initialized to the number of nodes, where a node refers to a server and the number of nodes is the number of servers in the server cluster. First, capacity expansion is required. In this embodiment, the capacity expansion strategy is to expand when the capacity usage number is greater than or equal to 0.75 of the initial value. The expansion is (2*OldSize+1), where OldSize refers to the number of nodes at the time of the last data write operation.

[0070] The addressing method is to first find the position of the linked list and hash(nodeName)%threadNum in the array according to the hash value of NodeName, and then search one by one in the list to find the final HashEntry.

[0071] 2. Hash Table 2 (HashTable2): Similar to HashTable1 in structure, except that its initialization size is the number of collection items, and the addressing method is hash(collectName)%collectNum.

[0072] The actual data stored in hash table 2 is compressed binary data, including MetricID, Data (actual collected data, such as CPU utilization and memory utilization), Lock (a lock is added when the data is written to the disk or InfluxDB to prevent dirty data from being written), and Status (whether it can be recycled, including 0: Draft, 1: Writing to disk, 2: Writing to InfluxDB, 3: Successfully written to InfluxDB, 4: Failed to write to InfluxDB, 5: Successfully written to disk, and 6: Failed to write to disk).

[0073] MetricID is the data's unique identifier, representing the data's uniqueness. It consists of: time + nodeName + collectName + DeviceId. Lock and Status can be changed based on the data's state before it was written to the database.

[0074] A brief overview of the use of the double hash table is as follows: hash table 1 records the node name and the name of the data to be written, and the node name and the name of the data to be written in hash table 1 are used to locate hash table 2 to obtain the specific data corresponding to the node name and the name of the data to be written.

[0075] The following example illustrates the process of storing data to be written into the cache queue:

[0076] First, complete the initial configuration of the cache queue based on the cluster size, including the queue length, the threshold time for writing data to disk files, the number of concurrent threads, and the number of collection items. Collection items are the specific server attributes to be collected, such as temperature and usage. The collection item name is the name of the data to be written.

[0077] The format of the data to be written is {nodeName=node1, collectName=CPU cpuTemp=80°C, idle=80%}, where cpuTemp and idle are the collection item names;

[0078] Retrieve Hashtable1 based on nodeName and write nodeName=node1 and collectName=CPU into HashEntry of Hashtable1.

[0079] Retrieve Hashtable2 based on nodeName+collectName and write the following into HashEntry of Hashtable2:

[0080] MetricID=20230101235959_node1_CPU_CPU0, Data={cpuTemp=80°C, idle=80%}, Lock=no lock, Status=0.

[0081] After the data to be written is stored in the cache queue, the data is stored in the cache queue by monitoring the cache queue. The data to be written is read concurrently from the cache queue, and the time difference between the collection time of the data to be written and the current time is calculated. The data to be written carries its collection time.

[0082] Step S23: Determine the time difference and the preset disk placement threshold to obtain a determination result.

[0083] The preset flush threshold is the time threshold for writing data to disk files and is set based on actual conditions. The time difference is compared with the preset flush threshold to determine whether the time difference is less than or greater than the preset flush threshold.

[0084] Step S24 , dividing the data to be written into first data to be written and second data to be written based on the judgment result, the storage area corresponding to the first data to be written is the target database, and the storage area corresponding to the second data to be written is the disk file.

[0085] When the judgment result is that the time difference is greater than the preset disk threshold, it means that the data to be written is expired, that is, the data has not been processed for a long time after being collected, and it is determined as the second data to be written. The storage area corresponding to the second data to be written is the disk file.

[0086] When the judgment result is that the time difference is not greater than the preset disk threshold, the data is determined as the first data to be written, and the storage area corresponding to the first data to be written is the target database, which refers to the database where the data needs to be stored.

[0087] Step S25: Process the data to be processed based on different storage areas and determine the processing result.

[0088] After the data to be written is divided and the storage area corresponding to each data is determined, each data is written into the corresponding database.

[0089] Specifically, step S25 includes the following steps:

[0090] S251, writing the first data to be written into the target database;

[0091] S252: Write the second data to be written into the disk file.

[0092] In some optional implementations, step S25 further includes: acquiring second data to be written from the disk file, and storing the second data to be written in the target database.

[0093] For the second data to be written stored in the disk file, data playback can be performed to concurrently read the second data to be written from the disk file and directly write it to the target database. If writing to the target database fails due to timeouts, network errors, or other issues, the data that failed to be written to the target database will be written to the disk file.

[0094] The above process is described in full below. Figure 4 As shown in the figure, using InfluxDB as the target database, concurrent reading is performed on the pending data in the write cache queue, which is divided into a direct write queue and a data replay queue. The direct write queue determines whether to write the pending data to disk based on the time difference and the preset disk write threshold. If the result is no, the first pending data is written to InfluxDB; if the result is yes, the second pending data is written to the disk file. The data replay queue concurrently reads the second pending data from the disk file and writes it to InfluxDB. If writing to InfluxDB fails, the data is stored to the disk file.

[0095] When writing data from the cache queue to the database, its status in the hash table is modified based on the data's actual state. Once the data has been stored in the database or disk file, the data's status in the cache queue is changed to "Written to InfluxDB Successfully" or "Written to Disk Successfully." The data in the cache queue is deleted simultaneously to prevent excessive data storage in the cache queue and reduce memory usage. Between the time data is extracted from the cache queue and before it is stored in the database or disk file, it is locked, allowing only one-way transfers.

[0096] The database data management method provided in this embodiment calculates the time difference between the acquisition time and the current time of the data to be written, determines the difference between the time difference and a preset disk write threshold, and then divides the data to be written based on the judgment result. The first data to be written is written to the target database, the second data to be written is written to a disk file, and the second data to be written is read from the disk file and stored in the target database. By dividing the data to be written, this method limits the amount of data written to the target database at one time, enabling batched data storage, avoiding the problem of target database memory overflow caused by large amounts of data written, and optimizing memory usage.

[0097] In this embodiment, a method for managing database data is provided. Figure 5 : is a flow chart of a method for managing database data according to an embodiment of the present invention. Figure 5As shown, if the data to be processed is data to be read, the time information includes the acquisition time and the query time of the data to be read, and the process includes the following steps:

[0098] Step S31: Obtain time information of the data to be processed.

[0099] For details, please see Figure 1 Step S11 of the illustrated embodiment will not be described in detail here.

[0100] Step S32 : storing index information corresponding to the to-be-read data whose acquisition time is less than a preset acquisition time threshold in the first reading area.

[0101] This embodiment is applied to scenarios involving reading data from a database. The data to be processed is the data to be read, which is stored in a target database. The time information for the data to be read includes the acquisition time and the query time. The query time includes the time of each query. The query time can be used to determine the query frequency of the data to be read.

[0102] The collection time threshold is set to a number of days, for example, 30 days. When the target database is started, it first loads the data to be read that was collected within the past 30 days. The index information for this data is stored in the first read area. The data to be read includes performance data for each server. For example, if the CPU temperature is 50°C, the corresponding index information is the CPU temperature and the server node name. The index information is used to query the data to be read and can be in the form of an identification number, node name, or other form.

[0103] Step S33, the index information corresponding to the data to be read in the first reading area whose query time is lower than the preset query time threshold is stored in the second reading area, and the data to be read corresponding to the index information of the data to be read in the first reading area and the index information of the data to be read in the second reading area are stored in the third reading area.

[0104] The preset query time threshold refers to the maximum query time. When the preset query time threshold is exceeded, it means that the data has not been queried for a long time. The data whose query time is lower than the preset query time threshold is the first data to be read, and the index information corresponding to the first data to be read is stored in the second reading area.

[0105] That is, the data to be read in the first read area is screened, and the index information of the data that is not frequently queried is transferred to the second read area.

[0106] In addition, a data screening algorithm may be used to screen the data in the first reading area, such as a least frequently used algorithm.

[0107] The first and second reading areas store index information of the data to be read, and the third reading area stores the data to be read corresponding to all the index information. The index information is used to query the data to be read in the third reading area.

[0108] like Figure 6 As shown, the first read area is direct memory, the second read area is replacement memory, and the third read area is replacement data. Using a double cache queue to store commonly used index information and using replacement memory to store data corresponding to the two cache queues can optimize memory usage.

[0109] After the database service starts, the database's index information is loaded into direct memory according to chronological order and the preset collection time threshold. Data in direct memory is then migrated to replacement memory using a least frequently used algorithm. Data corresponding to index information in both direct and replacement memory is stored in replacement data. Data in replacement memory can also be eliminated, which deletes data. This also deletes the data corresponding to the index information stored in the replacement data.

[0110] The size of the cache queue can be defined, the default is 1M*10*2, and the default size of the replacement data is 16M*20.

[0111] Step S34: Process the data to be processed based on different storage areas and determine the processing result.

[0112] Specifically, step S34 includes the following steps:

[0113] Step S341: Obtain a query request.

[0114] The user can issue a query request on the front-end page. The query request may include index information such as the name of the server node, the name of the data collection item, and the collection time, that is, the target index.

[0115] Step S342 : determining whether a target index corresponding to the query request is stored in the first read area based on the query request.

[0116] A query is performed in the first read area according to the index information in the query request to determine whether the first read area stores the target index included in the query request.

[0117] Step S343 : when the target index corresponding to the query request is stored in the first read area, the target data to be read corresponding to the target index is obtained from the third read area.

[0118] When the target index is stored in the first read area, data corresponding to the target index, ie, target data to be read, is acquired from the third read area.

[0119] In some optional implementations, step S34 further includes the following steps:

[0120] Step S344: When the target index corresponding to the query request is not stored in the first read area, determining whether the target index corresponding to the query request is stored in the second read area;

[0121] If the target index is not stored in the first read area, a search is performed in the second read area to determine whether the target index is stored in the second read area.

[0122] Step S345 : When the target index corresponding to the query request is stored in the second read area, the target to-be-read data corresponding to the target index is obtained from the third read area.

[0123] When the target index is stored in the second read area, data corresponding to the target index, ie, target data to be read, is acquired from the third read area.

[0124] The database data management provided by the embodiments of the present invention divides the data to be read based on time information and stores it in a first read area and a second read area, respectively. When processing data, the first read area is searched first. If the data does not exist, the second read area is searched. The dual storage queue approach simplifies data query time and improves data query efficiency. It also eliminates the need to load all data, allowing for on-demand loading and memory optimization.

[0125] The query process of this method embodiment is as follows Figure 7 As shown in the figure, taking InfluxDB as the target database, the first read area is direct memory, the second read area is replacement memory, and the third read area is replacement data. TSM (Time-Structured Merge Tree) is InfluxDB's storage engine, and InfluxDB uses the TSM storage engine to store all data. At startup, the database loads the TSM file and reads it into memory for subsequent query operations. Data retrieval based on a query request begins with a query in direct memory. If the index information is stored in direct memory, the query determines whether the replacement data contains the corresponding data. If so, the query completes. If the replacement data does not contain the corresponding data, the TSM is searched and the data retrieved from the TSM is stored in the replacement data. If the index information is not found in direct memory, the replacement memory is searched. If the replacement memory contains the corresponding data, the replacement data is determined to contain the corresponding data. If not, the TSM is searched and the retrieved index information is written to direct memory, and the corresponding data is written to the replacement memory.

[0126] The following uses InfluxDB as an example target database to provide an embodiment of a data management method for the database. Specific application scenarios include writing data to the database and querying and reading data. These two scenarios involve frequent changes in index data and second-level data writing scenarios.

[0127] InfluxDB is an open-source distributed time series, time, and metrics database written in Go and requiring no external dependencies. Designed for distribution and horizontal scalability, it is a core product of InfluxData. It is used in artificial intelligence development platforms to store monitoring and reporting data. When the service starts, InfluxDB adds the following information to its memory:

[0128] (1) Meta data: When InfluxDB starts, it reads metadata stored in the metadata store, including information such as database, data preprocessor, continuous query, etc. This metadata store is usually saved on disk to ensure that InfluxDB can maintain its state after restart.

[0129] (2)Write Ahead Log (WAL): When InfluxDB starts, it reads the WAL log, which includes write operations and query operations. WAL logs are usually used to persist data and ensure data consistency.

[0130] (3) TSM files (Time-Structured Merge Tree storage engine): InfluxDB uses the TSM storage engine to store all data. It loads TSM files at startup and reads them into memory for subsequent query operations.

[0131] InfluxDB uses an LSM-tree storage engine and a memory cache mechanism to optimize write performance. However, under high write loads, large-scale data collection, and insufficient server configuration, InfluxDB may face memory overflow issues.

[0132] The application scenarios involved in this embodiment include data writing to the database and data query and reading. These two scenarios involve frequent changes in index data and data writing in seconds:

[0133] (1) Scenario with frequent changes in index data: 400 physical nodes, each node generates 256 tasks (each node has 256 logical cores, each task has 1 core), each task runs for 2 hours, each task has 3 label values ​​(task name, task category, node), each label value is 20 characters, then 400*256*24 / 2*3*20 / 1024 / 1024=70Mb of memory data is added every day (25GB per year).

[0134] (2) Second-level data writing scenario: 400 physical nodes, 200+ collection items per node, 400*200=80,000 data items written concurrently per second, and each data item is 0.4 KB on average, so 400*200*0.4 KB / 1024=31 MB is written per second.

[0135] This method provides the database with a data writing component WriteCache and a data reading component ReadCache.

[0136] The data writing component includes: receiving write requests, temporarily writing the data to be written to the disk file according to the memory limit requirements, and then completing the data storage in batches through background tasks, mainly including: writing cache queue, data playback listener, and configuration items. The specific process is as follows Figure 8 As shown:

[0137] Write data is data to be written. After receiving the data to be written, it is stored in a cache queue. Part of the data to be written is written to InfluxDB, and part is written to disk (written to a disk file). If writing to InfluxDB fails, the data is marked as failed. If writing to InfluxDB succeeds, the data in the cache queue is removed. For data written to disk, if writing to disk fails, it is marked as failed. If writing to disk succeeds, the data in the cache queue is removed. Data written to disk files can be read and written to InfluxDB concurrently. WriteCache deployment includes: defining the configuration item size based on the cluster scale, completing nested HashTable initialization based on the configuration items, and starting the data playback listener.

[0138] The data reading component includes: receiving read requests, modifying InfluxDB's own index (index information) loading strategy to optimize on-demand loading; using dual cache queues to store frequently used indexes, and using buffer cache (third read area) to store temporarily replaced and frequently accessed data, which not only optimizes memory usage but also improves data query performance.

[0139] The dual cache queues are the first read area and the second read area. ReadCache deployment includes: initializing the dual index cache queues, initializing the data buffer cache, and loading the index within the last 30 days into the index cache queue by default.

[0140] This component adds a TSM cache queue to InfluxDB, loading index data according to different strategies to replace the memory overflow problem caused by full loading of InfluxDB; it optimizes the data writing logic through sharding technology, reducing memory usage and data write loss.

[0141] This embodiment proposes a method for resolving memory overflow issues with a single InfluxDB node under massive amounts of data. This method not only resolves the InfluxDB memory overflow problem but also improves query performance through a dual index queue + data buffer queue. It is integrated into the product via a plug-in, without intruding on existing services and minimizing the impact on InfluxDB itself. This solution not only improves the stability of InfluxDB itself and reduces InfluxDB resource usage, but also reduces maintenance costs and technical risks, thereby enhancing the competitiveness of the AI ​​platform among similar products.

[0142] In this embodiment, a database data management device is also provided. The device is used to implement the above-mentioned embodiments and preferred embodiments. Details already described will not be repeated here. As used below, the term "module" may refer to a combination of software and / or hardware that implements a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, implementation using hardware, or a combination of software and hardware, is also possible and contemplated.

[0143] This embodiment provides a data management device for a database, such as Figure 9 As shown, including:

[0144] Information acquisition module 81, used to obtain time information of the data to be processed;

[0145] an area determination module 82 for determining a storage area corresponding to the data to be processed based on the time information;

[0146] The data processing module 83 is used to process the data to be processed based on the different storage areas and determine the processing results.

[0147] In some optional implementations, if the data to be processed is data to be written, the time information includes the acquisition time of the data to be written, and the storage area includes a target database and a disk file, the area determination module 82 includes:

[0148] A time calculation unit, configured to calculate the time difference between the acquisition time of the data to be written and the current time;

[0149] A time judgment unit is used to judge the size of the time difference and a preset disk placement threshold to obtain a judgment result;

[0150] The data writing unit is used to divide the data to be written into first data to be written and second data to be written based on the judgment result, the storage area corresponding to the first data to be written is the target database, and the storage area corresponding to the second data to be written is the disk file.

[0151] In some optional implementations, the data processing module 83 includes:

[0152] a first writing unit, configured to write the first data to be written into the target database;

[0153] The second writing unit is configured to write the second data to be written into the disk file.

[0154] In some optional implementations, the data processing module 83 includes:

[0155] A third writing unit is configured to obtain the second data to be written from the disk file, and store the second data to be written into the target database.

[0156] In some optional implementations, if the data to be processed is data to be read, the time information includes the acquisition time and the query time of the data to be read, and the area determination module includes:

[0157] A first storage unit is configured to store index information corresponding to the to-be-read data whose acquisition time is less than a preset acquisition time threshold in a first reading area;

[0158] The second storage unit is used to store the index information corresponding to the to-be-read data in the first reading area whose query time is lower than the preset query time threshold in the second reading area, and the to-be-read data corresponding to the index information of the to-be-read data in the first reading area and the index information of the to-be-read data in the second reading area are stored in the third reading area.

[0159] In some optional implementations, the data processing module 83 includes:

[0160] A request obtaining unit, used for obtaining a query request;

[0161] a first index retrieval unit, configured to determine, based on the query request, whether the first read area stores a target index corresponding to the query request;

[0162] The first data reading unit is configured to, when the first reading area stores a target index corresponding to the query request, obtain target data to be read corresponding to the target index from the third reading area.

[0163] In some optional implementations, the data processing module 83 includes:

[0164] a second index retrieval unit, configured to determine whether the target index corresponding to the query request is stored in the second read area when the target index corresponding to the query request is not stored in the first read area;

[0165] The second data reading unit is configured to, when the target index corresponding to the query request is stored in the second reading area, obtain target data to be read corresponding to the target index from the third reading area.

[0166] The data management device of the database in this embodiment is presented in the form of a functional unit, where the unit refers to an ASIC circuit, a processor and memory that executes one or more software or fixed programs, and / or other devices that can provide the above functions.

[0167] The further functional description of each of the above modules and units is the same as that of the above corresponding embodiments and will not be repeated here.

[0168] The embodiment of the present invention also provides a computer device having the above Figure 9 The data management device of the database shown.

[0169] See also Figure 10 , Figure 10 is a structural diagram of a computer device provided by an optional embodiment of the present invention, such as Figure 10 As shown, the computer device includes: one or more processors 10, memory 20, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. Various components utilize different buses to communicate with each other and can be installed on a common mainboard or installed in other ways as needed. The processor can process the instructions executed in the computer device, including instructions stored in the memory or on the memory to display the graphical information of the GUI on an external input / output device (such as, a display device coupled to the interface). In some optional embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Equally, multiple computer devices can be connected, and each device provides part of the necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Figure 10 A processor 10 is taken as an example.

[0170] The processor 10 may be a central processing unit, a network processor, or a combination thereof. The processor 10 may further include a hardware chip. The hardware chip may be an application-specific integrated circuit, a programmable logic device, or a combination thereof. The programmable logic device may be a complex programmable logic device, a field programmable gate array, a general purpose array logic, or any combination thereof.

[0171] The memory 20 stores instructions that can be executed by at least one processor 10, so that the at least one processor 10 executes the method shown in the above embodiment.

[0172] The memory 20 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created based on the use of a computer device for displaying a small program landing page, etc. In addition, the memory 20 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk file storage device, a flash memory device, or other non-transient solid-state storage device. In some optional embodiments, the memory 20 may optionally include a memory remotely located relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of the above-mentioned networks include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0173] The memory 20 may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a flash memory, a hard disk or a solid-state drive; the memory 20 may also include a combination of the above types of memory.

[0174] The computer device further includes a communication interface 30 for the computer device to communicate with other devices or a communication network.

[0175] The embodiment of the present invention also provides a computer-readable storage medium. The above-mentioned method according to the embodiment of the present invention can be implemented in hardware, firmware, or implemented as a computer code that can be recorded in a storage medium, or implemented as a computer code that is originally stored in a remote storage medium or a non-temporary machine-readable storage medium and downloaded through a network and will be stored in a local storage medium, so that the method described herein can be stored in such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only storage memory, a random access memory, a flash memory, a hard disk or a solid-state drive, etc.; further, the storage medium can also include a combination of the above-mentioned types of memory. It can be understood that a computer, a processor, a microprocessor controller or programmable hardware includes a storage component that can store or receive software or computer code. When the software or computer code is accessed and executed by a computer, a processor or hardware, the method shown in the above embodiment is implemented.

[0176] Although the embodiments of the present invention have been described with reference to the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present invention. Such modifications and variations are all within the scope defined by the appended claims.

Claims

1. A method for managing database data, characterized in that: The method comprises: Get the time information of the data to be processed; Determining a storage area corresponding to the data to be processed based on the time information; Processing the data to be processed based on the different storage areas to determine a processing result; the time information corresponding to the data to be processed in different scenarios is different, and the scenarios include writing and querying database data; If the data to be processed is data to be written, the time information includes the acquisition time of the data to be written, and the storage area includes a target database and a disk file, determining the storage area corresponding to the data to be processed based on the time information includes: calculating a time difference between the acquisition time of the data to be written and the current time; determining a difference between the time difference and a preset disk write threshold to obtain a determination result; and dividing the data to be written into first data to be written and second data to be written based on the determination result, the storage area corresponding to the first data to be written being the target database, and the storage area corresponding to the second data to be written being the disk file; If the data to be processed is data to be read, the time information includes an acquisition time and a query time of the data to be read, and determining a storage area corresponding to the data to be processed based on the time information includes: storing index information corresponding to the data to be read whose acquisition time is less than a preset acquisition time threshold in a first read area; storing index information corresponding to the data to be read in the first read area whose query time is less than a preset query time threshold in a second read area, and storing the data to be read corresponding to the index information of the data to be read in the first read area and the index information of the data to be read in the second read area in a third read area.

2. The method according to claim 1, characterized in that The processing of the data to be processed based on the different storage areas to determine a processing result includes: Writing the first data to be written into the target database; The second data to be written is written into the disk file.

3. The method according to claim 2, characterized in that The processing of the data to be processed based on the different storage areas and determining the processing result further includes: The second data to be written is obtained from the disk file, and the second data to be written is stored in the target database.

4. The method according to claim 1, wherein The processing of the data to be processed based on the different storage areas to determine a processing result includes: Get query request; Determining, based on the query request, whether the first read area stores a target index corresponding to the query request; When the target index corresponding to the query request is stored in the first read area, the target to-be-read data corresponding to the target index is acquired from the third read area.

5. The method according to claim 4, characterized in that The processing of the data to be processed based on the different storage areas and determining the processing result further includes: When the target index corresponding to the query request is not stored in the first read area, determining whether the target index corresponding to the query request is stored in the second read area; When the target index corresponding to the query request is stored in the second read area, the target to-be-read data corresponding to the target index is acquired from the third read area.

6. A data management device for a database, characterized in that: The device comprises: An information acquisition module is used to obtain time information of the data to be processed; an area determination module, configured to determine a storage area corresponding to the data to be processed based on the time information; A data processing module, configured to process the data to be processed based on the different storage areas and determine a processing result; The time information corresponding to the data to be processed in different scenarios is different, and the scenarios include writing and querying database data; If the data to be processed is data to be written, the time information includes the acquisition time of the data to be written, and the storage area includes a target database and a disk file, the area determination module includes: a time calculation unit for calculating the time difference between the acquisition time of the data to be written and the current time; a time judgment unit for judging the difference between the time difference and a preset disk write threshold to obtain a judgment result; and a data writing unit for dividing the data to be written into first data to be written and second data to be written based on the judgment result, the storage area corresponding to the first data to be written being the target database, and the storage area corresponding to the second data to be written being the disk file; If the data to be processed is data to be read, the time information includes the acquisition time and the query time of the data to be read, and the area determination module includes: The first storage unit is used to store the index information corresponding to the to-be-read data whose acquisition time is less than the preset acquisition time threshold in the first reading area; the second storage unit is used to store the index information corresponding to the to-be-read data whose query time in the first reading area is less than the preset query time threshold in the second reading area, and the to-be-read data corresponding to the index information of the to-be-read data in the first reading area and the index information of the to-be-read data in the second reading area are stored in the third reading area.

7. A computer device, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the database data management method according to any one of claims 1 to 5 by executing the computer instructions.

8. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the database data management method according to any one of claims 1 to 5.

Images