Data processing method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention provides a data processing method and device, electronic equipment and a storage medium, belongs to the technical field of data processing, and is suitable for financial science and technology scenes. The method comprises the steps that a data table scanning request is obtained, a service scene is determined, and the data table scanning request is used for scanning at least one data table in a target database; creating a temporary cache queue in a cache pool according to the service scene; wherein the temporary cache queue comprises a temporary hot cache region and a temporary cold cache region; loading a target data page from a target database based on the data table scanning request, and loading the target data page to a temporary cold cache region; in response to the condition that a target data page in the temporary cold cache region meets a preset access condition, transferring the target data page to the temporary hot cache region; wherein the temporary hot cache region forbids data page replacement. According to the embodiment of the invention, the data processing efficiency can be improved.

Description

Technical Field

[0001] The present application relates to the field of data processing technology and is applicable to financial technology scenarios, and in particular to a data processing method and device, an electronic device, and a storage medium. Background Art

[0002] Data processing is a data management technology that extracts and analyzes targeted information from massive amounts of data to support business decision-making. Data processing can be applied in a variety of scenarios. For example, in FinTech, it can be used to process data for auto insurance, property and casualty insurance, and banking products to meet the needs of financial business decision-making.

[0003] Currently, data processing primarily utilizes a dynamic partitioning buffer management strategy, dividing the buffer into cold and hot data partitions, with a ratio of 37:63 for example. The innodb_old_blocks_time (default 1 second) is also introduced to delay cold data promotion. However, in real-world applications, performing a full table scan requires storing the data table in the cold data partition, occupying storage space in the hot data partition and causing some hot data to be eliminated. This eliminated hot data then needs to be retrieved when needed, impacting overall data processing efficiency.

[0004] Therefore, how to improve the efficiency of data processing has become a technical problem that needs to be solved urgently. Summary of the Invention

[0005] The main purpose of the embodiments of the present application is to provide a data processing method and device, an electronic device and a storage medium, aiming to improve the efficiency of data processing.

[0006] To achieve the above objectives, a first aspect of an embodiment of the present application provides a data processing method, the method comprising:

[0007] Obtaining a data table scanning request and determining a business scenario, wherein the data table scanning request is used to scan at least one data table in a target database;

[0008] Creating a temporary cache queue in the cache pool according to the business scenario; wherein the temporary cache queue includes a temporary hot cache area and a temporary cold cache area;

[0009] Loading a target data page from the target database based on the data table scan request, and loading the target data page into the temporary cold buffer area;

[0010] In response to the target data page in the temporary cold cache area meeting a predetermined access condition, the target data page is transferred to the temporary hot cache area; wherein the temporary hot cache area prohibits data page replacement.

[0011] In some embodiments, after transferring the target data page in the temporary cold cache area to the temporary hot cache area in response to the target data page in the temporary cold cache area meeting a predetermined access condition, the method further includes:

[0012] Create a primary cache area in the cache pool according to the business scenario;

[0013] In response to the temporary hot cache having no free storage space, the target data page in the temporary hot cache is transferred to the primary cache.

[0014] In some embodiments, creating a temporary cache queue in the cache pool according to the business scenario includes:

[0015] Determine a queue priority based on the business scenario; wherein the queue priority indicates the memory size required to create the queue;

[0016] Obtaining the data processing load status of the cache pool;

[0017] Determine a temporary queue memory ratio based on the queue priority and the data processing load status;

[0018] Memory is allocated to the cache pool according to the memory proportion of the temporary queue to obtain the temporary cache queue.

[0019] In some embodiments, the temporary cache queue further includes a temporary unallocated cache area, and allocating memory to the cache pool according to the memory proportion of the temporary queue to obtain the temporary cache queue includes:

[0020] Determine the free space ratio and the hot and cold data memory ratio according to the temporary queue memory ratio;

[0021] Allocating memory to the cache pool based on the memory ratio of hot and cold data to obtain the temporary hot cache area and the temporary cold cache area;

[0022] Allocating memory to the cache pool based on the free space ratio to obtain the temporary unallocated cache area;

[0023] The temporary cache queue is generated according to the temporary hot cache area, the temporary cold cache area, and the temporary unallocated cache area.

[0024] In some embodiments, loading the target data page into the temporary cold cache area includes:

[0025] Determining the storage space occupied by the data page of the target data page;

[0026] Obtaining the available storage space and total storage space of the temporary cold cache area;

[0027] If the available storage space is smaller than the storage space occupied by the data page, and the total storage space is larger than the storage space occupied by the data page, calculating a target released storage space based on the available storage space and the storage space occupied by the data page; wherein the target released storage space is the size of the memory that needs to be released;

[0028] Memory is released for the temporary cold cache area based on the target released storage space, and the target data page is loaded into the temporary cold cache area after the memory is released.

[0029] In some embodiments, releasing memory of the temporary cold cache area based on the target released storage space includes:

[0030] Obtaining a storage timestamp, an access frequency attribute, and a data page identifier for each data page in the temporary cold cache area;

[0031] Performing a weighted sum based on the storage timestamp and the access frequency attribute to obtain a data page access index;

[0032] sorting each data page based on the data page access index to obtain an initial data page sequence;

[0033] Adjusting the initial data page sequence according to the data page identifier of each data page to obtain a target data page sequence;

[0034] Based on the target released storage space, candidate data pages are determined from the target data page sequence, and the candidate data pages are released.

[0035] In some embodiments, after obtaining the available storage space and the total storage space of the temporary cold cache area, the method further includes:

[0036] If the available storage space is smaller than the storage space occupied by the data page, and the total storage space is smaller than the storage space occupied by the data page, calculating the gap storage space based on the available storage space and the storage space occupied by the data page;

[0037] Comparing the unallocated space of the temporary unallocated buffer area with the gap storage space,

[0038] If the unallocated space is smaller than the gap storage space, calculating the requested memory ratio based on the gap storage space;

[0039] A memory request is made to the cache pool based on the requested memory ratio to obtain increased memory, the increased memory is allocated to the temporary cold cache area, and the target data page is loaded into the temporary cold cache area after the memory is increased.

[0040] To achieve the above-mentioned purpose, a second aspect of an embodiment of the present application provides a data processing device, comprising:

[0041] a scan request acquisition module, configured to acquire a data table scan request and determine a business scenario, wherein the data table scan request is used to scan at least one data table in a target database;

[0042] A temporary queue creation module, configured to create a temporary cache queue in the cache pool according to the business scenario; wherein the temporary cache queue includes a temporary hot cache area and a temporary cold cache area;

[0043] a data loading module, configured to load a target data page from the target database based on the data table scan request, and load the target data page into the temporary cold buffer area;

[0044] The data transfer module is configured to transfer the target data page in the temporary cold cache area to the temporary hot cache area in response to the target data page in the temporary cold cache area meeting a predetermined access condition; wherein the temporary hot cache area prohibits data page replacement.

[0045] To achieve the above-mentioned purpose, the third aspect of an embodiment of the present application proposes an electronic device, which includes a memory and a processor, wherein the memory stores a computer program, and the processor implements the method described in the first aspect when executing the computer program.

[0046] To achieve the above-mentioned purpose, the fourth aspect of the embodiments of the present application proposes a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the method described in the first aspect.

[0047] The data processing method and device, electronic device, and storage medium proposed in this application obtain a data table scan request and determine a business scenario. Based on the business scenario, a temporary cache queue is dynamically created in the cache pool. The temporary cache queue includes a temporary hot cache area and a temporary cold cache area, which can avoid cache contamination of the cache pool by scanning the data table, and the hot and cold cache partitioning mechanism can achieve data isolation. Furthermore, based on the data table scan request, the target data page is loaded from the target database and loaded into the temporary cold cache area, which can avoid hot data being "contaminated" by cold data and reduce the risk of cache hit rate reduction. Finally, the policy of prohibiting data page replacement in the temporary hot cache area can ensure that high-value data (such as continuously accessed data pages) resides for a long time, avoiding the I / O overhead caused by repeated loading. By accurately identifying business needs and optimizing cache allocation strategies, unnecessary I / O operations are reduced, and the hot zone protection mechanism ensures fast access to key data, which can effectively improve the efficiency of data processing and enhance the efficiency and resource utilization of data table scanning. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 is a flow chart of the data processing method provided in an embodiment of the present application;

[0049] Figure 2 yes Figure 1 Flowchart of step S102 in FIG.

[0050] Figure 3 yes Figure 2 Flowchart of step S204 in FIG.

[0051] Figure 4 is a flow chart of a data processing method provided by another embodiment of the present application;

[0052] Figure 5 yes Figure 4 Flowchart of step S404 in FIG.

[0053] Figure 6 is a flow chart of a data processing method provided by another embodiment of the present application;

[0054] Figure 7 is a structural diagram of a data processing device provided in an embodiment of the present application;

[0055] Figure 8 This is a schematic diagram of the hardware structure of the electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0056] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0057] It should be noted that although the device schematics illustrate functional module divisions and the flowcharts illustrate logical sequences, in certain circumstances, the steps shown or described may be performed in a sequence that differs from the module divisions in the device or the sequence in the flowcharts. The terms "first," "second," and so on, in the specification, claims, and drawings, are used to distinguish similar items and are not necessarily used to describe a specific sequence or precedence.

[0058] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which this application pertains. The terms used herein are for the purpose of describing the embodiments of this application only and are not intended to limit this application.

[0059] First, let’s analyze some of the terms used in this application:

[0060] Data processing is a data management technology that extracts target information through a series of operations such as collecting, cleaning, converting, storing and analyzing massive data to provide support for business decisions.

[0061] MySQL is an open-source relational database management system that uses the Structured Query Language (SQL) for data storage and management. Known for its high performance, reliability, and ease of use, MySQL is widely used in web applications, data warehouses, and embedded systems.

[0062] LRU linked list, LRU (Least Recently Used linked list) is a data structure used to manage the Buffer Pool in the MySQL InnoDB storage engine. The LRU linked list records the access time of the data pages, keeps the most recently used data pages at the head of the linked list, and moves the data pages that have not been used for a long time to the end of the linked list. When the Buffer Pool space is insufficient, InnoDB will give priority to eliminating the "cold" data pages at the end of the linked list, thereby optimizing memory usage efficiency, reducing disk I / O operations, and improving database performance. In addition, InnoDB has optimized the traditional LRU algorithm and introduced a hot and cold data separation mechanism. The innodb_old_blocks_pct parameter is used to control the proportion of cold data areas to avoid cache pollution caused by pre-reading or full table scans.

[0063] Cold data refers to data in the database that is accessed very infrequently, usually including historical archive records, logs that have not been updated for a long time, or backup data.

[0064] Hot data refers to data in the database that is frequently accessed and has strict real-time requirements, such as real-time transaction records, user activity information, or data required for recent business analysis.

[0065] A full table scan is a less efficient data retrieval method used in database queries. When executing a query, the database engine doesn't use indexes, but instead reads all the data pages in the table row by row, checking each row to see if it meets the query criteria. This method works well for small data volumes or when the query criteria can't utilize indexes. However, it can lead to performance bottlenecks in large data volumes because it consumes significant I / O resources and CPU time to process irrelevant data.

[0066] OLTP (Online Transaction Processing) is a system focused on real-time transaction processing, primarily used to support daily business operations such as banking transactions, order processing, and inventory management. OLTP's core features include support for high concurrency and short transactions, and an emphasis on data consistency and real-time performance. It typically utilizes a relational database structure to ensure transaction atomicity, consistency, isolation, and durability.

[0067] OLAP (Online Analytical Processing) is a system focused on data analysis, primarily used to support complex queries and decision-making analysis, such as sales trend analysis and financial report generation. Its core features include support for multidimensional data analysis, complex calculations, and trend analysis. Typically based on data warehouses or data marts, OLAP utilizes a star or snowflake schema, allowing users to flexibly explore data from multiple perspectives and discover potential business insights.

[0068] Data processing can be applied to a variety of application scenarios. For example, in financial technology scenarios, in banks, business queries and decision analysis can be performed on financial products and bank transaction data; in insurance companies, sales of auto insurance products, life insurance products and customer groups can be analyzed and reports generated.

[0069] Currently, in MySQL, data processing mainly adopts a dynamic partitioning buffer management strategy, namely the LRU linked list, by dividing the buffer into cold data partitions and hot data partitions, for example, the cold and hot ratio is 37:63, and introducing innodb_old_blocks_time (default 1 second) to delay the promotion of cold data.

[0070] However, in actual application scenarios, when performing full table scan operations, large data table queries, and other scenarios, the data table needs to be stored in the cold data partition, occupying the storage space of the hot data partition, causing some hot data to be eliminated, reducing the cache hit rate, and when these eliminated hot data need to be used, they need to be retrieved again, affecting the overall efficiency of data processing.

[0071] Furthermore, because data tables are stored in cold data partitions, they may not be used in subsequent queries, occupying cache space and causing cache pollution. If these pages are accessed multiple times within a short period of time, cold data may be mistakenly identified as hot data, ultimately eliminating the truly hot data and affecting overall data processing efficiency.

[0072] Based on this, the embodiments of the present application provide a data processing method and device, an electronic device and a storage medium, aiming to improve the efficiency of data processing.

[0073] The data processing method and device, electronic device, and storage medium provided in the embodiments of the present application are specifically illustrated through the following embodiments. First, the data processing method in the embodiments of the present application is described.

[0074] The data processing method provided in the embodiment of the present application relates to the field of data processing technology. The data processing method provided in the embodiment of the present application can be applied to a terminal, can be applied to a server side, or can be software running in a terminal or a server side. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, etc.; the server side can be configured as an independent physical server, or can be configured as a server cluster or a distributed system composed of multiple physical servers, or can be configured as a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application that implements the data processing method, etc., but is not limited to the above forms.

[0075] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, and the like. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, and the like that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments in which tasks are performed by remote processing devices connected via a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media, including storage devices.

[0076] It should be noted that in each specific embodiment of the present application, when it comes to the need to perform relevant processing based on data related to the user's identity or characteristics, such as user information, user behavior data, user historical data, and user location information, the user's permission or consent will be obtained first, and the collection, use, and processing of such data will comply with relevant laws, regulations, and standards. In addition, when the embodiment of the present application needs to obtain the user's sensitive personal information, the user's separate permission or consent will be obtained through a pop-up window or by jumping to a confirmation page. After clearly obtaining the user's separate permission or consent, the necessary user-related data for the normal operation of the embodiment of the present application will be obtained.

[0077] Figure 1 This is an optional flowchart of the data processing method provided in the embodiment of the present application. Figure 1 The method may include but is not limited to steps S101 to S104.

[0078] Step S101: obtaining a data table scanning request and determining a business scenario, wherein the data table scanning request is used to scan at least one data table in a target database;

[0079] Step S102: creating a temporary cache queue in the cache pool according to the business scenario; wherein the temporary cache queue includes a temporary hot cache area and a temporary cold cache area;

[0080] Step S103: loading a target data page from a target database based on a data table scan request, and loading the target data page into a temporary cold buffer area;

[0081] Step S104 , in response to the target data page in the temporary cold cache area meeting a predetermined access condition, the target data page is transferred to the temporary hot cache area; wherein the temporary hot cache area prohibits data page replacement.

[0082] In the steps S101 to S104 shown in the embodiment of the present application, by obtaining a data table scan request and determining a business scenario, a temporary cache queue is dynamically created in the cache pool based on the business scenario. The temporary cache queue includes a temporary hot cache area and a temporary cold cache area, which can avoid the cache pollution of the cache pool caused by scanning the data table, and the hot and cold cache partitioning mechanism can achieve data isolation. Furthermore, based on the data table scan request, the target data page is loaded from the target database and loaded into the temporary cold cache area, which can avoid the hot data being "polluted" by the cold data and reduce the risk of a decrease in the cache hit rate. Finally, the policy of prohibiting data page replacement in the temporary hot cache area can ensure that high-value data (such as continuously accessed data pages) resides for a long time, avoiding the I / O overhead caused by repeated loading. By accurately identifying business needs and optimizing cache allocation strategies, unnecessary I / O operations are reduced, and the hot zone protection mechanism ensures fast access to key data, which can effectively improve the efficiency of data processing and improve the efficiency and resource utilization of data table scanning.

[0083] The data table scan request is used to instruct to scan at least one data table in the target database; wherein the target database may include one database or multiple databases, and the number of data tables is at least one.

[0084] In some embodiments, step S101 may include but is not limited to the following steps:

[0085] receiving data processing requests;

[0086] Perform type analysis on the data processing request to obtain the data processing type;

[0087] Requests whose data processing type is a data scan type are filtered out from the data processing requests to obtain data table scan requests.

[0088] It is understandable that in a database, data processing requests may include numerous types of data processing, including but not limited to full table scans, large table scans, single table queries, data calculations, and data reading and writing. Therefore, it is necessary to filter out requests of the data processing type of data table scans, such as full table scans and large table scans, from the numerous data processing requests to obtain data table scan requests.

[0089] It should be noted that requests for full table scans and large table scans need to be processed through a temporary cache queue; other types of requests are processed using the existing mechanisms of the cache pool.

[0090] Among them, large table scan refers to: determining the total size of the data table to be processed based on the data processing request. If the total size of the data table exceeds the preset memory threshold, it will be identified as a large table scan type and needs to be processed through a temporary cache queue.

[0091] Business scenarios are used to indicate data processing scenarios, including OLTP and OLAP. In FinTech scenarios, data processing such as bank transaction data and insurance product orders falls under OLTP, while data processing such as generating quarterly reports and conducting statistical analysis on insurance products falls under OLAP.

[0092] Understandably, in OLTP scenarios, data is frequently updated, and full or large table scans are less common. In OLAP scenarios, however, extensive data analysis is required, leading to a high incidence of full or large table scans. Therefore, the memory size of the temporary cache queue needs to be dynamically adjusted based on the business scenario.

[0093] In step S102 of some embodiments, the buffer pool is a buffer pool of a database, such as a buffer pool of a MySQL database, which can provide memory space for data processing.

[0094] It should be noted that the existing mechanism of the cache pool has a main cache area, namely the LRU linked list, which is used to handle regular data processing requests of the database, focusing on caching data pages corresponding to regular data processing requests, such as single-table query, data storage and other requests.

[0095] However, due to insufficient optimization of hot and cold partitions in the current LRU linked list, data processing efficiency is poor in full table scan scenarios. Therefore, it is necessary to add a newly managed, independently managed temporary cache queue to the cache pool to cache temporary data pages generated by operations such as full table scans and large table scans, as well as data that is expected to remain in memory. This can isolate the main cache area from temporary data, helping to improve data processing efficiency in scenarios such as full table scans.

[0096] See also Figure 2 In some embodiments, step S102 may include but is not limited to steps S201 to S204:

[0097] Step S201, determining a queue priority based on a business scenario; wherein the queue priority indicates the memory size required to create a queue;

[0098] Step S202, obtaining the data processing load status of the buffer pool;

[0099] Step S203, determining the temporary queue memory ratio based on the queue priority and data processing load status;

[0100] Step S204: Allocate memory to the buffer pool according to the temporary queue memory ratio to obtain a temporary buffer queue.

[0101] In the steps S201 to S204 shown in the embodiment of the present application, the queue priority is determined according to the business scenario. The queue priority is used to indicate the memory size required for the associated creation queue, so that the memory allocation can fit the needs of different business scenarios, avoid waste or insufficiency of resources, and ensure the efficiency of business operation. Furthermore, by obtaining the data processing load status of the cache pool, the operating status of the database cache pool can be grasped in real time, providing a reference for memory allocation, and avoiding the situation where more memory is allocated when the memory occupancy rate is high. Furthermore, the temporary queue memory ratio is determined based on the queue priority and the data processing load status. It can take into account both the memory requirements of the business scenario and the actual operating status of the database cache pool, making the memory allocation decision more scientific and reasonable and achieving balance. Finally, the cache pool is allocated memory according to the temporary queue memory ratio to obtain a temporary cache queue, which ensures that the temporary cache queue can operate efficiently, effectively improves the system's adaptability to different business scenarios and data processing capabilities, and improves the overall performance and stability of the system.

[0102] In step S201 of some embodiments, since OLTP scenarios involve frequent data updates and full table scan scenarios are less common, the priority of the temporary cache queue can be lowered and less memory can be allocated. OLAP scenarios involve a large amount of data analysis and a large number of full table scan scenarios, so the priority of the temporary cache queue needs to be increased and more memory needs to be allocated.

[0103] For example, the queue priority corresponding to the OLTP scenario is the first priority, and the memory size required to create the queue is the first memory ratio; the queue priority corresponding to the OLAP scenario is the second priority, and the memory size required to create the queue is the second memory ratio, where the second memory ratio is larger than the first memory ratio.

[0104] In some embodiments, the specific sizes of the first memory ratio and the second memory ratio can be set according to different business requirements, for example, the first memory ratio is 10% and the second memory ratio is 20%, but it is not limited thereto.

[0105] Furthermore, if the cache pool's memory usage is already high, creating a temporary cache queue might directly take up the main cache's already used memory space, causing hot data in the main cache to be eliminated, further impacting data processing efficiency. Therefore, it's necessary to monitor the cache pool's data processing load and dynamically adjust the size of the temporary cache queue based on the load to improve data processing flexibility.

[0106] In step S203 of some embodiments, the memory ratio corresponding to the queue priority is obtained, the available memory ratio of the buffer pool is determined based on the data processing load status, and the memory ratio corresponding to the queue priority is compared with the available memory ratio. If the memory ratio corresponding to the queue priority is less than the available memory ratio, the memory ratio corresponding to the queue priority is used as the temporary queue memory ratio; if the available memory ratio is less than the memory ratio corresponding to the queue priority, the available memory ratio is used as the temporary queue memory ratio, thereby realizing a flexible and dynamic memory allocation mechanism.

[0107] In other embodiments, the memory ratio of the main cache area can be dynamically adjusted according to the memory ratio of the temporary cache queue, thereby ensuring the data processing efficiency of the temporary cache queue without frequently eliminating data in the queue.

[0108] In some embodiments, the temporary cache queue includes a temporary hot cache area, a temporary cold cache area and a temporary unallocated cache area; wherein, the temporary cold cache area is used to store data tables for full table scans or large table scans; the temporary hot cache area is used to store data tables resident in memory, or data pages in the temporary cold cache area that meet predetermined access conditions will be transferred to the temporary hot cache area; the temporary unallocated cache area is idle content and can be flexibly allocated to the temporary hot cache area and the temporary cold cache area.

[0109] See also Figure 3 In some embodiments, step S204 may include but is not limited to steps S301 to S304:

[0110] Step S301, determining the free space ratio and the hot and cold data memory ratio according to the temporary queue memory ratio;

[0111] Step S302: Allocate memory to the cache pool based on the memory ratio of hot and cold data to obtain a temporary hot cache area and a temporary cold cache area;

[0112] Step S303: Allocate memory to the cache pool based on the free space ratio to obtain a temporary unallocated cache area;

[0113] Step S304 : generating a temporary buffer queue according to the temporary hot buffer area, the temporary cold buffer area, and the temporary unallocated buffer area.

[0114] In steps S301 to S304, as shown in the embodiment of the present application, by determining the free space ratio and the hot and cold data memory ratio based on the temporary queue memory ratio, memory resources can be reasonably allocated according to actual usage. Furthermore, based on the hot and cold data memory ratio and the free space ratio, a temporary hot cache area, a temporary cold cache area, and a temporary unallocated cache area are divided from the cache pool, and a temporary cache queue is generated by combining the temporary hot cache area, the temporary cold cache area, and the temporary unallocated cache area, which is conducive to more efficient use of memory resources when processing data.

[0115] In step S301 of some embodiments, the temporary queue memory ratio may be divided according to a preset partition ratio to obtain the free space ratio, the cold data memory ratio, and the hot data memory ratio;

[0116] For example, the temporary queue memory accounts for 20% of the cache pool, and the preset partition ratio stipulates that the free space: cold data memory: hot data memory is 2:5:3;

[0117] The free space ratio is 20% * 20% = 4%, which is 4% of the cache pool.

[0118] The cold data memory ratio is 20% * 50% = 10%, which is 10% of the cache pool.

[0119] The hot data memory ratio is 20% * 30% = 6%, which is 6% of the cache pool.

[0120] In some embodiments, after determining the free space ratio, cold data memory ratio and hot data memory ratio, the cache pool is memory allocated based on the free space ratio, cold data memory ratio and hot data memory ratio to obtain three partitions: a temporary hot cache area, a temporary cold cache area and a temporary unallocated cache area. These three partitions constitute a temporary cache queue.

[0121] It should be noted that the preset partition ratio needs to be set in combination with the actual application scenario, and the partition ratio can also be dynamically adjusted in a manner similar to an LRU linked list, but is not limited thereto.

[0122] In step S103 of some embodiments, according to at least one data table in the target database specified in the data table scan request, a corresponding data table is searched from the target database and is confirmed as a target data page.

[0123] It should be noted that the cache pool stores data in pages.

[0124] See also Figure 4 In some embodiments, the step of "loading the target data page into the temporary cold cache area" in step S103 may include but is not limited to steps S401 to S404:

[0125] Step S401, determining the storage space occupied by the data page of the target data page;

[0126] Step S402, obtaining the available storage space and total storage space of the temporary cold cache area;

[0127] Step S403: If the available storage space is smaller than the storage space occupied by the data page, and the total storage space is larger than the storage space occupied by the data page, a target released storage space is calculated based on the available storage space and the storage space occupied by the data page; wherein the target released storage space is the size of the memory to be released;

[0128] Step S404 : releasing memory of the temporary cold buffer area based on the target released storage space, and loading the target data page into the temporary cold buffer area after the memory is released.

[0129] In steps S401 to S404 shown in the embodiment of the present application, the storage space occupied by the data page of the target data page is determined, thereby clarifying the size of the data page to be processed. By obtaining the available storage space and the total storage space of the temporary cold cache area, the resource status of the temporary cold cache area can be fully understood. When the available storage space is less than the storage space occupied by the data page, but the total storage space is greater than the storage space occupied by the data page, the size of the memory required to be released in the temporary cold cache area is calculated based on the available storage space and the storage space occupied by the data page, and the target released storage space is obtained to avoid blind release leading to waste or insufficient resources. Finally, the temporary cold cache area is released in memory based on the target released storage space, which can effectively release enough space to accommodate the target data page, and then the target data page is loaded into the temporary cold cache area after the memory is released, ensuring that the target data page can smoothly enter the cache area, ensuring the integrity of the data table storage, and no need to store it in multiple times, which helps to improve the efficiency of data processing.

[0130] It is understandable that the target data pages of full table scans or large table scans may be accessed less frequently and may be used once and then no longer. Therefore, storing the target data pages in a temporary cold cache area can avoid storing a large amount of data in a temporary hot cache area or a main cache area, which would cause the data in the temporary hot cache area or the main cache area to be eliminated, thereby helping to improve the cache hit rate and thus improve data processing efficiency.

[0131] In step S404 of some embodiments, a first-in-first-out (FIFO) method may be used to release memory for data pages in the temporary cold buffer area, as long as the released storage space is greater than or equal to the target released storage space.

[0132] In addition, you can also use a method based on access frequency to release memory for data pages with low access frequency.

[0133] See also Figure 5 In some embodiments, step S404 may include but is not limited to steps S501 to S505:

[0134] Step S501, obtaining the storage timestamp, access frequency attribute and data page identifier of each data page in the temporary cold cache;

[0135] Step S502: performing a weighted sum based on the storage timestamp and the access frequency attribute to obtain a data page access index;

[0136] Step S503, sorting each data page based on the data page access index to obtain an initial data page sequence;

[0137] Step S504, adjusting the initial data page sequence according to the data page identifier of each data page to obtain a target data page sequence;

[0138] Step S505 : determining candidate data pages from the target data page sequence based on the target released storage space, and releasing the candidate data pages.

[0139] In the embodiment of the present application, steps S501 to S505 are obtained by obtaining the storage timestamp, access frequency attribute and data page identifier of each data page in the temporary cold cache area; performing weighted sum based on the storage timestamp and access frequency attribute to obtain the data page access index, and sorting each data page based on the target data page access index to obtain the initial data page sequence, which can comprehensively consider the storage time and usage popularity of the data page, and can more accurately measure the value of the data page, thereby more accurately reflecting the usage of the data page. Furthermore, the initial data page sequence is adjusted according to the data page identifier of each data page to obtain the target data page sequence, and the order of those data pages with low usage frequency and early storage time but marked by the data page identifier is adjusted to avoid being eliminated when the memory is released. Finally, based on the target storage space release, alternative data pages are determined from the target data page sequence and released. This hybrid memory release strategy not only avoids the risk of data loss caused by releasing based on a single dimension of time or frequency, but also efficiently releases memory space, ensuring that while meeting memory release requirements, valuable data pages are retained as much as possible, ensuring data access efficiency and data processing efficiency.

[0140] In some embodiments, the storage timestamp of a data page is used to characterize the time when the data page is stored in the temporary cold cache area, which can reflect the newness of the data page. If the data page is stored in the temporary cold cache area earlier, the data page may not be used in subsequent data processing. If the data page is stored in the temporary cold cache area later, the data page may still be needed for current data processing.

[0141] The access frequency attribute is used to characterize the number of times the data page is accessed, which can reflect the activity level of the data page. A higher access frequency may indicate that the data page is a public data page, while a lower access frequency may indicate that the data page is only used in a certain data processing task.

[0142] The data page identifier is used to indicate whether the data page is required for a specific data processing task. If it is, it cannot be eliminated. It can be represented by 0 or 1. A data page identifier of 0 indicates that the data page is not required for the specific data processing task, and a data page identifier of 1 indicates that the data page is required for the specific data processing task.

[0143] Understandably, if memory release relies solely on storage timestamps, frequently accessed data pages may be accidentally deleted. If memory release relies solely on access frequency, some data with low timeliness but frequently accessed during use may be retained. By assigning different weights to different factors using a weighted sum approach, we can balance the impact of time and frequency, achieving both precision and flexibility in caching strategies, effectively improving cache hit rates and storage space utilization, and ultimately, enhancing data processing efficiency.

[0144] In step S502 of some embodiments, the weight of the storage timestamp may be set to W1, and the weight of the access frequency attribute may be set to W2. Then, the calculation process of the data page access index is:

[0145] Data page access index = storage timestamp * W1 + access frequency attribute * W2.

[0146] In step S503 of some embodiments, after calculating the data page access index corresponding to each data page, the data pages are sorted based on the data page access index to obtain an initial data page sequence for the temporary cold buffer. The sorting can be performed in ascending order or descending order, depending on whether the data pages that require memory release are sorted in ascending order or descending order.

[0147] In step S504 of some embodiments, if the order of sorting is from small to large, a positive order screening method is used to eliminate the data pages in the previous order, and then the data pages that need to be marked as 1 are placed at the end of the initial data page sequence. If the order of sorting is from large to small, a reverse order screening method is used to eliminate the data pages in the previous order, and then the data pages that need to be marked as 1 are placed at the beginning of the initial data page sequence.

[0148] It is understandable that the data page with data page identifier 1 is the data required for a specific data processing task and needs to be retained to avoid the need to re-acquire it from the target database when executing the specific data processing task, which affects the efficiency of data processing.

[0149] In step S505 of some embodiments, data pages to be eliminated are determined from the target data page sequence based on the target released storage space, candidate data pages are obtained, and the candidate data pages are eliminated to achieve memory release.

[0150] In some embodiments, after step S402 , if the available storage space is larger than the storage space occupied by the data page, the target data page is loaded into a temporary cold cache area.

[0151] See also Figure 6 In some embodiments, after step S402, the data processing method may further include but is not limited to steps S601 to S604:

[0152] Step S601, if the available storage space is smaller than the storage space occupied by the data page, and the total storage space is smaller than the storage space occupied by the data page, calculate the gap storage space based on the available storage space and the storage space occupied by the data page;

[0153] Step S602: compare the unallocated space in the temporary unallocated buffer area with the gap storage space.

[0154] Step S603: If the unallocated space is smaller than the gap storage space, the requested memory ratio is calculated based on the gap storage space;

[0155] Step S604: Request memory from the cache pool based on the requested memory ratio to obtain additional memory, allocate the additional memory to the temporary cold cache area, and load the target data page into the temporary cold cache area after the memory is increased.

[0156] In steps S601 to S604 shown in the embodiment of the present application, when both the available storage space and the total storage space are less than the space required for the data page, it means that the temporary cold cache area cannot store the target data page, and it is necessary to calculate the gap storage space based on the available storage space and the storage space occupied by the data page, so as to accurately obtain the required increased memory space. Then, the unallocated space of the temporary unallocated cache area is compared with the gap storage space, and it can be intuitively judged whether the unallocated space of the temporary unallocated cache area in the temporary cache queue can meet the demand. If the unallocated space is insufficient, the requested memory ratio is calculated based on the gap storage space, and memory is requested from the cache pool, and the increased memory is allocated to the temporary cold cache area. The size of the memory space can be dynamically adjusted according to the data processing requirements, and the storage space of the temporary cold cache area is replenished in time to ensure that the temporary cache queue can normally receive and store data pages. Finally, the target data page is loaded into the temporary cold cache area after the memory is increased, which effectively prevents data processing interruptions or errors caused by insufficient storage space, and improves the accuracy and reliability of data processing.

[0157] It is understandable that if both the available storage space and the total storage space are smaller than the space required by the data page, it is possible to obtain the required memory from a temporary unallocated buffer area or buffer pool and allocate it to a temporary cold buffer area to improve the flexibility of data processing.

[0158] Among them, it can be preferred to obtain the required memory from the temporary unallocated buffer area. If the unallocated space in the temporary unallocated buffer area still cannot meet the gap, the required memory is obtained from the buffer pool.

[0159] After step S602 in some embodiments, the data processing method further includes:

[0160] If the unallocated space is greater than or equal to the gap storage space, a memory request is made to the temporary unallocated cache area based on the gap storage space to obtain increased memory, and the increased memory is allocated to the temporary cold cache area, and the target data page is loaded into the temporary cold cache area after the memory is increased.

[0161] In step S603 of some embodiments, the requested memory ratio is calculated based on the gap storage space. For example, assuming that the memory of the cache pool is 1024MB and the gap storage space is 32MB, the requested memory ratio is calculated to be 3.125%, which means that 3.125% of memory needs to be requested from the cache pool.

[0162] It should be noted that if the target data page is subsequently eliminated to release memory in the temporary cold cache area, the increased memory needs to be returned to the temporary unallocated cache area or cache pool to ensure memory resource allocation and efficient utilization of subsequent data processing tasks.

[0163] In some embodiments, the data processing method further includes:

[0164] Processing the target data page in the temporary cold buffer area according to the data table request to obtain a data processing result;

[0165] Store the data processing results in the target database.

[0166] This helps to view the data processing results from the target database at any time according to business needs, thereby improving the reusability of data processing.

[0167] In addition, after completing data processing on the target data page, the range frequency of the target data page needs to be updated, which can dynamically reflect the data popularity and facilitate the rapid determination of data to be eliminated when memory release is required, thereby improving memory resource utilization and business processing efficiency.

[0168] In step S104 of some embodiments, the predetermined access condition is an access frequency threshold; if the access frequency of the target data page in the temporary cold cache area is greater than or equal to the access frequency threshold, it means that the target data page can be identified as hot data, and the target data page can be transferred to the temporary hot cache area.

[0169] It's important to note that data page replacement is prohibited in the temporary hot cache, meaning that memory cannot be freed up using methods like first-in-first-out (FIFO) or access frequency-based methods. When the temporary hot cache runs out of free space, the entire cache must be eliminated or moved before it can be freed to receive new data pages.

[0170] Therefore, in some embodiments, after step S104, the data processing method may further include but is not limited to:

[0171] In response to the temporary hot cache having no free storage space, the target data page in the temporary hot cache is transferred to the main cache.

[0172] By transferring the target data page in the temporary hot cache area to the main cache area, the temporary hot cache space is released, ensuring the normal operation of the temporary cache queue, which is conducive to improving the efficiency of data processing.

[0173] In some embodiments, the data processing method provided by the embodiments of the present application is applied in financial technology scenarios and has been verified by banking services: after setting up a temporary cache queue in the cache pool to isolate full table / large table scan data, the hot data space in the main cache area is more stable and the hit rate is improved by 10%-30% (depending on the business load). In addition, because the data tables corresponding to the full table / large table scan are stored in the temporary cache queue, it is possible to avoid repeatedly obtaining and loading the same temporary data tables from the target database.

[0174] In addition, the temporary cache queue is allocated from the cache pool, and there is no need to modify the LRU mechanism of the MySQL kernel. It only requires expanding the queue management and modifying the routing logic of the data table, which is very compatible with the existing database system architecture.

[0175] See also Figure 7 , an embodiment of the present application further provides a data processing device that can implement the above-mentioned data processing method, the device comprising:

[0176] A scan request acquisition module 701 is configured to acquire a data table scan request and determine a business scenario, wherein the data table scan request is used to scan at least one data table in a target database;

[0177] A temporary queue creation module 702 is used to create a temporary cache queue in the cache pool according to the business scenario; wherein the temporary cache queue includes a temporary hot cache area and a temporary cold cache area;

[0178] A data loading module 703 is configured to load a target data page from a target database based on a data table scan request, and load the target data page into a temporary cold buffer area;

[0179] The data transfer module 704 is configured to transfer the target data page in the temporary cold cache area to the temporary hot cache area in response to the target data page in the temporary cold cache area meeting a predetermined access condition; wherein the temporary hot cache area prohibits data page replacement.

[0180] The specific implementation of the data processing device is basically the same as the specific embodiment of the above-mentioned data processing method, and will not be repeated here.

[0181] The present application also provides an electronic device comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the above-mentioned data processing method when executing the computer program. The electronic device can be any smart terminal including a tablet computer, an in-vehicle computer, or the like.

[0182] See also Figure 8 , Figure 8 The hardware structure of an electronic device according to another embodiment is shown. The electronic device includes:

[0183] The processor 801 can be implemented as a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;

[0184] The memory 802 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 802 can store an operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 802 and is called by the processor 801 to execute the data processing method of the embodiments of this application.

[0185] Input / output interface 803, used to implement information input and output;

[0186] Communication interface 804, used to implement communication interaction between this device and other devices, which can be achieved through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WiFi, Bluetooth, etc.);

[0187] Bus 805 , which transmits information between various components of the device (e.g., processor 801 , memory 802 , input / output interface 803 , and communication interface 804 );

[0188] The processor 801 , the memory 802 , the input / output interface 803 and the communication interface 804 are connected to each other in communication within the device via a bus 805 .

[0189] An embodiment of the present application further provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the above-mentioned data processing method is implemented.

[0190] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely arranged relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0191] The data processing method and device, electronic device and storage medium provided by the embodiments of the present application obtain a data table scan request and determine a business scenario, and dynamically create a temporary cache queue in the cache pool based on the business scenario. The temporary cache queue includes a temporary hot cache area and a temporary cold cache area, which can avoid the cache pollution of the cache pool caused by scanning the data table, and the hot and cold cache partitioning mechanism can achieve data isolation. Furthermore, based on the data table scan request, the target data page is loaded from the target database and loaded into the temporary cold cache area, which can avoid the hot data being "contaminated" by the cold data and reduce the risk of a decrease in the cache hit rate. Finally, the policy of prohibiting data page replacement in the temporary hot cache area can ensure that high-value data (such as continuously accessed data pages) resides for a long time, avoiding the I / O overhead caused by repeated loading. By accurately identifying business needs and optimizing cache allocation strategies, unnecessary I / O operations are reduced, and the hot zone protection mechanism ensures fast access to key data, which can effectively improve the efficiency of data processing and improve the efficiency and resource utilization of data table scanning.

[0192] The embodiments described in the embodiments of this application are intended to more clearly illustrate the technical solutions of the embodiments of this application and do not constitute a limitation on the technical solutions provided by the embodiments of this application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of this application are also applicable to similar technical problems.

[0193] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0194] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, i.e., they may be located in one place or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of this embodiment.

[0195] Those skilled in the art will appreciate that all or some of the steps in the methods, systems, and functional modules / units in the devices disclosed above may be implemented as software, firmware, hardware, or appropriate combinations thereof.

[0196] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0197] It should be understood that in this application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following items" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0198] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the above-mentioned units is only a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0199] The units described above as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0200] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0201] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes multiple instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: various media that can store programs, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0202] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but are not intended to limit the scope of the present invention. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and essence of the present invention should be within the scope of the present invention.

Claims

1. A data processing method, characterized in that: The method comprises: Obtaining a data table scanning request and determining a business scenario, wherein the data table scanning request is used to scan at least one data table in a target database; Creating a temporary cache queue in the cache pool according to the business scenario; wherein the temporary cache queue includes a temporary hot cache area and a temporary cold cache area; Loading a target data page from the target database based on the data table scan request, and loading the target data page into the temporary cold buffer area; In response to the target data page in the temporary cold cache area meeting a predetermined access condition, the target data page is transferred to the temporary hot cache area; wherein the temporary hot cache area prohibits data page replacement.

2. The method according to claim 1, characterized in that After transferring the target data page in the temporary cold cache area to the temporary hot cache area in response to the target data page in the temporary cold cache area meeting a predetermined access condition, the method further includes: Create a primary cache area in the cache pool according to the business scenario; In response to the temporary hot cache having no free storage space, the target data page in the temporary hot cache is transferred to the primary cache.

3. The method according to claim 1, characterized in that The step of creating a temporary cache queue in the cache pool according to the business scenario includes: Determine a queue priority based on the business scenario; wherein the queue priority indicates the memory size required to create the queue; Obtaining the data processing load status of the cache pool; Determine a temporary queue memory ratio based on the queue priority and the data processing load status; Memory is allocated to the cache pool according to the memory proportion of the temporary queue to obtain the temporary cache queue.

4. The method according to claim 3, characterized in that The temporary cache queue also includes a temporary unallocated cache area. The memory allocation of the cache pool according to the memory proportion of the temporary queue to obtain the temporary cache queue includes: Determine the free space ratio and the hot and cold data memory ratio according to the temporary queue memory ratio; Allocating memory to the cache pool based on the memory ratio of hot and cold data to obtain the temporary hot cache area and the temporary cold cache area; Allocating memory to the cache pool based on the free space ratio to obtain the temporary unallocated cache area; The temporary cache queue is generated according to the temporary hot cache area, the temporary cold cache area, and the temporary unallocated cache area.

5. The method according to claim 4, characterized in that The step of loading the target data page into the temporary cold cache area includes: Determining the storage space occupied by the data page of the target data page; Obtaining the available storage space and total storage space of the temporary cold cache area; If the available storage space is smaller than the storage space occupied by the data page, and the total storage space is larger than the storage space occupied by the data page, calculating a target released storage space based on the available storage space and the storage space occupied by the data page; wherein the target released storage space is the size of the memory that needs to be released; Memory is released for the temporary cold cache area based on the target released storage space, and the target data page is loaded into the temporary cold cache area after the memory is released.

6. The method according to claim 5, characterized in that The releasing memory of the temporary cold cache area based on the target released storage space includes: Obtaining a storage timestamp, an access frequency attribute, and a data page identifier for each data page in the temporary cold cache area; Performing a weighted sum based on the storage timestamp and the access frequency attribute to obtain a data page access index; sorting each data page based on the data page access index to obtain an initial data page sequence; Adjusting the initial data page sequence according to the data page identifier of each data page to obtain a target data page sequence; Based on the target released storage space, candidate data pages are determined from the target data page sequence, and the candidate data pages are released.

7. The method according to claim 5, characterized in that After obtaining the available storage space and the total storage space of the temporary cold cache area, the method further includes: If the available storage space is smaller than the storage space occupied by the data page, and the total storage space is smaller than the storage space occupied by the data page, calculating the gap storage space based on the available storage space and the storage space occupied by the data page; Comparing the unallocated space of the temporary unallocated buffer area with the gap storage space, If the unallocated space is smaller than the gap storage space, calculating the requested memory ratio based on the gap storage space; A memory request is made to the cache pool based on the requested memory ratio to obtain increased memory, the increased memory is allocated to the temporary cold cache area, and the target data page is loaded into the temporary cold cache area after the memory is increased.

8. A data processing device, characterized in that: The device comprises: a scan request acquisition module, configured to acquire a data table scan request and determine a business scenario, wherein the data table scan request is used to scan at least one data table in a target database; A temporary queue creation module, configured to create a temporary cache queue in the cache pool according to the business scenario; wherein the temporary cache queue includes a temporary hot cache area and a temporary cold cache area; a data loading module, configured to load a target data page from the target database based on the data table scan request, and load the target data page into the temporary cold buffer area; The data transfer module is configured to transfer the target data page in the temporary cold cache area to the temporary hot cache area in response to the target data page in the temporary cold cache area meeting a predetermined access condition; wherein the temporary hot cache area prohibits data page replacement.

9. An electronic device, characterized in that: The electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images