Data processing method and device, electronic equipment and storage medium

Abstract

Embodiments of the present application provide a data processing method and device, electronic equipment and storage medium, and relate to the technical field of blockchain. The method comprises: obtaining target data, the target data being data obtained from a storage medium other than a cache in response to a corresponding read request in a message queue, the cache comprising a first cache queue and at least one second cache queue; if it is determined that the first cache queue meets data cleaning conditions, determining data to be cleaned in the first cache queue and the number of reference read requests corresponding to the data to be cleaned; determining the sequence number of the second cache queue in which the data to be cleaned is stored according to the number, deleting the data to be cleaned from the first cache queue and storing it in the second cache queue with the corresponding sequence number, and storing the target data in the first cache queue. Embodiments of the present application can achieve load balancing between channels and ultimately reduce the average response time of read requests.

Description

Technical Field

[0001] This application relates to the field of blockchain technology, and more specifically, to a data processing method, apparatus, electronic device, and storage medium. Background Technology

[0002] When a user sends a query request to the database for the first time, the database first looks for the data in the cache. If the data to be accessed happens to be in the cache (usually called a Cache Hit), then the data is read directly from the cache. Otherwise, if the data to be queried by the user is not in the cache, this situation is called a Cache Miss. In this case, the database will first read the data to be queried by the user from the disk and put it into the cache, and then the user will read the data from the cache.

[0003] Flash-based SSDs are a common type of hard disk. These disks contain multiple independent channels, and the load on these channels can be uneven over time. Channels containing more "hot" data (i.e., data accessed frequently) will have a higher load, while other channels containing only "cold" data (data accessed infrequently) will be difficult to access. In this situation, read requests sent to the more heavily loaded channels will experience long latency.

[0004] Traditional cache cleanup strategies assume that the latency of missing data (i.e., the latency caused by retrieving data from the SSD when the data is not stored in the cache) is consistent across all data, and use the hit rate (number of hits / (number of hits + number of misses), where a hit is defined as data accessed that is in the cache) as the main performance metric. This approach often results in uneven load distribution across channels and a high average response time for data read requests. Summary of the Invention

[0005] The present invention provides a data processing method, apparatus, electronic device and storage medium that overcomes or at least partially solves the above problems.

[0006] Firstly, a data processing method is provided, the method comprising:

[0007] Retrieve target data, which is data obtained from storage media other than the cache in response to a read request in the message queue. The cache includes a first-level cache queue and at least one second-level cache queue.

[0008] If it is determined that the first-level cache queue meets the data cleanup conditions, then determine the data to be cleaned up in the first-level cache queue and the number of reference read requests corresponding to the data to be cleaned up;

[0009] Determine the sequence number of the secondary cache queue to store the data to be cleaned based on the quantity, delete the data to be cleaned from the primary cache queue and store it in the secondary cache queue with the corresponding sequence number, and store the target data in the primary cache queue.

[0010] The sequence number indicates the priority at which data in the corresponding secondary cache queue is cleared from the cache; the reference read request is a read request that is stored in the message queue before the read request corresponding to the data to be cleared, and is read after the read request corresponding to the data to be cleared.

[0011] Secondly, a data processing apparatus is provided, comprising:

[0012] The target data acquisition module is used to acquire target data, which is data obtained from storage media other than cache in response to the corresponding read request in the message queue. The cache includes a first-level cache queue and at least one second-level cache queue.

[0013] The module for determining data to be cleaned is used to determine the data to be cleaned in the first-level cache queue and the number of reference read requests corresponding to the data to be cleaned if the first-level cache queue is determined to meet the data cleanup conditions.

[0014] The data transfer module is used to determine the sequence number of the secondary cache queue for storing data to be cleaned based on the quantity, delete the data to be cleaned from the primary cache queue and store it in the secondary cache queue with the corresponding sequence number, and store the target data in the primary cache queue.

[0015] The sequence number indicates the priority at which data in the corresponding secondary cache queue is cleared from the cache; the reference read request is a read request that is stored in the message queue before the read request corresponding to the data to be cleared, and is read after the read request corresponding to the data to be cleared.

[0016] Thirdly, embodiments of the present invention provide an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the method provided in the first aspect.

[0017] Fourthly, embodiments of the present invention provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method provided in the first aspect.

[0018] Fifthly, embodiments of the present invention provide a computer program including computer instructions stored in a computer-readable storage medium. When a processor of a computer device reads the computer instructions from the computer-readable storage medium, the processor executes the computer instructions, causing the computer device to perform steps implementing the method provided in the first aspect.

[0019] The data processing method, apparatus, electronic device, and storage medium provided in this invention obtain target data. If it is determined that the first-level cache queue meets the data cleanup conditions, the number of data to be cleaned in the first-level cache queue and the number of reference read requests corresponding to the data to be cleaned are determined. The number of reference read requests can characterize the load of the data to be cleaned on the channel of the storage medium other than the cache. If it is determined that the first-level cache queue meets the data cleanup conditions, the number of data to be cleaned in the first-level cache queue and the number of corresponding reference read requests are further determined. Based on the number, the sequence number of the second-level cache queue storing the data to be cleaned is determined. This sequence number is used to indicate the priority of the data in the corresponding second-level cache queue being cleaned out of the cache. Thus, the data in the second-level cache queue with higher priority can be deleted from the cache first, realizing load balancing between channels and ultimately reducing the average response time of read requests. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments of this application will be briefly introduced below.

[0021] Figure 1 This application provides a schematic diagram of an interactive process for reading file data via a file access request.

[0022] Figure 2 This is a schematic diagram of the computing environment according to an embodiment of this application;

[0023] Figure 3 This is a flowchart illustrating the data processing method according to an embodiment of this application;

[0024] Figure 4 A flowchart illustrating the process of determining a reference read request in an embodiment of this application;

[0025] Figure 5 This is a schematic diagram of the cache data structure in an embodiment of this application;

[0026] Figure 6 This is an embodiment of the first-level cache queue in this application;

[0027] Figure 7 This is a schematic diagram of the structure of a data processing device provided in an embodiment of this application;

[0028] Figure 8 A schematic diagram of an optional structure for applying a distributed system to a blockchain system, as provided in an embodiment of the present invention;

[0029] Figure 9 An optional schematic diagram of the block structure provided in an embodiment of the present invention;

[0030] Figure 10 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0031] The embodiments of this application are described in detail below. Examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain this application, and should not be construed as limiting the invention.

[0032] Those skilled in the art will understand that, unless explicitly stated otherwise, the singular forms “a,” “an,” and “the” used herein may also include the plural forms. It should be further understood that the term “comprising” as used in the specification of this application means the presence of features, integers, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. It should be understood that when we say an element is “connected” or “coupled” to another element, it can be directly connected or coupled to the other element, or there may be intermediate elements. Furthermore, “connected” or “coupled” as used herein can include wireless connections or wireless coupling. The term “and / or” as used herein includes all or any units and all combinations of one or more associated listed items.

[0033] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0034] First, let's introduce and explain several terms used in this application:

[0035] Blockchain is a novel application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanisms, and cryptographic algorithms. Essentially, a blockchain is a decentralized database, a chain of data blocks linked together using cryptographic methods. Each data block contains information about a batch of network transactions, used to verify the validity of the information (anti-counterfeiting) and generate the next block. A blockchain can include an underlying platform, a platform product service layer, and an application service layer.

[0036] The underlying blockchain platform can include processing modules such as user management, basic services, smart contracts, and operational monitoring. The user management module is responsible for managing the identity information of all blockchain participants, including maintaining public and private key generation (account management), key management, and maintaining the correspondence between user real identities and blockchain addresses (access management). Furthermore, under authorization, it monitors and audits transactions of certain real identities and provides risk control rule configuration (risk control audit). Basic microservices are deployed on all blockchain node devices to verify the validity of business requests. After consensus is reached on valid requests, they are recorded in storage. For a new business request, the basic services first perform interface adaptation parsing and authentication (interface adaptation), and then encrypt the business information using a consensus algorithm (consensus management). After encryption, the data is transmitted completely and consistently to the shared ledger (network communication) and recorded and stored. The smart contract module is responsible for contract registration, issuance, triggering, and execution. Developers can define contract logic using a programming language and publish it to the blockchain (contract registration). According to the contract terms, the key or other events are invoked to trigger execution and complete the contract logic. It also provides functions for contract upgrades and cancellations. The operation monitoring module is mainly responsible for deployment, configuration modification, contract settings, cloud adaptation, and real-time status visualization output during product release, such as alarms, monitoring network conditions, and monitoring the health status of node devices.

[0037] The platform's product service layer provides the basic capabilities and implementation frameworks for typical applications. Developers can leverage these basic capabilities, along with the specific characteristics of their business needs, to implement blockchain-based business logic. The application service layer provides blockchain-based application services to business stakeholders.

[0038] Big data refers to data sets that cannot be captured, managed, and processed within a certain timeframe using conventional software tools. It represents massive, rapidly growing, and diverse information assets that require new processing models to achieve stronger decision-making, insightful discovery, and process optimization capabilities. With the advent of the cloud era, big data has attracted increasing attention. Big data requires specialized technologies to effectively process large amounts of data over a tolerable timeframe.

[0039] Please see as follows Figure 1This illustration demonstrates an interactive flow diagram of a file access request for reading file data according to an embodiment of this application. The data processing method provided in this embodiment can be used in a (local or distributed) file system based on a large-capacity mechanical hard disk. The file system includes program instructions for file access logic and program instructions for caching. File access logic refers to the organization of files from the user's perspective; it is the data and its structure that the user can directly process. It can also be understood as how the data is logically organized within the file. The program instructions for file access logic are executed by the terminal's processor. The program instructions for caching are used to manage the cache, which can refer to memory in the form of random access memory (RAM). Based on the program instructions for file access logic, the processor can manage memory to implement the corresponding functions of the embodiments of this application.

[0040] When a file access request (read request or write request) arrives at the processor, it requests the file system to search for the corresponding file in the cache. If the file is found, the target data can be read directly from the cache. If the file is not found, the target data needs to be read from the disk and stored in the cache. This process includes reading from and writing to the disk. After the target data is stored in the cache, it can be quickly read from the cache.

[0041] This application's embodiments can be applied to terminals, specifically executed by the terminal's processor. The terminal here can include, but is not limited to, smartphones, tablets, laptops, and desktop computers. The terminal's processor can read instructions (e.g., file access requests) from memory and local cache, place them in the instruction register, and issue control instructions to complete the execution of an instruction. However, the processor cannot directly read programs or data from the disk. Therefore, memory, as the component that directly communicates with the processor, serves as a bridge between the processor and the disk, temporarily storing processor-processed data and data exchanged with the disk. When the terminal is running, the processor writes some data from the disk into the cache, resolving the issue of the high-speed CPU accessing the slow memory.

[0042] Please see Figure 2 The diagram illustrates a computing environment according to an embodiment of this application. Host 110 can submit input / output (I / O) requests to storage controller 120 via server 130 to access data (e.g., tracks, logical block addresses, memory cells, cell groups (e.g., columns, rows, or arrays of cells), sectors, fields, etc.) in storage device 140.

[0043] Storage controller 120 includes one or more processors 1201 and a cache 1202 for caching data for storage device 140. Processor 1201 may include a single central processing unit (CPU), a single core or a group of cores on a single CPU, or a group of processing resources on one or more CPUs. Cache 1202 caches data transferred between host 110 and storage device 140, and cache 1202 is divided into a level 1 cache queue and a level 2 cache queue.

[0044] In response to a read request, processor 1201 retrieves the target data from storage device 140. If it determines that the Level 1 cache queue meets the data cleanup conditions, it determines the data to be cleaned in the Level 1 cache queue and the number of reference read requests corresponding to the data to be cleaned. Based on the number, it determines the sequence number of the Level 2 cache queue storing the data to be cleaned, deletes the data to be cleaned from the Level 1 cache queue, and stores it in the corresponding Level 2 cache queue. The target data is stored in the Level 1 cache queue. The sequence number indicates the priority of data in the corresponding Level 2 cache queue being cleaned out of cache 1202. That is, the cleanup order of data in cache 1202 is determined according to the "last-in, first-out" count of read requests (the number of read requests that were stored in the message queue before the read request corresponding to the data to be cleaned and were read after the read request corresponding to the data to be cleaned), effectively solving the problem of unbalanced load on storage device 140. When other read requests request data stored in cache 1202, the data is retrieved directly from cache 1202 to improve data retrieval efficiency.

[0045] Optionally, some or all of the functions may be implemented as microcode or firmware in the hardware device of the storage controller 120, for example in an application-specific integrated circuit (ASIC).

[0046] Storage device 110 may include one or more storage devices 140 known in the art, such as solid-state storage devices 140 (SSDs) composed of solid-state electronic devices, NAND flash memory cells, electrically erasable programmable read-only memory (EEPROM), flash memory, flash drives, random access memory (RAM) drives, storage-class memory (SCMs), phase-change memory (PCMs), resistive random access memory, spin-transfer torque memory, conductive bridged RAM, magnetic hard disk drives, optical discs, magnetic tapes, etc. Storage device 140 may be further configured as a device array, such as simple disk clustering, direct access storage device 140, redundant array of independent disks, virtualization devices, etc. Furthermore, storage device 140 may include heterogeneous storage devices 140 from different vendors or from the same vendor.

[0047] The memory may include a suitable volatile or non-volatile storage device 140.

[0048] The server 130 in this embodiment can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms.

[0049] The server execution method in this application embodiment can be implemented in the form of cloud computing. Cloud computing is a computing model that distributes computing tasks across a resource pool composed of a large number of computers, enabling various application systems to obtain computing power, storage space, and information services as needed. The network providing resources is called the "cloud." From the user's perspective, resources in the "cloud" are infinitely scalable, readily available, on-demand, expandable, and pay-as-you-go.

[0050] As a provider of fundamental cloud computing capabilities, a cloud resource pool (referred to as a cloud platform, generally called an IaaS (Infrastructure as a Service) platform) is established. Various types of virtual resources are deployed in the resource pool for external customers to choose from. The cloud resource pool mainly includes: computing devices (virtualized machines containing operating systems), storage devices, and network devices.

[0051] Based on logical function, a PaaS (Platform as a Service) layer can be deployed on top of the IaaS (Infrastructure as a Service) layer, and a SaaS (Software as a Service) layer can be deployed on top of the PaaS layer. Alternatively, SaaS can be deployed directly on top of IaaS. PaaS is a platform for running software, such as databases and web containers. SaaS refers to various types of business software, such as web portals and bulk SMS senders. Generally speaking, SaaS and PaaS are upper layers compared to IaaS.

[0052] Alternatively, host 110 may be connected to storage controller 120 via a bus interface (e.g., a peripheral component interconnect (PCI) bus interface and other interfaces known in the art).

[0053] The connection method of channels and chips in storage device 140 has evolved from single-channel to multi-channel. In a single-channel structure, all chips are connected to the same channel, while in a multi-channel structure, several NAND flash chips are mounted on different channels. Taking SSDs as an example, generally speaking, SSD performance increases proportionally with the number of channels; the more channels, the more concurrent read and write commands can be executed. Chips can share the same channel using a command interleaving method. However, the above channel and chip connection method has the following problem: when the application load is uneven, the utilization rate of chips belonging to different channels differs, causing some channels to be constantly busy while others remain idle due to lower chip utilization. This results in uneven channel utilization, failing to fully utilize the read and write performance of the SSD.

[0054] The data processing methods, apparatus, electronic devices, and computer-readable storage media provided in this application are intended to solve the above-mentioned technical problems of the prior art.

[0055] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.

[0056] Please see Figure 3 The figure illustrates a flowchart of a data processing method according to an embodiment of this application, as shown in the figure, including:

[0057] S101. Obtain target data. Target data is data obtained from storage media other than cache in response to the corresponding read request in the message queue.

[0058] When the data requested by a read request is already stored in the cache, the data can be quickly read and returned to the requester. However, when the data requested by a read request is not stored in the cache, it needs to be retrieved from other storage media (e.g., ...). Figure 2 Data is retrieved from the storage device shown. In this case, the data reading speed is slow, so when the data is obtained, it needs to be stored in the cache.

[0059] The cache in this embodiment includes a first-level cache queue and at least one second-level cache queue, that is, the cache is divided into two levels. When data is obtained from other storage media, it needs to be stored in the first-level cache queue. The data in the first-level cache queue is also continuously deleted according to preset rules and stored in the second-level cache queue. The data in the second-level cache queue is also continuously deleted according to preset rules, thereby ensuring the dynamic balance of the cache storage space.

[0060] S102. If it is determined that the first-level cache queue meets the data cleanup conditions, then determine the data to be cleaned up in the first-level cache queue and the number of reference read requests corresponding to the data to be cleaned up.

[0061] Specifically, in this embodiment, the access frequency of each data in the first-level cache queue is determined; if there is data in the first-level cache queue with an access frequency lower than a preset threshold, the first-level cache queue is determined to meet the data cleanup conditions; accordingly, the data in the first-level cache queue with an access frequency lower than the preset threshold is regarded as data to be cleaned up.

[0062] Because cache cleanup requires identifying which data originates from idle channels, but the cache itself is unaware of the internal structure and parallelism of other storage media. Read requests point to logical addresses, while parallelism is determined by physical addresses. Since the physical and logical addresses of storage media are dynamic, current technology cannot determine which physical channel the data originates from based solely on the logical address.

[0063] As discovered in the embodiments of this application, the latency experienced by read requests sent to an idle channel is relatively recent. Once a read request can be returned quickly, it indicates that the channel is an idle channel. Specifically, a read request is first added to the tail of the message queue. When the data corresponding to the request is returned from the SSD, the request is removed from the message queue. That is, read requests are added to the queue in the order of requests and leave the queue in an out-of-order manner. For a request R sent to an idle channel, the time spent in the data queue is very short. When request R leaves the queue, there are still many read requests that entered the queue before request R but are still waiting. These requests can be considered to come from busy channels, and their requested data has not yet been returned. Relative to request R, these requests are called reference read requests. The more reference read requests there are, the more idle the channel where request R is located is. Therefore, these reference read requests can be used as a measure of the idleness of the channel.

[0064] Based on this discovery, embodiments of this application measure the idle level of a channel by the number of reference read requests corresponding to the data. Reference read requests in these embodiments are read requests that are stored in a message queue before read requests corresponding to data to be cleaned, and are read after read requests corresponding to data to be cleaned.

[0065] Please see Figure 4 This example illustrates a flowchart of the process for determining a reference read request according to an embodiment of this application. Each message bit can store one read request. Whenever a read request is received, it is added to the tail of the queue, and its number in the queue is marked. Figure 4 As can be seen, a total of 10 read requests are stored in the message queue. The message queue has 10 message bits. The message bit marked with the letter "C" indicates that the data corresponding to the read request stored in that message bit has been returned from the SSD, and the message bit marked with the letter "W" indicates that the data corresponding to the read request stored in that message bit has not yet been returned from the SSD.

[0066] In (a), the data corresponding to the earliest read requests numbered 0-2 that entered the message queue has been returned from the SSD, while the data corresponding to read requests numbered 3-9 has not yet been returned from the SSD. In (b), the data corresponding to read request number 9 returned from the SSD earlier than read requests numbered 3-8, so the marker for the 10th message bit is updated to "C". Furthermore, for read request number 9, since 6 read requests (numbered 3-8) that entered the message queue earlier than it have not yet returned from the SSD, the number of reference read requests for read request number 9 is 6. In (c), the data corresponding to read request number 7 returned from the SSD, so the number of reference read requests corresponding to read request number 7 is 4 (numbered 3-6). In (d), the data corresponding to read request number 8 returned from the SSD, having entered the queue earlier than this read request, but read requests numbered 3-6 remain unprocessed. Therefore, the number of reference read requests corresponding to read request number 8 is 4.

[0067] S103. Determine the sequence number of the secondary cache queue for storing the data to be cleaned based on the quantity, delete the data to be cleaned from the primary cache queue and store it in the secondary cache queue with the corresponding sequence number, and store the target data in the primary cache queue.

[0068] Before data is stored in the cache in this embodiment, it is necessary to determine whether the cache queue meets the data cleanup conditions. For example, the data cleanup conditions may be that the storage space in the first-level cache queue is insufficient to store the data, or that the storage slots in the first-level cache queue are full. In this embodiment, the first-level cache queue can be divided into multiple storage slots, each used to store one piece of data. At the beginning of data processing, the first-level cache queue does not meet the data cleanup conditions, so the data can be stored directly in the first-level cache queue. As data processing progresses and the storage queue begins to meet the data cleanup conditions, it is necessary to delete some data in the first-level cache queue before the target data can be stored. This application does not directly delete this part of the data, that is, the data to be cleaned, from the cache, but stores this part of the data in the second-level cache queue first.

[0069] As can be seen from the above embodiments, the more reference read requests corresponding to each piece of data, the more idle the SSD channel where that data is located. Therefore, in this embodiment, at least one secondary cache queue can be set according to the number of reference read requests. For example, secondary cache queue 1 is used to store data with 1 to 3 reference read requests, secondary cache queue 2 is used to store data with 4 to 6 reference read requests, and so on. This obtains the correspondence between the sequence number of the secondary cache queue and the load level of the SSD channel. The sequence number has the meaning of indicating the priority of data in the corresponding secondary cache queue being cleared from the cache. Subsequently, the secondary cache queues with low load levels can be deleted first, thereby balancing the load between channels and reducing the average response time of read requests.

[0070] The data processing method of this application embodiment obtains target data. If it is determined that the first-level cache queue meets the data cleanup conditions, it determines the data to be cleaned in the first-level cache queue and the number of reference read requests corresponding to the data to be cleaned. The number of reference read requests can characterize the load of the data to be cleaned on the channel of the storage medium other than the cache. If it is determined that the first-level cache queue meets the data cleanup conditions, it further determines the data to be cleaned in the first-level cache queue and the number of corresponding reference read requests, and determines the sequence number of the second-level cache queue storing the data to be cleaned based on the number. The sequence number is used to indicate the priority of the data in the corresponding second-level cache queue being cleaned out of the cache. Thus, the data in the second-level cache queue with higher priority can be deleted from the cache first, realizing load balancing between channels and ultimately reducing the average response time of read requests.

[0071] Based on the above embodiments, as an optional embodiment, the sequence number of the secondary cache queue storing the data to be cleaned is determined according to the quantity, including:

[0072] Input the quantity into the preset increment function to obtain the sequence number of the secondary cache queue storing the data to be cleared, as output by the preset increment function; or

[0073] Input the quantity into the preset decrement function to obtain the sequence number of the secondary cache queue that stores the data to be cleared, which is output by the preset increment function.

[0074] It is understandable that if the sequence number of the second-level cache queue is obtained through a preset increment function, it means that the larger the sequence number of the second-level cache queue, the higher the priority for data to be deleted from the cache; if the sequence number of the second-level cache queue is obtained through a preset decrement function, it means that the smaller the sequence number of the second-level cache queue, the higher the priority for data to be deleted from the cache.

[0075] Specifically, the preset increasing function in this application embodiment can be a logarithmic function with a base greater than 1, such as log2N, where N represents the number of reference read requests. Figure 4 Taking data number 9 as an example, the number of reference read requests for this data is 6. After calculating log26 and taking the integer part, it is determined that this data is stored in the second-level cache queue with sequence number 2.

[0076] Please see Figure 5 The figure illustrates a schematic diagram of the cache data structure of an embodiment of this application. As shown, the cache includes a first-level cache queue and n+1 second-level cache queues. The sequence number of the second-level cache queue indicates the priority of data deletion from the cache in the sequence. As can be seen from the figure, the larger the sequence number, the higher the priority. In this embodiment, the sequence number is obtained according to a preset increment function. In the figure, the data bits in each cache queue (including the first-level cache queue and the second-level cache queue) are represented by small rectangular boxes. Filled data bits indicate that data is stored in the data bit, and unfilled data bits indicate that no data is stored in the data bit. Therefore, it can be seen that data can be deleted from the cache queue without following the data bit number sequentially.

[0077] Based on the above embodiments, as an optional embodiment, the data processing method further includes: if it is determined that the available capacity of the cache is less than a preset threshold, then data cleanup is performed starting from the non-empty and highest priority secondary cache queue until the available capacity of the cache is not less than the preset threshold.

[0078] Specifically, if the sequence number of the secondary cache queue is obtained based on a preset increment function, then data cleanup begins from the secondary cache queue with the largest non-empty sequence number, until the available cache capacity is not less than a preset threshold; or

[0079] If the sequence number of the secondary cache queue is obtained according to a preset subtraction function, then data cleanup will start from the secondary cache queue with the smallest non-empty sequence number until the available capacity of the cache is not less than the preset threshold.

[0080] In this embodiment of the application, when cleaning up data in the secondary cache queue, all data in the secondary cache may be deleted, or only some data may be deleted. This embodiment of the application does not make any specific limitations.

[0081] In one optional embodiment, obtaining the target data according to this application embodiment may include the following steps:

[0082] Receive read requests and store them in the message queue;

[0083] When a read request is read from the message queue, if at least one data corresponding to the read request is obtained from storage media other than the cache, then that data is used as the target data.

[0084] In this embodiment of the application, when a read request is received, the read request is stored in the message queue. When the data corresponding to the read request has been pre-stored in the cache, it can be retrieved quickly. That is to say, the time of the read request in the message queue is much shorter than the time of the read request corresponding to the data retrieved from the storage medium outside the cache. Therefore, it will not affect the calculation of the number of reference data requests for the data retrieved from the storage medium outside the cache.

[0085] Based on the above embodiments, the first-level cache queue of this application includes a Most Recently Used (MRU) end and a Least Recently Used (LRU) end.

[0086] In this embodiment, since the first-level cache queue can store multiple data items, it is necessary to sort the data in the first-level cache queue. This embodiment can sort the data according to the time of its last access, thus reflecting the time-domain nature of access. Specifically, the data in the first-level cache queue whose last access time is closest to the current time is designated as MRU data, and this MRU data is located at the MRU end. The data whose last access time is far from the current time is designated as LRU data, and this LRU data is located at the LRU end.

[0087] Further, identify the data to be cleaned up in the first-level cache queue, including:

[0088] Identify at least one piece of data to be cleaned from the LRU end of the first-level cache queue;

[0089] Store the target data in a first-level cache queue, including:

[0090] Store the target data at the MRU end of the first-level cache queue.

[0091] Please see Figure 6 This example illustrates an embodiment of a first-level cache queue 200 according to an embodiment of this application, which has an MRU end 201 and an LRU end 202. The MRU end 201 represents data recently added to or recently accessed in the first-level cache queue 200. Starting from the LRU end 202, data identified at the LRU end is selected to be deleted from the first-level cache queue 200. When data is added to the MRU end 201, other data moves down to the LRU end 202. If there is not enough space to add data to the MRU end 201, data can be deleted from the LRU end 202 to make room for new data to be added to the first-level cache queue 200.

[0092] Based on the above embodiments, as an optional embodiment, the secondary cache queue includes an MRU end and an LRU end.

[0093] Data cleanup begins with the highest-priority, non-empty secondary cache queue, including:

[0094] Data cleanup is performed on the LRU end of the non-empty and highest-priority secondary cache queue.

[0095] Specifically, data cleanup begins with the non-empty L2 cache queue with the largest sequence number, including: data cleanup from the LRU end of the non-empty L2 cache queue with the largest sequence number; and data cleanup begins with the non-empty L2 cache queue with the smallest sequence number, including: data cleanup from the LRU end of the non-empty L2 cache queue with the smallest sequence number.

[0096] In other words, when the L2 cache queue for storing data is determined using an increasing function, data is cleaned up from the LRU end of the non-empty L2 cache queue with the largest sequence number. When the L2 cache queue for storing data is determined using a decreasing function, data is cleaned up from the LRU end of the non-empty L2 cache queue with the smallest sequence number. For example, when the L2 cache queue for storing data is determined using an increasing function, if the total number of sequence numbers in the L2 cache queue is N, when the L2 cache queue with sequence number N is empty, data is cleaned up from the L2 cache queue with sequence number N-1, until the available space in the cache meets the preset conditions.

[0097] Since the location of data in a secondary cache queue is related to the number of reference read requests corresponding to the data, and data is constantly moving from the primary cache queue to the secondary cache queue, if only the highest priority secondary cache queue is deleted, data in lower priority secondary cache queues may not be deleted for a long time. Therefore, this application embodiment also includes:

[0098] Determine the access frequency of each data item in at least one secondary cache queue;

[0099] If there is data in the at least one secondary cache queue with an access frequency lower than a preset threshold, the data with an access frequency lower than the preset threshold is deleted from its corresponding secondary cache queue and stored in a secondary cache queue with a higher priority than the corresponding secondary cache queue.

[0100] Through this method, the embodiments of this application can realize dynamic management of the storage space data of the secondary cache queue and further improve the balance of responsibility between channels.

[0101] This application provides a data processing apparatus, such as... Figure 7 As shown, the device may include: a target data acquisition module 101, a data to be cleaned determination module 102, and a data transfer module 103, specifically:

[0102] The target data acquisition module 101 is used to acquire target data, which is data acquired from a storage medium other than the cache in response to a corresponding read request in the message queue. The cache includes a first-level cache queue and at least one second-level cache queue.

[0103] The data to be cleaned determination module 102 is used to determine the data to be cleaned in the first-level cache queue and the number of reference read requests corresponding to the data to be cleaned if it is determined that the first-level cache queue meets the data cleanup conditions.

[0104] The data transfer module 103 is used to determine the sequence number of the secondary cache queue for storing the data to be cleaned according to the quantity, delete the data to be cleaned from the primary cache queue and store it in the secondary cache queue with the corresponding sequence number, and store the target data in the primary cache queue.

[0105] Wherein, the sequence number is used to indicate the priority at which data in the corresponding secondary cache queue is cleared from the cache; the reference read request is a read request that is stored in the message queue before the read request corresponding to the data to be cleared, and is read after the read request corresponding to the data to be cleared.

[0106] The data processing apparatus provided in this embodiment of the invention specifically executes the process described in the above-described method embodiment. For details, please refer to the content of the above-described data processing method embodiment; further details will not be repeated here. The data processing apparatus provided in this embodiment of the invention, by acquiring target data, determines, if it is determined that the first-level cache queue meets the data cleanup conditions, the number of data to be cleaned in the first-level cache queue and the number of corresponding reference read requests for that data. The number of reference read requests characterizes the load of the data to be cleaned on the channel of the storage medium other than the cache. If it is determined that the first-level cache queue meets the data cleanup conditions, it further determines the number of data to be cleaned in the first-level cache queue and the number of corresponding reference read requests, and determines the sequence number of the second-level cache queue storing the data to be cleaned based on this number. This sequence number indicates the priority of data in the corresponding second-level cache queue being cleaned from the cache, thus allowing data in higher-priority second-level cache queues to be deleted from the cache first, achieving load balancing between channels and ultimately reducing the average response time of read requests.

[0107] Based on the above embodiments, as an optional embodiment, the data to be cleaned determination module includes:

[0108] The first sequence number determination submodule inputs the quantity into a preset increment function to obtain the sequence number of the secondary cache queue storing the data to be cleared, as output by the preset increment function; or

[0109] The second sequence number determination submodule inputs the quantity into the preset decrement function and obtains the sequence number of the secondary cache queue storing the data to be cleaned, which is output by the preset increment function.

[0110] Based on the above embodiments, as an optional embodiment, the data processing apparatus further includes:

[0111] The cleanup module is used to clean up data starting from the non-empty and highest-priority second-level cache queue if it is determined that the available capacity of the cache is less than a preset threshold, until the available capacity of the cache is not less than the preset threshold.

[0112] Based on the above embodiments, as an optional embodiment, the target data acquisition module includes:

[0113] The request receiving submodule is used to receive read requests and store them in the message queue.

[0114] The filtering module is used to select data as target data if at least one data corresponding to a read request is obtained from storage media other than the cache when a read request is read from the message queue.

[0115] Based on the above embodiments, as an optional embodiment, the first-level cache queue includes a Most Recently Used (MRU) end and a Least Recently Used (LRU) end;

[0116] The module for determining the data to be cleaned includes:

[0117] The data to be cleaned determination submodule is used to determine at least one piece of data to be cleaned from the LRU end of the first-level cache queue;

[0118] The data transfer module includes:

[0119] The first data transfer submodule is used to store the target data to the MRU end of the first-level cache queue.

[0120] Based on the above embodiments, as an optional embodiment, the preset increasing function is a logarithmic function with a preset value greater than 1 as the base.

[0121] Based on the above embodiments, as an optional embodiment, the secondary cache queue includes an MRU end and an LRU end;

[0122] The data transfer module includes:

[0123] The second data transfer submodule is used to store the data to be cleaned into the MRU end of the secondary cache queue with the corresponding sequence number;

[0124] The cleanup module is specifically used to clean up data from the LRU end of the non-empty and highest-priority secondary cache queue.

[0125] Based on the above embodiments, as an optional embodiment, the data processing apparatus further includes:

[0126] The access frequency determination module is used to determine the access frequency of each piece of data in at least one secondary cache queue;

[0127] The secondary queue update module is used to delete data with an access frequency lower than a preset threshold from its corresponding secondary cache queue and store it in a secondary cache queue with a higher priority than the corresponding secondary cache queue if at least one secondary cache queue contains data with an access frequency lower than a preset threshold.

[0128] The apparatus involved in the embodiments of the present invention can be a distributed system formed by connecting a client and multiple nodes (any form of computing device in the network, such as a server or a user terminal) through network communication.

[0129] Taking a distributed system as an example, see blockchain system. Figure 8 , Figure 8 This is an optional structural diagram of the distributed system 300 provided in this embodiment of the invention applied to a blockchain system. It consists of multiple nodes 400 (any form of computing device in the network, such as servers or user terminals) and clients 500. The nodes form a peer-to-peer (P2P) network. The P2P protocol is an application layer protocol running on top of the Transmission Control Protocol (TCP). In the distributed system, any machine, such as a server or terminal, can join and become a node. A node includes a hardware layer, a middleware layer, an operating system layer, and an application layer.

[0130] See Figure 8 The functions of each node in the blockchain system shown include:

[0131] 1) Routing: A basic function of nodes used to support communication between nodes.

[0132] In addition to routing capabilities, nodes can also have the following functions:

[0133] 2) Applications are deployed in the blockchain to implement specific business needs. They record data related to the implementation of functions to form record data, carry digital signatures in the record data to indicate the source of the task data, and send the record data to other nodes in the blockchain system. When other nodes successfully verify the source and integrity of the record data, they add the record data to a temporary block.

[0134] For example, the business logic implemented by the application includes:

[0135] 2.1) A wallet is used to provide the function of conducting electronic currency transactions, including initiating transactions (i.e., sending the transaction record of the current transaction to other nodes in the blockchain system; after other nodes successfully verify the transaction, they store the transaction record data in the temporary block of the blockchain as a response to acknowledge the validity of the transaction; of course, the wallet also supports querying the remaining electronic currency in the electronic currency address;

[0136] 2.2) Shared ledger, used to provide functions such as storage, query and modification of ledger data. It sends the record data of the operation on the ledger data to other nodes in the blockchain system. After the other nodes verify the validity, as a response to acknowledge the validity of the ledger data, they store the record data in a temporary block. They can also send confirmation to the node that initiated the operation.

[0137] 2.3) Smart contracts are computerized protocols that can execute the terms of a contract. They are implemented through code deployed on a shared ledger that executes when certain conditions are met. Based on actual business needs, the code is used to complete automated transactions, such as querying the logistics status of goods purchased by a buyer and transferring the buyer's electronic money to the merchant's address after the buyer signs for the goods. Of course, smart contracts are not limited to executing contracts for transactions; they can also execute contracts for processing received information.

[0138] 3) A blockchain consists of a series of blocks that are sequentially generated. Once a new block is added to the blockchain, it will not be removed. The blocks contain the data submitted by the nodes in the blockchain system.

[0139] When the embodiments of this application are applied to a blockchain system, the data will be represented by blocks. When a node in the blockchain system sends the recorded data to other nodes in the blockchain system, the node acts as the sender and the other nodes act as the receivers. The methods for data processing by the sender and receivers can be specifically referred to the above embodiments, and will not be repeated here.

[0140] See Figure 9 , Figure 9 This is an optional schematic diagram of the block structure provided in this embodiment of the invention. Each block includes the hash value of the transaction records stored in this block (the hash value of this block) and the hash value of the previous block. The blocks are connected through their hash values ​​to form a blockchain. Additionally, the block may include information such as a timestamp when it was generated. A blockchain is essentially a decentralized database, a chain of data blocks linked together using cryptographic methods. Each data block contains relevant information used to verify the validity of the information (anti-counterfeiting) and to generate the next block.

[0141] This application provides an electronic device comprising: a memory and a processor; at least one program stored in the memory, which, when executed by the processor, can achieve the following compared to the prior art: by acquiring target data, if it is determined that the first-level cache queue meets the data cleanup conditions, the number of data to be cleaned in the first-level cache queue and the number of reference read requests corresponding to the data to be cleaned are determined. The number of reference read requests can characterize the load of the data to be cleaned on the channel of the storage medium other than the cache. If it is determined that the first-level cache queue meets the data cleanup conditions, the number of data to be cleaned in the first-level cache queue and the number of corresponding reference read requests are further determined, and the sequence number of the second-level cache queue storing the data to be cleaned is determined according to the number. The sequence number is used to indicate the priority of the data in the corresponding second-level cache queue being cleaned out of the cache, so that the data in the second-level cache queue with higher priority can be deleted from the cache first, thereby achieving load balancing between channels and ultimately reducing the average response time of read requests.

[0142] In one alternative embodiment, an electronic device is provided, such as Figure 10 As shown, Figure 10 The illustrated electronic device 4000 includes a processor 4001 and a memory 4003. The processor 4001 and the memory 4003 are connected, for example, via a bus 4002. Optionally, the electronic device 4000 may also include a transceiver 4004. It should be noted that in practical applications, the transceiver 4004 is not limited to one type, and the structure of this electronic device 4000 does not constitute a limitation on the embodiments of this application.

[0143] Processor 4001 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It can implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. Processor 4001 may also be a combination that implements computational functions, such as including one or more microprocessor combinations, a combination of a DSP and a microprocessor, etc.

[0144] Bus 4002 may include a pathway for transmitting information between the aforementioned components. Bus 4002 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. Bus 4002 can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 10 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0145] The memory 4003 may be ROM (Read Only Memory) or other types of static storage devices capable of storing static information and instructions, RAM (Random Access Memory) or other types of dynamic storage devices capable of storing information and instructions, or EEPROM (Electrically Erasable Programmable Read Only Memory), CD-ROM (Compact Disc Read Only Memory) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited thereto.

[0146] The memory 4003 stores application code that executes the scheme of this application, and its execution is controlled by the processor 4001. The processor 4001 executes the application code stored in the memory 4003 to implement the content shown in the foregoing method embodiments.

[0147] This application provides a computer-readable storage medium storing a computer program that, when run on a computer, enables the computer to execute the corresponding content in the aforementioned method embodiments. Compared with the prior art, by acquiring target data, if it is determined that the first-level cache queue meets the data cleanup conditions, the number of data to be cleaned in the first-level cache queue and the number of reference read requests corresponding to the data to be cleaned are determined. The number of reference read requests can characterize the load of the data to be cleaned on the channel of the storage medium other than the cache. If it is determined that the first-level cache queue meets the data cleanup conditions, the number of data to be cleaned in the first-level cache queue and the number of corresponding reference read requests are further determined, and the sequence number of the second-level cache queue storing the data to be cleaned is determined according to the number. This sequence number is used to indicate the priority of the data in the corresponding second-level cache queue being cleaned out of the cache, so that the data in the second-level cache queue with higher priority can be deleted from the cache first, realizing load balancing between channels and ultimately reducing the average response time of read requests.

[0148] This application provides a computer program including computer instructions stored in a computer-readable storage medium. When a processor of a computer device reads the computer instructions from the computer-readable storage medium, the processor executes the computer instructions, causing the computer device to perform the content shown in the foregoing method embodiments. Compared with the prior art, by acquiring target data, if it is determined that the first-level cache queue meets the data cleanup conditions, the number of data to be cleaned in the first-level cache queue and the number of reference read requests corresponding to the data to be cleaned are determined. The number of reference read requests can characterize the load of the data to be cleaned on the channel of the storage medium other than the cache. If it is determined that the first-level cache queue meets the data cleanup conditions, the number of data to be cleaned in the first-level cache queue and the number of corresponding reference read requests are further determined, and the sequence number of the second-level cache queue storing the data to be cleaned is determined according to the number. This sequence number is used to indicate the priority of the data in the corresponding second-level cache queue being cleaned out of the cache, so that the data in the second-level cache queue with higher priority can be deleted from the cache first, realizing load balancing between channels and ultimately reducing the average response time of read requests.

[0149] It should be understood that although the steps in the flowcharts of the accompanying figures are shown sequentially as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the accompanying figures may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0150] The above are only some embodiments of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A data processing method, characterized by, The method comprises: acquiring target data, the target data being data acquired from a storage medium other than a cache in response to a corresponding read request in a message queue, the cache comprising a primary cache queue and at least one secondary cache queue; if it is determined that the primary cache queue meets a data cleaning condition, determining data to be cleaned in the primary cache queue and a quantity of reference read requests corresponding to the data to be cleaned; determining a serial number of a secondary cache queue in which the data to be cleaned is stored according to the quantity, deleting the data to be cleaned from the primary cache queue and storing the data to be cleaned in the secondary cache queue with the corresponding serial number, and storing the target data in the primary cache queue; wherein the serial number is used to indicate a priority of data in the corresponding secondary cache queue being cleaned out of the cache; the reference read request is a read request stored in the message queue before the read request corresponding to the data to be cleaned and read after the read request corresponding to the data to be cleaned.

2. The data processing method of claim 1, wherein, The determining of the serial number of the secondary cache queue in which the data to be cleaned is stored according to the quantity comprises: inputting the quantity into a preset increasing function to obtain the serial number of the secondary cache queue in which the data to be cleaned is stored output by the preset increasing function; or inputting the quantity into a preset decreasing function to obtain the serial number of the secondary cache queue in which the data to be cleaned is stored output by the preset increasing function.

3. The data processing method according to claim 1 or 2, characterized in that, The method further comprises: if it is determined that an available capacity of the cache is less than a preset threshold, starting data cleaning from a secondary cache queue that is not empty and has the highest priority until the available capacity of the cache is not less than the preset threshold.

4. The data processing method of claim 1, wherein, The acquiring of the target data comprises: receiving a read request and storing the read request in the message queue; when the read request is read from the message queue, if data corresponding to the at least one read request is acquired from a storage medium other than the cache, the data is taken as the target data.

5. The data processing method of claim 1, wherein, The primary cache queue comprises a most recently used (MRU) end and a least recently used (LRU) end; The determining of the data to be cleaned in the primary cache queue comprises: determining at least one data to be cleaned from the LRU end of the primary cache queue; The storing of the target data in the primary cache queue comprises: storing the target data in the MRU end of the primary cache queue.

6. The data processing method according to claim 2, characterized in that, The preset increasing function is a logarithmic function with a preset value greater than 1 as a base.

7. The data processing method according to claim 3, characterized in that, The secondary cache queue comprises an MRU end and an LRU end; The storing in the secondary cache queue with the corresponding serial number comprises: storing the data to be cleaned in the MRU end of the secondary cache queue with the corresponding serial number; The starting of the data cleaning from the secondary cache queue that is not empty and has the highest priority comprises: cleaning data from the LRU end of the secondary cache queue that is not empty and has the highest priority.

8. The data processing method according to claim 2, characterized in that, The method further comprises: determining access frequencies of data in the at least one secondary cache queue; If there is data with an access frequency lower than a preset threshold in the at least one secondary cache queue, the data with the access frequency lower than the preset threshold is deleted from the corresponding secondary cache queue and stored in a secondary cache queue with a higher priority than the corresponding secondary cache queue.

9. A data processing apparatus, characterized by, Comprise: A target data obtaining module, configured to obtain target data, the target data being data obtained from a storage medium other than a cache in response to a corresponding read request in a message queue, the cache comprising a primary cache queue and at least one secondary cache queue; A to-be-cleaned data determining module, configured to, if it is determined that the primary cache queue meets a data cleaning condition, determine data to be cleaned in the primary cache queue and a quantity of reference read requests corresponding to the data to be cleaned; A data storing module, configured to determine a serial number of a secondary cache queue in which the data to be cleaned is stored according to the quantity, delete the data to be cleaned from the primary cache queue and store the data to be cleaned in the secondary cache queue with the corresponding serial number, and store the target data in the primary cache queue; The serial number is used to indicate a priority of data in the corresponding secondary cache queue being cleaned out of the cache. The reference read request is a read request stored in the message queue before the read request corresponding to the data to be cleaned and read after the read request corresponding to the data to be cleaned.

10. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor implements the steps of the data processing method of any one of claims 1 to 8 when executing the program.

11. A computer readable storage medium, characterized in that, The computer readable storage medium stores computer instructions, and the computer instructions cause the computer to execute the steps of the data processing method of any one of claims 1 to 8.

Images