Data processing method and device, equipment and computer readable storage medium

Abstract

The application provides a data processing method, device and equipment and a computer readable storage medium. The method comprises: obtaining a target key value name to be queried; determining a data throughput corresponding to the target key value name in a current statistical period; determining a throughput threshold corresponding to a specific quantile based on the data throughput corresponding to each key value name queried in the current statistical period in the current statistical period; the specific quantile is dynamically adjusted based on the amount of data currently cached in the local shared memory; and in the case where it is determined that the data throughput corresponding to the target key value name in the current statistical period reaches the throughput threshold, the target key value name and the key value corresponding to the target key value name are cached in the local shared memory. Through the application, appropriate data caching services can be provided for different business scenarios and systems with different caching capabilities, and better versatility is achieved.

Description

Technical Field

[0001] The present application relates to computer storage technology, and in particular to a data processing method, apparatus, device, and computer-readable storage medium. Background Art

[0002] When performing data queries in high-concurrency scenarios, in order to reduce the access pressure on data sources such as databases or lower-level data services, it is usually necessary to cache the queried data. In the related art, in order to reduce the access pressure on the network bandwidth, it is usually determined whether the data to be queried needs to be cached based on whether the query frequency of the data to be queried reaches a preset frequency threshold, and the data to be cached is cached in the local memory of the server in the form of key-value pairs. However, as the amount of cached data increases, the occupancy rate of the local memory of the server will become larger and larger, which will affect the performance of data queries and even cause the server system to crash. In addition, the solution in the related art that determines whether the data to be queried needs to be cached by using a preset frequency threshold is not well applicable to systems with different business scenarios and different caching capabilities, and is not universal. Summary of the Invention

[0003] The embodiments of the present application provide a data processing method, apparatus, device and computer-readable storage medium, which, on the one hand, can automatically cache data corresponding to key-value names with large data throughput into local shared memory, thereby reducing the occupancy of local memory on the server side, improving server performance, and reducing the pressure on the database and network, thereby improving data query performance; on the other hand, it can provide appropriate data caching services for systems with different business scenarios and different caching capabilities, and has better versatility.

[0004] The technical solution of the embodiment of the present application is implemented as follows:

[0005] The present invention provides a data processing method, including:

[0006] Get the target key value name to be queried;

[0007] Determine the data throughput corresponding to the target key name in the current statistical period;

[0008] Determining a throughput threshold corresponding to a specific quantile based on the data throughput corresponding to each key value name queried during the current statistical period; the specific quantile is dynamically adjusted based on the amount of data currently cached in the local shared memory;

[0009] When it is determined that the data throughput corresponding to the target key name in the current statistical period reaches the throughput threshold, the target key name and the key value corresponding to the target key name are cached in the local shared memory.

[0010] In some embodiments, determining the throughput threshold corresponding to a specific quantile based on the data throughput corresponding to each key value name queried within the current statistical period includes: determining the data throughput corresponding to each key value name queried within the current statistical period; and determining the data throughput corresponding to the specific quantile in the results of sorting the data throughput corresponding to each key value name from small to large as the throughput threshold.

[0011] In some embodiments, after caching the target key name and the key value corresponding to the target key name in the local shared memory, the method further includes: sending a data cache notification to at least one slave agent so that the at least one slave agent caches the target key name and the key value corresponding to the target key name in the local shared memory of the at least one slave agent.

[0012] In some embodiments, determining the data throughput corresponding to the target key name in the current statistical period includes: querying the local shared memory to obtain the total number of queries for the target key name in the current statistical period and the average data volume of the key value corresponding to the target key name; based on the total number of queries and the average data volume, determining the data throughput corresponding to the target key name in the current statistical period.

[0013] In some embodiments, the method further includes: obtaining the number of single-machine queries for each key-value name queried and the average single-machine data volume of the key value corresponding to each key-value name reported by at least one slave agent according to a reporting period; obtaining the number of single-machine queries for each key-value name queried by the master agent in the current statistical period and the average single-machine data volume of the key value corresponding to each key-value name; summarizing the number of single-machine queries for each key-value name queried by the master agent and each key-value name reported by each slave agent in the current statistical period and the average single-machine data volume of the key value corresponding to each key-value name to obtain the total number of queries for each key-value name queried in the current statistical period and the average data volume of the key value corresponding to each key-value name; storing the total number of queries for each key-value name queried in the current statistical period and the average data volume of the key value corresponding to each key-value name in the local shared memory.

[0014] In some embodiments, when it is determined that the data throughput corresponding to the target key name in the current statistical period reaches the throughput threshold, the target key name and the key value corresponding to the target key name are cached in the local shared memory, including: determining the amount of data currently cached in the local shared memory; determining the current cache usage rate based on the currently cached amount of data and the total memory size of the local shared memory; when it is determined that the data throughput corresponding to the target key name in the current statistical period reaches the throughput threshold, and the current cache usage rate is less than the usage threshold, caching the target key name and the key value corresponding to the target key name in the local shared memory.

[0015] In some embodiments, when it is determined that the data throughput corresponding to the target key name in the current statistical period reaches the throughput threshold and the current cache usage is less than the usage threshold, the target key name and the key value corresponding to the target key name are cached in the local shared memory, including: querying the local shared memory to obtain the total number of queries for each key name queried in the current statistical period in the current statistical period; summing the total number of queries for each key name in the current statistical period to obtain the total number of requests in the current statistical period; when it is determined that the data throughput corresponding to the target key name in the current statistical period reaches the throughput threshold, the current cache usage is less than the usage threshold and the total number of requests reaches the number threshold, the target key name and the key value corresponding to the target key name are cached in the local shared memory.

[0016] In some embodiments, the method further includes: adding the target key name to a cached key name list; for each key name in the cached key name list, when it is determined that the key value corresponding to the key name has been updated in the data source, updating the updated key value to the local shared memory.

[0017] In some embodiments, when it is determined that the key value corresponding to the key value name has been updated in the data source, the updated key value is updated to the local shared memory, including: by periodically querying the data source, determining that the key value corresponding to the key value name has been updated, obtaining the updated key value; updating the updated key value to the local shared memory; sending a data update notification to at least one slave agent, so that after receiving the data update notification, the at least one slave agent obtains the updated key value corresponding to each key value name in the cached key value name list from the master agent.

[0018] In some embodiments, the method further includes: detecting the amount of data currently cached in the local shared memory according to a specific detection period; and dynamically adjusting the size of the specific quantile based on the amount of cached data.

[0019] In some embodiments, the cached data volume includes the number of cached key-value names, and the dynamic adjustment of the size of the specific quantile based on the cached data volume includes at least one of the following: when it is detected that the number of cached key-value names is less than a first quantity threshold, the specific quantile is reduced by a first adjustment step; when it is detected that the number of cached key-value names is greater than a second quantity threshold, the specific quantile is increased by a second adjustment step.

[0020] In some embodiments, caching the target key name and the key value corresponding to the target key name in the local shared memory includes: determining the duration adjustment parameter corresponding to the target key name based on the data throughput corresponding to the target key name in the current statistical period; determining the cache expiration duration of the target key name based on the duration adjustment parameter and a preset cache duration range; and caching the target key name and the key value corresponding to the target key name in the local shared memory according to the cache expiration duration.

[0021] An embodiment of the present application provides a data processing device, including:

[0022] The first acquisition module is used to obtain the target key value name to be queried;

[0023] A first determining module is used to determine the data throughput corresponding to the target key name in the current statistical period;

[0024] A second determination module is configured to determine a throughput threshold corresponding to a specific quantile based on the data throughput corresponding to each key name queried during the current statistical period; the specific quantile is dynamically determined based on the amount of data currently cached in the local shared memory;

[0025] The cache module is used to cache the target key name and the key value corresponding to the target key name in the local shared memory when it is determined that the data throughput corresponding to the target key name in the current statistical period reaches the throughput threshold.

[0026] In some embodiments, the second determination module is further used to: determine the data throughput corresponding to each key value name queried within the current statistical period within the current statistical period; and determine the data throughput corresponding to a specific quantile in the sorting results of the data throughput corresponding to each key value name from small to large as the throughput threshold.

[0027] In some embodiments, the device also includes: a first notification module, used to send a data cache notification to at least one slave agent, so that the at least one slave agent caches the target key name and the key value corresponding to the target key name into the local shared memory of the at least one slave agent.

[0028] In some embodiments, the first determination module is also used to: query the local shared memory to obtain the total number of queries for the target key name in the current statistical period and the average data volume of the key value corresponding to the target key name; based on the total number of queries and the average data volume, determine the data throughput corresponding to the target key name in the current statistical period.

[0029] In some embodiments, the device also includes: a second acquisition module, used to obtain the number of single-machine queries for each key-value name queried and the single-machine average data volume of the key value corresponding to each key-value name reported by at least one slave agent according to a reporting period; a third acquisition module, used to obtain the number of single-machine queries for each key-value name queried by the master agent in the current statistical period and the single-machine average data volume of the key value corresponding to each key-value name; a summary module, used to summarize the number of single-machine queries for each key-value name queried by the master agent and each key-value name reported by each slave agent in the current statistical period and the single-machine average data volume of the key value corresponding to each key-value name, and obtain the total number of queries for each key-value name queried in the current statistical period and the average data volume of the key value corresponding to each key-value name; a storage module, used to store the total number of queries for each key-value name queried in the current statistical period and the average data volume of the key value corresponding to each key-value name in the local shared memory.

[0030] In some embodiments, the cache module is also used to: determine the amount of data currently cached in the local shared memory; determine the current cache usage rate based on the amount of data currently cached and the total memory size of the local shared memory; and cache the target key name and the key value corresponding to the target key name in the local shared memory when it is determined that the data throughput corresponding to the target key name in the current statistical period reaches the throughput threshold and the current cache usage rate is less than the usage threshold.

[0031] In some embodiments, the cache module is also used to: query the local shared memory to obtain the total query times of each key value name queried in the current statistical period in the current statistical period; sum the total query times of each key value name in the current statistical period to obtain the total request times in the current statistical period; when it is determined that the data throughput corresponding to the target key value name in the current statistical period reaches the throughput threshold, the current cache usage rate is less than the usage threshold and the total request times reaches the number threshold, cache the target key value name and the key value corresponding to the target key value name in the local shared memory.

[0032] In some embodiments, the device further includes: an adding module for adding the target key value name to a cached key value name list; an updating module for updating each key value name in the cached key value name list, when it is determined that the key value corresponding to the key value name has been updated in the data source, the updated key value to the local shared memory.

[0033] In some embodiments, the update module is also used to: periodically query the data source to determine if the key value corresponding to the key value name is updated, and then obtain the updated key value; update the updated key value to the local shared memory; send a data update notification to at least one slave agent, so that after receiving the data update notification, the at least one slave agent obtains the updated key value corresponding to each key value name in the cached key value name list from the master agent.

[0034] In some embodiments, the device further includes: a detection module for detecting the amount of data currently cached in the local shared memory according to a specific detection period; and an adjustment module for dynamically adjusting the size of the specific quantile based on the amount of cached data.

[0035] In some embodiments, the amount of cached data includes the number of cached key-value names, and the adjustment module is further used to perform at least one of the following: when it is detected that the number of cached key-value names is less than a first quantity threshold, the specific quantile is reduced by a first adjustment step; when it is detected that the number of cached key-value names is greater than a second quantity threshold, the specific quantile is increased by a second adjustment step.

[0036] In some embodiments, the cache module is also used to: determine the duration adjustment parameter corresponding to the target key name based on the data throughput corresponding to the target key name in the current statistical period; determine the cache expiration duration of the target key name based on the duration adjustment parameter and a preset cache duration range; and cache the target key name and the key value corresponding to the target key name into local shared memory according to the cache expiration duration.

[0037] An embodiment of the present application provides a data processing device, comprising: a memory for storing executable instructions; and a processor for implementing the method provided in the embodiment of the present application when executing the executable instructions stored in the memory.

[0038] An embodiment of the present application provides a computer-readable storage medium storing executable instructions for causing a processor to execute the method provided in the embodiment of the present application.

[0039] The embodiments of the present application have the following beneficial effects:

[0040] Based on the current service's historical access data, the specific quantile is dynamically adjusted based on the amount of data currently cached in the local shared memory. The throughput threshold corresponding to the specific quantile is determined by counting the data throughput corresponding to each key-value name queried during the current statistical period. If the data throughput corresponding to the target key-value name during the current statistical period reaches the throughput threshold, the target key-value name and the key value corresponding to the target key-value name are cached in the local shared memory. This automatically caches data corresponding to key-value names with higher data throughput into the local shared memory, thereby reducing server-side local memory usage, improving server performance, and reducing database and network pressure, thereby improving data query performance. Furthermore, since the throughput threshold can be dynamically adjusted based on the distribution of throughput corresponding to each key-value name queried during the current statistical period and the current usage of the local shared memory, it can provide appropriate data caching services for different business scenarios and systems with different caching capabilities, thus achieving better versatility. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1A This is an optional architectural diagram of a data query system provided in an embodiment of the present application;

[0042] Figure 1B This is an optional structural diagram of the data query system provided in the embodiment of the present application applied to the blockchain system;

[0043] Figure 1C This is an optional schematic diagram of the block structure provided in the embodiment of the present application;

[0044] Figure 2 This is an optional structural diagram of a data processing device provided in an embodiment of the present application;

[0045] Figure 3 This is an optional flowchart of the data processing method provided in the embodiment of the present application;

[0046] Figure 4This is an optional flowchart of the data processing method provided in the embodiment of the present application;

[0047] Figure 5 This is an optional flowchart of the data processing method provided in the embodiment of the present application;

[0048] Figure 6 This is an optional flowchart of the data processing method provided in the embodiment of the present application;

[0049] Figure 7 This is an optional flowchart of the data processing method provided in the embodiment of the present application;

[0050] Figure 8 This is an optional flowchart of the data processing method provided in the embodiment of the present application;

[0051] Figure 9 A schematic diagram of an optional implementation architecture of the data query method provided in an embodiment of the present application. DETAILED DESCRIPTION

[0052] In order to make the purpose, technical solutions and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limiting this application. All other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0053] In the following description, reference is made to “some embodiments”, which describes a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0054] If similar descriptions of "first / second" appear in the application documents, the following explanation is added. In the following description, the terms "first\second\third" involved are merely used to distinguish similar objects and do not represent a specific order for the objects. It can be understood that "first\second\third" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0055] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which this application pertains. The terms used herein are for the purpose of describing the embodiments of this application only and are not intended to limit this application.

[0056] Before further describing the embodiments of the present application in detail, the nouns and terms involved in the embodiments of the present application are explained. The nouns and terms involved in the embodiments of the present application are subject to the following interpretations.

[0057] 1) Large key data: This refers to data with a large key-value data size in the key-value (KV) format, such as data with a key-value data size of 100Kb, 200Kb, 1Mb, or 2Mb.

[0058] 2) Hot key data: refers to the data corresponding to keys with a large number of concurrent requests in high-concurrency scenarios.

[0059] In order to better understand the data processing method provided in the embodiments of the present application, the data caching solution in the related art is first described below.

[0060] In related technologies, in high-concurrency scenarios, general key-value data is centrally stored in memory caches such as Remote Dictionary Server (Redis), Tencent Cloud KV (CKV), DCache, and Memcached. Small key data is organized as much as possible, and hot key data is avoided as much as possible. If the amount of data corresponding to a key is large, more IO bandwidth will be consumed during data transmission. Both large and hot key data require memory caching on the server side. Memory caching on the server side can lead to higher local memory usage on the server side. In a distributed environment, data inconsistency between multiple machines can occur, and centralized cache failures can lead to request breakdown. The impact of cache failures is even more severe for large or hot key data.

[0061] In addition, the related art determines the key that needs to be cached through a preset threshold. Since the query frequency of data in different business scenarios can vary greatly, and the different caching capabilities of different systems will also lead to different amounts of cached data that can be supported, the data caching solution in the above-mentioned related art cannot be well adapted to different systems and is not universal.

[0062] The embodiments of the present application provide a data processing method, apparatus, device and computer-readable storage medium. On the one hand, it can automatically cache the data corresponding to the key-value name with a large data throughput into the local shared memory, thereby reducing the occupancy of the local memory of the server, improving the performance of the server, and reducing the pressure on the database and network, and improving the data query performance; on the other hand, it can provide appropriate data caching services for different business scenarios and systems with different caching capabilities, and has better versatility. The following describes an exemplary application of the data processing device provided by the embodiment of the present application. The data processing device provided by the embodiment of the present application can be implemented as various types of user terminals such as laptops, tablet computers, desktop computers, set-top boxes, mobile devices (for example, mobile phones, portable music players, personal digital assistants, dedicated messaging devices, portable gaming devices), and can also be implemented as a server. The following will describe an exemplary application when the device is implemented as a server.

[0063] See also Figure 1A , Figure 1A This is an optional architectural diagram of the data query system 100 provided in an embodiment of the present application, which can realize data query in a high-concurrency scenario. The terminal (terminal 400-1 and terminal 400-2 are shown as examples) is connected to the server 200 through the network 300. The network 300 can be a wide area network or a local area network, or a combination of the two.

[0064] The terminal is used to: display an interactive interface for users to perform data query on a graphical interface (graphic interface 410 - 1 and graphical interface 410 - 2 are shown as examples), receive data query operations from users, and send data query requests to the data query service 201 in the server 200 .

[0065] Server 200 is used to run data query service 201 and data processing service 202. Data query service 201 is used to receive a data query request; query the key value corresponding to the target key name to be queried in the data query request from local shared memory; if the key value corresponding to the target key name is not found in the local shared memory, query the key value corresponding to the target key name from data source 500; and return the queried key value. Data processing service 202 is used to obtain the target key name to be queried; determine the data throughput corresponding to the target key name in the current statistical period; determine a throughput threshold corresponding to a specific quantile based on the data throughput corresponding to each key name queried in the current statistical period; the specific quantile is dynamically adjusted based on the amount of data currently cached in the local shared memory; if it is determined that the data throughput corresponding to the target key name in the current statistical period reaches the throughput threshold, cache the target key name and the key value corresponding to the target key name in the local shared memory.

[0066] In some embodiments, the server 200 can be an independent physical server, or it can be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The data query service 201 and data processing service 202 running in the server 200 can be two independent processes or threads, or different threads in the same process, or two tasks executed by the same thread. The terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, etc., but is not limited to this. The terminal and the server can be directly or indirectly connected via wired or wireless communication, which is not limited in the embodiments of the present invention.

[0067] The data query system 100 involved in the embodiment of the present application may also be a distributed system 101 of a blockchain system, see Figure 1B , Figure 1B This is an optional structural diagram of a data query system 100 provided in an embodiment of the present application, applied to a blockchain system. The distributed system 101 can be composed of multiple nodes 102 (any type of computing device connected to the network, such as a server or user terminal) and clients 103, forming a peer-to-peer (P2P) network. The P2P protocol is an application layer protocol running on top of the Transmission Control Protocol (TCP). In a distributed system, any machine, such as a server or terminal, can join and become a node. Nodes include a hardware layer, an intermediate layer, an operating system layer, and an application layer.

[0068] See also Figure 1B The functions of each node in the blockchain system shown include:

[0069] 1) Routing: A basic function of a node, used to support communication between nodes.

[0070] In addition to the routing function, nodes can also have the following functions:

[0071] 2) Applications are deployed in the blockchain to implement specific services based on actual business needs. They record data related to the implementation of the functions to form record data. The record data carries a digital signature to indicate the source of the task data. The record data is sent to other nodes in the blockchain system for other nodes to add to the temporary block when they successfully verify the source and integrity of the record data.

[0072] For example, the services implemented by the application include:

[0073] 2.1) Wallet: This provides the functionality for conducting electronic currency transactions, including initiating transactions (i.e., sending the current transaction record to other nodes in the blockchain system. Upon successful verification by other nodes, the transaction record data is stored in a temporary block of the blockchain as a response to acknowledge the transaction validity). The wallet also supports querying the remaining electronic currency in an electronic currency address.

[0074] 2.2) Shared ledgers are used to store, query, and modify account data. Records of operations on account data are sent to other nodes in the blockchain system. After verification, other nodes acknowledge the validity of the account data by storing the recorded data in a temporary block. They can also send confirmation to the node that initiated the operation.

[0075] 2.3) Smart contracts are computerized protocols that can enforce the terms of a contract. They are implemented through code deployed on a shared ledger that is executed when certain conditions are met. Based on actual business needs, the code is used to complete automated transactions, such as querying the logistics status of a buyer's purchased goods and transferring the buyer's electronic currency to the merchant's address after the buyer signs for the goods. Of course, smart contracts are not limited to executing contracts for transactions, but can also execute contracts that process received information.

[0076] 3) Blockchain, including a series of blocks that are connected to each other in the order of their generation. Once a new block is added to the blockchain, it will not be removed. The block records the record data submitted by the nodes in the blockchain system.

[0077] 4) Consensus is a process in a blockchain network used to reach agreement on transactions in a block among multiple nodes involved. The agreed-upon block is appended to the end of the blockchain. Mechanisms for achieving consensus include Proof of Work (PoW), Proof of Stake (PoS), Delegated Proof-of-Stake (DPoS), and Proof of Elapsed Time (PoET).

[0078] See also Figure 1C , Figure 1CThis is an optional schematic diagram of the block structure provided by an embodiment of the present application. Each block includes the hash value of the transaction records stored in the block (the hash value of the current block) and the hash value of the previous block. The blocks are connected by hash values ​​to form a blockchain. In addition, the block may also include information such as the timestamp when the block was generated. Blockchain is essentially a decentralized database, a series of data blocks generated using cryptographic methods. Each data block contains relevant information used to verify the validity of the information (anti-counterfeiting) and generate the next block.

[0079] See also Figure 2 , Figure 2 is a structural diagram of a data processing device 200 provided in an embodiment of the present application, Figure 2 The data processing device 200 shown includes: at least one processor 210, a memory 250, at least one network interface 220 and a user interface 230. The various components in the data processing device 200 are coupled together via a bus system 240. It is understood that the bus system 240 is used to achieve connection and communication between these components. In addition to including a data bus, the bus system 240 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, the bus system 240 is not described in detail. Figure 2 Various buses are labeled as bus system 240 .

[0080] The processor 210 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., where the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0081] The user interface 230 includes one or more output devices 231 that enable presentation of media content, including one or more speakers and / or one or more visual display screens. The user interface 230 also includes one or more input devices 232, including user interface components that facilitate user input, such as a keyboard, mouse, microphone, touch screen display, camera, other input buttons and controls.

[0082] The memory 250 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state memory, hard drives, optical drives, etc. The memory 250 may optionally include one or more storage devices that are physically remote from the processor 210.

[0083] The memory 250 includes volatile memory or nonvolatile memory, or may include both volatile and nonvolatile memory. The nonvolatile memory may be read-only memory (ROM), and the volatile memory may be random access memory (RAM). The memory 250 described in the embodiments of the present application is intended to include any suitable type of memory.

[0084] In some embodiments, the memory 250 can store data to support various operations, examples of which include programs, modules, and data structures, or a subset or superset thereof, as exemplified below.

[0085] Operating system 251, including system programs for processing various basic system services and performing hardware-related tasks, such as the framework layer, core library layer, and driver layer, which are used to implement various basic services and process hardware-based tasks;

[0086] A network communication module 252 for reaching other computing devices via one or more (wired or wireless) network interfaces 220 , exemplary network interfaces 220 including Bluetooth, WiFi, and USB;

[0087] a presentation module 253 for enabling presentation of information via one or more output devices 231 (e.g., a display screen, a speaker, etc.) associated with the user interface 230 (e.g., a user interface for operating peripheral devices and displaying content and information);

[0088] The input processing module 254 is configured to detect one or more user inputs or interactions from one of the one or more input devices 232 and to translate the detected inputs or interactions.

[0089] In some embodiments, the data processing device provided in the embodiments of the present application can be implemented in software. Figure 2 A data processing device 255 stored in the memory 250 is shown, which can be software in the form of programs and plug-ins, including the following software modules: a first acquisition module 2551, a first determination module 2552, a second determination module 2553 and a cache module 2554. These modules are logical and can therefore be arbitrarily combined or further split according to the functions implemented.

[0090] The functions of each module will be described below.

[0091] In other embodiments, the data processing device provided in the embodiments of the present application can be implemented in hardware. As an example, the data processing device provided in the embodiments of the present application can be a processor in the form of a hardware decoding processor, which is programmed to execute the storage method provided in the embodiments of the present application. For example, the processor in the form of a hardware decoding processor can adopt one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), or other electronic components.

[0092] The data processing method provided in the embodiment of the present application will be described below in conjunction with the exemplary application and implementation of the terminal or server provided in the embodiment of the present application.

[0093] See also Figure 3 , Figure 3 This is an optional flow chart of the data processing method provided in the embodiment of the present application. Figure 3 The following steps are explained with reference to the steps shown in the figure. The execution subject of the following steps can be the terminal or server mentioned above.

[0094] In step S101, the target key name to be queried is obtained;

[0095] Here, the target key name to be queried refers to the key name to be queried by the data query requester. During implementation, the target key name to be queried can be obtained in an appropriate manner according to the actual situation. For example, after receiving a data query request, the current server or terminal can obtain the target key name to be queried from the data query request; other servers or other terminals can also obtain the target key name to be queried in the data query request after receiving the data query request, and report the target key name to the current server or terminal; the current server or terminal can also read the target key name to be queried from a database or historical access data.

[0096] In step S102, the data throughput corresponding to the target key name in the current statistical period is determined;

[0097] Here, the total amount of data corresponding to the key value queried for a key name within a specific time period is the data throughput corresponding to the key name within that time period. The specific time period can be user-preset or system-default, and is not limited here. The data throughput of the key name can be counted according to a specific time period. The current statistical period can be the time period in which the current query time of the target key name occurs. The data throughput corresponding to the target key name within the current statistical period can be the sum of the amount of data queried for the target key name each time within the current statistical period.

[0098] During implementation, the data throughput corresponding to the target key name in the current statistical period can be determined by querying historical access data, or each time a query request is received, the statistical information used to count the data throughput corresponding to each key name in the current statistical period can be updated based on the data volume of the key value corresponding to the target key name queried by each query request. By querying the statistical information corresponding to the target key name, the data throughput corresponding to the target key name in the current statistical period can be determined. The statistical information corresponding to the key-value name can be determined according to the actual situation. For example, the statistical information corresponding to the key-value name can directly include the data throughput of the key-value name in the current statistical period. The data throughput of the key-value name in the current statistical period can be directly obtained by obtaining the statistical information corresponding to the key-value name; the statistical information corresponding to the key-value name can also include the number of queries for the key-value name in the current statistical period and the amount of data corresponding to the key value of the key-value name obtained in each query. The sum of the amount of data corresponding to the key value of the key value obtained in each query in the current statistical period can be determined as the data throughput of the key-value name in the current statistical period; the statistical information corresponding to the key-value name can also include the number of queries for the key-value name in the current statistical period and the average amount of data corresponding to the key value of the key-value name. The data throughput of the key-value name in the current statistical period can be determined based on the number of queries and the average amount of data. The statistical information corresponding to the key-value name can be stored in any suitable place, including but not limited to local disk, local memory, local shared memory or database.

[0099] In some embodiments, when the server providing the data query service is an independent server, the data throughput corresponding to the target key name in the current statistical period may be the data throughput counted on the independent server in the current statistical period.

[0100] In some embodiments, when the server providing data query services is a server cluster, the data throughput corresponding to the target key name in the current statistical period can be the sum of the data throughput counted on each stand-alone server in the server cluster in the current statistical period.

[0101] In step S103, based on the data throughput corresponding to each key name queried in the current statistical period, a throughput threshold corresponding to a specific quantile is determined; the specific quantile is dynamically adjusted based on the amount of data currently cached in the local shared memory;

[0102] Here, you can determine the throughput threshold corresponding to a specific quantile by analyzing the distribution of data throughput corresponding to each key-value name queried during the current statistical period. The specific quantile can be a percentage quantile, such as the 80th or 90th percentile, or a decile or quartile.

[0103] The local shared memory may be the shared memory in the aforementioned terminal or server. The data stored in the local shared memory may include, but is not limited to, cached key names, key values ​​corresponding to each key name, and statistical information corresponding to the key name. The amount of data currently cached in the local shared memory may be the amount of data already stored in the local shared memory. The specific quantile may be dynamically adjusted based on the amount of data currently cached in the local shared memory. During implementation, those skilled in the art may dynamically adjust the specific quantile in an appropriate manner based on actual circumstances, which is not limited here. For example, an initial value for the specific quantile may be pre-set, and the value of the specific quantile may be increased as the amount of data currently cached in the local shared memory increases, and the value of the specific quantile may be decreased as the amount of data currently cached in the local shared memory decreases. Alternatively, each time a target key name to be queried is obtained, the value of the specific quantile may be increased if it is determined that the amount of data currently cached in the local shared memory is greater than a preset data volume threshold, and the value of the specific quantile may be increased if it is determined that the amount of data currently cached in the local shared memory is less than the preset data volume threshold.

[0104] In step S104, when it is determined that the data throughput corresponding to the target key name in the current statistical period reaches the throughput threshold, the target key name and the key value corresponding to the target key name are cached in the local shared memory.

[0105] Here, if the data throughput corresponding to the target key-value name reaches the throughput threshold during the current statistical period, query data for the target key-value name needs to be cached. The target key-value name and the key value corresponding to the target key-value name can be cached in local shared memory. The current service or other services running on the server can obtain the cached key value corresponding to the target key-value name by accessing the local shared memory, thereby reducing the number of data query requests sent directly to the data source.

[0106] In some embodiments, step S102 can be implemented via the following steps S121 to S122: In step S121, the local shared memory is queried to obtain the total number of queries for the target key name in the current statistical period and the average data volume of the key value corresponding to the target key name; in step S122, the data throughput corresponding to the target key name in the current statistical period is determined based on the total number of queries and the average data volume. Here, the local shared memory can store the total number of queries for each key name queried in the current statistical period and the average data volume of the key value corresponding to each key name. By querying the local shared memory, the total number of queries for the target key name in the current statistical period and the average data volume of the key value corresponding to the target key name can be obtained. The data throughput corresponding to the target key name in the current statistical period can be obtained by multiplying the total number of queries for the target key name in the current statistical period by the average data volume of the key value corresponding to the target key name. In this way, the data throughput corresponding to the target key name in the current statistical period can be determined simply and quickly.

[0107] In some embodiments, when determining the throughput threshold corresponding to a specific percentile based on the data throughput corresponding to each key value name queried in the current statistical period in the above-mentioned step S103, it can be implemented through the following steps S131 to S132: in step S131, determine the data throughput corresponding to each key value name queried in the current statistical period in the current statistical period; in step S132, determine the data throughput corresponding to the specific percentile in the sorting results of the data throughput corresponding to each key value name from small to large as the throughput threshold.

[0108] In some embodiments, in the above step S104, when the target key name and the key value corresponding to the target key name are cached in the local shared memory, it can be achieved through the following steps S141 to S143:

[0109] In step S141, based on the data throughput corresponding to the target key name in the current statistical period, a duration adjustment parameter corresponding to the target key name is determined;

[0110] Here, the cache expiration time of the target key name to be cached can be adjusted according to the data throughput corresponding to the target key name in the current statistical period. The time adjustment parameters may include but are not limited to the adjustment ratio of the cache expiration time to extend or shorten, the step size of the cache expiration time to increase or decrease, etc. During implementation, the data throughput corresponding to the target key name in the current statistical period can be used to query a specific mapping relationship to determine the time adjustment parameter corresponding to the target key name. The time adjustment parameter corresponding to the target key name can also be calculated based on the data throughput corresponding to the target key name in the current statistical period according to a specific algorithm. Those skilled in the art can determine the time adjustment parameter corresponding to the target key name in a suitable manner according to actual conditions, which is not limited here.

[0111] In step S142, based on the duration adjustment parameter and the preset cache duration range, the cache expiration duration of the target key name is determined;

[0112] Here, the cache duration range is the value range of the cache expiration duration, which may include the minimum and maximum values ​​of the cache expiration duration. The cache expiration duration of the target key name may be determined from the preset cache duration range according to the duration adjustment parameter and a specific value selection strategy. For example, the duration adjustment parameter may be a percentage quantile, and the value corresponding to the percentage quantile within the preset cache duration range may be determined as the cache expiration duration of the target key name; the duration adjustment parameter may also be an offset, and the offset between the minimum value or maximum value of the cache expiration duration within the cache duration range and the value of the offset may be determined as the cache expiration duration of the target key name.

[0113] In step S143, the target key name and the key value corresponding to the target key name are cached in a local shared memory according to the cache expiration time.

[0114] When caching the target key name and the key value corresponding to the target key name in the local shared memory, you can set a cache expiration time for the cached target key name and the key value corresponding to the target key name. When the cache expiration time reaches the cache expiration time, the cached target key name and the key value corresponding to the target key name will become invalid. The system can clear the expired cache data from the local shared memory according to a specific invalidation cache cleanup policy.

[0115] Through steps S141 to S143, the cache expiration time of the target key name can be determined based on the data throughput corresponding to the target key name in the current statistical period, and the target key name and the key value corresponding to the target key name can be cached in the local shared memory according to the determined cache expiration time. In this way, different cache expiration times can be set for key names with different data throughputs, thereby fully utilizing the local shared memory when improving the performance of the data query service through caching, thereby improving the utilization rate of the local shared memory.

[0116] In an embodiment of the present application, a specific percentile is dynamically adjusted based on the amount of data currently cached in the local shared memory, and the throughput threshold corresponding to the specific percentile is determined by counting the data throughput corresponding to each key name queried in the current statistical period. When it is determined that the data throughput corresponding to the target key name in the current statistical period reaches the throughput threshold, the target key name and the key value corresponding to the target key name are cached in the local shared memory. In this way, on the one hand, the data corresponding to the key name with a larger data throughput can be automatically cached in the local shared memory, thereby reducing the occupancy of the local memory of the server, improving the performance of the server, and reducing the pressure on the database and network, improving the data query performance; on the other hand, because the throughput threshold can be dynamically adjusted according to the distribution of the throughput corresponding to each key name queried in the current statistical period and the current usage of the local shared memory, it can provide appropriate data caching services for systems with different business scenarios and different caching capabilities, and has better versatility.

[0117] In some embodiments, see Figure 4 , Figure 4 This is an optional flow chart of the data processing method provided in the embodiment of the present application, based on Figure 3 , after step S104, step S401 may be further executed. The following will be described in conjunction with each step, and the execution subject of the following steps may be the terminal or server mentioned above.

[0118] In step S401, a data cache notification is sent to at least one slave agent, so that the at least one slave agent caches the target key name and the key value corresponding to the target key name into the local shared memory of the at least one slave agent.

[0119] Here, in a distributed scenario, the data query service may be provided by a cluster, which may include a master agent and at least one slave agent registered with the master agent, and the data query service may be implemented in a master-slave agent manner. The current terminal or server may serve as the master agent. After receiving a data query request from the slave agent, the target key name to be queried in the data query request may be reported to the master agent. After the master agent caches the target key name and the key value corresponding to the target key name in the local shared memory, the master agent may notify the registered slave agents to cache the target key name and the key value corresponding to the target key name in the local shared memory of each slave agent by sending a data cache notification.

[0120] In an embodiment of the present application, after the target key name and the key value corresponding to the target key name are cached in the local shared memory, the slave agent can be notified by sending a data cache notification to cache the target key name and the key value corresponding to the target key name in the local shared memory of each slave agent. In this way, the slave agent does not need to perform the process of determining whether the target key name needs to be cached, which can reduce the consumption of computing resources of the slave agent and thereby improve the performance of the data query service.

[0121] In some embodiments, see Figure 5 , Figure 5 This is an optional flow chart of the data processing method provided in the embodiment of the present application, based on Figure 3 The method may further perform the following steps S501 to S504. The following steps will be described in conjunction with each other. The execution subject of the following steps may be the terminal or server mentioned above.

[0122] In step S501, at least one of the single-machine query times for each key-value name being queried and the single-machine average data volume of the key value corresponding to each key-value name, reported from the proxy according to a reporting period, is obtained.

[0123] Here, the number of single-machine queries for a key-value name reported by the slave agent refers to the number of data query requests corresponding to the key-value name received by the slave agent during the reporting period, and the average single-machine data volume of the key value corresponding to the key-value name refers to the average data volume of the key value corresponding to the key-value name queried by the slave agent during the reporting period. Each slave agent can report to the master agent the number of single-machine queries for each key-value name queried in the data query requests received during the reporting period and the average single-machine data volume of the key value corresponding to each key-value name according to the reporting period. The reporting period can be a preset value, a system default value, or a value dynamically determined by each slave agent based on the frequency of received data query requests. The reporting periods corresponding to different slave agents can be the same or different, and are not limited here.

[0124] In step S502, the number of single-machine queries for each key-value name queried by the main agent in the current statistical period and the single-machine average data volume of the key value corresponding to each key-value name are obtained.

[0125] Here, the master agent can also receive data query requests. The number of single-machine queries for the key value name queried by the master agent during the current statistical period refers to the number of data query requests corresponding to the key value name received by the master agent during the current statistical period. The average single-machine data volume of the key value corresponding to the key value name refers to the average data volume of the key value corresponding to the key value name queried by the master agent during the current statistical period.

[0126] During implementation, based on the historical access data of the main agent, the number of single-machine queries for each key-value name queried by the main agent in the current statistical period and the average single-machine data volume of the key value corresponding to each key-value name can be counted in real time. After each data query request is received in the current statistical period, the number of single-machine queries for the key-value name corresponding to the data query request and the average single-machine data volume of the key value corresponding to the key-value name can be updated and stored. Through a simple data query, the number of single-machine queries for each key-value name queried by the main agent in the current statistical period and the average single-machine data volume of the key value corresponding to each key-value name can be obtained.

[0127] In step S503, the number of single-machine queries for each key-value name queried by the master agent and each key-value name reported by each slave agent during the current statistical period, as well as the average single-machine data volume of the key value corresponding to each key-value name are summarized to obtain the total number of queries for each key-value name queried during the current statistical period and the average data volume of the key value corresponding to each key-value name.

[0128] Here, the total number of queries for each key-value name queried during the current statistical period refers to the sum of the number of standalone queries for each key-value name queried on the master agent and the number of standalone queries for each key-value name queried on each slave agent during the current statistical period. The total number of queries for each key-value name queried during the current statistical period can be obtained by summing the standalone query counts for each key-value name queried on the master agent and the standalone query counts for each key-value name reported by each slave agent during the current statistical period, based on the key-value name.

[0129] For each key-value name queried in the current statistical period, the average data volume of the key value corresponding to the key-value name refers to the average data volume of the key value corresponding to the key-value name queried each time at the master agent and each slave agent in the current statistical period. The data volume of the key value corresponding to the key-value name queried each time at the master agent and the data volume of the key value corresponding to the key-value name queried each time at the slave agent in the current statistical period can be averaged to obtain the average data volume of the key value corresponding to the key-value name in the current statistical period. During implementation, the single-machine average data volume of the key value corresponding to each key-value name queried at the master agent in the current statistical period and the single-machine average data volume of the key value corresponding to each key-value name reported by each slave agent can be summarized by key-value name to obtain the average data volume of the key value corresponding to each key-value name in the current statistical period.

[0130] In step S504, the total number of queries for each key name queried in the current statistical period and the average data volume of the key value corresponding to each key name are stored in the local shared memory.

[0131] Here, the total number of queries for each key-value name queried in the current statistical period and the average data volume of the key value corresponding to each key-value name may be stored in the local shared memory.

[0132] It should be noted that the above steps S501 to S504 are not limited to the following when they are implemented. Figure 5 The execution order shown, for example, the above steps S501 to S504 may also be executed after step S101.

[0133] In an embodiment of the present application, by summarizing the number of single-machine queries for each key-value name queried by the master agent in the current statistical period and the number of single-machine queries for each key-value name reported by each slave agent according to the reporting period, as well as the single-machine average data volume of the key value corresponding to each key-value name, the total number of queries for each key-value name queried in the current statistical period and the average data volume of the key value corresponding to each key-value name are obtained, and the total number of queries for each key-value name queried in the current statistical period and the average data volume of the key value corresponding to each key-value name are stored in the local shared memory. In this way, the total number of queries for each key-value name queried in the current statistical period and the average data volume of the key value corresponding to each key-value name can be directly obtained from the local shared memory, so that the data throughput corresponding to each key-value name queried in the current statistical period can be simply and quickly obtained.

[0134] In some embodiments, see Figure 6 , Figure 6 This is an optional flow chart of the data processing method provided in the embodiment of the present application, based on Figure 3The above step S104 can be implemented by the following steps S601 to S603. The following steps will be described in combination with each other. The execution subject of the following steps can be the terminal or server mentioned above.

[0135] In step S601, the amount of data currently cached in the local shared memory is determined;

[0136] Here, the amount of data currently cached in the local shared memory is the total amount of data currently stored in the local shared memory. The data currently cached in the local shared memory may include, but is not limited to, each cached key name, the key value corresponding to each key name, and statistical information used to calculate the data throughput corresponding to each key name in the current statistical period.

[0137] In step S602, a current cache usage rate is determined based on the amount of data currently cached and the total memory size of the local shared memory;

[0138] Here, the current cache usage rate may be the current usage rate of the local shared memory. The current cache usage rate may be determined based on the current amount of cached data and the total memory size of the local shared memory. For example, the current cache usage rate may be determined as the ratio of the current amount of cached data to the total memory size of the local shared memory. Alternatively, the current free memory size in the local shared memory may be determined based on the current amount of cached data and the total memory size of the local shared memory, and the current cache usage rate may be determined based on the current free memory size and the total memory size of the local shared memory.

[0139] In step S603, when it is determined that the data throughput corresponding to the target key name in the current statistical period reaches the throughput threshold and the current cache usage is less than the usage threshold, the target key name and the key value corresponding to the target key name are cached in the local shared memory.

[0140] Here, the usage rate threshold may be a value pre-set by the user according to actual conditions, or a default value, or may be dynamically adjusted according to the data volume of the key value corresponding to the target key value name, and is not limited here.

[0141] In some embodiments, the above step S603 may be implemented by the following steps S631 to S633:

[0142] In step S631, the local shared memory is queried to obtain the total number of queries for each key name queried in the current statistical period.

[0143] Here, the local shared memory may store the total number of queries for each key-value name queried in the current statistical period.

[0144] In step S632, the total number of queries for each key name in the current statistical period is summed to obtain the total number of requests in the current statistical period;

[0145] Here, the total number of requests in the current statistical period may be the total number of data query requests received by the data query service in the current statistical period.

[0146] In step S633, when it is determined that the data throughput corresponding to the target key name in the current statistical period reaches the throughput threshold, the current cache usage is less than the usage threshold and the total number of requests reaches the number threshold, the target key name and the key value corresponding to the target key name are cached in the local shared memory.

[0147] Here, the usage rate threshold may be a value pre-set by the user according to actual conditions, or a default value, or may be dynamically adjusted according to the total number of requests, which is not limited here.

[0148] In an embodiment of the present application, when it is determined that the data throughput corresponding to the target key name in the current statistical period reaches the throughput threshold, and the current cache usage is less than the usage threshold, the target key name and the key value corresponding to the target key name are cached in the local shared memory. In this way, when performing data caching, the data throughput corresponding to the key name in the current statistical period and the current cache usage can be comprehensively considered to determine whether to cache the key name, thereby ensuring the stable operation of the system and avoiding the performance of the data query service being affected by excessive cached data. Further, when performing data caching, the data throughput corresponding to the key name in the current statistical period, the current cache usage, and the total number of data query requests received by the data query service in the current statistical period can be comprehensively considered to determine whether to cache the key name, thereby avoiding caching key names with large key values ​​but low query frequency, thereby improving the utilization rate of the local shared memory, and further improving the performance of the data query service.

[0149] In some embodiments, see Figure 7 , Figure 7 This is an optional flow chart of the data processing method provided in the embodiment of the present application, based on Figure 3 The method may further perform the following steps S701 to S702. The following steps will be described in conjunction with each other. The execution subject of the following steps may be the terminal or server mentioned above.

[0150] In step S701, the target key-value name is added to the cached key-value name list;

[0151] Here, the cached key-value name list may include all currently cached key-value names. After the target key-value name and the key value corresponding to the target key-value name are cached in the local shared memory, the target key-value name may be added to the cached key-value name list.

[0152] In step S702, for each key name in the cached key name list, if it is determined that the key value corresponding to the key name is updated in the data source, the updated key value is updated to the local shared memory.

[0153] Here, the data source can be any suitable data provider, such as a database, a data warehouse or a service that provides data query. The current terminal or server can directly query the data source to obtain the key value corresponding to the target key name, or can indirectly obtain the key value corresponding to the target key name in the data source through other processes or services.

[0154] During implementation, it is possible to determine whether each key-value name in the cached key-value name list has been updated in the data source by any appropriate method such as polling or detection. For example, the key value of each key-value name in the cached key-value name list in the data source can be periodically queried, and by comparing the key value in the data source with the key value in the local shared memory, it can be determined whether the key value has been updated; a version number can also be set for the key value corresponding to each key-value name, and by querying the data source for a new version of the key value for each key-value name, it can be determined whether the key value has been updated; it is also possible to receive a notification sent by the data source when a key-value update operation corresponding to the key-value name is executed, and determine whether the key value has been updated.

[0155] In some embodiments, the above-mentioned step S702 can be implemented by the following steps S721 to S723: in step S721, by periodically querying the data source and determining that the key value corresponding to the key name has been updated, the updated key value is obtained; in step S722, the updated key value is updated to the local shared memory; in step S723, a data update notification is sent to at least one slave agent, so that after receiving the data update notification, the at least one slave agent obtains the updated key value corresponding to each key name in the cached key name list from the master agent. Here, after obtaining the updated key value, the registered slave agent can be notified to perform data synchronization update by sending a data update notification. After receiving the data update notification, the slave agent can send a data acquisition request to the master agent to obtain the updated key value corresponding to each key name in the cached key name list from the master agent.

[0156] In some embodiments, at least one next-level slave agent can be registered on the slave agent, and the data cached in the local shared memory of each machine in the distributed scenario can be synchronously updated through a master-slave cascade, thereby reducing the access pressure of the slave agent to the master agent and improving the stability of the overall service in the distributed scenario.

[0157] In an embodiment of the present application, for each cached key name, when it is determined that the key value corresponding to the key name in the data source has been updated, the updated key value is obtained, and the updated key value is updated to the local shared memory. In this way, it can be ensured that the data cached in the local shared memory is updated in a timely manner, thereby maintaining the validity of the cached data, avoiding cache invalidation breakdown problems, and thus improving the reliability of data queries. Furthermore, after obtaining the updated key value of each cached key name, a data update notification is sent to at least one slave agent, so that after receiving the data update notification, at least one slave agent obtains the updated key value corresponding to each key name in the cached key name list from the master agent. In this way, the consistency of the data cached in the local shared memory of different machines in a distributed scenario can be guaranteed, and the master-slave synchronization method can increase the speed of data update propagation, thereby improving the efficiency of data synchronization.

[0158] In some embodiments, see Figure 8 , Figure 8 This is an optional flow chart of the data processing method provided in the embodiment of the present application, based on Figure 3 The method may further perform the following steps S801 to S802. The following steps will be described in conjunction with each other. The execution subject of the following steps may be the terminal or server mentioned above.

[0159] In step S801, the amount of data currently cached in the local shared memory is detected according to a specific detection cycle;

[0160] Here, the detection period may be a value preset by the user, a default value, or a value dynamically determined according to the amount of data currently cached in the local shared memory, and is not limited here.

[0161] In step S802 , the size of the specific quantile is dynamically adjusted based on the amount of cached data.

[0162] Here, the size of the specific quantile can be dynamically adjusted based on the amount of data currently cached in the local shared memory. During implementation, those skilled in the art can dynamically adjust the specific quantile in an appropriate manner according to actual conditions, which is not limited here. For example, the initial value of the specific quantile can be pre-set, and the value of the specific quantile can be increased as the amount of data currently cached in the local shared memory increases, and the value of the specific quantile can be decreased as the amount of data currently cached in the local shared memory decreases; or each time the target key value name to be queried is obtained, the value of the specific quantile can be increased when it is determined that the amount of data currently cached in the local shared memory is greater than a preset data volume threshold, and the value of the specific quantile can be increased when it is determined that the amount of data currently cached in the local shared memory is less than a preset data volume threshold.

[0163] It should be noted that the above steps S801 to S802 are not limited to the following when being implemented. Figure 8 The execution order shown, for example, the above steps S801 to S802 may also be executed before step S101.

[0164] In some embodiments, the cached data amount includes the number of cached key-value names, and the above step S802 may include at least one of the following steps S821 and S822:

[0165] In step S821, when it is detected that the number of cached key-value names is less than a first quantity threshold, the specific quantile is reduced by a first adjustment step size;

[0166] Here, the amount of cached data may include the number of cached key-value names, and the first quantity threshold may be a value preset by the user, a default value, or a value dynamically determined based on the amount of data currently cached in the local shared memory. The first adjustment step may be a value preset by the user, a default value, or a value dynamically determined based on the amount of data currently cached in the local shared memory. When it is detected that the number of cached key-value names is less than the first quantity threshold, the specific quantile may be reduced by the first adjustment step. For example, assuming that the first quantity threshold is 50 and the first adjustment step is 5%, when it is detected that the number of cached key-value names is less than 50, the specific quantile may be reduced by 5%.

[0167] In step S822 , when it is detected that the number of cached key-value names is greater than a second number threshold, the specific quantile is increased by a second adjustment step.

[0168] Here, the second quantity threshold can be a value preset by the user, a default value, or a value dynamically determined based on the amount of data currently cached in the local shared memory. The second adjustment step can be a value preset by the user, a default value, or a value dynamically determined based on the amount of data currently cached in the local shared memory. When it is detected that the number of cached key-value names is greater than the second quantity threshold, the specific quantile can be increased by the second adjustment step. For example, assuming that the second quantity threshold is 1000 and the second adjustment step is 3%, when it is detected that the number of cached key-value names is greater than 1000, the specific quantile can be increased by 3%.

[0169] In the embodiments of the present application, the amount of data currently cached in the local shared memory is detected according to a specific detection cycle, and the size of the specific quantile is dynamically adjusted based on the amount of cached data. In this way, the amount of data cached in the local shared memory can be controlled within an appropriate range, thereby fully utilizing the local shared memory for data caching. The upper limit of the local shared memory usage can be taken into account, and thus appropriate data caching services can be provided for systems with different cache capabilities, thereby achieving better versatility.

[0170] Below, an exemplary application of an embodiment of the present application in an actual application scenario will be described. Taking data caching in a high-concurrency scenario as an example, an embodiment of the present application provides a data query method that can automatically determine hot key data and large key data, cache the data corresponding to the hot key data and large key data into a local shared memory, and can synchronize large key data or hot key data in a high-concurrency scenario to ensure data consistency. This method can be applied to high-concurrency scenarios where a single key in the background data corresponds to an overall submitted content, and it is necessary to ensure overall data consistency, it is inconvenient to split it into smaller pieces, and the value data size is large or hot. In this scenario, reading data from the data provider every time will affect system performance, and caching data on each machine of a distributed service is difficult to ensure data consistency. This method can also be applied to scenarios of data synchronization between data providers and maintainers such as general data configuration and content management background and high-concurrency services.

[0171] See also Figure 9 , Figure 9A schematic diagram of the implementation architecture of the data query method provided in an embodiment of the present application. The method interacts with the data provider 910 through the synchronization module 920 to obtain the original KV data, and saves the original KV data with the version number to the database 930. The master agent 940 and the slave agent (exemplarily shown as 941, 942 and 943) are separate process services running on different servers (exemplarily shown as 950, 951, 952 and 953) in the distributed service, and can serve as agents of the server (exemplarily shown as 960, 961, 962 and 963). The master agent 940 will poll or query the database 930 at regular intervals, and determine whether there is any data update based on the data version. If there is any data update, the local data will be updated and saved in the local shared memory 970. Then the master agent 940 will notify each slave agent to pull the latest data and update it. After receiving notifications from the master agent 940, the slave agent can pull updated data from the master agent 940 through an interface and update the cached data in the local shared memory (illustrated by 971, 972, and 973). Furthermore, the slave agent can periodically pull the latest data to ensure eventual data consistency across all machines. Newly added slave agents must register with the master agent, and the master agent regularly checks the availability of each slave agent. The master and slave agents can provide a KV data query interface similar to the Get method, which the server can call to retrieve data cached in the local shared memory. The server also includes a memory cache (illustrated by 980, 981, 982, and 983), which enables caching of query data in the server's memory. The memory cache's expiration rules can be configured through other configuration platforms. This data query interface has a built-in fallback policy. When the memory cache or shared memory cache expires, it will fall back to a direct connection to the database and cache the data retrieved from the database. Among them, the interaction information between the master agent 940 and the slave agent, the slave agent and the local shared memory, and the server and the local shared memory, such as the number of cache calls, the number of cache invalidations, the number of return to the source, etc., can all be detected through the detection platform 990 (such as Tencent Cloud Monitor Platform, etc.).

[0172] In some implementations, access statistics can be added to the data query interface. Statistics can be collected for individual machines or across all machines. The statistics include the amount of data corresponding to each key and the frequency of queries. When a key's corresponding data is found to be high-value data within a certain time interval, or when the key's access frequency reaches a certain threshold, the slave agent reports the key to the master agent, instructing the master agent to update and synchronize the key's value. Upon receiving the request, the master agent adds the key to an update list. The next time the master agent updates, the key's value is updated and sent to the slave agent. The master agent also counts the value of each key in the update list. Furthermore, for each key in the update list, if the master agent does not receive a message reporting that key from a slave agent within a certain period of time, it automatically deletes the key from the update list. Small or non-hot key data can be cached or cached locally without global cache synchronization. The slave agent can also register a lower-level agent, thus forming a tree-like master-slave agent structure.

[0173] In some implementations, considering the different query frequencies of keys in different systems and the different system cache capacities, the traditional solution of determining the keys that need to be cached by setting a frequency threshold cannot effectively adapt to different systems and is not universal. Therefore, from a statistical perspective, by examining the relationship between the actual cached data size and the system's tolerance, and comprehensively considering the situation of hot keys / large keys, it is possible to adaptively determine whether the key currently requested for query needs to be cached. Assuming the system memory size is M, the total cached data volume C and the statistical information data volume D should satisfy the following formula 1-1:

[0174] (1-1);

[0175] Among them, n is the number of keys currently cached, is the size of the key-value pair corresponding to the i-th key in the cache; a is a coefficient used to represent the upper limit of the usage of local shared memory that should not be exceeded, such as 80%, 90%, etc. The amount of data occupied by the statistical information of the i-th key in the shared memory. The statistical information can include the number of queries and data volume of the i-th key.

[0176] Set a time window T, for example, within the last 5 minutes, and count the number of visits to each key in this time window T , and calculate the quantile of the data throughput size of each key queried within the time window T. Let the i-th key size be , the number of queries in the time window T is , the throughput of the system input / output (IO) data required from the start to the current for the i-th key within the time window T is , denoted as , the total number of queries for each key queried within the time window T is , denoted as . For a request for a current key, the conditions for determining whether caching is required may include: when the total number of queries for each key within the time window T reaches a certain threshold, that is, when , where b is the query count threshold, caching is considered.

[0177] In addition, assuming that within the time window T, the data throughput of each key queried follows an empirical distribution function P, the q-quantile of this distribution function can be calculated as , such as 80%, 90%, etc. When the data throughput of the i-th key satisfies the following formula 1-2, the data corresponding to the current key is cached, and the number of currently cached keys is incremented by 1:

[0178] (1-2);

[0179] To prevent the number of caches from being too small, the number of cached keys can be periodically detected here, and the quantile q can be dynamically adjusted. When n < N, then q = q - step can be adjusted, where step is the step size, which can be preset, such as setting step = 5%.

[0180] When caching each key, the corresponding cache expiration time can be dynamically adjusted according to the data throughput of each key within the time window T.

[0181] In the embodiments of the present application, for scenarios with high concurrency and large keys / hot keys, the pressure on the database and / or network bandwidth can be reduced. At the same time, for multi-threaded / multi-process services, each machine has only one local cache data copy placed in shared memory, which can ensure data consistency. In addition, from a statistical perspective, by checking the relationship between the actual amount of cached data and the system's bearing capacity, and comprehensively considering the situation of hot keys / large keys, it can adaptively determine whether the key queried by the current request needs to be cached, which can better adapt to systems with different business scenarios and caching capabilities and has better generality.

[0182] Next, the implementation of the data processing device 255 provided in the embodiments of the present application as an exemplary structure of software modules will be continued. In some embodiments, as Figure 2 shown, the software modules stored in the data processing device 255 in the memory 250 may include:

[0183] The first acquisition module 2551 is used to obtain the target key value name to be queried;

[0184] A first determining module 2552 is configured to determine the data throughput corresponding to the target key name in the current statistical period;

[0185] The second determining module 2553 is configured to determine a throughput threshold corresponding to a specific quantile based on the data throughput corresponding to each key name queried in the current statistical period; the specific quantile is dynamically determined based on the amount of data currently cached in the local shared memory;

[0186] The cache module 2554 is used to cache the target key name and the key value corresponding to the target key name in the local shared memory when it is determined that the data throughput corresponding to the target key name in the current statistical period reaches the throughput threshold.

[0187] In some embodiments, the second determination module is further used to: determine the data throughput corresponding to each key value name queried within the current statistical period within the current statistical period; and determine the data throughput corresponding to a specific quantile in the sorting results of the data throughput corresponding to each key value name from small to large as the throughput threshold.

[0188] In some embodiments, the device also includes: a first notification module, used to send a data cache notification to at least one slave agent, so that the at least one slave agent caches the target key name and the key value corresponding to the target key name into the local shared memory of the at least one slave agent.

[0189] In some embodiments, the first determination module is also used to: query the local shared memory to obtain the total number of queries for the target key name in the current statistical period and the average data volume of the key value corresponding to the target key name; based on the total number of queries and the average data volume, determine the data throughput corresponding to the target key name in the current statistical period.

[0190] In some embodiments, the device also includes: a second acquisition module, used to obtain the number of single-machine queries for each key-value name queried and the single-machine average data volume of the key value corresponding to each key-value name reported by at least one slave agent according to a reporting period; a third acquisition module, used to obtain the number of single-machine queries for each key-value name queried by the master agent in the current statistical period and the single-machine average data volume of the key value corresponding to each key-value name; a summary module, used to summarize the number of single-machine queries for each key-value name queried by the master agent and each key-value name reported by each slave agent in the current statistical period and the single-machine average data volume of the key value corresponding to each key-value name, and obtain the total number of queries for each key-value name queried in the current statistical period and the average data volume of the key value corresponding to each key-value name; a storage module, used to store the total number of queries for each key-value name queried in the current statistical period and the average data volume of the key value corresponding to each key-value name in the local shared memory.

[0191] In some embodiments, the cache module is also used to: determine the amount of data currently cached in the local shared memory; determine the current cache usage rate based on the amount of data currently cached and the total memory size of the local shared memory; and cache the target key name and the key value corresponding to the target key name in the local shared memory when it is determined that the data throughput corresponding to the target key name in the current statistical period reaches the throughput threshold and the current cache usage rate is less than the usage threshold.

[0192] In some embodiments, the cache module is also used to: query the local shared memory to obtain the total query times of each key value name queried in the current statistical period in the current statistical period; sum the total query times of each key value name in the current statistical period to obtain the total request times in the current statistical period; when it is determined that the data throughput corresponding to the target key value name in the current statistical period reaches the throughput threshold, the current cache usage rate is less than the usage threshold and the total request times reaches the number threshold, cache the target key value name and the key value corresponding to the target key value name in the local shared memory.

[0193] In some embodiments, the device further includes: an adding module for adding the target key value name to a cached key value name list; an updating module for updating each key value name in the cached key value name list, when it is determined that the key value corresponding to the key value name has been updated in the data source, the updated key value to the local shared memory.

[0194] In some embodiments, the update module is also used to: periodically query the data source to determine if the key value corresponding to the key value name is updated, and then obtain the updated key value; update the updated key value to the local shared memory; send a data update notification to at least one slave agent, so that after receiving the data update notification, the at least one slave agent obtains the updated key value corresponding to each key value name in the cached key value name list from the master agent.

[0195] In some embodiments, the device further includes: a detection module for detecting the amount of data currently cached in the local shared memory according to a specific detection period; and an adjustment module for dynamically adjusting the size of the specific quantile based on the amount of cached data.

[0196] In some embodiments, the amount of cached data includes the number of cached key-value names, and the adjustment module is further used to perform at least one of the following: when it is detected that the number of cached key-value names is less than a first quantity threshold, the specific quantile is reduced by a first adjustment step; when it is detected that the number of cached key-value names is greater than a second quantity threshold, the specific quantile is increased by a second adjustment step.

[0197] In some embodiments, the cache module is also used to: determine the duration adjustment parameter corresponding to the target key name based on the data throughput corresponding to the target key name in the current statistical period; determine the cache expiration duration of the target key name based on the duration adjustment parameter and a preset cache duration range; and cache the target key name and the key value corresponding to the target key name into local shared memory according to the cache expiration duration.

[0198] The present invention provides a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the data processing method described above in the present invention.

[0199] The embodiment of the present application provides a computer-readable storage medium storing executable instructions, wherein the executable instructions are stored. When the executable instructions are executed by a processor, the processor will execute the data processing method provided by the embodiment of the present application, for example, Figure 3 The method shown.

[0200] In some embodiments, the computer-readable storage medium may be a memory such as FRAM, ROM, PROM, EP-ROM, EEPROM, flash memory, magnetic surface storage, optical disk, or CD-ROM; or various devices including one or any combination of the above memories.

[0201] In some embodiments, executable instructions may be in the form of a program, software, software module, script, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0202] As an example, executable instructions may, but need not, correspond to a file in a file system, may be stored as part of a file that stores other programs or data, such as in one or more scripts in a HyperText Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple coordinating files (e.g., files storing one or more modules, subroutines, or code portions).

[0203] By way of example, executable instructions may be deployed to be executed on one computing device, or on multiple computing devices at one site, or on multiple computing devices distributed across multiple sites and interconnected by a communication network.

[0204] To sum up, through the embodiments of the present application, on the one hand, the data corresponding to the key-value names with larger data throughput can be automatically cached in the local shared memory, thereby reducing the occupancy of the local memory on the server side, improving the performance of the server side, and reducing the pressure on the database and network, thereby improving data query performance; on the other hand, for systems with different business scenarios and different caching capabilities, appropriate data caching services can be provided, which has better versatility.

[0205] The above description is merely an embodiment of the present application and is not intended to limit the scope of protection of the present application. Any modifications, equivalent replacements, and improvements made within the spirit and scope of the present application are included in the scope of protection of the present application.

Claims

1. A data processing method, characterized in that: include: Get the target key value name to be queried; Determine the data throughput corresponding to the target key name in the current statistical period; Determining a throughput threshold corresponding to a specific quantile based on the data throughput corresponding to each key value name queried during the current statistical period; the specific quantile is dynamically adjusted based on the amount of data currently cached in the local shared memory; When it is determined that the data throughput corresponding to the target key name in the current statistical period reaches the throughput threshold, caching the target key name and the key value corresponding to the target key name in the local shared memory; Among them, determining the data throughput corresponding to the target key name in the current statistical period includes: querying the local shared memory to obtain the total number of queries for the target key name in the current statistical period and the average data volume of the key value corresponding to the target key name; based on the total number of queries and the average data volume, determining the data throughput corresponding to the target key name in the current statistical period.

2. The method according to claim 1, characterized in that The determining, based on the data throughput corresponding to each key name queried in the current statistical period in the current statistical period, a throughput threshold corresponding to a specific quantile includes: Determine the data throughput corresponding to each key value name queried in the current statistical period in the current statistical period; The data throughput corresponding to the specific quantile in the results of sorting the data throughput corresponding to each key name from small to large is determined as the throughput threshold.

3. The method according to claim 1, characterized in that After caching the target key name and the key value corresponding to the target key name in the local shared memory, the method further includes: A data cache notification is sent to at least one slave agent, so that the at least one slave agent caches the target key name and the key value corresponding to the target key name in a local shared memory of the at least one slave agent.

4. The method according to claim 1, wherein The method further comprises: Obtain at least one single-machine query count for each key-value name being queried and a single-machine average data volume of the key value corresponding to each key-value name, reported from the proxy terminal according to a reporting period; Obtain the number of single-machine queries for each key-value name queried by the main agent in the current statistical period and the single-machine average data volume of the key value corresponding to each key-value name; Summarizing the number of single-machine queries for each key-value name queried by the master agent and each key-value name reported by each slave agent during the current statistical period, as well as the average single-machine data volume of the key value corresponding to each key-value name, to obtain the total number of queries for each key-value name queried during the current statistical period and the average data volume of the key value corresponding to each key-value name; The total number of queries for each key name queried in the current statistical period and the average data volume of the key value corresponding to each key name are stored in the local shared memory.

5. The method according to claim 1, wherein The method of caching the target key name and the key value corresponding to the target key name in the local shared memory when it is determined that the data throughput corresponding to the target key name in the current statistical period reaches the throughput threshold includes: Determining the amount of data currently cached in the local shared memory; Determining a current cache usage rate based on the amount of data currently cached and the total memory size of the local shared memory; When it is determined that the data throughput corresponding to the target key name in the current statistical period reaches the throughput threshold and the current cache usage is less than the usage threshold, the target key name and the key value corresponding to the target key name are cached in the local shared memory.

6. The method according to claim 5, characterized in that The method of caching the target key name and the key value corresponding to the target key name in the local shared memory, when determining that the data throughput corresponding to the target key name in the current statistical period reaches the throughput threshold and the current cache usage is less than the usage threshold, includes: Query the local shared memory to obtain the total query times of each key value name queried in the current statistical period in the current statistical period; Sum the total number of queries for each key name in the current statistical period to obtain the total number of requests in the current statistical period; When it is determined that the data throughput corresponding to the target key name in the current statistical period reaches the throughput threshold, the current cache usage is less than the usage threshold and the total number of requests reaches the number threshold, the target key name and the key value corresponding to the target key name are cached in the local shared memory.

7. The method according to any one of claims 1 to 6, characterized in that The method further comprises: Add the target key value name to the cached key value name list; For each key-value name in the cached key-value name list, when it is determined that the key value corresponding to the key-value name is updated in the data source, the updated key value is updated to the local shared memory.

8. The method according to claim 7, characterized in that When it is determined that the key value corresponding to the key value name is updated in the data source, updating the updated key value to the local shared memory includes: By periodically querying the data source, if it is determined that the key value corresponding to the key value name is updated, obtaining the updated key value; Updating the updated key value into the local shared memory; A data update notification is sent to at least one slave agent, so that after receiving the data update notification, the at least one slave agent obtains an updated key value corresponding to each key value name in the cached key value name list from the master agent.

9. The method according to any one of claims 1 to 6, characterized in that The method further comprises: Detecting the amount of data currently cached in the local shared memory according to a specific detection cycle; The size of the specific quantile is dynamically adjusted based on the amount of cached data.

10. The method according to claim 9, characterized in that The cached data volume includes the number of cached key-value names, and dynamically adjusting the size of the specific quantile based on the cached data volume includes at least one of the following: When it is detected that the number of cached key-value names is less than a first quantity threshold, reducing the specific quantile by a first adjustment step size; When it is detected that the number of cached key-value names is greater than a second number threshold, the specific quantile is increased by a second adjustment step.

11. The method according to any one of claims 1 to 6, characterized in that: The step of caching the target key name and the key value corresponding to the target key name in the local shared memory includes: Determining a duration adjustment parameter corresponding to the target key name based on the data throughput corresponding to the target key name in the current statistical period; Determining a cache expiration time for the target key name based on the time adjustment parameter and a preset cache time range; According to the cache expiration time, the target key name and the key value corresponding to the target key name are cached in the local shared memory.

12. A data processing device, characterized in that: include: The first acquisition module is used to obtain the target key value name to be queried; A first determining module is used to determine the data throughput corresponding to the target key name in the current statistical period; A second determination module is configured to determine a throughput threshold corresponding to a specific quantile based on the data throughput corresponding to each key name queried during the current statistical period; the specific quantile is dynamically adjusted based on the amount of data currently cached in the local shared memory; a cache module configured to cache the target key name and the key value corresponding to the target key name in the local shared memory when it is determined that the data throughput corresponding to the target key name in the current statistical period reaches the throughput threshold; Among them, determining the data throughput corresponding to the target key name in the current statistical period includes: querying the local shared memory to obtain the total number of queries for the target key name in the current statistical period and the average data volume of the key value corresponding to the target key name; based on the total number of queries and the average data volume, determining the data throughput corresponding to the target key name in the current statistical period.

13. The device according to claim 12, characterized in that The second determining module is further configured to: determine the data throughput corresponding to each key value name queried in the current statistical period in the current statistical period; The data throughput corresponding to the specific quantile in the results of sorting the data throughput corresponding to each key value name from small to large is determined as the throughput threshold.

14. A data processing device, characterized in that: include: a memory for storing executable instructions; A processor, configured to implement the method according to any one of claims 1 to 11 when executing the executable instructions stored in the memory.

15. A computer-readable storage medium, characterized in that Executable instructions are stored, and when executed by a processor, they are used to implement the method described in any one of claims 1 to 11.

16. A computer program product comprising a computer program or instructions, characterized in that When the computer program or instruction is executed by a processor, the method according to any one of claims 1 to 11 is implemented.

Images