A data processing method, apparatus, electronic device, and storage medium
By introducing distributed cache and local cache in the cloud computing environment and adopting GuavaCache asynchronous cache strategy, the performance bottlenecks and information asymmetry of traditional databases in massive data processing are solved, efficient data storage and query, and memory is slowed down Pressure, improve system performance.
Patent Information
- Application Number
- CN202111215630.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-10-19
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2041-10-19
AI Technical Summary
Traditional databases face performance bottlenecks when storing massive data and processing requests in cloud computing environments, especially due to information asymmetry and memory leaks caused by untimely data synchronization between distributed caches and local caches.
A data processing method is proposed. By introducing distributed cache and local cache in the cluster, and using GuavaCache asynchronous cache strategy, it ensures that the new data is synchronized to the local cache of each node within the valid data time, and the valid data time is the first time period 2 times, query the distributed cache first, and query the local cache if it is not stored to avoid memory leakage.
Effectively slows down memory pressure, improves system performance, ensures data consistency and avoids memory leaks, and improves the efficiency of data query.
Smart Images

Figure CN114048234B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of computer technology, and in particular to a data processing method, device, electronic device and storage medium. Background Art
[0002] In the cloud computing environment, the storage and processing requests of massive data have put great pressure on the traditional database structure, and traditional databases face the performance bottleneck problem of supporting access to large amounts of data. Summary of the invention
[0003] The present application provides a data processing method, device, electronic device and storage medium to at least solve the above technical problems existing in the prior art.
[0004] The first aspect of the present application provides a data processing method, comprising:
[0005] Responding to a target data query request from a first node in the cluster;
[0006] Confirming whether the target data corresponding to the target data query request is stored in the distributed cache of the cluster;
[0007] If the target data is not stored in the distributed cache, querying the target data in the local cache corresponding to the first node;
[0008] Among them, the newly added data of each node in the cluster is stored in the distributed cache within the data validity period, and the newly added data is synchronized to the local cache of each node in a first time period; the data validity period is greater than the first time period.
[0009] In the above solution, the length of the data valid time is twice the length of the first time period.
[0010] In the above solution, the newly added data is synchronized to the local cache of each node in the first time period, including:
[0011] The local cache of each node in the cluster is based on the GuavaCache asynchronous cache strategy, and performs data synchronization in the first time period.
[0012] In the above solution, if the target data is stored in the distributed cache, the target data is sent.
[0013] A second aspect of the present application provides a data processing device, including:
[0014] A response unit, configured to respond to a target data query request from a first node in the cluster;
[0015] A confirmation unit, used to confirm whether the target data corresponding to the target data query request is stored in the distributed cache of the cluster;
[0016] a query unit, configured to query the target data in a local cache corresponding to the first node if the target data is not stored in the distributed cache;
[0017] Among them, the newly added data of each node in the cluster is stored in the distributed cache within the data validity period, and the newly added data is synchronized to the local cache of each node in a first time period; the data validity period is greater than the first time period.
[0018] In the above solution, the length of the data valid time is twice the length of the first time period.
[0019] In the above solution, the device further comprises:
[0020] The synchronization unit is used to synchronize data of the local cache of each node in the cluster in the first time period based on the GuavaCache asynchronous cache strategy.
[0021] In the above solution, the device further comprises:
[0022] A sending unit is used to send the target data if the target data is stored in the distributed cache.
[0023] The third aspect of the present application provides an electronic device, including a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; the memory is used to store computer programs; and the processor is used to implement the method steps described in the above data processing method when executing the program stored in the memory.
[0024] A fourth aspect of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method steps described in the above data processing method are implemented.
[0025] The data processing method provided by the present application responds to a target data query request of a first node in a cluster; confirms whether the target data corresponding to the target data query request is stored in the distributed cache of the cluster; if the target data is not stored in the distributed cache, queries the target data in the local cache corresponding to the first node; wherein, the newly added data of each node in the cluster is stored in the distributed cache within the data validity time, and the newly added data is synchronized to the local cache of each node in a first time period; the data validity time is greater than the first time period. The data processing method proposed by the present invention effectively alleviates memory pressure and improves system performance. BRIEF DESCRIPTION OF THE DRAWINGS
[0026] Figure 1 An optional flow chart of the data processing method provided in the embodiment of the present application is shown;
[0027] Figure 2 Another optional flow chart of the data processing method provided in the embodiment of the present application is shown;
[0028] Figure 3 An optional structural diagram of a data processing device provided in an embodiment of the present application is shown;
[0029] Figure 4 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application is shown. DETAILED DESCRIPTION
[0030] In order to make the purpose, features, and advantages of the present application more obvious and easy to understand, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present application.
[0031] In a cloud computing environment, the cache mechanism in the database includes at least the local cache of each node in the cluster and the distributed cache in the cluster. The local cache is the cache corresponding to each node in the cluster. The local cache of each node is broadcast in the cluster at a certain time period to complete data synchronization. The distributed cache is shared by the cluster.
[0032] In the solution based on caching data on local nodes in the cluster, it may happen that after node A adds data a, the cluster has not yet had time to broadcast the corresponding synchronization data information to other nodes in the cluster, and node B receives a data query request for data a. Therefore, the data cannot be found in the local cache of node B, resulting in data asymmetry between nodes in the cluster.
[0033] Although distributed cache can solve this problem well, when the volume of new data in each node is large, it will put pressure on the distributed cache and cause memory leaks.
[0034] In view of the defects in data storage and acquisition in the prior art, the present application provides a data processing method that can at least solve some or all of the above-mentioned technical problems.
[0035] Figure 1 An optional flow chart of the data processing method provided in an embodiment of the present application is shown, and will be explained according to each step.
[0036] Step S101, responding to a target data query request from a first node in a cluster.
[0037] The cluster includes at least one node, and the first node may be any node included in the cluster.
[0038] In some embodiments, a data processing device (hereinafter referred to as the device) executes step S102 in response to a target data query request from a first node in the cluster.
[0039] Step S102: confirm whether the target data corresponding to the target data query request is stored in the distributed cache of the cluster.
[0040] In some embodiments, the device confirms whether the target data corresponding to the target data query request is stored in the distributed cache of the cluster; if the target data is stored in the distributed cache of the cluster, the target data is sent; or, if the target data is not stored in the distributed cache of the cluster, step S103 is executed.
[0041] Among them, the newly added data of any node in the cluster is stored in the distributed cache within the data validity period, and the newly added data is synchronized to the local cache of each node in a first time period; the data validity period is greater than the first time period.
[0042] In some embodiments, when new data is added to any node in the cluster, the new data is stored in the database and distributed cache of the cluster; the local cache of each node in the cluster synchronizes data with the database in a first time period, thereby completing data synchronization of all nodes in the cluster.
[0043] In some optional embodiments, the length of the data valid time is twice the first time period; optionally, the quotient between the length of the data valid time and the first time period can also be other values greater than 1, and this application does not make specific restrictions.
[0044] Step S103: If the target data is not stored in the distributed cache, query the target data in the local cache corresponding to the first node.
[0045] In some embodiments, if the target data is not stored in the distributed cache, it means that the target data has been stored in the distributed cache for a period longer than the data validity period, and the target data has been deleted from the distributed cache; the device queries the target data in the local cache corresponding to the first node.
[0046] In some optional embodiments, after the device queries the target data in the local cache corresponding to the first node, it may also send the target data.
[0047] In this way, through the data processing method provided by the embodiment of the present application, the data validity time of the data in the distributed cache is greater than the first time period, so that when querying data, the distributed cache is queried first, and if the target data does not exist in the distributed cache, the local cache corresponding to the first node is queried. The distributed cache is queried first, and the local cache is queried only if the target data is not stored in the distributed cache. This ensures that the information in the local cache has been updated at least once before querying the local cache, preventing the problem of information asymmetry caused by untimely broadcasting. At the same time, the data in the distributed cache has a data validity time, and the data outside the data validity time is deleted, which can avoid memory leaks in the distributed cache due to excessive data volume.
[0048] Figure 2 Another optional flow chart of the data processing method provided in the embodiment of the present application is shown.
[0049] Figure 2 In the embodiment, taking the newly added cache of the first node in the cluster as an example, data storage and acquisition are performed. Those skilled in the art should know that the data processing method provided in the embodiment of the present application is applicable to data storage and acquisition of any node in any cluster.
[0050] Step S201: in response to a request from a first node to add first data, the first data is stored in a database of the cluster and a distributed cache of the cluster respectively.
[0051] In some embodiments, a data processing device (hereinafter referred to as the device) stores the first data in a database of the cluster and a distributed cache of the cluster in response to a request from the first node to add the first data, respectively. The distributed cache is a cache shared by each node in the cluster.
[0052] In some optional embodiments, the device may further set a data validity period for storing the first data in the distributed cache; within the data validity period, the first data is stored in the distributed cache; outside the data validity period, the device or the cluster deletes the first data from the distributed cache. Optionally, the data validity periods of different data may be the same or different.
[0053] In some optional embodiments, the device or the cluster may periodically delete data outside the data validity period in the distributed cache; the device or the cluster may also monitor the data validity period of each piece of data, and delete the data from the distributed cache when the data validity period is reached.
[0054] The first data may be the target data involved in step S101 to step S103, or other data.
[0055] Step S202: synchronize first data in the database of the cluster to the local cache of the first node in a first time period.
[0056] In some embodiments, the device synchronizes first data in a database of the cluster to a local cache of the first node at a first time period.
[0057] In specific implementation, the device can synchronize the first data in the database to the local cache of the first node in the first time period based on the GuavaCache asynchronous cache strategy. In some optional embodiments, the device can also synchronize the first data in the database of the cluster to the local cache of at least two nodes of the cluster in the first time period.
[0058] In specific implementation, the device can synchronize the first data in the database to the local cache of at least two nodes in the cluster in the first time period based on the GuavaCache asynchronous caching strategy, so as to avoid blocking of the data updating thread and other threads (such as the thread corresponding to the data query request or other data synchronization threads).
[0059] Among them, the first time period is less than the data validity period; optionally, the length of the data validity period may be twice the first time period. For example, the data validity period of the first data stored in the distributed cache is 10 minutes, and the first period is 5 minutes. Those skilled in the art should understand that the above 10 minutes and 5 minutes are only examples, and may also be any other value (such as 6 minutes, 3 minutes), etc., and this application does not make specific limitations.
[0060] Step S203 , in response to a target data query request from a second node in the cluster, confirming whether target data corresponding to the target data query request is stored in the distributed cache of the cluster.
[0061] In some embodiments, the device responds to a target data query request from a second node in the cluster; confirms whether the target data corresponding to the target data query request is stored in the distributed cache of the cluster; if the target data is stored in the distributed cache of the cluster, sends the target data; or, if the target data is not stored in the distributed cache of the cluster, it means that the target data has been stored in the distributed cache for a period longer than the data validity period, and the target data has been deleted from the distributed cache.
[0062] In some optional embodiments, after the device queries the local cache corresponding to the target node or the distributed cache for the target data, it may also send the target data.
[0063] The data between two synchronizations of nodes in the cluster cannot be queried in the local cache (e.g., the local cache of the node is synchronized once every 5 minutes, the first synchronization is performed at the 1st minute, and the second synchronization is performed at the 6th minute. In the time zone from the 1st minute to the 6th minute, some data is only stored in the database and the distributed cache, and not in the local cache). Assuming that the data is added in the 2nd minute, the data query request that appears within 2 to 12 minutes can obtain the result in the distributed cache; after 12 minutes, the data in the distributed cache is lost. Since the local cache has been synchronized with each node in the cluster, it can be queried from the local cache of the node where the data query request is initiated. Therefore, in response to the data query request, the distributed cache is queried first. On the one hand, it alleviates the problem of large-scale data access faced by the local cache of the database or each node in the cluster; on the other hand, since the data validity time of the data in the distributed cache is greater than the first time period, it can be guaranteed that the corresponding data can be queried, that is, the data is not lost in the cluster (for example, the data validity time has expired, but the node has not synchronized the corresponding data).
[0064] In some optional embodiments, after the device queries the local cache corresponding to the target node or the distributed cache for the target data, it may also send the target data.
[0065] In this way, through the data processing method provided by the embodiment of the present application, the data validity time of the data in the distributed cache is greater than the first time period, so that when querying data, the distributed cache is queried first. If the target data does not exist in the distributed cache, the local cache corresponding to the target node is queried. On the one hand, through the distributed cache, the problem of large-scale data access faced by the database or the local cache of each node in the cluster is alleviated; on the other hand, the data in the distributed cache has a data validity time, and the data outside the data validity time is deleted, which can avoid memory leaks caused by excessive data volume in the distributed cache; on the third hand, the distributed cache is queried first, and the local cache is queried if the target data is not stored in the distributed cache. After the node synchronizes the data, the cluster can broadcast the information of the synchronized data to other nodes. After the node receives the data query request, it can know the specific node of the corresponding data storage, avoiding the problem of information asymmetry between nodes in the cluster.
[0066] Figure 3 An optional structural diagram of a data processing device provided in an embodiment of the present application is shown, and will be explained according to each part.
[0067] In some embodiments, the data processing device 300 includes: a response unit 301 , a confirmation unit 302 , and a query unit 303 .
[0068] The response unit 301 is used to respond to the target data query request of the first node in the cluster;
[0069] The confirmation unit 302 is used to confirm whether the target data corresponding to the target data query request is stored in the distributed cache of the cluster;
[0070] The query unit 303 is configured to query the target data in a local cache corresponding to the first node if the target data is not stored in the distributed cache;
[0071] Among them, the local cache of each node in the cluster synchronizes data in a first time period; the data stored in the distributed cache has a data validity period; the data validity period of the target data stored in the distributed cache is greater than the first time period.
[0072] In some embodiments, the length of the data valid time is twice the length of the first time period.
[0073] In some embodiments, the data processing device 300 may further include a synchronization unit 304 .
[0074] The synchronization unit 304 is used to synchronize data of the local cache of each node in the cluster in the first time period based on the GuavaCache asynchronous cache strategy.
[0075] In some embodiments, the data processing device 300 may further include a sending unit 305 .
[0076] The sending unit 305 is configured to send the target data if the target data is stored in the distributed cache.
[0077] In some embodiments, the data processing device 300 may further include a cache unit 306 .
[0078] The cache unit 306 is used to store the first data in the database of the cluster and the distributed cache of the cluster respectively in response to the request of the first node to add the first data.
[0079] The cache unit 306 is specifically configured to store the first data in the database of the cluster and the distributed cache of the cluster in response to the request of the first node to add the first data, respectively. The distributed cache may be a cache shared by each node in the cluster.
[0080] The cache unit 306 is specifically used to set the data validity period of the first data stored in the distributed cache; within the data validity period, the first data is stored in the distributed cache; outside the data validity period, the first data is deleted from the distributed cache. Optionally, the data validity periods of different data can be the same or different.
[0081] Figure 4 The schematic diagram of the hardware structure of the electronic device provided by the embodiment of the present application is shown. The electronic device 700 includes: at least one processor 701, a memory 702 and at least one network unit 704. The various components in the electronic device 700 are coupled together through a bus system 705. It can be understood that the bus system 705 is used to realize the connection and communication between these components. In addition to the data bus, the bus system 705 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, the bus system 705 is not used in the following examples. Figure 4 Various buses are labeled as bus system 705.
[0082] It can be understood that the memory 702 can be a volatile memory or a non-volatile memory, and can also include both volatile and non-volatile memories. Among them, the non-volatile memory can be ROM, programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), magnetic random access memory (FRAM), ferromagnetic random access memory, flash memory, magnetic surface memory, optical disk, or compact disc read-only memory (CD-ROM); the magnetic surface memory can be a disk memory or a tape memory. The volatile memory can be a random access memory (RAM), which is used as an external cache. By way of example but not limitation, many forms of RAM are available, such as static random access memory (SRAM), synchronous static random access memory (SSRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), direct memory bus random access memory (DRRAM). The memory 702 described in the embodiments of the present application is intended to include but is not limited to these and any other suitable types of memory.
[0083] The memory 702 in the embodiment of the present application is used to store various types of data to support the operation of the electronic device 700. Examples of such data include: any computer program for operating on the electronic device 700, such as the application 722. The program for implementing the method of the embodiment of the present application may be included in the application 722.
[0084] The method disclosed in the embodiment of the present application can be applied to the processor 701, or implemented by the processor 701. The processor 701 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the method can be completed by the hardware integrated logic circuit or software instructions in the processor 701. The processor 701 may be a general-purpose processor, a digital signal processor (DSP, Digital Signal Processor), or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. The processor 701 can implement or execute the methods, steps and logic block diagrams disclosed in the embodiment of the present application. The general-purpose processor may be a microprocessor or any conventional processor, etc. In combination with the steps of the method disclosed in the embodiment of the present application, it can be directly embodied as a hardware decoding processor to execute, or it can be executed by a combination of hardware and software modules in the decoding processor. The software module can be located in a storage medium, which is located in the memory 702, and the processor 701 reads the information in the memory 702 and completes the steps of the aforementioned method in combination with its hardware.
[0085] In an exemplary embodiment, the electronic device 700 can be implemented by one or more application specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), FPGAs, general-purpose processors, controllers, MCUs, MPUs, or other electronic components to execute the aforementioned method.
[0086] In addition to the above-mentioned methods and devices, an embodiment of the present application may also be a computer program product, which includes computer program instructions, which, when executed by a processor, enable the processor to execute the steps of the method according to various embodiments of the present application described in the above-mentioned "Exemplary Method" section of this specification.
[0087] The computer program product may be written in any combination of one or more programming languages to write program codes for performing the operations of the embodiments of the present application, including object-oriented programming languages, such as Java, C++, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as an independent software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server.
[0088] In addition, an embodiment of the present application may also be a computer-readable storage medium on which computer program instructions are stored. When the computer program instructions are executed by a processor, the processor executes the steps of the method according to various embodiments of the present application described in the above "Exemplary Method" section of this specification.
[0089] The computer readable storage medium can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can include, for example, but is not limited to, a system, device or device of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.
[0090] The basic principles of the present application are described above in conjunction with specific embodiments. However, it should be noted that the advantages, strengths, effects, etc. mentioned in the present application are only examples and not limitations, and it cannot be considered that these advantages, strengths, effects, etc. are required by each embodiment of the present application. In addition, the specific details disclosed above are only for the purpose of illustration and ease of understanding, not for limitation, and the above details do not limit the present application to being implemented by adopting the above specific details.
[0091] The block diagrams of the devices, apparatuses, equipment, and systems involved in this application are only illustrative examples and are not intended to require or imply that they must be connected, arranged, and configured in the manner shown in the block diagram. As will be appreciated by those skilled in the art, these devices, apparatuses, equipment, and systems can be connected, arranged, and configured in any manner. Words such as "including", "comprising", "having", etc. are open words, referring to "including but not limited to", and can be used interchangeably with them. The words "or" and "and" used here refer to the words "and / or" and can be used interchangeably with them, unless the context clearly indicates otherwise. The words "such as" used here refer to the phrase "such as but not limited to", and can be used interchangeably with them.
[0092] It should also be noted that in the apparatus, device and method of the present application, each component or each step can be decomposed and / or recombined. Such decomposition and / or recombination should be regarded as equivalent solutions of the present application.
[0093] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use the present application. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of the present application. Therefore, the present application is not intended to be limited to the aspects shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.
[0094] The above description has been given for the purpose of illustration and description. In addition, this description is not intended to limit the embodiments of the present application to the forms disclosed herein. Although multiple example aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, changes, additions and sub-combinations thereof.
Claims
1. A data processing method, characterized in that: The method comprises: In response to a request from the first node to add first data, storing the first data in a database of the cluster and a distributed cache of the cluster respectively; Synchronizing first data in a database of the cluster to local caches of at least two nodes in a first time period; In response to a target data query request from a second node in the cluster, confirming whether target data corresponding to the target data query request is stored in a distributed cache of the cluster; If the target data is not stored in the distributed cache, querying the target data in the local cache corresponding to the second node; Among them, the newly added data of any node in the cluster is stored in the distributed cache within the data validity period, and the newly added data is synchronized to the local cache of each node in a first time period; the data validity period is greater than the first time period.
2. The method according to claim 1, characterized in that: The data valid time is twice the first time period.
3. The method according to claim 1, characterized in that The newly added data is synchronized to the local cache of each node in a first time period, including: The local cache of each node in the cluster performs data synchronization based on the GuavaCache asynchronous cache strategy.
4. The method according to claim 1, characterized in that: If the target data is stored in the distributed cache, the target data is sent.
5. A data processing device, characterized in that: The device comprises: a cache unit, configured to respond to a request from the first node to add first data, and store the first data in a database of the cluster and a distributed cache of the cluster respectively; A synchronization unit, configured to synchronize first data in a database of the cluster to local caches of at least two nodes in a first time period; a response unit, configured to respond to a target data query request from a second node in the cluster; A confirmation unit, used to confirm whether the target data corresponding to the target data query request is stored in the distributed cache of the cluster; a query unit, configured to query the target data in a local cache corresponding to the second node if the target data is not stored in the distributed cache; Among them, the newly added data of any node in the cluster is stored in the distributed cache within the data validity period, and the newly added data is synchronized to the local cache of each node in a first time period; the data validity period is greater than the first time period.
6. The device according to claim 5, characterized in that The data valid time is twice the first time period.
7. The device according to claim 5, characterized in that The device also includes: The synchronization unit is used to synchronize data in the local cache of each node in the cluster based on the GuavaCache asynchronous cache strategy.
8. The device according to claim 5, characterized in that The device also includes: A sending unit is used to send the target data if the target data is stored in the distributed cache.
9. An electronic device, characterized in that: The invention comprises a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; the memory is used to store computer programs; and the processor is used to implement the method steps described in any one of claims 1 to 4 when executing the program stored in the memory.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method steps described in any one of claims 1 to 4 are implemented.
