Data processing method and apparatus, and device, database and computer program product
By sharing the data storage layer with the database read and write nodes, obtaining and updating data, the data consistency problem between the database and the cache is solved, stable and reliable data processing is achieved, and the database load is reduced.
Patent Information
- Application Number
- PCT/IB2024/062177
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2024-01-26
- Filing Date
- 2024-12-04
- Publication Date
- 2025-07-31
AI Technical Summary
In the prior art, data consistency between the database and the cache is difficult to guarantee, especially in asynchronous operations, data inconsistency is likely to occur, and achieving final consistency through middleware requires additional costs.
Through the cache read node and the database read and write node, the data storage layer is shared by the cache read and write node, the data read request is obtained, the target cache area is determined, and when the target cache area is empty, the target data is obtained from the data storage layer and updated to the cache area to ensure that the data between the data storage layer and the cache area is consistent.
It ensures data consistency between the data storage layer and the cache area without additional development costs, improves the stability and reliability of data processing, and reduces the risk of database breakdown and penetration.
Smart Images

Figure IB2024062177_31072025_PF_FP_ABST
Abstract
Description
[0001]TECHNICAL FIELD The present disclosure relates to the field of data processing technology, and more particularly to a data processing method, apparatus, device, database, and computer program product. Background: With the rapid development of database technology, database applications are becoming increasingly widespread. When using databases for data read and write operations, data consistency between the database and the cache presents a challenge in current application scenarios. Currently, to achieve data consistency between the database and the cache, a widely used approach is to use middleware to ensure consistency between the database and the cache. Specifically, middleware can be used to subscribe to database logs, then generate messages based on database read and write operations, which are posted to a message queue and used to invalidate the cache using the messages in the message queue. However, in such an implementation, database updates and cache updates are performed asynchronously, thus only ensuring eventual consistency between the database and cache data and incurring additional costs. SUMMARY: Embodiments of the present disclosure provide a data processing method, apparatus, device, database, and computer program product that achieve data consistency between the database and the cache, ensuring the accuracy and reliability of data processing operations. In a first aspect, an embodiment of the present disclosure provides a data processing method, comprising: obtaining a data read request through the cache read node, wherein the cache read node is located in a database, and the cache read node in the database shares a data storage layer with the database read-write node; determining a target cache area corresponding to the data read request in the cache read node; when the target cache area is empty, obtaining target data corresponding to the data read request stored in the data storage layer; and updating and storing the target data in the target cache area to make the data between the data storage layer and the cache area consistent. In a second aspect, an embodiment of the present disclosure provides a data processing device, comprising: a first acquisition module, configured to acquire a data read request through the cache read node, wherein the cache read node is located in a database, and the cache read node in the database shares a data storage layer with the database read-write node; a first determination module, configured to determine a target cache area corresponding to the data read request in the cache read node; the first acquisition module is also used to acquire target data corresponding to the data read request stored in the data storage layer when the target cache area is empty; a first processing module, configured to update and store the target data in the target cache area to make the data between the data storage layer and the cache area consistent.In a third aspect, embodiments of the present disclosure provide an electronic device, comprising: a memory and a processor; wherein the memory is configured to store one or more computer instructions, wherein the one or more computer instructions, when executed by the processor, implement the data processing method described in the first aspect. In a fourth aspect, embodiments of the present disclosure provide a computer storage medium, configured to store a computer program, wherein the computer program causes a computer to implement the data processing method described in the first aspect when executed. In a fifth aspect, embodiments of the present disclosure provide a computer program product, comprising: a computer program, which, when executed by a processor of an electronic device, causes the processor to perform the steps of the data processing method described in the first aspect. In a sixth aspect, an embodiment of the present disclosure provides a database, comprising: a cache read node for performing read operations on a cache area and a database read-write node for performing read and write operations on a database storage layer, wherein the cache read node and the database read-write node in the database share a data storage layer; the cache read node is used to: obtain a data read request; determine a target cache area corresponding to the data read request in the cache read node; when the target cache area is empty, obtain target data corresponding to the data read request stored in the data storage layer; update and store the target data in the target cache area to make the data between the data storage layer and the cache area consistent. The data processing method, apparatus, device, database, and computer program product provided in this embodiment obtain a data read request through a cache read node, determine a target cache area corresponding to the data read request in the cache read node, obtain target data corresponding to the data read request stored in the data storage layer, and update and store the target data in the target cache area to ensure data consistency between the data storage layer and the cache area. This effectively ensures data consistency between the data storage layer and the cache area through database and cache integration without additional development costs, that is, ensuring data consistency between the cache area and the data storage layer. This not only improves the stability and reliability of data processing, but also allows direct data read operations by directly accessing the data storage layer when the target cache area is empty. This allows the cache read node to help the database read / write node block some traffic, while reducing access traffic to the database read / write node, thereby reducing the risk of database penetration and penetration, further improving the practicality of the method.To more clearly illustrate the embodiments of the present disclosure or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or prior art descriptions. Obviously, the drawings described below represent some embodiments of the present disclosure. Persons skilled in the art can also derive other drawings based on these drawings without inventive effort. Figure 1 is a schematic diagram illustrating the principles of a data write operation provided by the related art; Figure 2 is a schematic diagram illustrating the principles of a data processing method provided by an embodiment of the present disclosure; Figure 3 is a flowchart illustrating a data processing method provided by an embodiment of the present disclosure; Figure 4 is a flowchart illustrating another data processing method provided by an embodiment of the present disclosure; Figure 5 is a flowchart illustrating yet another data processing method provided by an embodiment of the present disclosure; Figure 6 is a schematic diagram illustrating the principles of a method for database and cache consistency provided by an application embodiment of the present disclosure; Figure 7 is a schematic diagram illustrating the structure of a data processing device provided by an embodiment of the present disclosure; Figure 8 is a schematic diagram illustrating the structure of an electronic device corresponding to the data processing device provided by the embodiment of Figure 7; and Figure 9 is a schematic diagram illustrating the structure of a database provided by an embodiment of the present disclosure. To further clarify the objectives, technical solutions, and advantages of the embodiments of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings. It should be understood that the described embodiments represent only a portion of the embodiments of the present disclosure, and are not intended to be exhaustive. All other embodiments devised by persons of ordinary skill in the art based on the embodiments of the present disclosure without inventive effort are intended to fall within the scope of protection of the present disclosure. The terms used in the embodiments of the present disclosure are intended solely to describe specific embodiments and are not intended to limit the present disclosure. The singular forms "a," "an," "the," and "the" used in the embodiments of the present disclosure and the appended claims are intended to include the plural forms, unless the context clearly indicates otherwise. "A plurality of" generally includes at least two, but does not exclude the inclusion of at least one. It should be understood that the term "and / or" as used herein is merely a description of an association relationship between associated objects, indicating that three relationships may exist. For example, A and / or B may represent three situations: A exists alone, A and B exist simultaneously, and B exists alone. In addition, the character " / " in this document generally indicates that the associated objects are in an "or" relationship. Depending on the context, the terms "if" and "if" as used herein may be interpreted as "when..." or "when..." or "in response to determination..." or "in response to detection."Similarly, depending on the context, the phrase "if determined" or "if (stated condition or event) is detected" can be interpreted as "when determined" or "in response to determining" or "when (stated condition or event) is detected" or "in response to detecting (stated condition or event)." It should also be noted that the terms "comprise," "include," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a product or system comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such product or system. Without further limitation, elements defined by the phrase "comprising a..." do not preclude the presence of additional identical elements in the product or system comprising the elements. Furthermore, the sequence of steps in the following method embodiments is provided as an example only and is not intended to be a strict limitation. To facilitate an understanding of the specific implementation of the technical solutions in this embodiment, the following briefly describes the relevant technologies: A database is a method for persistently storing data, typically used to store structured and long-term data. A database can communicate with a database management system, which manages database operations such as creation, storage, access, updates, and deletions. Applications communicating with the database can perform efficient data retrieval and processing. A cache is a temporary data storage method primarily used to improve data access speed. The cache is typically located between the application and the database. When an application needs to access data, it first checks whether the data exists in the cache. If so, it retrieves the data directly from the cache, avoiding a database access. If the data does not exist in the cache, it reads the data from the database and stores it in the cache so that it can be retrieved directly from the cache the next time it is accessed. This caching reduces database access times, improving system responsiveness and concurrency. With the rapid development of database technology, database applications are becoming increasingly widespread. When using databases for data read and write operations, data consistency between the database and the cache presents a challenge in current application scenarios. Specifically, the consistency issue between the database and cache layers is primarily "double write," where a copy of the data is stored in both the database and the cache.For data in a database and cache, consistency between the two can be maintained on the application side. This includes logic such as the order of database (DB) and cache operations, cache selection for updates and deletions, ensuring atomicity of DB and cache operations, retrying after failures, and maintaining data consistency. In addition to leveraging application logic to maintain consistency between database and cache data, eventual data consistency can also be achieved through complex middleware (including message queues and consumers). Specifically, as shown in Figure 1, when a user issues a data write request, the database data can be updated based on the write request. To maintain consistency between the database and cache data, the corresponding cache data can be deleted. When network anomalies or high cache loads occur, deleting data from the corresponding cache area can easily fail. To retry deleting the data from the cache area, a deletion failure message can be generated and sent to a message queue. The cache key that failed to be deleted is then sent to the consumer, allowing the consumer to retry the cache deletion operation through the message consumer to achieve data consistency between the database and cache. As can be seen above, related technologies can achieve weak data consistency through application logic, which can easily lead to data inconsistencies when reading and writing data between the database and cache. Using complex middleware can only achieve eventual consistency between the database and cache, which incurs additional costs. To address the above technical issues, this embodiment provides a data processing method, apparatus, device, database, and computer program product. Referring to FIG. 2 , the data processing method provided in this embodiment may be executed by a data processing device. It should be noted that the data processing device may be implemented as any device or database capable of providing database services. When the data processing device is implemented as a database, the database may be embedded in a device capable of providing database services, such as a terminal device, a personal computer, a tablet computer, a local server, or a cloud server. In this case, when the data processing device is implemented as a cloud server, the data processing method may be executed in the cloud. The cloud may include several computing nodes (cloud servers), each of which has processing resources such as computing and storage. In the cloud, multiple computing nodes may be organized to provide a particular service, and a single computing node may also provide one or more services.The cloud can provide this service by providing a service interface, which users can call to access the corresponding service. Service interfaces include software development kits (SDKs) and application programming interfaces (APIs). The data processing device is connected to a client, where the client is used by users to perform data processing operations. The client can be any computing device with a certain data transmission capability. In specific implementations, the client can be a mobile phone, a personal computer (PC), a tablet computer, a configuration application, etc. In addition, the basic structure of the client may include at least one processor. The number of processors depends on the configuration and type of the client. The client may also include memory, which can be volatile, such as random access memory (RAM), non-volatile, such as read-only memory (ROM), flash memory, etc., or a combination of both types. The memory typically stores an operating system (OS), one or more application programs, and may also store program data. In addition to the processing unit and memory, the client also includes some basic configurations, such as a network card chip, an I / O bus, a display component, and some peripheral devices. Optionally, some peripheral devices may include, for example, a keyboard, a mouse, a stylus, a printer, etc. Other peripheral devices are well known in the art and are not described in detail here. A data processing device refers to a device that can provide data processing operations in a network virtual environment, typically a device that utilizes a network for information planning and data processing operations. In physical implementation, a data processing device can be any device that can provide computing services, respond to data processing requests, and perform data processing operations based on the data processing requests. For example, it can be a cluster server, a conventional server, a cloud server, a cloud host, a virtual center, etc. The data processing device primarily comprises a processor, a hard disk, memory, a system bus, etc., similar to a general-purpose computer architecture. In the above embodiment, the client establishes a network connection with the data processing device, and the network connection may be a wireless or wired network connection.If the client can be communicatively connected to the data processing device, the mobile network standard can be any one of 2G (GSM), 2.5G (GPRS), 3G (WCDMA, TD-SCDMA, CDMA2000, UTMS), 4G (LTE), 4G+ (LTE+), Wi-Fi Max, 5G, 6G, etc. In an embodiment of the present disclosure, the client is configured to generate or obtain a data read request corresponding to a cache read node. Specifically, the client can display a human-computer interaction interface, obtain an execution operation input by a user in the human-computer interaction interface, and generate or obtain a data read request corresponding to the cache read node based on the execution operation. The cache read node is located in a database, which may also include a database read / write node. The cache read node and the database read / write node share a data storage layer. To implement the data read operation through the cache read node, the data read request can be sent to the data processing device. A data processing device is configured to obtain a data read request corresponding to a cache read node. In order to accurately implement a data read operation, a target cache area corresponding to the data read request in the cache read node can be determined. When the target cache area is not empty, the target data corresponding to the data read request can be directly obtained through the target cache area. When the target cache area is empty, it means that the target data corresponding to the data read request does not exist in the cache read node. At this time, the target data corresponding to the data read request can be obtained through the data storage layer in the database. Since the target cache area in the cache read node does not contain the target data corresponding to the data read request, in order to ensure data consistency between the cache area in the cache read node and the data storage layer, the target data read through the data read request can be updated and stored in the target cache area, thereby achieving strong data consistency or consistency between the data storage layer and the cache area. This effectively ensures data consistency between the data storage layer and the cache area through database and cache integration without requiring additional development. This not only improves the stability and reliability of data processing, but also, because when the target data does not exist in the target cache area, it can be directly retrieved through the data storage layer without going through the database. This effectively reduces the number of data read requests the database must process, thereby reducing the risk of database corruption and penetration, further improving the practicality of this method. The following detailed description of some embodiments of the present disclosure is provided in conjunction with the accompanying drawings. The following embodiments and features may be combined unless they conflict with each other. Furthermore, the sequence of steps in the following method embodiments is provided for illustrative purposes only and is not intended to be a strict limitation.FIG3 is a flow chart illustrating a data processing method provided by an embodiment of the present disclosure. Referring to FIG3 , this embodiment provides a data processing method. The method may be executed by a data processing device. It is understood that the data processing device may be implemented as software or a combination of software and hardware. Specifically, when the data processing device is implemented as hardware, in some instances, the data processing device may be implemented as a database. The database may be embedded in various electronic devices capable of performing data processing operations, including but not limited to tablet computers, personal computers (PCs), servers, and the like. When the data processing device is implemented as software, it may be installed in the electronic devices listed above. Based on the aforementioned data processing device, the data processing method in this embodiment may include the following steps: Step S301: Obtaining a data read request via a cache read node. The cache read node is located in a database, and the cache read node in the database shares a data storage layer with the database read / write nodes. Step S302: Determining a target cache area corresponding to the data read request in the cache read node. Step S303: If the target cache area is empty, obtaining target data corresponding to the data read request stored in the data storage layer. Step S304: Update the target data and store it in the target cache area to ensure data consistency between the data storage layer and the cache area. The specific implementation principles and effects of each of the above steps are described in detail below: Step S301: Obtain a data read request via a cache read node, wherein the cache read node is located in a database, and the cache read node and database read / write nodes in the database share the data storage layer. The data processing device is provided with a database, which may be a structure with separate computing and storage functions. Specifically, the database may include: a data storage layer, and database read / write nodes and cache read nodes communicatively connected to the data storage layer. The data storage layer is used to implement data storage operations, the database read / write nodes are used to implement read and write operations on the data storage layer, and the cache read nodes are used to implement read operations on the cache area. Furthermore, the database read / write nodes and the cache read nodes share the data storage layer. When a user has a data read request, in order to implement the data read operation, the data processing device can obtain a data read request through a cache read node. In some instances, the data read request can be obtained through a human-computer interaction operation. In this case, obtaining the data read request through the cache read node may include: displaying a human-computer interaction interface; obtaining a data read operation input by the user in the human-computer interaction interface; and obtaining the data read request based on the data read operation and the cache read node.In other instances, a data read request may be obtained not only through human-computer interaction but also through a client or a third device. In this case, obtaining the data read request through the cache read node may include: determining a client or a third device in communication with the data processing apparatus; and actively or passively sending the data read request to the cache read node through the client or the third device. This effectively enables the cache read node to actively or passively obtain the data read request, thereby effectively ensuring accurate and reliable acquisition of the data read request. In some other examples, the database includes cache read nodes and database read / write nodes. The cache read nodes support key-value requests, while the database read / write nodes support Structured Query Language (SQL) requests. These key-value and SQL requests are different types of requests. Therefore, to implement data read or write operations based on different types of requests in different application scenarios, the data processing device may be configured with a communication interface for implementing data read operations through the cache read node or data read / write operations through the database read / write node. For example, the database read / write node may be communicatively connected to a database interface, while the cache read node may be communicatively connected to a cache interface. In this case, the data processing device may obtain data read requests through the cache interface and the cache read node, thus ensuring accurate and reliable acquisition of data read requests. Similarly, the data processing device may obtain data read / write requests through the database interface and the database read / write node. Step S302: Determine a target cache area in the cache read node corresponding to the data read request. After receiving a data read request, in order to implement the data read operation, a target cache region corresponding to the data read node in the cache read node may be determined. In some instances, the target cache region may be determined based on a request region identifier in the data read request, or a request region identifier associated with the data read request. In this case, determining the target cache region corresponding to the data read request in the cache read node may include: determining the request region identifier based on the data read request; and determining the target cache region based on the request region identifier. For example, if the cache regions in the cache read node include region 1, region 2, and region 3, after receiving the data read request, the request region identifier is obtained based on the data read request. If the request region identifier corresponds to region 3, region 3 may be determined as the target cache region. This effectively ensures accurate and reliable determination of the target cache region.Step S303: If the target cache area is empty, the target data corresponding to the data read request stored in the data storage layer is retrieved. After determining the target cache area, it is determined whether the target data corresponding to the data read request can be retrieved through the target cache area. In some instances, whether the target data corresponding to the data read request can be retrieved from the target cache area can be determined by determining whether the target cache area is empty. Specifically, if the target cache area is not empty, it indicates that the target data corresponding to the data read request is stored in the target cache area, and the target data can be directly retrieved from the cache area. Conversely, if the target cache area is empty, it indicates that no data is stored in the target cache area. In this case, to ensure accurate reading of the target data, the target data corresponding to the data read request stored in the data storage layer can be retrieved through the data storage layer, effectively ensuring accurate and reliable acquisition of the target data. Furthermore, if the target cache area is empty, to ensure stable and reliable acquisition of the target data through the data storage layer, the cache read node may include a second database engine in communication with the data storage layer. The second database engine can then retrieve the target data corresponding to the data read request from the data storage layer, effectively ensuring accurate and reliable determination of the target data. Step S304: Update the target data and store it in the target cache area to ensure data consistency between the data storage layer and the cache area. Since the target cache area is empty, this indicates that the cache area in the cache read node does not store the target data corresponding to the data read request. To ensure data consistency between the cache area in the cache read node and the data storage layer (in some instances, data between the cache area and the data storage layer can be maintained in real-time or strong consistency), the target data can be updated and stored in the target cache area. This effectively enables direct updates to the cache area in the cache read node through the data storage layer, improving the quality and efficiency of data updates.The data processing method provided in this embodiment obtains a data read request through a cache read node and determines a target cache area corresponding to the data read request in the cache read node. If the target cache area is empty, the method obtains the target data corresponding to the data read request stored in the data storage layer. The method then updates and stores the target data in the target cache area, thereby ensuring data consistency between the data storage layer and the cache area. This method effectively ensures data consistency between the data storage layer and the cache area through database and cache integration without additional development costs. This ensures data consistency between the cache area and the data storage layer, thereby ensuring data consistency between the cache area and the data storage layer. This not only improves the stability and reliability of data processing, but also allows data read operations to be performed directly by directly accessing the data storage layer when the target cache area is empty. This allows the cache read node to help the database read / write node block some traffic, thereby reducing access traffic to the database read / write node, thereby reducing the risk of database penetration and further improving the practicality of the method. FIG4 is a flow chart of another data processing method provided by an embodiment of the present disclosure. Based on the above embodiment, referring to FIG4 , after a data read request is received by a cache read node, in order to stably implement a data read operation, the method may first identify whether the target data has been written to the data storage layer, and then perform a corresponding data read operation based on the identification result. Specifically, the method in this embodiment further includes: Step S401: Identifying whether the target data corresponding to the data read request has been written to the data storage layer. Regarding the data storage layer in a database, a user may write multiple data to the data storage layer as needed, and the multiple data may include the target data corresponding to the data read request, or the multiple databases may not include the target data corresponding to the data read request. If the multiple data include the target data corresponding to the data read request, the target data read operation can be accurately implemented; if the multiple data do not include the target data corresponding to the data read request, the target data read operation cannot be accurately implemented.Based on the above statements, it can be seen that after obtaining a data read request, in order to ensure the stability and reliability of the data processing operation, after obtaining the data read request through the cache read node, it can be identified whether the target data corresponding to the data read request is written to the data storage layer. In some instances, identifying whether the target data corresponding to the data read request is written to the data storage layer can be achieved through a data matching operation. In this case, identifying whether the target data corresponding to the data read request is written to the data storage layer may include: obtaining the target data corresponding to the data read request and all data stored in the data storage layer; identifying whether the target data exists in all the data; if the target data exists in all the data, determining that the target data corresponding to the data read request has been written to the data storage layer; if the target data does not exist in all the data, determining that the target data corresponding to the data read request has not been written to the data storage layer. In other instances, whether the target data corresponding to a data read request is written to the data storage layer can be identified not only through a data matching operation but also through write operation records. In this case, identifying whether the target data corresponding to the data read request is written to the data cache layer may include: obtaining a write operation record corresponding to the data storage layer through a database read / write node; determining a keyword corresponding to the data read request; and identifying whether the target data corresponding to the data read request is written to the data storage layer based on the write operation record and the keyword. Specifically, when writing data to the data storage layer using a database read / write node, the database read / write node may store a write operation record, or the database read / write node may correspond to a write operation record. The write operation record may include relevant records of data written to the data storage layer using the database read / write node at any time and in any scenario. It should be noted that the data corresponding to any write operation in the write operation record may have already been written to the data storage layer, or the data corresponding to any write operation in the write operation record may be in the process of being written to the data storage layer and has not yet been stored in the data storage layer. Therefore, in order to accurately identify whether the target data corresponding to the data read request is written to the data cache layer, after obtaining the data read request, the write operation record corresponding to the data storage layer can be obtained through the database read / write node.In addition, after obtaining the data read request, the data read request can be analyzed and processed to obtain a keyword key corresponding to the data read request, and the keyword key is used to identify the target data corresponding to the data read request. After obtaining the write operation record and the keyword, the write operation record and the keyword can be analyzed and processed to identify whether the target data corresponding to the data read request is written to the data storage layer. In some instances, identifying whether the target data corresponding to the data read request is written to the data storage layer can be achieved by a pre-trained machine learning model or a neural network model. In this case, identifying whether the target data corresponding to the data read request is written to the data storage layer may include: obtaining a pre-trained machine learning model or a neural network model; inputting the write operation record and the keyword into the machine learning model or the neural network model, and obtaining a recognition result output by the machine learning model or the neural network model. The recognition result may be a first recognition result used to identify that the target data corresponding to the data read request has been written or is being written to the data storage layer; or the recognition result may be a second recognition result used to identify that the target data corresponding to the data read request has not been written to the data storage layer. In other instances, not only can a pre-trained machine learning model or neural network model be used to identify whether the target data corresponding to a data read request is written to the data storage layer, but write operation records and keywords can also be directly analyzed and processed to identify whether the target data corresponding to the data read request is written to the data storage layer. In this case, based on the write operation record and keyword, identifying whether the target data corresponding to the data read request is written to the data storage layer may include: if the write operation record includes value information corresponding to the keyword, determining that the target data has been written or is being written to the data storage layer; if the write operation record does not include value information corresponding to the keyword, determining that the target data has not been written to the data storage layer. Specifically, since data in the data storage layer is often stored in key-value pairs, after obtaining the write operation record and keyword, the value information corresponding to the keyword is first obtained, and then the value information corresponding to the keyword is identified. If the write operation record includes value information corresponding to the keyword, it can be determined that the target data has been written or is being written to the data storage layer, thereby enabling accurate data read operations based on the data storage layer. In the case that the write operation record does not include value information corresponding to the keyword, it can be determined that the target data is not written into the data storage layer, and thus accurate data reading operations cannot be implemented based on the data storage layer.Step S402: If the target data has been written or is being written to the data storage layer, the target cache area corresponding to the data read request in the cache read node is determined. Step S403: If the target data has not been written to the data storage layer, the process waits until the target data is written to the data storage layer. After obtaining the recognition result, different strategies can be used to perform the data read operation based on different recognition results. Specifically, if the recognition result indicates that the target data has been written or is being written to the data storage layer, it indicates that the target data corresponding to the data read request may be stored in the data storage layer at the current time or at a future time. In this case, the data read operation can be directly performed through the data storage layer, and the target cache area corresponding to the data read request in the cache read node is determined. If the target data has not been written to the data storage layer, this indicates that the data storage layer currently does not store the target data corresponding to the data read request. To ensure stable data read operations, the determination of the target cache area corresponding to the data read request in the cache read node can be temporarily suspended, and a wait period can be maintained until the target data is written to the data storage layer. If it is determined that the target data has been written to the data storage layer, the target cache area corresponding to the data read request in the cache read node can be determined, allowing the corresponding data read operation to be performed based on the target cache area, thereby ensuring stable data processing operations. In this embodiment, by identifying whether the target data corresponding to the data read request has been written to the data storage layer, the target cache area corresponding to the data read request in the cache read node can be determined if the target data has been written to or is being written to the data storage layer; otherwise, the wait period is maintained until the target data is written to the data storage layer. This effectively implements the use of different strategies for data read operations based on whether the target data has been written to the data storage layer, further improving the practicality of the method. FIG5 is a flow chart of another data processing method provided by an embodiment of the present disclosure. Based on any of the above embodiments, with reference to FIG5 , the method in this embodiment can not only implement data read operations, but can also implement data write operations using database read / write nodes. In this case, before obtaining a data read request through a cache read node, the method in this embodiment can further include: Step S501: Obtaining a data write request corresponding to the data to be executed through a database read / write node.When a user requests data writing, the data processing device can obtain a data write request corresponding to the data to be executed through the database read / write node. The method for obtaining the data write request in this embodiment is similar to the specific method for obtaining the data read request in the above-mentioned embodiment. For details, please refer to the above description and will not be repeated here. Step S502: Based on the data write request, the data to be executed is written to the data storage layer to obtain the database data stored in the data storage layer. After obtaining the data write request, the data to be executed can be written to the data storage layer based on the data write request, thereby obtaining the database data stored in the data storage layer. In some instances, the data write operation can be performed by a first database engine. In this case, the database read / write node can include the first database engine in communication with the data storage layer. The data to be executed can then be written to the data storage layer through the first database engine, effectively ensuring the stability and reliability of the data write operation. Step S503: Based on the database data, the cache area in the cache read node is updated to obtain updated cache data. After writing the data to be executed to the data storage layer, to ensure data consistency between the data storage layer and the cache area of the cache read node, after obtaining the database data stored in the data storage layer, a data update operation can be performed on the cache area of the cache read node based on the database data, thereby obtaining updated cache data stored in the cache area. In some instances, the data update operation can be implemented through a data synchronization operation. In this case, updating the cache area of the cache read node based on the database data to obtain updated cache data can include: generating a data synchronization instruction based on the database data; and sending the database data to the cache area of the cache read node based on the data synchronization instruction and performing update storage, thereby obtaining updated cache data. In other instances, the data update operation may be implemented not only through a data synchronization operation but also through a data invalidation operation. In this case, updating the cache area in the cache read node based on the database data to obtain updated cache data may include: obtaining a keyword of the database data; determining a target cache area corresponding to the keyword in the cache area in the cache read node; generating a data invalidation instruction corresponding to the target cache area; and invalidating the data in the target cache area based on the data invalidation instruction, thereby obtaining the updated cache data.To accurately update the cache area in the cache read node based on database data, after acquiring the database data, the database data can be analyzed and processed to obtain a keyword corresponding to the database data. A target cache area corresponding to the keyword can then be determined within the cache area in the cache read node. Because the cache data in the target cache area in the cache read node is data cached at a historical moment, this cached data differs from the database data currently stored in the data storage layer. Therefore, to ensure data consistency between the data storage layer and the cache area, a data invalidation instruction corresponding to the target cache area can be generated. In some instances, the data invalidation instruction can be generated using a pre-trained machine learning model or neural network model, or based on preset prompt information. Generating the data invalidation instruction corresponding to the target cache area can include: obtaining, from the data storage layer, prompt information indicating that the database read / write node is writing the pending data to the data storage layer; and generating the data invalidation instruction corresponding to the target cache area based on the prompt information. Specifically, when a database read / write node writes pending data to the data storage layer, the data storage layer may generate prompt information to identify that the database read / write node has written the pending data to the data storage layer, so that the cache read node can promptly learn of the pending data being written to the data storage layer. The prompt information may include at least one of the following: a keyword corresponding to the pending data, an operation time corresponding to the pending data, an identifier of the database read / write node corresponding to the pending data, a storage area of the pending data in the data storage layer, and the like. After the data storage layer generates the prompt information, the prompt information may be actively or passively sent to the cache read node. After the cache read node receives the prompt information, it may generate a data invalidation instruction corresponding to the target cache area based on the prompt information. Specifically, the second database engine in the cache read node receives the prompt information and may generate a data invalidation instruction corresponding to the target cache area based on the prompt information. The data invalidation instruction includes an identifier of the target cache area corresponding to the database data. After obtaining the data invalidation instruction, the data in the target cache area can be invalidated based on the data invalidation instruction. For example, the data in the target cache area can be deleted, or the data in the target cache area can be replaced with the database data in the data invalidation instruction, so that the updated cache data can be obtained, ensuring the stability and reliability of the data update operation on the cache area.In this embodiment, a database read / write node obtains a data write request corresponding to pending data. Based on the data write request, the pending data is written to the data storage layer, obtaining the database data stored in the data storage layer. Based on the database data, the cache area of the cache read node is updated to obtain updated cache data. This effectively ensures data consistency between the cache area and the data storage layer, further improving the accuracy and reliability of data processing operations. In specific applications, referring to FIG6 , this application embodiment provides a method for achieving strong data consistency between a database and a cache. The method can be performed by a database that integrates a data storage layer and a cache area. The database includes a data computing layer and a data storage layer. The data computing layer may include a proxy module, a database read / write node RW in communication with the proxy module, and a cache read node R0. The proxy module includes a database interface in communication with the database read / write node RW and a cache interface in communication with the cache read node R0. The database read / write node RW is used to perform data read and write operations on the data storage layer, while the cache read node R0 is used to perform data read operations on the cache area. For database read / write nodes RW, they may include an SQL parsing module, an executor engine, and a first database engine. The database read / write node RW can communicate with the data storage layer via the first database engine. The SQL parsing module is used to parse, plan, or optimize data read / write requests to translate them into machine language recognizable by the device. The executor engine is used to determine a corresponding data read operation based on a data read request, or a corresponding data write operation based on a data write request. The first database engine is used to read relevant data from the data storage layer based on the data read operation determined by the executor engine, or is configured to write relevant data to the data storage layer based on the data write operation determined by the executor engine. For cache read node R0, cache read node R0 may include an parsing module, a cache area, an executor engine, and a second database engine. The cache read node R0 can communicate with the data storage layer via the second database engine. The parsing module is used to parse, plan, or optimize data read requests to translate them into machine language recognizable by the device. The executor engine is used to determine a corresponding data read operation based on the data read request. The second database engine is used to read relevant data in the data storage layer based on the data read operation determined by the executor engine.It should be noted that the first and second database engines share a data storage layer to ensure strong data consistency between the data storage layer and cache read node R0. A method for achieving strong database and cache consistency based on the aforementioned database can include the following steps: Step 1: Cache read node R0 receives a data read request via a cache interface. Step 2: After cache read node R0 receives the data read request, it can determine the key corresponding to the data read request. Step 3: Based on the key, the target cache area corresponding to the data read request is determined in the cache read node. Step 4: If the target cache area is not empty, the data in the target cache area can be directly read based on the data read request. Step 5: If the target cache area is empty, the target data corresponding to the data read request is obtained from the data storage layer by the second database engine. Step 6: The target data is updated and stored in the target cache area to ensure strong data consistency between the data storage layer and the cache area. Furthermore, the method in this embodiment can also include the following steps: Step 11: Database read / write node RW receives a data write request via the database interface. Step 12: Based on the data write request, the data to be executed is written to the data storage layer to obtain the database data stored in the data storage layer. Step 13: Based on the database data, the cache area in the cache read node is updated to obtain updated cache data. It should be noted that the first database engine in the RW node may include a logging engine for implementing log management operations. This logging engine can store log information corresponding to the database read / write nodes in the data storage layer and perform update management operations on the log information based on the operation. As can be seen from the above, the database in this embodiment can include a first database engine, a logging engine, and a second database engine. This database can support any data update and data management operations of the aforementioned heterogeneous engines, thereby effectively ensuring the stability and reliability of data processing operations. In addition, a strong consistency mechanism (SCC) strategy can be configured in the RW nodes in the database. This can ensure strong consistency of data in the RW nodes, R0 nodes, and the data storage layer. That is, during the dual-write operation on the database, for the data storage layer and the cache area, the dual-write operation can both succeed or fail, and there will be no data inconsistency, further improving the stability and reliability of data read and write operations.The technical solution provided by this application embodiment, specifically, during a data read operation through the cache read node R0, strong data consistency between the data storage layer and the cache area can be achieved through the first database engine and the second database engine of the shared data storage layer. In addition, when a data write operation is performed through the RW node, strong data consistency between the RW node and the R0 node can also be achieved through the data storage layer. This effectively enables the database to maintain strong data consistency between the R0 node, the RW node, and the data storage layer without the need for additional development, and can also solve operational maintenance issues such as cache updates and expiration, while reducing the risk of database breakdown and penetration after cache expiration. Data consistency is ensured through the integration of the database and the cache, further improving the stability and reliability of the method. FIG7 is a schematic structural diagram of a data processing device provided by an embodiment of the present disclosure. Referring to FIG7 , this embodiment provides a data processing device, which is used to execute the data processing method shown in FIG3 . The data processing device may include: a first acquisition module 11, configured to acquire a data read request through a cache read node, wherein the cache read node is located in a database, and the cache read node in the database and the database read-write node share a data storage layer; a first determination module 12, configured to determine a target cache area corresponding to the data read request in the cache read node; the first acquisition module 11 is further configured to, when the target cache area is empty, acquire target data corresponding to the data read request stored in the data storage layer; a first processing module 13, configured to update and store the target data in the target cache area to ensure data consistency between the data storage layer and the cache area. In some instances, after receiving a data read request from a cache read node, the first processing module 13 in this embodiment is configured to: identify whether the target data corresponding to the data read request has been written to the data storage layer; if the target data has been written or is being written to the data storage layer, allow the target cache area corresponding to the data read request in the cache read node to be determined; if the target data has not been written to the data storage layer, wait until the target data is written to the data storage layer. In some instances, after the first processing module 13 identifies whether the target data corresponding to the data read request has been written to the data cache layer, the first processing module 13 is configured to: obtain a write operation record corresponding to the data storage layer from a database read / write node; determine a keyword corresponding to the data read request; and, based on the write operation record and the keyword, identify whether the target data corresponding to the data read request has been written to the data storage layer.In some instances, when the first processing module 13 identifies whether the target data corresponding to the data read request has been written to the data storage layer based on the write operation record and the keyword, the first processing module 13 is configured to: determine that the target data has been written to or is being written to the data storage layer if the write operation record includes value information corresponding to the keyword; and determine that the target data has not been written to the data storage layer if the write operation record does not include value information corresponding to the keyword. In some instances, before obtaining a data read request from a cache read node, the first acquisition module 11 and the first processing module 13 in this embodiment are configured to: first acquisition module 11 is configured to obtain a data write request corresponding to the data to be executed from a database read / write node; first processing module 13 is configured to write the data to be executed to the data storage layer based on the data write request to obtain database data stored in the data storage layer; and update the cache area in the cache read node based on the database data to obtain updated cache data. In some instances, when the first processing module 13 updates the cache area of the cache read node based on database data to obtain updated cache data, the first processing module 13 is configured to: obtain a keyword from the database data; determine a target cache area corresponding to the keyword within the cache area of the cache read node; generate a data invalidation instruction corresponding to the target cache area; and invalidate the data in the target cache area based on the data invalidation instruction to obtain updated cache data. In some instances, when the first processing module 13 generates the data invalidation instruction corresponding to the target cache area, the first processing module 13 is configured to: obtain, from the data storage layer, prompt information indicating that the database read / write node is writing the pending data to the data storage layer; and generate a data invalidation instruction corresponding to the target cache area based on the prompt information. The apparatus shown in FIG7 can execute the method of the embodiments shown in FIG2-FIG6. For portions not described in detail in this embodiment, reference is made to the relevant description of the embodiments shown in FIG2-FIG6. The implementation process and technical effects of this technical solution are described in the embodiments shown in FIG2-FIG6 and are not further elaborated here. In one possible design, the structure of the data processing device shown in FIG7 can be implemented as an electronic device, which can be a controller, a personal computer, a partition, or other devices. As shown in FIG8 , the electronic device may include a first processor 21 and a first memory 22. OThe first memory 22 is used to store a program for the electronic device to execute the data processing method provided in the embodiments shown in Figures 2-6 . The first processor 21 is configured to execute the program stored in the first memory 22. The program includes one or more computer instructions. When executed by the first processor 21, the one or more computer instructions can implement the following steps: obtaining a data read request through a cache read node, wherein the cache read node is located in a database, and the cache read node and the database read / write nodes in the database share a data storage layer; determining a target cache area corresponding to the data read request in the cache read node; if the target cache area is empty, obtaining target data corresponding to the data read request stored in the data storage layer; and updating and storing the target data in the target cache area to ensure data consistency between the data storage layer and the cache area. Furthermore, the first processor 21 is further configured to execute all or part of the steps in the embodiments shown in Figures 2-6 . The electronic device may also include a first communication interface 23 configured to enable the electronic device to communicate with other devices or a communication network. In addition, embodiments of the present disclosure provide a computer storage medium configured to store computer software instructions for use by an electronic device, including a program for executing the data processing methods described in the embodiments illustrated in Figures 2 through 6 . Furthermore, embodiments of the present disclosure provide a computer program product comprising: a computer-readable storage medium storing computer instructions. When the computer instructions are executed by one or more processors, the one or more processors are caused to perform the steps of the data processing methods described in the embodiments illustrated in Figures 2 through 6 . FIG9 is a schematic diagram of the structure of a database provided by an embodiment of the present disclosure. Referring to FIG9 , the database in this embodiment may execute the data processing method shown in FIG2 . Specifically, the database may include: a cache read node 301 for performing read operations on a cache area, and a database read / write node 300 for performing read / write operations on a database storage layer 302. The cache read node 301 and the database read / write node 300 in the database share a data storage layer 302. The cache read node 301 is configured to: obtain a data read request; determine a target cache area corresponding to the data read request in the cache read node 301; if the target cache area is empty, obtain target data corresponding to the data read request stored in the data storage layer 302; and update and store the target data in the target cache area to ensure data consistency between the data storage layer 302 and the cache area.In some examples, the database read / write node 300 is communicatively connected to the data storage layer 302 via the included first database engine, and the cache read node 301 is communicatively connected to the data storage layer 302 via the included second database engine. The first and second database engines share data via the data storage layer 302. The database shown in FIG9 can execute the method of the embodiments shown in FIG2-FIG6. For portions not described in detail in this embodiment, reference can be made to the relevant description of the embodiments shown in FIG2-FIG6. The execution process and technical effects of this technical solution are described in the embodiments shown in FIG2-FIG6 and will not be repeated here. The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they may be located in one location or distributed across multiple network units. Some or all of these modules may be selected to achieve the objectives of this embodiment according to actual needs. Persons of ordinary skill in the art can understand and implement the present invention without inventive effort. Through the description of the above embodiments, those skilled in the art will clearly understand that each embodiment can be implemented by adding the necessary general-purpose hardware platform, or of course, by a combination of hardware and software. Based on this understanding, the essence of the above technical solution, or the portion that contributes to the prior art, can be embodied in the form of a computer product. The present disclosure can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. The present disclosure is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present disclosure. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable device to produce a machine, so that the instructions executed by the processor of the computer or other programmable device produce a device for implementing the functions specified in one or more processes in the flowchart and / or one or more blocks in the block diagram.These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable device to operate in a specific manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means that implement the functions specified in one or more flowcharts and / or one or more blocks in a block diagram. These computer program instructions may also be loaded onto a computer or other programmable device, causing the computer or other programmable device to execute a series of operational steps to produce a computer-implemented process, such that the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one or more flowcharts and / or one or more blocks in a block diagram. In a typical configuration, a computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory. Memory may include non-volatile memory in a computer-readable medium, random access memory (RAM), and / or non-volatile memory, such as read-only memory (ROM) or flash memory. Memory is an example of a computer-readable medium. Computer-readable media, including both permanent and non-permanent, removable and non-removable media, may be implemented using any method or technology for information storage. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory computer-readable media, such as modulated data signals and carrier waves. Finally, it should be noted that the above embodiments are merely illustrative of the technical solutions of the present disclosure and are not intended to limit the present disclosure. Although the present disclosure has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they may modify the technical solutions described in the aforementioned embodiments or replace some of the technical features therein with equivalents. However, such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present disclosure.
Claims
Claims 1. A data processing method, comprising: Obtain a data reading request through a cache reading node, where the cache reading node is located in a database, and a data storage layer is shared between the cache reading node and a database reading and writing node in the database; determine a target cache area in the cache reading node corresponding to the data reading request; in the case that the target cache area is empty, obtain target data corresponding to the data reading request stored in the data storage layer; update and store the target data in the target cache area to make the data between the data storage layer and the cache area consistent.
2. The method according to claim 1, wherein After obtaining the data reading request through the cache reading node, the method further includes: identifying whether the target data corresponding to the data reading request has been written to the data storage layer; in the case that the target data has been written or is being written to the data storage layer, allow determining the target cache area in the cache reading node corresponding to the data reading request; in the case that the target data has not been written to the data storage layer, wait until the target data is written to the data storage layer.
3. The method according to claim 2, wherein Identifying whether the target data corresponding to the data reading request has been written to the data cache layer includes: obtaining a write operation record corresponding to the data storage layer through the database reading and writing node; determining a keyword corresponding to the data reading request; based on the write operation record and the keyword, identifying whether the target data corresponding to the data reading request has been written to the data storage layer.
4. The method according to claim 3, wherein Based on the write operation record and the keyword, identifying whether the target data corresponding to the data reading request has been written to the data storage layer includes: in the case that the write operation record includes value information corresponding to the keyword, determining that the target data has been written or is being written to the data storage layer; in the case that the write operation record does not include value information corresponding to the keyword, determining that the target data has not been written to the data storage layer.
5. The method according to any one of claims 1-4, wherein Before obtaining the data reading request through the cache reading node, the method further includes: obtaining a data write request corresponding to the data to be executed through the database reading and writing node; Based on the data write request, write the data to be executed to the data storage layer to obtain database data stored in the data storage layer; based on the database data, update the cache area in the cache reading node to obtain updated cache data.
6. The method according to claim 5, wherein Updating the cache area in the cache read node based on the database data to obtain updated cache data, including: obtaining the keyword of the database data; determining, in the cache area of the cache read node, the target cache area corresponding to the keyword; generating a data invalidation instruction corresponding to the target cache area; invalidating the data in the target cache area based on the data invalidation instruction to obtain the updated cache data.
7. The method according to claim 6, wherein Generating a data invalidation instruction corresponding to the target cache area, including: obtaining, through the data storage layer, hint information for indicating that the database read / write node writes the data to be executed to the data storage layer; generating a data invalidation instruction corresponding to the target cache area based on the hint information.
8. The method according to claim 1, wherein The data storage layer is used to implement data storage operations, the database read / write node is used to implement read / write operations on the data storage layer, and the cache read node is used to implement read operations on the cache area.
9. The method according to claim 1, wherein The target cache area is determined by the request area identifier in the data read request or a request area identifier associated with the data read request.
10. The method according to claim 5, wherein Writing the data to be executed to the data storage layer based on the data write request to obtain the database data stored in the data storage layer, including: writing the data to be executed to the data storage layer through a first database engine to obtain the database data.
11. A data processing device, comprising: A first acquisition module, configured to obtain a data read request through a cache read node, where the cache read node is located in a database, and the cache read node in the database shares a data storage layer with a database read / write node; a first determination module, configured to determine the target cache area in the cache read node corresponding to the data read request; the first acquisition module is further configured to, when the target cache area is empty, obtain the target data corresponding to the data read request stored in the data storage layer; a first processing module, configured to update and store the target data to the target cache area to make the data between the data storage layer and the cache area consistent. 19 To make the data between the data storage layer and the cache area consistent.
12. A database, comprising: A cache read node for performing read operations on the cache area and a database read / write node for performing read / write operations on the database storage layer, where the cache read node in the database shares a data storage layer with the database read / write node; the cache read node is used to: obtain a data read request; determine the target cache area in the cache read node corresponding to the data read request; When the target cache area is empty, obtain the target data corresponding to the data read request stored in the data storage layer; Update and store the target data to the target cache area to make the data between the data storage layer and the cache area consistent.
13. The database according to claim 12, wherein The database read / write node is communicatively connected to the data storage layer through the included first database engine, the cache read node is communicatively connected to the data storage layer through the included second database engine, and data sharing is performed between the first database engine and the second database engine through the data storage layer.
14. An electronic device, comprising: A memory, a processor; wherein, the memory is used to store one or more computer instructions, and when the one or more computer instructions are executed by the processor, the method according to any one of the above claims 1-10 is implemented.
15. A computer program product, comprising: A computer program, when the computer program is executed by a processor of an electronic device, causes the processor to execute the steps in the method according to any one of the above claims 1-10.
Citation Information
Patent Citations
Database caching
CN110502534A
Data query method and device, medium and electronic equipment
CN110633296A
LUN data replication method and device
CN113986128A
File synchronization method and device, file updating method and device, electronic equipment and medium
CN116975012A