Data processing method and device, equipment, storage medium and product

Abstract

Embodiments of the present application disclose data processing methods and devices, equipment, storage media and products, and relate to the technical field of computers. The method comprises: receiving a snapshot query request sent by a business side for real-time data stored in a preset in-memory database, determining a target instruction before a snapshot time corresponding to a query object according to historical instruction data, wherein the historical instruction data comprises data corresponding to historical instructions obtained by reading a preset persistent file of the preset in-memory database, the historical instructions are executed by the business side on the preset in-memory database to generate the real-time data, the target instruction is executed on the preset in-memory database to make the preset in-memory database generate snapshot data corresponding to the snapshot query request, and the business side is notified to read the snapshot data from the preset in-memory database. By adopting the above technical solution, snapshot data of a specified time corresponding to real-time data can be obtained while saving storage space of the business side.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to data processing methods, apparatus, devices, storage media, and products. Background Technology

[0002] In-memory databases, such as Redis, are characterized by high data read and write speeds and are often used for processing real-time data, such as processing leaderboard data in various business scenarios.

[0003] Currently, real-time data typically supports real-time display but lacks snapshot functionality. If historical snapshot data is required, data collection code must be written in advance on the business side, and a collection cycle must be set to periodically collect and back up real-time data from the in-memory database. If the collection cycle is too short, it will consume a massive amount of memory space; if the collection cycle is too long, the continuity of snapshot versions cannot be guaranteed, and the accuracy of snapshot backtracking is difficult to ensure. Summary of the Invention

[0004] This application provides data processing methods, apparatus, devices, storage media, and products that can acquire snapshot data of a specified time corresponding to real-time data while saving storage space on the business side.

[0005] According to one aspect of this application, a data processing method is provided, the method comprising:

[0006] Receive a snapshot query request sent by the service side for real-time data stored in a preset memory database, wherein the snapshot query request includes the query object and the snapshot time;

[0007] The target instruction corresponding to the query object before the snapshot time is determined based on the historical instruction data. The historical instruction data includes data corresponding to the historical instructions obtained by reading the preset persistent file of the preset memory database. The historical instructions are executed by the business side against the preset memory database to generate the real-time data.

[0008] The target instruction is executed on the preset memory database to cause the preset memory database to generate snapshot data corresponding to the snapshot query request.

[0009] The service side is notified to read the snapshot data from the preset memory database.

[0010] According to another aspect of this application, a data processing apparatus is provided, the apparatus comprising:

[0011] The snapshot query request receiving module is used to receive snapshot query requests sent by the service side for real-time data stored in a preset memory database, wherein the snapshot query request includes the query object and the snapshot time;

[0012] The target instruction determination module is used to determine the target instruction corresponding to the query object before the snapshot time based on historical instruction data. The historical instruction data includes data corresponding to historical instructions obtained by reading a preset persistent file of the preset memory database. The historical instructions are executed by the business side against the preset memory database to generate the real-time data.

[0013] The target instruction execution module is used to execute the target instruction on the preset memory database, so that the preset memory database generates snapshot data corresponding to the snapshot query request;

[0014] The notification module is used to notify the service side to read the snapshot data from the preset memory database.

[0015] According to another aspect of this application, a data processing apparatus is provided, the data processing apparatus comprising:

[0016] At least one processor; and

[0017] A memory communicatively connected to the at least one processor; wherein,

[0018] The memory stores a computer program that can be executed by the at least one processor, which enables the at least one processor to perform the data processing method described in any embodiment of this application.

[0019] According to another aspect of this application, a computer-readable storage medium is provided, the computer-readable storage medium storing a computer program for causing a processor to execute and implement the data processing method described in any embodiment of this application.

[0020] According to another aspect of this application, a computer program product is provided, the computer program product including a computer program that, when executed by a processor, implements the data processing method described in any embodiment of this application.

[0021] The data processing scheme provided in this embodiment receives a snapshot query request from the service side for real-time data stored in a preset memory database. It determines the target instruction corresponding to the snapshot time prior to the query object based on historical instruction data. The historical instruction data includes data corresponding to historical instructions obtained by reading a preset persistent file in the preset memory database. These historical instructions are executed by the service side against the preset memory database to generate real-time data. The target instruction is then executed against the preset memory database to generate snapshot data corresponding to the snapshot query request. The service side is then notified to read the snapshot data from the preset memory database. By adopting this technical solution, a device outside the service side responds to the snapshot query request sent by the service side, determines the target instruction executed by the service side against the preset memory database prior to the snapshot time corresponding to the query object based on historical instruction data, re-executes the target instruction to generate snapshot data corresponding to the snapshot time in the preset memory database, and notifies the service side to read it from the preset memory database. The service side does not need to collect and store snapshot data in advance, and can obtain snapshot data corresponding to a specified time from real-time data while saving storage space on the service side, facilitating the relevant applications of the snapshot data.

[0022] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this application, nor is it intended to limit the scope of this application. Other features of this application will become readily apparent from the following description. Attached Figure Description

[0023] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1 A scenario architecture diagram illustrating the application scenario to which a data processing method provided in this application embodiment is applicable;

[0025] Figure 2 A flowchart illustrating a data processing method provided in an embodiment of this application;

[0026] Figure 3 A flowchart illustrating another data processing method provided in an embodiment of this application;

[0027] Figure 4 This application provides a schematic diagram of a historical instruction data acquisition process.

[0028] Figure 5 This application provides a schematic diagram of the data flow of instruction data in an embodiment.

[0029] Figure 6 A structural block diagram of a data processing apparatus provided in an embodiment of this application;

[0030] Figure 7 This is a structural block diagram of a data processing device provided in an embodiment of this application. Detailed Implementation

[0031] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0032] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0033] Figure 1 This is a scenario architecture diagram illustrating the application scenario to which the data processing method provided in this embodiment of the application is applicable. Specifically, refer to... Figure 1This application scenario can include a business side 101, a preset in-memory database 102, and a data processing side 103. The specific type of the preset in-memory database 102 is not limited; for example, it could be Redis or Codis. The preset in-memory database 102 can be used to store real-time data. The types and data structure types of real-time data can be set according to actual business needs. Taking live streaming as an example, it typically involves various styles of leaderboards, such as points leaderboards, reward leaderboards, competition (PK) leaderboards, and contribution leaderboards. Using Redis as an example, leaderboard data can be stored in Redis-zset (ordered list, abbreviated as zset) type. The data processing side 103 can also be called the acquisition side, and can be implemented by the data processing device (also called acquisition device) provided in this application embodiment. The data processing device can deploy data processing (acquisition) services, which can be used to execute the data processing methods provided in this application embodiment.

[0034] For example, during operation, the business side 101 performs operations on the preset memory database 102 by executing operation instructions, such as data write operations. The preset memory database 102 responds to the operations of the business side 101 and generates real-time data. The preset memory database 102 has a data persistence function. After the persistence function is enabled, a corresponding preset persistence file can be generated. The preset persistence file can store the instructions executed by the business side on the preset memory database. Taking Redis as an example, the Append Only File (AOF) function can be enabled. This function can record the write instructions executed by the business side to obtain an AOF file. This AOF file can be considered as a kind of preset persistence file in this application embodiment. In this application embodiment, when the business side needs to obtain snapshot data at a certain moment, such as the snapshot data of the points leaderboard at 11:00 AM, it can send a snapshot query request to the data processing side. The data processing side determines the target instruction to be executed on the preset memory database based on historical instruction data, and then the preset memory database generates the snapshot data required by the business side for the business side to read.

[0035] Figure 2 This is a flowchart illustrating a data processing method provided in an embodiment of this application. This embodiment is applicable to situations where snapshot data of real-time data is acquired. The method can be executed by a data processing device, which can be implemented in hardware and / or software and can be configured within a data processing equipment. Figure 2 As shown, the method includes:

[0036] Step 201: Receive a snapshot query request sent by the service side for real-time data stored in a preset memory database, wherein the snapshot query request includes the query object and the snapshot time.

[0037] For example, when a business needs to obtain snapshot data at a specific moment, it can send a snapshot query request to the data processing service. The query object can include the identifier of the real-time data to be queried. This identifier can be set according to actual needs. For example, real-time data is stored in a preset in-memory database in the form of key-value pairs. The query object can correspond to the key name, and the key value can correspond to the specific data content of the real-time data, such as the user identifier and points value in a points leaderboard. The snapshot time can be understood as the time when the snapshot data to be obtained needs to be obtained, such as 11:00 AM in the example above, which can be represented in the form of a timestamp.

[0038] Step 202: Determine the target instruction corresponding to the query object before the snapshot time based on the historical instruction data, wherein the historical instruction data includes data corresponding to the historical instructions obtained by reading the preset persistent file of the preset memory database, and the historical instructions are executed by the business side against the preset memory database to generate the real-time data.

[0039] For example, the data processing service can pre-configure address information corresponding to a preset memory database, such as Internet Protocol (IP) and port number, to read preset persistent files from the preset memory database. After receiving a snapshot query request from the service side, the data processing service can read the preset persistent files to obtain historical instruction data; alternatively, the data processing service can read the preset persistent files at a preset frequency, save historical instruction data, and retrieve the pre-stored historical instruction data after receiving a snapshot query request from the service side. Optionally, after reading the preset persistent files, the data processing service can notify the preset memory database to delete the preset persistent files that have been read by the data processing device, thereby saving storage space on the device hosting the preset memory database.

[0040] For example, each instruction in the historical instruction data can contain the corresponding operation object and operation time. After obtaining the historical instruction data, the instruction data whose operation object is the query object and whose operation time is before the snapshot time can be filtered out. That is, the operation instruction data that the business side has executed on the query object between the snapshot time, and thus obtain the target instruction.

[0041] Step 203: Execute the target instruction for the preset memory database to cause the preset memory database to generate snapshot data corresponding to the snapshot query request.

[0042] For example, after determining the target instruction, the data processing service operates on the preset memory database by executing the target instruction, so that the preset memory database can re-execute the relevant operations between the snapshot times for the query object to obtain the real-time data corresponding to the snapshot time as snapshot data.

[0043] For example, to avoid affecting real-time data at the current moment, such as preventing errors in real-time data that needs to be displayed in real time, operations on the preset in-memory database can be re-executed from the initial state. For example, if the query object is today's points leaderboard, it starts from an empty leaderboard and executes the points accumulation operation before 11:00 AM according to the target instruction. Specifically, assuming the points leaderboard includes user A, user A obtains 200 points at 10:00 AM, 100 points at 10:30 AM, and 300 points at 1:00 PM, the business side performs data write operations on the preset in-memory database at the above three time points to accumulate points. If the current time is 2:00 PM, then in the current real-time points leaderboard, user A's points are 600. If the snapshot time is 11:00 AM, the target instruction includes operations to accumulate 200 points and 100 points. When the data processing service executes the target instruction, the preset in-memory database accumulates 200 points and 100 points sequentially from 0, resulting in user A's points being 300. Optionally, to facilitate the distinction between snapshot data and real-time data at the current moment, the query object can be renamed when executing the target instruction. This can be done by adding a suffix to the query object, where the suffix can be associated with the snapshot time. Alternatively, if the query object corresponds to a key name, the key name in the target instruction is determined based on the query object and the snapshot time. For example, the snapshot time can be added as a suffix to the original key name (query object) to obtain the new key name in the target instruction. For instance, the original key name is xx, and the new key name is xx-11am.

[0044] Step 204: Notify the service side to read the snapshot data from the preset memory database.

[0045] For example, after executing the target instruction, the data processing service can instruct the business side to read snapshot data from a preset in-memory database. Optionally, if the query object has been renamed, the business side can read the corresponding snapshot data from the preset in-memory database according to pre-agreed renaming rules or according to the new query object notified by the data processing service.

[0046] The data processing method provided in this application embodiment involves a device outside the service side responding to a snapshot query request sent by the service side. Based on historical instruction data, the target instruction executed by the service side on a preset memory database before the snapshot time corresponding to the query object is determined. By re-executing the target instruction, the preset memory database generates snapshot data corresponding to the snapshot time, and the service side is notified to read from the preset memory database. The service side does not need to collect and store snapshot data in advance, and can obtain snapshot data of a specified time corresponding to real-time data while saving storage space on the service side, so as to facilitate the relevant applications of snapshot data.

[0047] In some embodiments, the method may further include: synchronizing a preset persistent file in the preset memory database to a local disk; filtering instruction data corresponding to a preset data structure and a preset object from the preset persistent file on the local disk to obtain first historical instruction data, wherein the preset data structure corresponds to the real-time data, and the preset object includes the query object. The advantage of this configuration is that pre-synchronizing the preset persistent file to the local disk of the data processing device and selectively filtering the instruction data can improve the efficiency of determining the target instruction, thereby increasing the response speed of snapshot query requests and improving the query efficiency of snapshot data.

[0048] For example, the data processing service can configure remote synchronization (rsync) to periodically synchronize a preset persistent file to the local disk. The service then reads the preset persistent file locally, filters out data containing operation instructions for operating on a preset data structure and targeting a preset object, and obtains first historical instruction data. This first historical instruction data can be understood as data from a backup persistent file for the preset data structure and preset object, backed up in the data processing device. Optionally, data other than the first historical instruction data in the preset persistent file on the local disk can be deleted to save storage space on the data processing device.

[0049] Optionally, determining the target instruction corresponding to the query object before the snapshot time based on historical instruction data includes: determining the target instruction corresponding to the query object before the snapshot time based on the first historical instruction data.

[0050] In some embodiments, the method further includes: constructing a corresponding first database operation statement based on the first historical instruction data; executing the first database operation statement on a preset database to store second historical instruction data corresponding to the first historical instruction data in the preset database. The advantage of this configuration is that using a preset database to store historical instruction data ensures the accuracy and reliability of historical instruction data storage and supports the storage of large amounts of data.

[0051] For example, a first database operation statement corresponding to the first historical instruction data can be constructed based on the mapping relationship between each field in the first historical instruction data and each field in the second historical instruction data. The specific construction method can be determined according to the type of the preset database.

[0052] Optionally, the preset database includes a columnar data warehouse. The advantages of this configuration are that it allows reading only the necessary columns, facilitates data aggregation operations on large datasets, and is conducive to data compression. Since historical instruction data is typically substantial, columnar data warehouses offer significant advantages over databases like MySQL. The specific type of columnar data warehouse is not limited; for example, it could be Clickhouse (CK), a columnar database with a Massively Parallel Processing (MPP) architecture that enables data querying using Structured Query Language (SQL) statements and supports offline services.

[0053] Optionally, after executing the first database operation statement against the preset database, the process may include deleting the first historical instruction data from the local disk. This arrangement frees up local disk storage space on the data processing device.

[0054] Optionally, determining the target instruction corresponding to the query object before the snapshot time based on historical instruction data includes: constructing a second database operation statement based on the query object and the snapshot time; executing the second database operation statement on a preset database to query the target data corresponding to the query object before the snapshot time in the second historical instruction data; and determining the corresponding target instruction based on the target data. For example, when it is necessary to determine the target instruction, a second database operation statement is constructed. This statement can retrieve database data before the snapshot time for the query object, i.e., the target data, and then, based on the target data, restore the corresponding instruction for the preset memory database, i.e., the target instruction, through field mapping or other methods.

[0055] Optionally, it may also include: clearing the first historical instruction data in the local disk of the data processing device using a preset cleaning cycle, thereby freeing up the local disk storage space of the data processing device.

[0056] For example, in some embodiments, while storing the first historical instruction data on the local disk, a second historical instruction data can be stored using a preset database to accommodate snapshot data queries under different circumstances. For instance, in a first query scenario, the target instruction corresponding to the query object before the snapshot time is determined based on the first historical instruction data; in a second query scenario, the target instruction corresponding to the query object before the snapshot time is determined based on the second historical instruction data. For example, the first query scenario is the default query scenario, that is, the target instruction is determined primarily based on the first historical instruction data; the second query scenario is a scenario where the target instruction cannot be determined based on the first historical instruction data, which can ensure the success rate of snapshot data queries.

[0057] In some embodiments, different query scenarios can be divided based on how close the current time is to the snapshot time. For example, short-term queries use the first historical instruction data, while long-term queries use the second historical instruction data.

[0058] In some embodiments, determining the target instruction preceding the snapshot time corresponding to the query object based on historical instruction data includes: determining whether the difference between the current time and the snapshot time is less than a preset time difference; if so, determining the target instruction preceding the snapshot time corresponding to the query object based on the first historical instruction data. The advantage of this setting is that for recent snapshot queries, determining the target instruction by reading the first historical instruction data from the local disk can improve query efficiency. Optionally, the preset time difference can correspond to a preset cleanup period; for example, if the preset cleanup period is 1 day, then the preset time difference can be 1 day.

[0059] In some embodiments, after determining whether the difference between the current time and the snapshot time is less than a preset time difference, the method further includes: if not, constructing a second database operation statement based on the query object and the snapshot time; executing the second database operation statement on a preset database to query target data corresponding to the query object before the snapshot time in the second historical instruction data; and determining the corresponding target instruction based on the target data. The advantage of this configuration is that for long-term snapshot queries, the second historical instruction data can be used, saving disk space on the data processing device and ensuring a high query success rate.

[0060] In some embodiments, the second database operation statement includes aggregation operation items for target fields corresponding to the same type of instruction. The advantage of this setup is that data aggregation allows for the merging of instructions of the same type, reducing the number of target instructions and improving backtracking efficiency. For example, the same type of instruction could be a zincby instruction, and the aggregation operation items could include field grouping items (such as group by field) and summation items (such as sum). As in the example above, if the target field could be the field corresponding to user a, then the operation instructions for accumulating 200 points and accumulating 100 points for user a can be merged into a target instruction accumulating 300 points.

[0061] In some embodiments, the preset in-memory database includes a master database and a slave database, with preset persistence enabled in the slave database, and the preset persistent file generated in the slave database. This configuration reduces the impact on the master database's performance and improves the overall performance of the preset in-memory database.

[0062] Figure 3 This is a flowchart illustrating another data processing method provided in this application embodiment. Based on the above-described optional embodiments, the method may include:

[0063] Step 301: Synchronize the preset persistent files in the preset memory database to the local disk.

[0064] In this embodiment of the application, historical instruction data can be collected by the data processing service during the business process. Figure 4 This is a schematic diagram illustrating the process of collecting historical instruction data, provided in an embodiment of this application. Figure 5 This is a schematic diagram of the data flow of instruction data provided in an embodiment of this application, which can be referred to. Figure 4 and Figure 5 To understand.

[0065] For example, taking the pre-defined in-memory database Redis as an example, it is a No-SQL non-relational in-memory database with good performance and high reliability, making it well-suited for storing real-time data. The Redis in this embodiment includes a Redis master and a Redis slave. AOF configuration can be enabled on the Redis slave. To save computing power, it can be set to everysec mode, i.e., write-back mode every second. During Redis operations on the business side, after each write command is executed, the log is first written to the memory buffer of the AOF file. Every second, the contents of the buffer are written to the disk to obtain the AOF file. Figure 4As shown, after the business side writes data to the Redis master, the Redis master synchronizes the data to the Redis slave. The Redis slave then writes the AOF file to disk, which is then collected by the acquisition side, i.e., the data processing side. The data processing service can be configured to periodically synchronize the AOF file to the local disk using rsync. Taking real-time data, such as ranking data, as an example, the real-time data is stored in Redis in zset format.

[0066] Optionally, since Redis machines are generally configured with large memory and small hard disks, after the data processing service synchronizes the AOF file via rsync, the AOF file of the Redis machine can be retained for only one day or a shorter period (which can be determined according to the frequency of rsync). This can solve the problem of not being able to save AOF files for a long time under the characteristics of large memory and small hard disks.

[0067] Step 302: Select instruction data corresponding to the preset data structure and preset object from the preset persistent files in the local disk to obtain the first historical instruction data.

[0068] The preset data structure corresponds to the real-time data, and the preset object includes the query object.

[0069] For example, after the data processing service reads the AOF file locally, it filters out the instruction data of the zset key specified by the business side according to the format of the AOF file to obtain the first historical instruction data.

[0070] Taking Redis 7.0's AOF file rules as an example, each line is a data unit, that is, a command data. For example, [*number] represents which command was executed, and [$number] represents the length of the next line. The last line of Redis 7.0 is timestamp information, accurate to the second. By matching operation commands such as "ZADD", "ZINCRBY" and "ZREM", zset-related operations can be filtered out. Then, by matching the zset key configured on the business side, the relevant operation commands can be filtered out to obtain the first historical command data.

[0071] like Figure 5 As shown, the full AOF file in the Redis slave database can be periodically cleaned up after being synchronized to the data processing device. The data processing device can also clean up the full AOF file by filtering it, resulting in a filtered backup AOF file, which is the first historical instruction data.

[0072] Step 303: Construct a corresponding first database operation statement based on the first historical instruction data, and execute the first database operation statement for the preset database to store the second historical instruction data corresponding to the first historical instruction data in the preset database.

[0073] For example, the default database is CK. The data processing service transforms each instruction in the first historical instruction data, such as into a corresponding INSERT SQL statement, to ensure the integrity of the CK table.

[0074] For example, the filtered AOF files on the local disk of the data processing device can also be cleaned periodically using a preset cleanup cycle, which corresponds to a preset time difference.

[0075] Step 304: Receive a snapshot query request from the service side for real-time data stored in the preset memory database.

[0076] The snapshot query request includes the query object and the snapshot time.

[0077] Step 305: Determine whether the difference between the current time and the snapshot time is less than the preset time difference. If yes, proceed to step 306; otherwise, proceed to step 307.

[0078] Step 306: Determine the target instruction before the snapshot time corresponding to the query object based on the first historical instruction data, and then execute step 308.

[0079] Step 307: Construct a second database operation statement based on the query object and snapshot time, execute the second database operation statement for the preset database, and query the target data before the snapshot time corresponding to the query object in the second historical instruction data, and determine the corresponding target instruction based on the target data.

[0080] Step 308: Execute the target instruction for the preset memory database so that the preset memory database generates snapshot data corresponding to the snapshot query request.

[0081] For example, the operation object in the target instruction, i.e. the key, is obtained by adding a suffix to the snapshot time of the query object.

[0082] Step 309: Notify the business side to read snapshot data from the preset memory database.

[0083] For example, the business side can query snapshot data by adding a snapshot time suffix to the query object. The snapshot time suffix of the query object can be used as the version identifier (ID) of the snapshot data.

[0084] The data processing method provided in this application embodiment synchronizes preset persistent files in the preset memory database to the local disk during normal operation of the preset memory database on the business side. It then filters and backs up the data structure corresponding to the real-time data specified by the business side, as well as the instruction data of preset objects. Simultaneously, it converts the backed-up historical instruction data into the preset database for storage. When a snapshot query request is received from the business side, the method selects the corresponding target instruction based on the proximity of the snapshot time to the current time, choosing from historical instruction data from different sources. This target instruction is then executed on the preset memory database, generating snapshot data for the business side to read. By adopting the above technical solution, snapshot backtracking at any time point can be supported, facilitating relevant analysis and optimization on the business side. This solves the machine pain point of large memory and small hard disk in memory databases, enabling the storage of historical instruction data for a relatively long period and targeted filtering to improve subsequent query efficiency, while ensuring the accuracy and success rate of snapshot queries.

[0085] Figure 6 This is a structural block diagram of a data processing apparatus provided in an embodiment of this application. The apparatus can be implemented by software and / or hardware, and is generally integrated into a data processing device. It can perform data processing by executing data processing methods. Figure 6 As shown, the device includes:

[0086] The snapshot query request receiving module 601 is used to receive a snapshot query request sent by the service side for real-time data stored in a preset memory database, wherein the snapshot query request includes the query object and the snapshot time.

[0087] The target instruction determination module 602 is used to determine the target instruction corresponding to the query object before the snapshot time based on historical instruction data. The historical instruction data includes data corresponding to historical instructions obtained by reading a preset persistent file of the preset memory database. The historical instructions are executed by the business side against the preset memory database to generate the real-time data.

[0088] The target instruction execution module 603 is used to execute the target instruction on the preset memory database so that the preset memory database generates snapshot data corresponding to the snapshot query request.

[0089] The notification module 604 is used to notify the service side to read the snapshot data from the preset memory database.

[0090] The data processing apparatus provided in this application embodiment allows a device outside the business side to respond to a snapshot query request sent by the business side. Based on historical instruction data, the target instruction executed by the business side on a preset memory database before the snapshot time corresponding to the query object is determined. By re-executing the target instruction, the preset memory database generates snapshot data corresponding to the snapshot time, and the business side is notified to read from the preset memory database. The business side does not need to collect and store snapshot data in advance, and can obtain snapshot data of a specified time corresponding to real-time data while saving storage space on the business side, so as to facilitate the relevant applications of snapshot data.

[0091] Optionally, the device may also include:

[0092] The file synchronization module is used to synchronize the preset persistent files in the preset memory database to the local disk;

[0093] The filtering module is used to filter instruction data corresponding to a preset data structure and a preset object from a preset persistent file on the local disk to obtain first historical instruction data, wherein the preset data structure corresponds to the real-time data, and the preset object includes the query object.

[0094] Optionally, the device may also include:

[0095] The statement construction module is used to construct the corresponding first database operation statement based on the first historical instruction data;

[0096] The statement execution module is used to execute the first database operation statement on a preset database to store the second historical instruction data corresponding to the first historical instruction data in the preset database.

[0097] Optionally, the target instruction determination module includes:

[0098] A time determination unit is used to determine whether the difference between the current time and the snapshot time is less than a preset time difference;

[0099] The first target instruction determination unit is used to determine the target instruction corresponding to the query object before the snapshot time based on the first historical instruction data, when the judgment result of the time determination unit is yes.

[0100] Optionally, the target instruction determination module further includes:

[0101] The second target instruction determination unit is used to construct a second database operation statement based on the query object and the snapshot time when the judgment result of the time determination unit is negative; execute the second database operation statement for a preset database to query the target data corresponding to the query object before the snapshot time in the second historical instruction data; and determine the corresponding target instruction according to the target data.

[0102] Optionally, the second database operation statement includes an aggregation operation item for the target field corresponding to the same type of instruction.

[0103] Optionally, the real-time data is stored in the preset memory database in the form of key-value pairs, the query object corresponds to the key name, and the key name in the target instruction is determined according to the query object and the snapshot time.

[0104] Optionally, the preset memory database includes a master database and a slave database, the slave database has a preset persistence function enabled, and the preset persistence file is generated in the slave database.

[0105] This application provides a data processing device, which can integrate the data processing apparatus provided in this application. Figure 7 This is a structural block diagram of a data processing device provided in an embodiment of this application. The data processing device 700 includes a processor 702 and a memory 701 communicatively connected to the processor 702. The memory 701 stores a computer program executable by the processor 702. The computer program is executed by the processor 702 to enable the processor 702 to perform the data processing method described in any embodiment of this application. The number of processors can be one or more. Figure 7 Let's take a processor as an example.

[0106] This application also provides a computer-readable storage medium storing a computer program that enables a processor to implement the data processing method described in any embodiment of this application when executed.

[0107] This application also provides a computer program product, which includes a computer program that, when executed by a processor, implements the data processing method provided in this application.

[0108] The data processing apparatus, device, storage medium, and product provided in the above embodiments can execute the data processing method provided in any embodiment of this application, and have the corresponding functional modules and beneficial effects for executing the method. Technical details not described in detail in the above embodiments can be found in the data processing method provided in any embodiment of this application.

Claims

1. A data processing method, characterized in that, include: Receive a snapshot query request sent by the service side for real-time data stored in a preset memory database, wherein the snapshot query request includes the query object and the snapshot time; The target instruction corresponding to the query object before the snapshot time is determined based on the historical instruction data. The historical instruction data includes data corresponding to the historical instructions obtained by reading the preset persistent file of the preset memory database. The historical instructions are executed by the business side against the preset memory database to generate the real-time data. The target instruction is executed on the preset memory database to cause the preset memory database to generate snapshot data corresponding to the snapshot query request. The service side is notified to read the snapshot data from the preset memory database; The method further includes: synchronizing a preset persistent file in the preset memory database to a local disk; First historical instruction data is obtained by filtering instruction data corresponding to a preset data structure and a preset object from a preset persistent file in the local disk, wherein the preset data structure corresponds to the real-time data and the preset object includes the query object; The first historical instruction data is the data of the backup persistent file for the preset data structure and preset object, which is backed up in the data processing device.

2. The method according to claim 1, characterized in that, Also includes: Construct the corresponding first database operation statement based on the first historical instruction data; The first database operation statement is executed on the preset database to store the second historical instruction data corresponding to the first historical instruction data in the preset database.

3. The method according to claim 2, characterized in that, The step of determining the target instruction corresponding to the query object before the snapshot time based on historical instruction data includes: Determine whether the difference between the current time and the snapshot time is less than a preset time difference; If so, then the target instruction corresponding to the query object before the snapshot time is determined based on the first historical instruction data.

4. The method according to claim 3, characterized in that, After determining whether the difference between the current time and the snapshot time is less than a preset time difference, the method further includes: If not, then construct a second database operation statement based on the query object and the snapshot time; The second database operation statement is executed for the preset database to query the target data before the snapshot time corresponding to the query object in the second historical instruction data; The corresponding target instruction is determined based on the target data.

5. The method according to claim 4, characterized in that, The second database operation statement includes aggregation operation items for the target fields corresponding to the same type of instruction.

6. The method according to claim 1, characterized in that, The real-time data is stored in the preset memory database in the form of key-value pairs. The query object corresponds to the key name, and the key name in the target instruction is determined according to the query object and the snapshot time.

7. The method according to claim 1, characterized in that, The preset memory database includes a master database and a slave database. The preset persistence function is enabled in the slave database, and the preset persistent file is generated in the slave database.

8. A data processing apparatus, characterized in that, include: The snapshot query request receiving module is used to receive snapshot query requests sent by the service side for real-time data stored in a preset memory database, wherein the snapshot query request includes the query object and the snapshot time; The target instruction determination module is used to determine the target instruction corresponding to the query object before the snapshot time based on historical instruction data. The historical instruction data includes data corresponding to historical instructions obtained by reading a preset persistent file of the preset memory database. The historical instructions are executed by the business side against the preset memory database to generate the real-time data. The target instruction execution module is used to execute the target instruction on the preset memory database, so that the preset memory database generates snapshot data corresponding to the snapshot query request; The notification module is used to notify the service side to read the snapshot data from the preset memory database; The file synchronization module is used to synchronize the preset persistent files in the preset memory database to the local disk; The filtering module is used to filter instruction data corresponding to a preset data structure and a preset object from a preset persistent file in the local disk to obtain first historical instruction data. The preset data structure corresponds to the real-time data, and the preset object includes the query object. The first historical instruction data is data from a backup persistent file for the preset data structure and preset object that is backed up in the data processing device.

9. A data processing device, characterized in that, The data processing device includes: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the data processing method according to any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the data processing method according to any one of claims 1-7.

11. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the data processing method according to any one of claims 1-7.

Images