A data processing method and device

Abstract

The present invention discloses a data processing method and device, and relates to the field of computer technology. A specific implementation of the method includes: receiving a request to flush data from a database into Redis, reading full data from the database according to the received request, and obtaining incremental data; using Lua language, processing the data in Redis according to the unique identifier of the incremental data, and generating an incremental identifier of the incremental data to complete incremental caching; using Lua language, judging whether there is an incremental identifier of the full data, if so, determining that the full data has been incrementally cached, if not, flushing the full data from the database into Redis to complete the full cache. This implementation ensures the accuracy of data flushing and the real-time update of cached data, and can also avoid the situation where data is overwritten, without the need for manual intervention.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a data processing method and device. Background Art

[0002] For scenarios that provide high-concurrency read and write services, the database cannot support the concurrency, and Redis needs to be used for data caching. If Redis data is lost or disordered, or if Redis is initially connected, cache initialization is required, that is, full cache.

[0003] At present, during the full cache period, the following solutions are mainly used to process incremental data: Solution 1, suspend incremental caching, specifically record the incremental data during the full cache period, and start the incremental cache service after the full cache is completed to flush the incremental data into Redis; Solution 2, after the full cache is completed, use the calibration program to query the data from the database and Redis in turn for comparison, and re-flush if they are not synchronized.

[0004] In the process of implementing the present invention, it was found that there are at least the following problems in the prior art: for solution one, it involves tedious manual operations, sacrificing the real-time nature of the cached data, and there is a possibility of a large backlog of cache synchronization requests, which puts pressure on the server; for solution two, the accuracy of the cached data cannot be guaranteed. If the data cache value is found to be abnormal later, manual intervention synchronization is required. Summary of the invention

[0005] In view of this, an embodiment of the present invention provides a data processing method and device, which can ensure the accuracy of data flushing and the real-time update of cached data, and can also avoid the situation where data is overwritten without the need for manual intervention.

[0006] To achieve the above objective, according to one aspect of an embodiment of the present invention, a data processing method is provided.

[0007] The data processing method of an embodiment of the present invention includes: receiving a request to flush data from a database into Redis, reading full data from the database and obtaining incremental data according to the received request; using Lua language to process the data in Redis according to a unique identifier of the incremental data, and generating an incremental identifier of the incremental data to complete incremental caching; using Lua language to determine whether there is an incremental identifier of the full data, and if so, determining that the full data has been incrementally cached, and if not, flushing the full data from the database into Redis to complete full caching.

[0008] Optionally, the using of Lua language to process the data in the Redis according to the unique identifier of the incremental data and generate the incremental identifier of the incremental data includes: executing the Lua language to operate on the data corresponding to the incremental data in the Redis according to the unique identifier of the incremental data and the change logic corresponding to the incremental data; generating the incremental identifier of the incremental data according to the unique identifier of the incremental data, and using the Redis hash tag to route the unique identifier of the incremental data and the incremental identifier of the incremental data to the same hash slot.

[0009] Optionally, before using the Lua language to process the data in the Redis according to the unique identifier of the incremental data, the method also includes: determining whether the full data flush switch is turned on; if so, using the Lua language to process the data in the Redis according to the unique identifier of the incremental data; if not, according to the unique identifier of the incremental data and the change logic corresponding to the incremental data, operating on the data corresponding to the incremental data in the Redis.

[0010] Optionally, before using the Lua language to determine whether there is an incremental identifier of the full data, the method also includes: recording a first location information, wherein the first location information is the data location information for the start of flushing the full data; and after completing the full cache, the method also includes: recording a second location information, wherein the second location information is the data location information for the end of flushing the full data.

[0011] Optionally, the method further includes: deleting, in the Redis, an incremental identifier generated between the first location information and the second location information.

[0012] Optionally, before using the Lua language to determine whether there is an incremental identifier of the full data, the method further includes: turning on a full data flush switch; and after completing the full cache, the method further includes: turning off the full data flush switch.

[0013] To achieve the above objective, according to another aspect of an embodiment of the present invention, a data processing device is provided.

[0014] The data processing device of an embodiment of the present invention includes: a data acquisition module, which is used to receive a request to flush data from a database into Redis, read the full data from the database according to the received request, and obtain the incremental data; an incremental cache module, which is used to use the Lua language to process the data in the Redis according to the unique identifier of the incremental data, and generate an incremental identifier of the incremental data to complete incremental caching; a full cache module, which is used to use the Lua language to determine whether there is an incremental identifier of the full data. If so, it is determined that the full data has been incrementally cached. If not, the full data is flushed from the database into the Redis to complete full caching.

[0015] Optionally, the incremental cache module is also used to: execute the Lua language, and according to the unique identifier of the incremental data and the change logic corresponding to the incremental data, operate on the data corresponding to the incremental data in the Redis; according to the unique identifier of the incremental data, generate an incremental identifier of the incremental data, and use the Redis hash tag to route the unique identifier of the incremental data and the incremental identifier of the incremental data to the same hash slot.

[0016] Optionally, the incremental cache module is also used to: determine whether the full data flush switch is turned on; if so, use the Lua language to process the data in the Redis according to the unique identifier of the incremental data; if not, according to the unique identifier of the incremental data and the change logic corresponding to the incremental data, operate on the data corresponding to the incremental data in the Redis.

[0017] Optionally, the full cache module is also used to: record the first location information, wherein the first location information is the data location information at the start of full data flushing; and record the second location information, wherein the second location information is the data location information at the end of full data flushing.

[0018] Optionally, the full cache module is also used to: delete the incremental identifier generated between the first location information and the second location information in the Redis.

[0019] Optionally, the full cache module is further used to: turn on a full data flushing switch; and turn off a full data flushing switch.

[0020] To achieve the above objective, according to another aspect of an embodiment of the present invention, an electronic device is provided.

[0021] An electronic device according to an embodiment of the present invention includes: one or more processors; a storage device for storing one or more programs, and when the one or more programs are executed by the one or more processors, the one or more processors implement the data processing method according to the embodiment of the present invention.

[0022] To achieve the above objective, according to another aspect of an embodiment of the present invention, a computer readable medium is provided.

[0023] A computer-readable medium according to an embodiment of the present invention stores a computer program, and when the program is executed by a processor, the data processing method according to the embodiment of the present invention is implemented.

[0024] One embodiment of the above invention has the following advantages or beneficial effects: when flushing incremental data into Redis, the atomicity of the Lua language is used to add an incremental identifier of the incremental data to the hash slot where the unique identifier of the incremental data is located, so that the incremental identifier can be used to represent the cache status of the incremental data; when flushing the full amount of data into Redis, Lua can be used to automatically detect whether the full amount of data has been incrementally cached, which ensures the accuracy of data flushing and avoids the situation where data is overwritten. In addition, during the process of flushing the full amount of data into the Redis cache, the incremental cache synchronization does not need to be paused, which ensures the real-time update of the cached data and avoids the backlog of cache synchronization requests. At the same time, the operation of flushing the full amount of data into Redis is completely automated, and does not require tedious manual intervention, avoiding the risk of misoperation that may occur during manual operation.

[0025] The further effects of the above-mentioned non-conventional optional manner will be described below in conjunction with the specific implementation manner. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] The accompanying drawings are used to better understand the present invention and do not constitute an improper limitation of the present invention.

[0027] Figure 1 is a schematic diagram of main steps of a data processing method according to an embodiment of the present invention;

[0028] Figure 2 is a schematic diagram of the main process of flushing incremental data into Redis according to an embodiment of the present invention;

[0029] Figure 3 It is a schematic diagram of the main process of flushing all data into Redis according to an embodiment of the present invention;

[0030] Figure 4 is a schematic diagram of main modules of a data processing device according to an embodiment of the present invention;

[0031] Figure 5is an exemplary system architecture diagram to which embodiments of the present invention may be applied;

[0032] Figure 6 It is a schematic diagram of the structure of a computer system of a terminal device or a server suitable for implementing an embodiment of the present invention. DETAILED DESCRIPTION

[0033] The following is a description of exemplary embodiments of the present invention in conjunction with the accompanying drawings, including various details of the embodiments of the present invention to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present invention. Similarly, for clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0034] Figure 1 FIG. 1 is a schematic diagram of the main steps of the data processing method according to an embodiment of the present invention. Figure 1 As shown, the main steps of the data processing method may include:

[0035] Step S101, receiving a request to flush data from a database into Redis, reading full data from the database according to the received request, and obtaining incremental data;

[0036] Step S102, using Lua language, according to the unique identifier of the incremental data, the data in Redis is processed, and an incremental identifier of the incremental data is generated to complete the incremental cache;

[0037] Step S103, using Lua language, determine whether there is an incremental mark for the full data. If so, determine that the full data has been incrementally cached. If not, flush the full data from the database into Redis to complete the full cache.

[0038] When using Redis for data caching, if Redis data is lost or disordered, or a data table in the database is initially connected to Redis, a cache initialization operation is required, that is, the table data needs to be read in full and flushed into Redis, that is, full caching is required. In step S101, a request to flush data from the database into Redis is received, and then the full amount of data can be read from the database according to the received request so as to flush the full amount of data into Redis. Since there is incremental data in the database, that is, changed data, during the process of flushing the full amount of data into Redis, the incremental data can be obtained so that the incremental data can be flushed into Redis later.

[0039] Among them, the database is a warehouse for storing data, and its essence is a file system. Data is stored in a specific format, and users can add, modify, delete and query data in the database. In an embodiment of the present invention, the database can be a MySQL database, that is, an open source and free database. The database stores data in tables as organizational units; according to the data type specified by the table field, data can be filled in one by one, and the information in each row of the table can be called a record. The full amount of data refers to all the data in the database. In an embodiment of the present invention, the full amount of data can be determined by the received request. For example, the received request specifies that all data in a certain data table A in the database needs to be cached, then the full amount of data is all the data in the data table A.

[0040] In addition, in the embodiment of the present invention, incremental data refers to data that has changed in the process of caching the full amount of data. If the database is a MySQL database, Canal can be used to simulate MySQL, read MySQL logs from the nodes, and monitor MySQL data changes in real time to synchronize Redis. Among them, the purpose of Canal is to provide incremental data subscription and consumption based on database incremental log analysis. Currently, it mainly supports MySQL database. Therefore, in the embodiment of the present invention, Canal can be used to monitor database data changes and obtain incremental data.

[0041] After obtaining the full data and incremental data through step S101, steps S102 and S103 illustrate the process of flushing the incremental data and the full data into Redis using the Lua language. Lua is a lightweight and compact scripting language written in standard C language and open in source code form. It is designed to be embedded in applications, thereby providing flexible expansion and customization functions for applications. In the prior art, both schemes 1 and 2 are non-atomic operations for flushing data into Redis, so the accuracy and real-time performance of data flushing cannot be guaranteed. Among them, atomic operation means that a business logic must not be split. In the embodiment of the present invention, the Lua language is used to perform data flushing, and the atomicity characteristics of the Lua language can be used to ensure the accuracy and real-time performance of data flushing and avoid data overwriting.

[0042] In step S102, Lua language can be used to process the data in Redis according to the unique identifier of the incremental data, and generate an incremental identifier of the incremental data to complete the incremental cache. If an incremental identifier of some incremental data is generated, it means that the incremental data is flushed into Redis, so the incremental identifier can be used to indicate the cache update of the incremental data.

[0043] Furthermore, using Lua language to process the data in Redis according to the unique identifier of the incremental data and generate the incremental identifier of the incremental data can include: executing Lua language, that is, executing Lua language, according to the unique identifier of the incremental data, according to the change logic corresponding to the incremental data, operating the data corresponding to the incremental data in Redis; generating the incremental identifier of the incremental data according to the unique identifier of the incremental data, and using Redis hash tag to route the unique identifier of the incremental data and the incremental identifier of the incremental data to the same hash slot.

[0044] That is to say, the Lua language is executed, and specific logical operations such as add, delete, modify and query operations are performed on the data corresponding to the incremental data in Redis according to the unique identifier of the incremental data, such as the key of the incremental data. In addition, when performing specific logical operations, the incremental identifier of the incremental data can be generated according to the unique identifier of the incremental data, such as the unique identifier is key, and the generated incremental identifier is updateFlag:{key}. Since the Lua language requires that all keys operated must be in the same hash slot, in an embodiment of the present invention, the Redis hash tag (i.e., HashTag, which is a way to ensure that two keys are in the same hash slot) can be used to route the unique identifier key of the incremental data and the incremental identifier updateFlag:{key} of the incremental data to the same hash slot. That is, the incremental identifier updateFlag:{key} of the incremental data is added in the hash slot where the unique identifier key is located. If the unique identifier key of the incremental data and the incremental identifier updateFlag:{key} of the incremental data are routed to different hash slots, then the Lua language will report an error when performing the operation. In addition, the reason why the key is enclosed in {} is that HashTag is used to ensure that it and the original key will be routed to the same hash slot. Of course, other forms of expression may also be used, and the embodiment of the present invention is not limited to this.

[0045] In addition, before flushing the incremental data into Redis, that is, before executing the incremental cache, you can first determine whether the full cache is currently being executed. If the full cache is not currently being executed, there will be no data overwriting, so the incremental data can be directly operated in Redis according to the database's add, delete, modify and query logic. Therefore, as an embodiment of the present invention, before using the Lua language to process the data in Redis according to the unique identifier of the incremental data, the data processing method may also include: determining whether the full data flush switch is turned on; if so, using the Lua language to process the data in Redis according to the unique identifier of the incremental data; if not, according to the unique identifier of the incremental data, according to the change logic corresponding to the incremental data, the data corresponding to the incremental data in Redis is operated.

[0046] Specifically, if the full data flush switch is turned on, it means that the full data is currently being flushed into Redis, so you can use the Lua language and the unique identifier of the incremental data to operate on the data in Redis. If the full data flush switch is turned off, that is, the full data flush switch is not turned on, it means that only the incremental data is currently being flushed into Redis, so you can operate on the data in Redis.

[0047] Figure 2 FIG. 1 is a schematic diagram of the main process of flushing incremental data into Redis according to an embodiment of the present invention. Figure 2 As shown in the figure, the main process of flushing incremental data into Redis may include:

[0048] Step S201, using Canal to obtain incremental data;

[0049] Step S202, determine whether the full data flash switch is turned on, if so, execute step S203, if not, execute step S205;

[0050] Step S203, executing Lua language, and operating the data corresponding to the incremental data in Redis according to the unique identifier of the incremental data and the change logic corresponding to the incremental data;

[0051] Step S204, generating an incremental identifier of the incremental data according to the unique identifier of the incremental data, and routing the unique identifier of the incremental data and the incremental identifier of the incremental data to the same hash slot by using a Redis hash tag;

[0052] Step S205, according to the unique identifier of the incremental data and the change logic corresponding to the incremental data, operations are performed on the data corresponding to the incremental data in Redis.

[0053] It should be noted that step S203 and step S204 are Lua atomic operations. When the embodiment of the present invention flushes the incremental data into Redis, the atomicity of the Lua language is used to add an incremental identifier of the incremental data to the hash slot where the unique identifier of the incremental data is located, so that the incremental identifier can be used to indicate the cache status of the incremental data.

[0054] In step S103, Lua language can be used to determine whether there is an incremental identifier for the full data. If so, it is determined that the full data has been incrementally cached. If not, the full data is flushed from the database to Redis to complete the full cache.

[0055] Before flushing the full data into Redis, the full data can be read from the database and converted into the data format of Redis to facilitate subsequent flushing of the full data into Redis. As described above, when flushing incremental data into Redis, an incremental identifier of the incremental data can be generated. Therefore, for the full data, if there is an incremental identifier of the full data, it means that the full data has been flushed into Redis through incremental caching. Therefore, when performing full caching, the Lua language can be used to check whether the incremental identifier of the full data already exists according to the unique identifier key of the full data. If it exists, it means that it has been flushed into Redis during incremental caching. Afterwards, if the full data changes again, the changed data can also be flushed into Redis through incremental caching. If it does not exist, it means that the full data has not been flushed into Redis, so there will be no data overwriting, and the full data can be flushed into Redis. Read the full data from the database in a loop, and then analyze the read full data according to the above process until the data in the database is read. It should be noted that, since the data in the database is continuously updated, the full data here refers to the full data corresponding to the request received in step S101.

[0056] Furthermore, before using the Lua language to determine whether there is an incremental identifier for the full amount of data, the data processing method may also include: recording the first point information, wherein the first point information is the data position information at the beginning of flushing the full amount of data; and after completing the full amount of caching, the data processing method also includes: recording the second position information, wherein the second position information is the data position information at the end of flushing the full amount of data.

[0057] Furthermore, the data processing method may also include: deleting the incremental identifier generated between the first location information and the second location information in Redis.

[0058] Before flushing the full data, record the location information of the start of the data flush and print the log, that is, the timestamp of the MySQL incremental log before the full data is flushed. And after the full data is flushed, record the location information of the end of the data flush. Through the start and end location information of the data flush, you can re-parse the MySQL database log to obtain the database incremental information, and traverse and delete all incremental data in Redis. If it crashes before cleaning, you can also manually call the cleanup method to clean it up.

[0059] In addition, before using the Lua language to determine whether there is an incremental mark of the full data, the data processing method also includes: turning on the full data flush switch; and, after completing the full cache, the data processing method also includes: turning off the full data flush switch. That is to say, before the full cache, the full data flush switch needs to be turned on. And, after completing the full cache, the full data flush switch needs to be turned off. After the full data flush switch is turned off, if incremental data is obtained, the incremental data can be directly flushed into Redis according to normal logic.

[0060] Figure 3 FIG. 1 is a schematic diagram of the main process of flushing all data into Redis according to an embodiment of the present invention. Figure 3 As shown in the figure, the main process of flushing all data into Redis may include:

[0061] Step S301, recording the first bit point information, wherein the first bit point information is the data bit position information at the start of full data flushing;

[0062] Step S302, turning on the full data flushing switch;

[0063] Step S303, read the full amount of data from the database, and convert the read full amount of data into a storage format corresponding to Redis;

[0064] Step S304, using Lua language, determine whether there is an incremental mark of the full data, if so, execute step S305, if not, execute step S306;

[0065] Step S305, determining that the full amount of data has been incrementally cached;

[0066] Step S306, flush all data from the database into Redis;

[0067] Step S307, determining whether all the data has been read from the database, if so, executing step S308;

[0068] Step S308: Determine that the full cache is completed, and turn off the full data flushing switch;

[0069] Step S309: Record the second location information, wherein the second location information is the data location information at which the full data flushing is completed;

[0070] Step S310: Delete the incremental identifier generated between the first location information and the second location information in Redis.

[0071] In the data processing method provided by the embodiment of the present invention, first, the full amount of data is read from the database, and the changes in the database are monitored to obtain the incremental data. Then, according to the method described in steps S201 to S205 above, the incremental data is flushed into Redis using the atomicity characteristics of the Lua language. And, when the incremental data is flushed into Redis, the incremental identifier of the incremental data is added to the hash slot where the unique identifier of the incremental data is located, and the incremental update is represented by the incremental identifier. It is also possible to flush the full amount of data into Redis using the Lua language according to the method described in steps S301 to S310 above. And, when the full amount of data is flushed into Redis, Lua can be used to automatically detect whether the full amount of data has been incrementally cached, ensuring the accuracy of the data flushing, and avoiding the situation where the data is overwritten. In addition, in the data processing method provided by the embodiment of the present invention, in the process of flushing the full amount of data into the Redis cache, the incremental cache synchronization does not need to be suspended, which ensures the real-time update of the cached data and avoids the backlog of cache synchronization requests. At the same time, the operation of flushing all data into Redis is completely automated and does not require tedious manual intervention, thus avoiding the risk of misoperation that may occur during manual operation.

[0072] Figure 4 Schematic diagram of main modules of a data processing device according to an embodiment of the present invention. Figure 4 As shown, the main modules of the data processing device 400 may include: a data acquisition module 401 , an incremental cache module 402 and a full cache module 403 .

[0073] Among them, the data acquisition module 401 can be used to: receive a request to flush data from the database into Redis, read the full data from the database according to the received request, and obtain the incremental data; the incremental cache module 402 can be used to: use the Lua language to process the data in Redis according to the unique identifier of the incremental data, and generate an incremental identifier of the incremental data to complete the incremental cache; the full cache module 403 can be used to: use the Lua language to determine whether there is an incremental identifier of the full data. If so, it is determined that the full data has been incrementally cached. If not, the full data is flushed from the database into Redis to complete the full cache.

[0074] As an embodiment of the present invention, the incremental cache module 402 can also be used to: execute Lua language, operate on the data corresponding to the incremental data in Redis according to the unique identifier of the incremental data and the change logic corresponding to the incremental data; generate an incremental identifier of the incremental data according to the unique identifier of the incremental data, and use the Redis hash tag to route the unique identifier of the incremental data and the incremental identifier of the incremental data to the same hash slot.

[0075] As an embodiment of the present invention, the incremental cache module 402 can also be used to: determine whether the full data flush switch is turned on; if so, use the Lua language to process the data in Redis according to the unique identifier of the incremental data; if not, according to the unique identifier of the incremental data and the change logic corresponding to the incremental data, operate on the data corresponding to the incremental data in Redis.

[0076] As an embodiment of the present invention, the full cache module 403 can also be used to: record the first point information, wherein the first point information is the data position information at the start of flushing the full data; and record the second point information, wherein the second point information is the data position information at the end of flushing the full data.

[0077] As an embodiment of the present invention, the full cache module 403 may also be used to delete the incremental identifier generated between the first location information and the second location information in Redis.

[0078] As an embodiment of the present invention, the full cache module 403 may also be used to: turn on a full data flushing switch; and turn off a full data flushing switch.

[0079] According to the data processing device of the embodiment of the present invention, when flushing incremental data into Redis, the atomicity of the Lua language is used to add an incremental identifier of the incremental data in the hash slot where the unique identifier of the incremental data is located, so that the incremental identifier can be used to represent the cache status of the incremental data; when flushing the full amount of data into Redis, Lua can be used to automatically detect whether the full amount of data has been incrementally cached, which ensures the accuracy of data flushing and avoids the situation where data is overwritten. Moreover, in the process of flushing the full amount of data into the Redis cache, the incremental cache synchronization does not need to be paused, which ensures the real-time update of the cached data and avoids the backlog of cache synchronization requests. At the same time, the operation of flushing the full amount of data into Redis is completely automated, and does not require tedious manual intervention, avoiding the risk of misoperation that may occur during manual operation.

[0080] Figure 5 An exemplary system architecture 500 is shown to which the data processing method or data processing device according to the embodiment of the present invention can be applied.

[0081] like Figure 5 As shown, system architecture 500 may include terminal devices 501, 502, 503, a network 504 and a server 505. Network 504 is used to provide a medium for communication links between terminal devices 501, 502, 503 and server 505. Network 504 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0082] Users can use terminal devices 501, 502, 503 to interact with server 505 through network 504 to receive or send messages, etc. Terminal devices 501, 502, 503 can be various electronic devices with display screens and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, etc.

[0083] Server 505 can be a server that provides various services, for example, a background management server that provides support when a user uses terminal devices 501, 502, and 503 to process data (only as an example); for another example, server 505 can complete the data processing of an embodiment of the present invention.

[0084] It should be noted that the data processing method provided in the embodiment of the present invention is generally executed by the server 505 , and accordingly, the data processing device is generally disposed in the server 505 .

[0085] It should be understood that Figure 5 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to the implementation requirements.

[0086] Reference below Figure 6 , which shows a schematic diagram of the structure of a computer system 600 of a terminal device suitable for implementing an embodiment of the present invention. Figure 6 The terminal device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0087] like Figure 6 As shown, the computer system 600 includes a central processing unit (CPU) 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage part 608 into a random access memory (RAM) 603. In the RAM 603, various programs and data required for the operation of the system 600 are also stored. The CPU 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0088] The following components are connected to the I / O interface 605: an input section 606 including a keyboard, a mouse, etc.; an output section 607 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 608 including a hard disk, etc.; and a communication section 609 including a network interface card such as a LAN card, a modem, etc. The communication section 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to the I / O interface 605 as needed. A removable medium 611, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 610 as needed, so that a computer program read therefrom is installed into the storage section 608 as needed.

[0089] In particular, according to the embodiments disclosed in the present invention, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present invention include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 609, and / or installed from the removable medium 611. When the computer program is executed by the central processing unit (CPU) 601, the above-mentioned functions defined in the system of the present invention are executed.

[0090] It should be noted that the computer-readable medium shown in the present invention may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present invention, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0091] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0092] The modules involved in the embodiments of the present invention may be implemented in software or hardware. The modules described may also be provided in a processor. For example, they may be described as follows: a processor includes a data acquisition module, an incremental cache module, and a full cache module. The names of these modules do not, in some cases, constitute limitations on the modules themselves. For example, the data acquisition module may also be described as a module that "receives a request to flush data from a database into Redis, reads full data from the database according to the received request, and obtains incremental data."

[0093] As another aspect, the present invention also provides a computer-readable medium, which may be included in the device described in the above embodiment; or it may exist independently without being assembled into the device. The above computer-readable medium carries one or more programs. When the above one or more programs are executed by a device, the device includes: receiving a request to flush data from a database into Redis, reading the full data from the database according to the received request, and obtaining the incremental data; using the Lua language, processing the data in Redis according to the unique identifier of the incremental data, and generating an incremental identifier of the incremental data to complete the incremental cache; using the Lua language, determining whether there is an incremental identifier of the full data, if so, determining that the full data has been incrementally cached, if not, flushing the full data from the database into Redis to complete the full cache.

[0094] According to the technical solution of the embodiment of the present invention, when the incremental data is flushed into Redis, the atomicity of the Lua language is utilized to add an incremental identifier of the incremental data to the hash slot where the unique identifier of the incremental data is located, so that the incremental identifier can be used to represent the cache status of the incremental data; when the full amount of data is flushed into Redis, Lua can be used to automatically detect whether the full amount of data has been incrementally cached, which ensures the accuracy of the data flushing and avoids the situation where the data is overwritten. Moreover, in the process of flushing the full amount of data into the Redis cache, the incremental cache synchronization does not need to be paused, which ensures the real-time update of the cached data and avoids the backlog of cache synchronization requests. At the same time, the operation of flushing the full amount of data into Redis is completely automated, and does not require tedious manual intervention, avoiding the risk of misoperation that may occur during manual operation.

[0095] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may occur depending on design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A data processing method, characterized in that: include: Receive a request to flush data from a database into Redis, read the full amount of data from the database according to the received request, and obtain incremental data; Using Lua language, according to the unique identifier of the incremental data, the data in the Redis is processed, and an incremental identifier of the incremental data is generated to complete incremental caching; wherein, the data corresponding to the incremental data in the Redis is operated; Using the Lua language, determine whether there is an incremental identifier for the full data. If so, determine that the full data has been incrementally cached. If not, flush the full data from the database into the Redis to complete the full cache.

2. The method according to claim 1, characterized in that The using of Lua language to process the data in the Redis according to the unique identifier of the incremental data and generate the incremental identifier of the incremental data includes: Execute the Lua language, and operate the data corresponding to the incremental data in the Redis according to the unique identifier of the incremental data and the change logic corresponding to the incremental data; According to the unique identifier of the incremental data, an incremental identifier of the incremental data is generated, and the unique identifier of the incremental data and the incremental identifier of the incremental data are routed to the same hash slot by using a Redis hash tag.

3. The method according to claim 2, characterized in that Before processing the data in the Redis according to the unique identifier of the incremental data using the Lua language, the method further includes: Determine whether the full data flush switch is turned on; If yes, use the Lua language to process the data in the Redis according to the unique identifier of the incremental data; If not, then according to the unique identifier of the incremental data and the change logic corresponding to the incremental data, operations are performed on the data corresponding to the incremental data in the Redis.

4. The method according to claim 1, characterized in that Before using the Lua language to determine whether there is an incremental identifier for the full amount of data, the method further includes: Recording the first bit position information, wherein the first bit position information is the data bit position information at the start of flushing the full amount of data; and After the full amount of caching is completed, the method further includes: Record the second location information, wherein the second location information is the data location information at which the full data flushing is completed.

5. The method according to claim 4, characterized in that The method further comprises: The incremental identifier generated between the first location information and the second location information is deleted in the Redis.

6. The method according to claim 1, characterized in that Before using the Lua language to determine whether there is an incremental identifier for the full amount of data, the method further includes: turning on a full amount of data flushing switch; and, After the full amount of cache is completed, the method further includes: turning off the full amount of data flushing switch.

7. A data processing device, characterized in that: include: The data acquisition module is used to receive a request to flush data from the database into Redis, read the full data from the database according to the received request, and obtain the incremental data; An incremental cache module, used to process the data in the Redis using the Lua language according to the unique identifier of the incremental data, and generate an incremental identifier of the incremental data to complete incremental caching; wherein, operations are performed on the data corresponding to the incremental data in the Redis; The full cache module is used to use the Lua language to determine whether there is an incremental identifier of the full data. If so, it is determined that the full data has been incrementally cached. If not, the full data is flushed from the database into the Redis to complete the full cache.

8. The device according to claim 7, characterized in that The incremental cache module is also used for: Execute a specific scripting language Lua, and operate the data corresponding to the incremental data in the Redis according to the unique identifier of the incremental data and the change logic corresponding to the incremental data; According to the unique identifier of the incremental data, an incremental identifier of the incremental data is generated, and the unique identifier of the incremental data and the incremental identifier of the incremental data are routed to the same hash slot by using a Redis hash tag.

9. An electronic device, characterized in that: include: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 6.

10. A computer readable medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images