Data processing method, device, system, medium and program product

Abstract

The present disclosure provides a data processing method, including: obtaining a target data set from a database according to an identification information set, wherein the data in the target data set is data decoupled from the business; storing the data in the target data set in a cache; traversing the data in the database to determine the target data, wherein the target data is the changed data in the target data set; and using the target data to update the associated data in the cache, wherein the associated data is the data already stored in the cache and having the same identification information as the target data. The present disclosure also provides a data processing device, a computer system, a readable storage medium, and a computer program product.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and more specifically, to a data processing method, device, computer system, readable storage medium, and computer program product. Background Art

[0002] In Internet systems, data is generally stored in a database. When the application server starts, data is called from the database through remote requests and loaded into the system memory in sequence.

[0003] In the process of implementing the concept of the present disclosure, the inventors found that there are at least the following problems in the related technology: the data and business are too deeply coupled, resulting in excessive data size, slow processing speed each time data is loaded, and frequent queries put pressure on the database. Summary of the invention

[0004] In view of this, the present disclosure provides a data processing method, apparatus, computer system, readable storage medium, and computer program product.

[0005] One aspect of the present disclosure provides a data processing method, comprising:

[0006] Acquire a target data set from a database according to the identification information set, wherein the data in the target data set is data decoupled from the business;

[0007] Store the data in the target data set into the cache;

[0008] Traversing the data in the database to determine target data, wherein the target data is the changed data in the target data set; and

[0009] The associated data in the cache is updated using the target data, wherein the associated data is data already stored in the cache and having the same identification information as the target data.

[0010] According to an embodiment of the present disclosure, the data processing method further includes:

[0011] storing the target identification information corresponding to the target data in a cache queue of the cache; and

[0012] Updates the data version used to characterize the target identification information of different batches in the cache queue.

[0013] According to an embodiment of the present disclosure, the data processing method further includes:

[0014] receiving an instruction for loading a target data set in the cache into the system memory; and

[0015] In response to the instruction, the target data set in the cache is loaded into the system memory.

[0016] According to an embodiment of the present disclosure, the data processing method further includes:

[0017] Determine whether the target data set has been fully loaded into the system memory;

[0018] When the target data set has been fully loaded into the system memory, query the data version;

[0019] According to the data version, determine whether the target data is generated;

[0020] When it is determined that target data is generated, obtaining target identification information in the cache queue; and

[0021] Based on the target identification information, the target data is loaded into the system memory.

[0022] According to an embodiment of the present disclosure, when the target data set has been completely loaded into the system memory, querying the data version includes:

[0023] When the target data set has been fully loaded into the system memory, the data version is queried cyclically at preset time intervals.

[0024] According to an embodiment of the present disclosure, loading a target data set into a system memory or loading target data into a system memory includes:

[0025] Processing a target data set or target data, wherein the processing includes classification and / or format conversion;

[0026] Load the processed data into system memory.

[0027] According to an embodiment of the present disclosure, traversing the data in the database to determine the target data includes:

[0028] Based on identification information corresponding to data in the target data set already stored in the cache, traversing the data in the database; and

[0029] Determine the target data.

[0030] According to an embodiment of the present disclosure, traversing data in a database includes:

[0031] Loop through the data in the database at preset time intervals.

[0032] According to an embodiment of the present disclosure, storing data in a target data set into a cache includes:

[0033] Storing the data in the target data set into the cache; and

[0034] Stores a collection of identification information in the cache.

[0035] Another aspect of the present disclosure provides a data processing device, comprising:

[0036] An acquisition module, used to acquire a target data set from a database according to an identification information set, wherein the data in the target data set is data decoupled from the business;

[0037] A storage module is used to store the data in the target data set into the cache;

[0038] A traversal module, used to traverse the data in the database and determine target data, wherein the target data is the changed data in the target data set; and

[0039] The updating module is used to update the associated data in the cache using the target data, wherein the associated data is data already stored in the cache and having the same identification information as the target data.

[0040] Another aspect of the present disclosure provides a computer system, comprising:

[0041] one or more processors;

[0042] a memory for storing one or more programs,

[0043] Among them, when one or more programs are executed by one or more processors, the one or more processors implement the method.

[0044] Yet another aspect of the present disclosure provides a computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, causes the processor to implement a method.

[0045] Yet another aspect of the present disclosure provides a computer program product, including a computer program, wherein the computer program includes computer executable instructions, and the instructions are used to implement the method when being executed.

[0046] According to the embodiments of the present disclosure, a target data set is obtained from a database according to an identification information set, wherein the data in the target data set is data decoupled from the business; the data in the target data set is stored in a cache; the data in the database is traversed to determine the target data, wherein the target data is the data in the target data set that has changed; and the target data is used to update the associated data in the cache, wherein the associated data is data that has been stored in the cache and has the same identification information as the target data. By decoupling the data in the target data set from the business, association with other businesses is avoided; and the target data is used to update the associated data in the cache, timely updates are achieved. Therefore, the technical problem of slow query speed and impact on other businesses caused by large data volume when directly querying data from a database and loading it into system memory in the prior art is at least partially overcome, thereby achieving the technical effect of decoupled query data and timely data update. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] The above and other objects, features and advantages of the present disclosure will become more apparent through the following description of the embodiments of the present disclosure with reference to the accompanying drawings, in which:

[0048] Figure 1 An exemplary system architecture to which the data processing method and apparatus of the present disclosure can be applied is schematically shown;

[0049] Figure 2 A flowchart schematically shows a data processing method according to an embodiment of the present disclosure;

[0050] Figure 3 A signaling diagram schematically illustrates a data processing method according to an embodiment of the present disclosure;

[0051] Figure 4 A signaling diagram schematically shows a data processing method according to another embodiment of the present disclosure;

[0052] Figure 5 A signaling diagram schematically shows a data processing method according to another embodiment of the present disclosure;

[0053] Figure 6 A block diagram schematically shows a data processing device according to an embodiment of the present disclosure; and

[0054] Figure 7 The block diagram schematically shows a computer system suitable for implementing the data processing method according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0055] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the present disclosure. In the following detailed description, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of the embodiments of the present disclosure. However, it is apparent that one or more embodiments may also be implemented without these specific details. In addition, in the following description, descriptions of known structures and technologies are omitted to avoid unnecessary confusion of the concepts of the present disclosure.

[0056] The terms used herein are only for describing specific embodiments and are not intended to limit the present disclosure. The terms "comprise", "include", etc. used herein indicate the existence of the features, steps, operations and / or components, but do not exclude the existence or addition of one or more other features, steps, operations or components.

[0057] All terms (including technical and scientific terms) used herein have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification, and should not be interpreted in an idealized or overly rigid manner.

[0058] In the case of using expressions such as "at least one of A, B, and C, etc.", it should generally be interpreted in accordance with the meaning of the expression generally understood by those skilled in the art (for example, "a system having at least one of A, B, and C" should include but is not limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, C, etc.). In the case of using expressions such as "at least one of A, B, or C, etc.", it should generally be interpreted in accordance with the meaning of the expression generally understood by those skilled in the art (for example, "a system having at least one of A, B, or C" should include but is not limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, C, etc.).

[0059] The embodiment of the present disclosure provides a data processing method. The method includes obtaining a target data set from a database according to an identification information set, wherein the data in the target data set is data decoupled from the business; storing the data in the target data set in a cache; traversing the data in the database to determine the target data, wherein the target data is the changed data in the target data set; and using the target data to update the associated data in the cache, wherein the associated data is the data already stored in the cache that has the same identification information as the target data.

[0060] According to the embodiments of the present disclosure, the data in the target data set is decoupled from the business to avoid association with other businesses; and the associated data in the cache is updated with the target data to achieve timely update.

[0061] Figure 1 The exemplary system architecture 100 to which the data processing method and apparatus according to the embodiment of the present disclosure can be applied is schematically shown. It should be noted that: Figure 1 What is shown is merely an example of a system architecture to which the embodiments of the present disclosure can be applied, in order to help those skilled in the art understand the technical content of the present disclosure, but it does not mean that the embodiments of the present disclosure cannot be used in other devices, systems, environments or scenarios.

[0062] like Figure 1 As shown, the system architecture 100 according to this embodiment may include terminal devices 101, 102, 103, a network 104 and a server 105. The network 104 is used to provide a medium for communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired and / or wireless communication links, etc.

[0063] Users can use terminal devices 101, 102, 103 to interact with server 105 through network 104 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 101, 102, 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients and / or social platform software, etc. (only as examples).

[0064] The terminal devices 101 , 102 , and 103 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, and desktop computers.

[0065] The server 105 may be a server that provides various services, such as a background management server (only an example) that provides support for websites browsed by users using the terminal devices 101, 102, and 103. The background management server may analyze and process the received data such as user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal device.

[0066] It should be noted that the data processing method provided in the embodiment of the present disclosure can generally be executed by the server 105. Accordingly, the data processing device provided in the embodiment of the present disclosure can generally be set in the server 105. The data processing method provided in the embodiment of the present disclosure can also be executed by a server or server cluster that is different from the server 105 and can communicate with the terminal devices 101, 102, 103 and / or the server 105. Accordingly, the data processing device provided in the embodiment of the present disclosure can also be set in a server or server cluster that is different from the server 105 and can communicate with the terminal devices 101, 102, 103 and / or the server 105.

[0067] For example, the data in the database may be originally stored in any one of the terminal devices 101, 102, or 103 (for example, the terminal device 101, but not limited thereto), or stored in an external storage device and may be imported into the terminal device 101. Then, the terminal device 101 may send the data to a database of another terminal device, server, or server cluster, and other servers or server clusters that may query the data in the database may execute the data processing method provided by the embodiments of the present disclosure.

[0068] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to the implementation requirements.

[0069] Figure 2 The flowchart of the data processing method according to the embodiment of the present disclosure is schematically shown.

[0070] like Figure 2 As shown, the method includes operations S210 to S240.

[0071] In operation S210, a target data set is acquired from a database according to the identification information set, wherein the data in the target data set is data decoupled from the business.

[0072] According to an embodiment of the present disclosure, before acquiring the target data set from the database according to the identification information set, a request for acquiring the target data set from the database may be received, wherein the request carries the identification information set.

[0073] According to an embodiment of the present disclosure, the identification information in the identification information set may be ID (identification) information representing the data, but is not limited thereto, and may also be other unique attribute identification information of the data.

[0074] According to an embodiment of the present disclosure, the data in the target data set may be large object data of the order of 100,000. More specifically, the target data set may include tens of thousands or hundreds of thousands of copies of data, each copy including tens of thousands or hundreds of thousands of sub-data; however, it is not limited thereto, and the target data set may also be data of ordinary magnitude. In the case where the target data set is large object data of the order of 100,000, the advantages of the data processing method of the present disclosure in terms of fast and stable data retrieval and loading can be better reflected.

[0075] According to an embodiment of the present disclosure, the database may be a Date base, and the data is stored in the database. When the application server is started, the data is searched from the database and loaded into the system memory through an RPC call (ie, remote procedure call).

[0076] According to the embodiments of the present disclosure, the data in the target data set is decoupled from the business, thus preventing the data from being too deeply coupled with the business.

[0077] In operation S220 , the data in the target data set is stored in a cache.

[0078] According to an embodiment of the present disclosure, the cache may be a redis cache (ie, a Key-Value database).

[0079] According to the embodiments of the present disclosure, the data of the target data set is stored in the cache, and the cache is used as an intermediate transmission medium, rather than being directly retrieved from the database and loaded into the system memory. On the one hand, when the data in the cache is loaded into the system memory, the entire target data set is loaded in seconds, and the loading speed is fast; on the other hand, when the amount of data retrieved is large, compared with directly querying from the database and loading into the system memory, the pressure on the database is reduced.

[0080] In operation S230 , the data in the database are traversed to determine target data, wherein the target data is the changed data in the target data set.

[0081] In operation S240, the associated data in the cache is updated using the target data, wherein the associated data is data already stored in the cache and having the same identification information as the target data.

[0082] According to an embodiment of the present invention, when data in a database is updated, target data is determined by traversing the data in the database, and associated data in the cache is updated using the target data, thereby achieving timely synchronization of data in the cache with data in the database, thereby avoiding the problem that the data loaded into the system memory is expired data rather than real-time updated data.

[0083] According to the embodiments of the present invention, the data in the target data set is decoupled from the business, and the cache is used as an intermediate transmission medium to improve the robustness of the system; the target data is determined in a timely manner, and the target data is used to update the associated data in the cache, so that the data in the cache and the data in the database are synchronized in time, thereby improving the response processing capability.

[0084] Reference below Figure 3 to Figure 5 , combined with specific embodiments Figure 2 The method shown is further explained.

[0085] Figure 3 A signaling diagram of a data processing method according to an embodiment of the present disclosure is schematically shown.

[0086] like Figure 3 As shown, the data processing method includes the following operations S310 to S370.

[0087] In operation S310, a target data set is acquired from a database according to an identification information set, wherein the data in the target data set is data decoupled from the business.

[0088] According to an embodiment of the present disclosure, after querying data from a database using a scheduling system, the queried data may be stored in a cache.

[0089] According to the embodiments of the present disclosure, the data in the target data set is data decoupled from the business, wherein the data in the target data set can be decoupled from the business using a scheduling system, and the implementation method can be to process a step or a stage of business processing, thereby achieving the effect of decoupling. However, it is not limited to this, and other methods of decoupling data from business can also be used for decoupling. As long as it is achieved that when one party's logic changes, it will not affect the whole, and the simultaneous retrieval of the associated data of the underlying business is avoided.

[0090] In operation S320 , data in the target data set is stored in a cache.

[0091] According to the embodiment of the present disclosure, the target data set contains a large amount of data, which may be a large object data of 100,000 levels. When the scheduling system is used to retrieve data, data can be extracted in a preset order, rather than in seconds.

[0092] In operation S330, the identification information set is stored in a cache.

[0093] According to an embodiment of the present disclosure, the data in the target data set can be stored in the cache while the identification information set can be stored in the cache, so that the business system can load the corresponding target data set into the system memory based on the identification information set.

[0094] In operation S340 , the data in the database are traversed to determine target data, wherein the target data is the changed data in the target data set.

[0095] According to an optional embodiment of the present disclosure, based on identification information corresponding to data in a target data set already stored in a cache, data in a database is traversed; and target data is determined.

[0096] According to an optional embodiment of the present disclosure, the changed data in the database can be queried based on the identification information corresponding to the data in the target data set already stored in the cache as retrieval information; however, it is not limited to this, and the changed data in the database can be queried based on a preset time period, and then the corresponding identification information of the changed data is matched with the identification information corresponding to the data in the target data set already stored in the cache, and the matched data can be determined as the target identification information.

[0097] According to an optional embodiment of the present disclosure, traversing the data in the database may include traversing the data in the database in a loop at a preset time interval.

[0098] According to an optional embodiment of the present disclosure, the preset time interval may be a time interval of 5 seconds to 10 seconds, but is not limited thereto, and may also be a time interval of 1 minute. It may be set according to specific actual conditions.

[0099] According to an optional embodiment of the present disclosure, data in the database is traversed in a loop at preset time intervals, so that data stored in the cache can be synchronized with data changed in the database in a timely manner.

[0100] In operation S350, the associated data in the cache is updated using the target data, wherein the associated data is data already stored in the cache and having the same identification information as the target data.

[0101] According to an embodiment of the present disclosure, the associated data may be understood as expired data. In the present disclosure, the target data may replace the associated data based on the target identification information, thereby replacing and updating the expired data in the cache.

[0102] According to the embodiments of the present disclosure, only the updated data is replaced, rather than re-retrieving the data in all target data sets. This avoids data backlogs and simplifies storage, alleviating the amount of data stored in the cache. In addition, a faster response is performed on the changed data in the database, and only the changed data is subsequently retrieved, reducing the pressure on the database.

[0103] In operation S360, the target identification information corresponding to the target data is stored in a cache queue of the cache.

[0104] According to an embodiment of the present disclosure, when the target data updates the associated data in the cache, the target identification information corresponding to the target data can be stored in the cache at the same time, so that the business system can load the updated data into the system memory based on the target identification information in the cache queue to achieve mutual correspondence and prevent the problem of erroneous loading.

[0105] In operation S370 , the data version of the target identification information representing different batches in the cache queue is updated.

[0106] According to an embodiment of the present disclosure, a cache queue and a data version may be set at the same time. The cache queue is implemented using key-value. The data version corresponds to target identification information of different batches in the cache queue. The data version may start from 0 and be a long integer number that increases one by one.

[0107] According to the embodiment of the present disclosure, each time the scheduling system writes data into the cache, a data version is generated according to the current cache queue. For example, the corresponding data version of the current cache queue is 5. When new target identification information is written into the cache queue, the corresponding data version is increased by 1 and updated to 6. In the subsequent process of loading the updated data into the system memory, the target identification information in the cache queue corresponding to the data version can be determined according to the data version, and the target data can be queried using the target identification information.

[0108] According to the embodiments of the present disclosure, using the data version as an identifier can effectively prevent data loss.

[0109] According to an optional embodiment of the present disclosure, an instruction for loading a target data set in a cache into a system memory is received; and in response to the instruction, the target data set in the cache is loaded into the system memory.

[0110] According to an embodiment of the present disclosure, the target data set stored in the cache can be loaded into the system memory through the constructed business system. Wherein, the target data set in the cache can be loaded into the system memory based on the target identification information set.

[0111] According to an optional embodiment of the present disclosure, the target data set may be processed before loading, and the processed data may be loaded into the system memory.

[0112] According to an embodiment of the present disclosure, data in a target data set may be classified and format converted so as to be converted into a format required by an application processor corresponding to a system memory.

[0113] Figure 4 A signaling diagram of a data processing method according to another embodiment of the present disclosure is schematically shown.

[0114] like Figure 4 As shown, the data processing method includes operations S410 to S470.

[0115] In operation S410 , an instruction for loading a target data set in a cache into a system memory is received.

[0116] According to an embodiment of the present disclosure, the target data set may be loaded from the cache into the system memory by using the business system.

[0117] According to an embodiment of the present disclosure, the instruction for loading the target data set in the cache into the system memory may be sent by the scheduling system to the business system, but is not limited thereto, and may also be sent by a user to the business system. The main purpose is to start the loading service.

[0118] In operation S420 , in response to the instruction, the target data set in the cache is loaded into the system memory.

[0119] According to an embodiment of the present disclosure, the operation of loading the target data set in the cache into the system memory can be loaded in seconds.

[0120] In operation S430 , it is determined whether the target data set has been completely loaded into the system memory.

[0121] In operation S440 , when the target data set has been completely loaded into the system memory, the data version is queried.

[0122] According to an embodiment of the present disclosure, the business system may be used to determine whether all the data in the target data set has been loaded into the system memory, and then query the data version if it is determined that all the data has been loaded.

[0123] In operation S450, it is determined whether target data is generated according to the data version.

[0124] According to an optional embodiment of the present disclosure, when all data sets have been loaded into the system memory, the data versions are queried cyclically at preset time intervals.

[0125] According to an optional embodiment of the present disclosure, the preset time interval may be a time interval of 5 seconds to 10 seconds, but is not limited thereto, and may also be a time interval of 1 minute. Matching settings may be made according to the preset time interval of the scheduling system traversing the data in the database, so as to better achieve timely acquisition of updated data from the database and store it in the cache, and timely loading from the cache to the system memory, thereby achieving coordinated processing.

[0126] According to the embodiments of the present disclosure, using the data version as an identifier to query and confirm the target data can effectively prevent data loss.

[0127] In operation S460, when it is determined that target data is generated, target identification information in the cache queue is obtained.

[0128] In operation S470, target data is loaded into the system memory based on the target identification information.

[0129] According to an optional embodiment of the present disclosure, the data version is cyclically queried at preset time intervals to load updated data into the system memory in a timely manner, thereby achieving a faster response processing effect and achieving the loading of a larger amount of data.

[0130] According to an optional embodiment of the present disclosure, the target data stored in the cache can be loaded into the system memory through the constructed business system. The target data in the cache can be loaded into the system memory based on the target identification information corresponding to the target data.

[0131] According to an optional embodiment of the present disclosure, the target data may be processed before loading, and the processed data may be loaded into the system memory. In the present disclosure, the target data may be classified and format converted so as to be converted into a format required by a processor corresponding to the system memory.

[0132] Figure 5 A signaling diagram of a data processing method according to another embodiment of the present disclosure is schematically shown.

[0133] like Figure 5 As shown, the data processing method relies on the scheduling system and the business system. Before loading the data in the database (DB) into the system memory, the cache (Redis) is used as an intermediate transmission medium. First, the scheduling system stores the data in the cache and periodically queries whether the corresponding data in the database has been updated. Subsequently, only the changed data needs to be replaced with the data in the cache to achieve timely updates. Then the business system service is started, the data in the cache is loaded into the system memory, and the data in the system memory is updated in time by periodically querying whether the corresponding data in the cache has been updated.

[0134] In summary, according to the embodiments of the present disclosure, by designing a scheduling system and a business system, tens of thousands or hundreds of thousands of data can be loaded into the system memory in seconds without affecting other businesses. This solves the problems existing in the prior art of directly loading data from the database into the system memory, such as: it takes a long time to load data, and the speed is slow when the amount of data is large; frequent operations put pressure on the database; data is coupled with business, and when data is retrieved, they are interdependent and need to be retrieved from the bottom layer.

[0135] Figure 6 The block diagram schematically shows a data processing device according to an embodiment of the present disclosure.

[0136] like Figure 6 As shown, the data processing device 600 includes an acquisition module 610 , a storage module 620 , a traversal module 630 and an update module 640 .

[0137] An acquisition module 610 is used to acquire a target data set from a database according to the identification information set, wherein the data in the target data set is data decoupled from the business;

[0138] A storage module 620, used to store the data in the target data set into a cache;

[0139] A traversal module 630 is used to traverse the data in the database and determine target data, wherein the target data is the changed data in the target data set; and

[0140] The updating module 640 is used to update the associated data in the cache using the target data, wherein the associated data is data already stored in the cache and having the same identification information as the target data.

[0141] According to an embodiment of the present disclosure, the data processing device 600 further includes an identification storage module and a version update module.

[0142] an identification storage module, used to store target identification information corresponding to the target data into a cache queue of the cache; and

[0143] The version update module is used to update the data version used to characterize the target identification information of different batches in the cache queue.

[0144] According to an embodiment of the present disclosure, the data processing device 600 further includes a receiving module and a responding module.

[0145] A receiving module, configured to receive an instruction for loading a target data set in a cache into a system memory; and

[0146] The response module is used to load the target data set in the cache into the system memory in response to the instruction.

[0147] According to an embodiment of the present disclosure, the data processing device 600 further includes a loading determination module, a version query module, a target determination module, and a data loading module.

[0148] A loading determination module is used to determine whether the target data set has been fully loaded into the system memory;

[0149] A version query module is used to query the data version when the target data set has been fully loaded into the system memory;

[0150] A target determination module is used to determine whether target data has been generated according to the data version;

[0151] an identification acquisition module, used to acquire target identification information in the cache queue when it is determined that target data is generated; and

[0152] The data loading module is used to load the target data into the system memory based on the target identification information.

[0153] According to an embodiment of the present disclosure, the version query module includes a version loop query unit.

[0154] The version loop query unit is used to loop query the data version at preset time intervals when the target data set has been fully loaded into the system memory.

[0155] According to an embodiment of the present disclosure, loading a target data set into a system memory or loading target data into a system memory includes processing the target data set or the target data, wherein the processing includes classification and / or format conversion; and loading the processed data into the system memory.

[0156] According to an embodiment of the present disclosure, the traversal module 630 includes a database traversal unit and a target data determination unit.

[0157] a database traversal unit, configured to traverse the data in the database based on identification information corresponding to the data in the target data set already stored in the cache; and

[0158] The target data determining unit is used to determine the target data.

[0159] According to an embodiment of the present disclosure, traversing the data in the database may include traversing the data in the database in a loop at a preset time interval.

[0160] According to an embodiment of the present disclosure, the storage module 620 includes a data storage unit and an identification storage unit.

[0161] A data storage unit, used to store the data in the target data set into a cache; and

[0162] The identification storage unit is used to store the identification information set into the cache.

[0163] According to the embodiments of the present invention, any one or more of the modules, submodules, units, and subunits, or at least part of the functions of any one of them can be implemented in one module. According to the embodiments of the present invention, any one or more of the modules, submodules, units, and subunits can be split into multiple modules for implementation. According to the embodiments of the present invention, any one or more of the modules, submodules, units, and subunits can be at least partially implemented as hardware circuits, such as field programmable gate arrays (FPGAs), programmable logic arrays (PLAs), systems on chips, systems on substrates, systems on packages, application specific integrated circuits (ASICs), or can be implemented by hardware or firmware in any other reasonable way of integrating or packaging the circuit, or implemented in any one of the three implementation methods of software, hardware, and firmware, or in any appropriate combination of any of them. Alternatively, according to the embodiments of the present invention, one or more of the modules, submodules, units, and subunits can be at least partially implemented as computer program modules, and when the computer program modules are run, the corresponding functions can be performed.

[0164] For example, any multiple of the acquisition module 610, the storage module 620, the traversal module 630, and the update module 640 can be combined in one module / unit / sub-unit for implementation, or any one of the modules / units / sub-units can be split into multiple modules / units / sub-units. Alternatively, at least part of the functions of one or more of these modules / units / sub-units can be combined with at least part of the functions of other modules / units / sub-units and implemented in one module / unit / sub-unit. According to an embodiment of the present disclosure, at least one of the acquisition module 610, the storage module 620, the traversal module 630, and the update module 640 can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application specific integrated circuit (ASIC), or can be implemented by hardware or firmware such as any other reasonable way of integrating or packaging the circuit, or implemented in any one of the three implementation methods of software, hardware, and firmware or in a suitable combination of any of them. Alternatively, at least one of the acquisition module 610 , the storage module 620 , the traversal module 630 , and the update module 640 may be at least partially implemented as a computer program module, and when the computer program module is executed, the corresponding function may be performed.

[0165] It should be noted that the data processing device part in the embodiments of the present disclosure corresponds to the data processing method part in the embodiments of the present disclosure. The description of the data processing device part specifically refers to the data processing method part, which will not be repeated here.

[0166] Figure 7A block diagram of a computer system suitable for implementing the method described above according to an embodiment of the present disclosure is schematically shown. Figure 7 The computer system shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0167] like Figure 7 As shown, the computer system 700 according to the embodiment of the present disclosure includes a processor 701, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) 702 or the program loaded from the storage part 708 to the random access memory (RAM) 703. The processor 701 may include, for example, a general-purpose microprocessor (such as a CPU), an instruction set processor and / or a related chipset and / or a special-purpose microprocessor (for example, an application-specific integrated circuit (ASIC)), etc. The processor 701 may also include an onboard memory for caching purposes. The processor 701 may include a single processing unit or multiple processing units for performing different actions of the method flow according to the embodiment of the present disclosure.

[0168] In RAM 703, various programs and data required for the operation of system 700 are stored. Processor 701, ROM 702 and RAM 703 are connected to each other through bus 704. Processor 701 performs various operations of the method flow according to the embodiment of the present disclosure by executing the program in ROM 702 and / or RAM 703. It should be noted that the program can also be stored in one or more memories other than ROM 702 and RAM 703. Processor 701 can also perform various operations of the method flow according to the embodiment of the present disclosure by executing the program stored in the one or more memories.

[0169] According to an embodiment of the present disclosure, the system 700 may further include an input / output (I / O) interface 705, which is also connected to the bus 704. The system 700 may further include one or more of the following components connected to the I / O interface 705: an input section 706 including a keyboard, a mouse, etc.; an output section 707 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 708 including a hard disk, etc.; and a communication section 709 including a network interface card such as a LAN card, a modem, etc. The communication section 709 performs communication processing via a network such as the Internet. A drive 710 is also connected to the I / O interface 705 as needed. A removable medium 711, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 710 as needed, so that a computer program read therefrom is installed into the storage section 708 as needed.

[0170] According to an embodiment of the present disclosure, the method flow according to an embodiment of the present disclosure can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable storage medium, and the computer program contains a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 709, and / or installed from the removable medium 711. When the computer program is executed by the processor 701, the above-mentioned functions defined in the system of the embodiment of the present disclosure are executed. According to an embodiment of the present disclosure, the system, equipment, device, module, unit, etc. described above can be implemented by a computer program module.

[0171] The present disclosure also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or may exist independently without being assembled into the device / apparatus / system. The above computer-readable storage medium carries one or more programs, and when the above one or more programs are executed, the method according to the embodiment of the present disclosure is implemented.

[0172] According to an embodiment of the present disclosure, the computer-readable storage medium may be a non-volatile computer-readable storage medium. For example, it may include, but is not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, apparatus, or device.

[0173] For example, according to an embodiment of the present disclosure, the computer-readable storage medium may include the ROM 702 and / or the RAM 703 described above and / or one or more memories other than the ROM 702 and the RAM 703 .

[0174] The embodiments of the present disclosure also include a computer program product, which includes a computer program, which contains program code for executing the method provided by the embodiments of the present disclosure. When the computer program product runs on an electronic device, the program code is used to enable the electronic device to implement the data processing method provided by the embodiments of the present disclosure.

[0175] When the computer program is executed by the processor 701, the above functions defined in the system / device of the embodiment of the present disclosure are executed. According to the embodiment of the present disclosure, the system, device, module, unit, etc. described above can be implemented by a computer program module.

[0176] In one embodiment, the computer program may rely on tangible storage media such as optical storage devices, magnetic storage devices, etc. In another embodiment, the computer program may also be transmitted and distributed in the form of signals on a network medium, and downloaded and installed through the communication part 709, and / or installed from the removable medium 711. The program code contained in the computer program may be transmitted using any appropriate network medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0177] According to an embodiment of the present disclosure, the program code for executing the computer program provided by the embodiment of the present disclosure can be written in any combination of one or more programming languages. Specifically, these computing programs can be implemented using high-level process and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, Java, C++, python, "C" language or similar programming languages. The program code can be executed entirely on the user computing device, partially on the user device, partially on the remote computing device, or entirely on the remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (for example, using an Internet service provider to connect through the Internet).

[0178] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram may represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of the code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box may also occur in an order different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions. It can be understood by those skilled in the art that the features recorded in the various embodiments and / or claims of the present disclosure can be combined and / or combined in a variety of ways, even if such a combination or combination is not explicitly recorded in the present disclosure. In particular, without departing from the spirit and teaching of the present disclosure, the features described in the various embodiments and / or claims of the present disclosure may be combined and / or combined in a variety of ways. All of these combinations and / or combinations fall within the scope of the present disclosure.

[0179] The embodiments of the present disclosure are described above. However, these embodiments are only for illustrative purposes and are not intended to limit the scope of the present disclosure. Although the embodiments are described above separately, this does not mean that the measures in the various embodiments cannot be used in combination to advantage. The scope of the present disclosure is defined by the attached claims and their equivalents. Without departing from the scope of the present disclosure, those skilled in the art may make a variety of substitutions and modifications, which should all fall within the scope of the present disclosure.

Claims

1. A data processing method, comprising: Acquire a target data set from a database according to the identification information set, wherein the data in the target data set is data decoupled from the business; Storing the data in the target data set into a cache; Traversing the data in the database to determine target data, wherein the target data is the changed data in the target data set; and The associated data in the cache is updated using the target data, wherein the associated data is data already stored in the cache and having the same identification information as the target data.

2. The method according to claim 1, further comprising: Storing target identification information corresponding to the target data in a cache queue of the cache; as well as The data version used to characterize the target identification information of different batches in the cache queue is updated.

3. The method according to claim 2, further comprising: receiving an instruction for loading the target data set in the cache into a system memory; as well as In response to the instruction, the target data set in the cache is loaded into the system memory.

4. The method according to claim 3, further comprising: Determining whether the target data set has been fully loaded into the system memory; In the case that the target data set has been completely loaded into the system memory, querying the data version; Determining whether the target data has been generated according to the data version; When it is determined that the target data is generated, obtaining the target identification information in the cache queue; as well as Based on the target identification information, the target data is loaded into the system memory.

5. The method according to claim 4, wherein when the target data set has been completely loaded into the system memory, querying the data version comprises: When the target data set has been completely loaded into the system memory, the data version is queried cyclically at preset time intervals.

6. The method according to claim 4, wherein: The loading of the target data set into the system memory or the loading of the target data into the system memory comprises: Processing the target data set or the target data, wherein the processing includes classification and / or format conversion; The processed data is loaded into the system memory.

7. The method according to claim 1, wherein: The traversing the data in the database to determine the target data comprises: traversing the data in the database based on identification information corresponding to the data in the target data set already stored in the cache; and The target data is determined.

8. The method according to claim 7, wherein traversing the data in the database comprises: The data in the database is traversed in a loop at preset time intervals.

9. The method according to claim 1, wherein: The storing the data in the target data set into the cache comprises: storing the data in the target data set into the cache; and The identification information set is stored in the cache.

10. A data processing device, comprising: An acquisition module, used to acquire a target data set from a database according to an identification information set, wherein the data in the target data set is data decoupled from the business; A storage module, used for storing the data in the target data set into a cache; a traversal module, configured to traverse the data in the database and determine target data, wherein the target data is the changed data in the target data set; and An updating module is used to update the associated data in the cache using the target data, wherein the associated data is data already stored in the cache and having the same identification information as the target data.

11. A computer system comprising: one or more processors; a memory for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 9.

12. A computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, enables the processor to implement the method according to any one of claims 1 to 9.

13. A computer program product, comprising a computer program, the computer program comprising computer executable instructions, the instructions being used to implement the method of any one of claims 1 to 9 when executed.

Images