Pipeline synchronous caching method, device, equipment, storage medium and program product

Abstract

The present application relates to a pipeline-type synchronous caching method, apparatus, computer equipment, storage medium and computer program product, and relates to the field of data synchronization technology. The present application can achieve a balance between performance and flexibility for complex business models, different synchronization requirements and large data volumes. The method comprises: obtaining real-time incremental data from a relational database, processing the real-time incremental data based on one or more pre-bound functions provided by a function expression engine to obtain data to be submitted, wherein the one or more functions at least include a merge function for merging data with the same primary key in the real-time incremental data, and finally submitting the data to be submitted to the cache database through pipeline technology.

Description

Technical Field

[0001] The present application relates to the field of data synchronization technology, and in particular to a pipeline synchronization cache method, apparatus, computer equipment, storage medium and computer program product. Background Art

[0002] As enterprises grow, business system access performance requirements become increasingly stringent. To improve system access speeds within applications, caching mechanisms are often implemented at the application level, storing query data in memory. Redis (remote dictionary server) is a cache database that manages cached data. Data synchronization between this relational database and the cache database is a key concern. Synchronous caching must ensure both data consistency and real-time performance. Furthermore, personalized data filtering is required based on business needs to meet actual data application requirements.

[0003] The current cache data synchronization method, based on database log increment technology, involves the relational database writing data changes to the binary log. The slave node then copies the master's log to its relay log. The slave node then replays the relay log events, reflecting the data changes to its own data. The incremental capture component simulates the slave node's interaction protocol, masquerading as a slave node and sending a dump protocol to the master node. Upon receiving the dump request, the master node begins pushing the binary log to the slave node (the incremental capture component). The incremental capture component parses the binary log object (originally a byte stream), parses, transforms, and synchronizes the captured log to the cache. However, this technology only supports simple table-level filtering and typically commits data to the cache using a single write. Therefore, it struggles to achieve flexible data filtering and cache commit performance for large data volumes, especially when faced with complex business models, diverse synchronization requirements, and large data volumes. Summary of the Invention

[0004] Based on this, it is necessary to provide a pipeline synchronous caching method, device, computer equipment, storage medium and computer program product to address the above technical problems.

[0005] In a first aspect, the present application provides a pipeline synchronous caching method. The method comprises:

[0006] Obtain real-time incremental data from relational databases;

[0007] Processing the real-time incremental data based on one or more pre-bound functions provided by a function expression engine to obtain data to be submitted; the one or more functions at least include a merge function; the merge function is used to merge data with the same primary key in the real-time incremental data;

[0008] The data to be submitted is submitted to the cache database through pipeline technology.

[0009] In one embodiment, the real-time incremental data is processed based on one or more pre-bound functions provided by the function expression engine to obtain the data to be submitted, including: when the one or more functions include the merge function and other functions, the real-time incremental data is processed according to the other functions to obtain the data to be merged, and the data to be merged is processed according to the merge function to obtain the data to be submitted.

[0010] In one embodiment, the other functions include event functions; the processing of the real-time incremental data according to the other functions to obtain the data to be merged includes: determining the target data field and target value bound in the event function; if the target data field in the real-time incremental data has the target value, processing the real-time incremental data according to the event corresponding to the event function to obtain the data to be merged.

[0011] In one embodiment, the real-time incremental data includes real-time incremental data of multiple tables; the real-time incremental data is processed based on one or more pre-bound functions provided by a function expression engine to obtain data to be submitted, including: for the real-time incremental data of each table, determining the one or more pre-bound functions provided by the function expression engine corresponding to the table, processing the real-time incremental data of the table based on the one or more functions to obtain the data to be submitted of the table; and obtaining the data to be submitted based on the data to be submitted of each table.

[0012] In one embodiment, determining the pre-bound one or more functions provided by the function expression engine corresponding to the table includes: obtaining one or more functions corresponding to the identifier from the function expression engine according to the identifier of the table as the pre-bound one or more functions provided by the function expression engine corresponding to the table.

[0013] In a second aspect, the present application further provides a pipeline-type synchronous caching device. The device comprises:

[0014] Data acquisition module, used to obtain real-time incremental data from relational databases;

[0015] a merge processing module, configured to process the real-time incremental data based on one or more pre-bound functions provided by a function expression engine to obtain data to be submitted; the one or more functions include at least a merge function; the merge function is configured to merge data with the same primary key in the real-time incremental data;

[0016] The pipeline submission module is used to submit the data to be submitted to the cache database through pipeline technology.

[0017] In one embodiment, the real-time incremental data includes real-time incremental data of multiple tables; the merge processing module is used to determine the pre-bound one or more functions provided by the function expression engine corresponding to the real-time incremental data of each table, and process the real-time incremental data of the table based on the one or more functions to obtain the data to be submitted of the table; and obtain the data to be submitted based on the data to be submitted of each table.

[0018] In a third aspect, the present application further provides a computer device. The computer device includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are performed:

[0019] Acquire real-time incremental data from a relational database; process the real-time incremental data based on one or more pre-bound functions provided by a function expression engine to obtain data to be submitted; the one or more functions include at least a merge function; the merge function is used to merge data with the same primary key in the real-time incremental data; and submit the data to be submitted to a cache database through pipeline technology.

[0020] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the following steps:

[0021] Acquire real-time incremental data from a relational database; process the real-time incremental data based on one or more pre-bound functions provided by a function expression engine to obtain data to be submitted; the one or more functions include at least a merge function; the merge function is used to merge data with the same primary key in the real-time incremental data; and submit the data to be submitted to a cache database through pipeline technology.

[0022] In a fifth aspect, the present application further provides a computer program product. The computer program product includes a computer program that, when executed by a processor, implements the following steps:

[0023] Acquire real-time incremental data from a relational database; process the real-time incremental data based on one or more pre-bound functions provided by a function expression engine to obtain data to be submitted; the one or more functions include at least a merge function; the merge function is used to merge data with the same primary key in the real-time incremental data; and submit the data to be submitted to a cache database through pipeline technology.

[0024] The above-mentioned pipeline synchronization cache method, device, equipment, storage medium and program product obtains the real-time incremental data of the relational database, processes the real-time incremental data based on one or more pre-bound functions provided by the function expression engine, and obtains the data to be submitted. The one or more functions at least include a merge function for merging the data with the same primary key in the real-time incremental data, and finally submits the data to be submitted to the cache database through pipeline technology. This solution can use database log incremental technology to parse the changed data, and process the real-time incremental data based on one or more pre-bound functions provided by the function expression engine during the data heterogeneous transformation process to achieve code-free parameterized configuration of data synchronization business needs, and one or more functions include a merge function to achieve disorder of the ordered data in the same batch, so that the pipeline submission cache can be used to improve the cache submission performance and reduce the synchronization cache delay, thereby truly achieving a balance between performance and flexibility for complex business models, different synchronization requirements, and large data volumes. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 This is an application environment diagram of the pipeline synchronous caching method in an embodiment of the present application;

[0026] Figure 2 Schematic diagram of the process of the pipeline synchronous caching method in the embodiment of the present application;

[0027] Figure 3 This is a schematic diagram of the merge processing in a specific example of this application;

[0028] Figure 4 This is a schematic diagram of pipeline submission processing in a specific example of this application;

[0029] FIG5( a ) is a schematic diagram of table-level filtering processing in the current technology;

[0030] FIG5( b ) is a schematic diagram of filtering processing based on a function expression engine in an embodiment of the present application;

[0031] Figure 6 This is a schematic diagram of event function processing in a specific example of this application;

[0032] Figure 7 This is a structural block diagram of a pipeline-type synchronous cache device in an embodiment of the present application;

[0033] Figure 8 This is a diagram of the internal structure of a computer device in an embodiment of the present application. DETAILED DESCRIPTION

[0034] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0035] The pipeline synchronization caching method provided in the embodiment of the present application can be applied to Figure 1 In the application environment shown, the application environment corresponds to the application environment of the existing real-time incremental data acquisition technology. The pipeline synchronous caching method provided by the embodiment of the present application re-implements the conversion and loading in the existing real-time incremental data acquisition technology. On the basis of fully realizing the original functions, it can add the screening and filtering of data heterogeneous conversion and the pipeline of cache loading to achieve heterogeneous accuracy and real-time loading. Specifically, the pipeline synchronous caching method provided by the embodiment of the present application can be specifically applied to Figure 1 In the conversion module and loading module of the data synchronization structure, the scheduler in the data synchronization structure can schedule and control data processing, and apply the pipeline synchronization caching method provided in the embodiment of the present application. A function expression engine is used in the conversion module to eliminate data without business significance, such as records of disassembly in the telecommunications industry, invalid package instances, etc., to achieve code-free parameterized configuration of data synchronization business needs. The loading performance is improved by combining the merge function with the pipeline submission used in the loading module. In a database with 50,000 database inserts and update operations per second, the pipeline submission loading link is used to improve the cache loading performance by 2.5 times. At the same time, due to the use of the merge function, there are no identical records in the same batch, avoiding the timing problem of the same batch of data in the prior art.

[0036] In addition to the current cache data synchronization method based on database log increment technology mentioned in the background technology above, the existing cache data synchronization methods include:

[0037] Data synchronization is an incremental, proactive, and strongly consistent technology: 1. When updating database data (adding, deleting, updating), the cache is eliminated (cache invalidation). 2. When reading data, the cache is updated. To avoid the avalanche problem caused by cache breakdown, synchronization processing is required. Only one thread is controlled to read data and update it to the cache, and other threads are blocked waiting. 3. Set a cache expiration time. If the cache update fails, the cache will be invalidated when the cache expiration time expires.

[0038] Data synchronization is an incremental, passive, and quasi-consistent technology: 1. When updating data, an MQ message is sent to update the cache after the database is updated. 2. The cache update service consumes the MQ update data message and reads the database data for related business processing. 3. The cache update service updates the business processing result data in the cache.

[0039] Task scheduling and updating technology: 1. This method updates the cache regularly through distributed scheduling tasks. 2. It is simple to implement and is suitable for business scenarios with low real-time requirements.

[0040] In this regard, data synchronization is an incremental, passive, and consistent technology, and task scheduling and update technology is suitable for business scenarios where real-time and consistency requirements are not high. However, data synchronization is an incremental, active, and highly consistent technology that will increase the complexity of business processes and the pressure on application servers. When updating large amounts of data, real-time performance cannot be guaranteed.

[0041] Therefore, based on the current cache data synchronization methods mentioned above, how to flexibly implement data screening and filtering and large data volume cache submission performance in the face of complex business models, different synchronization requirements, and huge data volumes is an urgent problem to be solved in this field.

[0042] The pipeline synchronization caching method provided in the embodiment of the present application can use database log incremental technology to parse the changed data, and process the real-time incremental data based on one or more pre-bound functions provided by the function expression engine during the data heterogeneous transformation, so as to realize the code-free parameterized configuration of data synchronization business needs, and one or more functions contain a merge function to realize the disordering of the ordered data of the same batch, so that the pipeline submission cache can be used to improve the cache submission performance and reduce the synchronization cache delay, thereby truly achieving a balance between performance and flexibility for complex business models, different synchronization requirements, and large data volumes.

[0043] Compared with the current cache data synchronization method mentioned above, the pipeline synchronization caching method provided by the embodiment of the present application adopts a merge function combined with a pipeline submission method in data conversion and loading to avoid the dependency between the same batch of data in Redis pipeline submission, reduce the interaction time in the TCP link, and reduce the synchronization cache delay; adopt a merge function in data conversion to add a merge algorithm to the data of the same batch, the same table and the same primary key, reduce invalid and repeated synchronization operations, reduce the total amount of synchronized data, thereby improving synchronization efficiency and reducing synchronization delay; adopt a function expression engine for screening and filtering in data conversion, and the selectable functions can include multiple functions and their combinations. These functions can be selected according to different data in different tables to realize code-free parameterized configuration of data synchronization business needs.

[0044] The following is Figure 1In the application environment shown, the pipeline synchronous caching method of the present application is further described in combination with various embodiments and corresponding drawings.

[0045] In one embodiment, Figure 2 As shown, a pipeline synchronous caching method is provided, which may include the following steps:

[0046] Step S201: Acquire real-time incremental data from a relational database.

[0047] like Figure 1 As shown, the real-time incremental data transmitted from the relational database to the database log can be obtained by the selection module and the extraction module, and then processed by the conversion module and the loading module.

[0048] Step S202: Process the real-time incremental data based on one or more pre-bound functions provided by the function expression engine to obtain data to be submitted.

[0049] Specifically, considering that the data in the synchronized cache maintains a controllable cache capacity scale on the basis of meeting business needs, loading commonly used query fields in the database table, and eliminating invalid business data in the cache in real time, step S202 in this application adopts a function expression engine to implement code-free parameterized configuration. The function expression engine can be used to provide a variety of functions, which may include event functions, row filtering functions, column filtering functions, merge functions, extensible personalized functions, etc. Users can pre-bind one or more functions for real-time incremental data. Multiple functions can be a combination of the aforementioned functions, wherein the one or more functions include at least a merge function, and the merge function is used to merge data with the same primary key in the real-time incremental data. As an example, the row filtering function: inserts a certain data into the database table to implement the function of whether to load the data into the cache, such as inserting pre-account opening data into the product instance information table. If the business does not need to use this data, this function is selected for filtering; the column filtering function inserts a certain data into the database table to implement the function of loading some fields of the data into the cache; in the function expression engine, supported expressions may include: relational operators (=, !=, >, >=, <, <=, in, isnull, isnotnull), logical operators (and, or, not), and expressions support combination and priority. Notin can be implemented by logical operator not + relational operator in, and isnot null can also be implemented by logical operator not + relational operator isnull. () is supported, and the priority is from left to right. The priority inside () and between () are also from left to right, which is in line with the priority understood at the normal mathematical level. In terms of specific implementation, the expression writing requirements are as follows: a string can start and end with "'", and in can be followed by parentheses. There must be at least one space on both sides of and, or, not, in, isnull, and isnotnull. and, or, not, in, isnull, and isnotnull are not case sensitive.

[0050] In this step, one or more pre-bound functions provided by the function expression engine process the real-time incremental data, and the one or more functions include at least a merge function. The merge function merges the data with the same primary key in the real-time incremental data so that there are no identical records in the data to be submitted, avoiding the timing problem of the same batch of data in the existing technology and realizing the disorder of the same batch of ordered data.

[0051] Specifically, for the merge function, combined with Figure 3 In this application, the merge function can use pkhash parallel loading + batch merging to break up the original database transactions, preprocess the data, and merge the insert / update / delete data. The specific merge algorithm can be referred to Figure 3The merge algorithm shown in , and then parallelize according to table+pk (for data in the same table, delete is executed first, then insert / update is executed, serial guarantee is made, and the problem of unique constraint data change is solved), and the SQL of the same table will be batch merged. Figure 3 In the example, the real-time incremental data input to the merge function can be merged with the relevant data of the product instance identified as 1001 under the action of the merge algorithm, so that there is no correlation between the merged data (data to be submitted). Then, the data to be submitted can be submitted to the cache database through the pipeline technology in the subsequent step S203. In contrast, when the real-time incremental data is directly submitted to the cache database through the pipeline technology, the consistency of the pipeline submitted data cannot be guaranteed due to the correlation and dependency between the data of the product instance identified as 1001.

[0052] Step S203: submit the data to be submitted to the cache database through pipeline technology.

[0053] Specifically, considering that data synchronization is performed in batches, the Redis cache database processes client network IO requests in a single-threaded manner. If a connection is created and disconnected every time a request is executed, it will consume too much time, resulting in reduced execution efficiency. The pipeline submission cache cannot guarantee the timing of the data submitted at the same time, that is, the consistency of the pipeline submission data can only be guaranteed if there is no association or dependency between the same batch of data. Therefore, combined with Figure 4 Because the merge function is used in the above steps to process the real-time incremental data to obtain the data to be submitted, this step can use the pipeline technology (Pipeline) provided by the Redis cache database to submit the data to be submitted to the Redis cache database during data cache loading. Using this pipeline technology, multiple requests can be sent to the Redis cache database at one time and multiple execution results can be returned. This saves the time of sending requests and creating connections, and greatly improves the efficiency of data synchronization execution.

[0054] The pipeline synchronization cache method of the above embodiment obtains real-time incremental data from a relational database, processes the real-time incremental data based on one or more pre-bound functions provided by a function expression engine, and obtains data to be submitted. The one or more functions at least include a merge function for merging data with the same primary key in the real-time incremental data, and finally submits the data to be submitted to the cache database through pipeline technology. This solution can use database log incremental technology to parse out the changed data, and process the real-time incremental data based on one or more pre-bound functions provided by a function expression engine during the data heterogeneous transformation process, to achieve code-free parameterized configuration of data synchronization business needs, and one or more functions include a merge function to achieve disordering of ordered data in the same batch, so that a pipeline submission cache can be used to improve cache submission performance and reduce synchronization cache latency, thereby truly achieving a balance between performance and flexibility for complex business models, different synchronization requirements, and large amounts of data.

[0055] In one embodiment, the real-time incremental data may include real-time incremental data of multiple tables. Based on this, the real-time incremental data is processed based on one or more pre-bound functions provided by the function expression engine in step S202 to obtain the data to be submitted, which may include:

[0056] For the real-time incremental data of each table, determine the pre-bound one or more functions provided by the function expression engine corresponding to the table, process the real-time incremental data of the table based on the one or more functions, and obtain the data to be submitted for the table; based on the data to be submitted for each table, obtain the data to be submitted.

[0057] Specifically, as shown in FIG5(a), current technology cannot flexibly implement data filtering when facing data synchronization business needs, and can only perform simple table-level filtering, such as whether the data to be filtered in the table needs to be cached synchronously. This will result in loading a large amount of data with no business significance, which greatly wastes cache storage, such as records of device disassembly and invalid package instances in the telecommunications industry. In this embodiment, as shown in FIG5(b), the real-time incremental data can include real-time incremental data of multiple tables, such as the real-time incremental data of table A and the real-time incremental data of table B. For the real-time incremental data of each table, one or more pre-bound functions provided by the function expression engine corresponding to the table are determined. For example, the one or more pre-bound functions corresponding to table A can be a merge function and an event function, and the one or more pre-bound functions corresponding to table B can be a merge function, a row filter function, and a column filter function. Then, based on the corresponding one or more functions, the real-time incremental data of the table is processed to obtain the data to be submitted for the table. The data to be submitted for each table can then be aggregated into the total data to be submitted for submission to the Redis cache database through the pipeline technology in step S203.

[0058] Based on this, in some embodiments, the one or more pre-bound functions provided by the function expression engine corresponding to the determination table in the above embodiments may include:

[0059] According to the table identifier, one or more functions corresponding to the identifier are obtained from the function expression engine as the one or more pre-bound functions corresponding to the table and provided by the function expression engine.

[0060] In this embodiment, an identifier for identifying the table can be set for each table, and then after one or more functions are bound to the table, the identifier of the table can be matched with the one or more functions, thereby pre-forming a correspondence between the two. In this way, in the stage of processing real-time incremental data, one or more functions corresponding to the identifier of the table can be obtained from the function expression engine according to the identifier of the table, and used as one or more pre-bound functions provided by the function expression engine corresponding to the table.

[0061] In one embodiment, the processing of the real-time incremental data based on one or more pre-bound functions provided by the function expression engine to obtain the data to be submitted in step S202 may include:

[0062] When one or more functions include a merge function and other functions, the real-time incremental data is processed according to the other functions to obtain the data to be merged, and the data to be merged is processed according to the merge function to obtain the data to be submitted.

[0063] In this embodiment, the pre-bound one or more functions provided by the function expression engine include at least a merge function. When the one or more functions also include other functions (such as event functions, row filtering functions, column filtering functions, extensible personalized functions, etc.), the real-time incremental data can be processed according to the other functions first, and the data obtained after the processing is used as the data to be merged. Finally, the data to be merged is processed according to the merge function to obtain the data to be submitted, and then submitted to the Redis cache database through pipeline technology in step S203, thereby ensuring that there is no correlation between the data to be submitted that needs to be submitted through pipeline technology in step S203.

[0064] Based on this, in some embodiments, the other functions include event functions. Based on this, in the above embodiments, the real-time incremental data is processed according to the other functions to obtain the data to be merged, specifically including:

[0065] Determine the target data field and target value bound in the event function; if the target data field in the real-time incremental data has a target value, process the real-time incremental data according to the event corresponding to the event function to obtain the data to be merged.

[0066] In this embodiment, the corresponding event can be bound to the target data field of the real-time incremental data and the target value can be set in the event function. Then, in the data conversion stage, for the real-time incremental data to be processed, if the target data field in the real-time incremental data has the target value, the real-time incremental data is processed according to the event corresponding to the event function, thereby obtaining the data to be merged. Among them, the event function can include the elimination function DtEvent, the generation function ItEvent, the retention function KtEvent, etc., and can also include the event function of the custom event. Specifically, taking the elimination function as an example, combined with Figure 6 When a target data field in a database table is updated to a target value, the function of deleting the data from the cache is implemented. For example, in the product instance information table, if the status field of a product instance is updated to 'invalid', the data needs to be eliminated from the cache. In the implementation, the configuration mapping selects the status field (target data field), binds the elimination (event) function, and enters the 'invalid' value (target value, 1100). Figure 6 As shown, execute the update status field statement shown, obtain binlog parsing data, and then perform binding event conversion to obtain the data content after the elimination function conversion, which can be submitted to the Redis cache database through pipeline technology in step S203 after being processed by the merge function.

[0067] Overall, the pipeline synchronization caching method of this application can achieve TB-level throughput and incremental data processing capabilities with low latency in seconds, and the database accuracy reaches 99.999%. Through parameterized configuration, including flexible database table to cache entity mapping relationships and rich function expressions to meet various personalized business needs, the R&D delivery time of data synchronization caching demand tasks can be shortened from weeks to minutes.

[0068] It should be understood that, although the various steps in the flowcharts involved in the various embodiments described above are displayed in sequence according to the instructions of the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and these steps can be executed in other orders. Moreover, at least a portion of the steps in the flowcharts involved in the various embodiments described above can include multiple steps or multiple stages, and these steps or stages are not necessarily executed and completed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a portion of steps or stages in other steps.

[0069] Based on the same inventive concept, embodiments of the present application also provide a pipeline-type synchronous caching device for implementing the pipeline-type synchronous caching method described above. The solution provided by this device is similar to the solution described in the method described above. Therefore, the specific limitations of one or more pipeline-type synchronous caching device embodiments provided below can be found in the limitations of the pipeline-type synchronous caching method described above and will not be repeated here.

[0070] In one embodiment, Figure 7 As shown, a pipeline synchronization cache device is provided, and the device 700 may include:

[0071] Data acquisition module 701, used to obtain real-time incremental data from the relational database;

[0072] A merge processing module 702 is configured to process the real-time incremental data based on one or more pre-bound functions provided by a function expression engine to obtain data to be submitted; the one or more functions include at least a merge function; the merge function is configured to merge data with the same primary key in the real-time incremental data;

[0073] The pipeline submission module 703 is used to submit the data to be submitted to the cache database through pipeline technology.

[0074] In one embodiment, the merge processing module 702 is used to process the real-time incremental data according to the other functions to obtain the data to be merged, and to process the data to be merged according to the merge function to obtain the data to be submitted when the one or more functions include the merge function and other functions.

[0075] In one embodiment, the other functions include event functions; a merge processing module 702 is used to determine the target data field and target value bound in the event function; if the target data field in the real-time incremental data has the target value, the real-time incremental data is processed according to the event corresponding to the event function to obtain the data to be merged.

[0076] In one embodiment, the real-time incremental data includes real-time incremental data of multiple tables; the merge processing module 702 is used to determine the pre-bound one or more functions provided by the function expression engine corresponding to the real-time incremental data of each table, and process the real-time incremental data of the table based on the one or more functions to obtain the data to be submitted of the table; and obtain the data to be submitted based on the data to be submitted of each table.

[0077] In one embodiment, the merging processing module 702 is used to obtain one or more functions corresponding to the identifier from the function expression engine according to the identifier of the table as one or more pre-bound functions corresponding to the table and provided by the function expression engine.

[0078] Each module in the aforementioned pipeline-type synchronous cache device can be implemented in whole or in part through software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in hardware form, or can be stored in a computer device memory in software form, so that the processor can call and execute the corresponding operations of each module.

[0079] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 8 As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, abbreviated as I / O) and a communication interface. The processor, memory and input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a pipeline synchronous caching method is implemented.

[0080] Those skilled in the art will understand that Figure 8 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0081] In one embodiment, a computer device is provided, including a memory and a processor. The memory stores a computer program, and the processor implements the steps in the above method embodiments when executing the computer program.

[0082] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0083] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0084] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, database or other media used in the embodiments provided in this application may include at least one of non-volatile and volatile memory. Non-volatile memory may include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory may include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in the various embodiments provided herein may include at least one of a relational database and a non-relational database. Non-relational databases may include, but are not limited to, distributed databases based on blockchains. The processor involved in the various embodiments provided herein may be, but are not limited to, a general-purpose processor, a central processing unit, a graphics processing unit, a digital signal processor, a programmable logic unit, a data processing logic unit based on quantum computing, and the like.

[0085] The technical features of the above embodiments can be combined arbitrarily. In order to make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0086] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present application. It should be noted that a person of ordinary skill in the art may make various modifications and improvements without departing from the spirit of the present application, and these modifications and improvements fall within the scope of protection of the present application. Therefore, the scope of protection of the present application shall be determined by the appended claims.

Claims

1. A pipeline synchronous caching method, characterized in that: The method comprises: Obtain real-time incremental data from relational databases; The real-time incremental data is processed based on one or more pre-bound functions provided by a function expression engine to obtain data to be submitted; the one or more functions include at least a merge function; the merge function is used to merge data with the same primary key in the real-time incremental data; the real-time incremental data includes real-time incremental data of multiple tables; the real-time incremental data is processed based on one or more pre-bound functions provided by a function expression engine to obtain data to be submitted, including: for the real-time incremental data of each table, determining the one or more pre-bound functions provided by the function expression engine corresponding to the table, processing the real-time incremental data of the table based on the one or more functions to obtain data to be submitted for the table; obtaining data to be submitted based on the data to be submitted of each table; The data to be submitted is submitted to the cache database through pipeline technology.

2. The method according to claim 1, characterized in that The processing of the real-time incremental data based on one or more pre-bound functions provided by the function expression engine to obtain data to be submitted includes: When the one or more functions include the merge function and other functions, the real-time incremental data is processed according to the other functions to obtain the data to be merged, and the data to be merged is processed according to the merge function to obtain the data to be submitted.

3. The method according to claim 2, characterized in that The other functions include event functions; and the processing of the real-time incremental data according to the other functions to obtain the data to be merged includes: Determine the target data field and target value bound in the event function; If the target data field in the real-time incremental data has the target value, the real-time incremental data is processed according to the event corresponding to the event function to obtain the data to be merged.

4. The method according to claim 1, wherein The determination table corresponds to one or more pre-bound functions provided by the function expression engine, including: According to the identifier of the table, one or more functions corresponding to the identifier are obtained from the function expression engine as one or more pre-bound functions corresponding to the table and provided by the function expression engine.

5. A pipeline synchronous cache device, characterized in that: The device comprises: Data acquisition module, used to obtain real-time incremental data from relational databases; A merge processing module is configured to process the real-time incremental data based on one or more pre-bound functions provided by a function expression engine to obtain data to be submitted; the one or more functions include at least a merge function; the merge function is configured to merge data with the same primary key in the real-time incremental data; the real-time incremental data includes real-time incremental data of multiple tables; the merge processing module is configured to determine, for each table's real-time incremental data, the one or more pre-bound functions provided by the function expression engine corresponding to the table, and process the table's real-time incremental data based on the one or more functions to obtain data to be submitted for the table; and obtain data to be submitted based on the data to be submitted for each table; The pipeline submission module is used to submit the data to be submitted to the cache database through pipeline technology.

6. The device according to claim 5, characterized in that The merge processing module is used to process the real-time incremental data according to the other functions to obtain the data to be merged when the one or more functions include the merge function and other functions, and to process the data to be merged according to the merge function to obtain the data to be submitted.

7. The device according to claim 5, characterized in that The merging processing module is used to obtain one or more functions corresponding to the identifier from the function expression engine according to the identifier of the table, as one or more pre-bound functions corresponding to the table and provided by the function expression engine.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 4 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 4 are implemented.

10. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 4 are implemented.

Images