A templated business cache processing method and system

Abstract

The present invention belongs to the technical field of computer technology server-side data processing, and discloses a templated business cache processing method and system. For a relational database using structured data, this method completes data storage in the relational database by defining a fixed data model as a cache data structure; the data stored in the cache is converted into a usable binary format in a serialized manner; an abstract interface for the data structure and an abstract interface for data operations are defined, and cache commands are performed on the detailed problem data used by the template class. The present invention uses a template to encapsulate the specific implementation of the operation steps into the template class, so that a general processing that supports multiple data types can be used to improve the code reuse rate and reduce redundancy. At the same time, the template class can be easily expanded to support new data types, and the unified maintenance method in the template class makes the code easier to maintain and expand.

Description

Technical Field

[0001] The present invention belongs to the technical field of computer technology server-side data processing, and in particular, relates to a templated business cache processing method and system. Background Art

[0002] Cache is an important concept in the computer field, and plays an important role in hardware devices, operating systems, databases, application software and many other fields. Cache in the conventional sense refers to a data storage device that provides fast access. Nowadays, the concept of cache has been expanded. There is not only cache between the CPU and the main memory, but also cache between the memory and the hard disk (disk cache), and even cache between the hard disk and the network in a certain sense, called Internet temporary folders or network content cache. Any structure located between two hardware with a large difference in speed and used to coordinate the difference in data transmission speed between the two can be called cache.

[0003] While providing high-speed access, cache also has the problem of high unit price. Therefore, in actual production use, data with higher value will be stored in the cache. The value here can be assessed based on various factors such as usage rate and importance. In the process of using the cache, the data in the cache is usually constantly updated. Some data with higher value will replace low-value data, and expired data needs to be cleaned up immediately. This way of loading and cleaning cache data is called a cache strategy.

[0004] In the field of application software, ensuring data consistency is an important non-functional requirement of most business systems. At the same time, cache components are introduced to achieve high performance. When using cache, you need to choose an appropriate strategy based on the characteristics of the system. Issues that usually need attention include but are not limited to cache breakdown, cache penetration, and cache avalanche. For systems with complex business, it is not easy to use cache at a low cost and solve various cache usage problems.

[0005] The traditional cache processing method, taking Spring Cache as an example, the Spring framework provides annotation-based declarative programming support for caching. In terms of methods, you only need to add corresponding annotations and increase the global cache configuration to easily implement basic cache operations.

[0006] However, the above cache operation support cannot adapt well to many business scenarios. For example, when there are many types of data connected to the cache, the above method needs to create many redundant classes or methods to implement the cache function. The cache operation will invade the specific implementation class. At the same time, due to the lack of linkage processing for related data items and the inability to support one operation affecting multiple cache key values, it is difficult to ensure cache consistency. In addition, there is a lack of compensation mechanism and it is impossible to flexibly manage the life cycle of various keys. Summary of the invention

[0007] In order to overcome the problems existing in the related art, the disclosed embodiments of the present invention provide a templated service cache processing method and system, and more particularly, a method for ensuring the consistency of cache and service data by using a templated solution.

[0008] The technical solution is as follows: a templated business cache processing method provides data structure encapsulation and templated definition of the operation process for cache components that only support non-relational data storage, realizes synchronization with relational database data, and achieves the purpose of improving query efficiency, specifically including the following steps:

[0009] S1, for a relational database using structured data, a fixed data model is defined as a cache data structure to complete data storage in the relational database;

[0010] S2, the data stored in the cache is serialized and converted into binary format;

[0011] S3, defines the abstract interface of data structure and standardizes the operable data structure; defines the abstract interface of data operation, provides public data processing functions, and provides external service usage functions;

[0012] S4, caches the detail problem data used by the template class, encapsulates the detail problems, uses a variety of cache operation instructions, arranges them into a fixed process, forms a template method, and exposes a simple and easy-to-use API for the access party; the detail problem data includes cache lifecycle management and miss problem data.

[0013] In step S1, data storage in a relational database is completed, including: the data in the table is stored in the dimension of rows using the redis string type, the key is the table name + primary key, and the value is the row data.

[0014] It also includes: the data relationship between tables is stored using a collection type, the key is the primary table name + foreign key table name + foreign key value, and the value is the primary table id set.

[0015] In step S2, the data stored in the cache is serialized, including: for large amounts of data, using Kryo serialization; for low read and write performance requirements, using a Json format serialization framework.

[0016] In step S3, an abstract interface of the data structure is defined, an abstract interface of data operations is defined, a persistent entity layer interface is defined, and a method for obtaining a primary key is defined.

[0017] Furthermore, methods for obtaining primary keys include:

[0018] Public methods include add, delete, modify and query operations. The parameters of the new and modified methods are entity objects, and the persistent entity objects are returned. The objects are compatible with the data structure in the relational database.

[0019] The query method supports returning data results based on a primary key or a set of primary keys, wherein the data results include data from only a relational database, or only from a cache, or a combination of a relational database and a cache;

[0020] The delete method parameter is an entity object, which performs synchronous deletion of data in the relational database and the data in the cache. Adding, modifying, deleting and querying all support batch operations.

[0021] For data with associated foreign key relationships, the relationship collection is maintained synchronously when the write operation is performed. This type of data operation additionally provides an interface for querying the associated data collection based on the relationship.

[0022] Furthermore, in the definition of the method for obtaining the primary key, the optimistic locking mechanism of the relational database is used to use the version number in high-concurrency scenarios to ensure that no overwriting scenarios occur. Based on the unique constraint, it is ensured that no duplicate additions are made when concurrent data is inserted. Since the cached writes are only submitted but not executed when writing database data, the cache commit transaction operation is executed after the database transaction is successfully submitted. For special scenarios under high concurrency, a new data comparison scheduled task is added to compare the cached data modified in the recent period with the database. The data in the relational database is used as the benchmark to keep the cache consistent with the relational database, clean up the dirty data, and wait for the next reload.

[0023] To address the problem of cached data occupying memory, we differentiate between hot and cold data based on business scenarios, and load hot data into the cache to improve reading speed. Hot data includes time-sensitive data and non-time-sensitive data. We increase the expiration time for time-sensitive data to prevent a large amount of cold data from occupying memory space. We set a memory usage threshold for non-time-sensitive data to limit the total memory usage, and specify an elimination algorithm to ensure normal operation when the usage limit is reached.

[0024] In step S4, a cache command is issued to the detail problem data used by the template class, and the detail problem is encapsulated, including:

[0025] S401, the logic of the query operation: first, the key in the cache is spliced ​​according to the primary key value, and the cache data is queried according to the key. If the data exists, it is directly returned; if the data does not exist, the Bloom filter of the specified type is queried. If a value exists, it means that it has been queried and it is directly returned;

[0026] S402, logic of adding and modifying operations: first, the key in the cache is spliced ​​according to the primary key value, and the write operation of the database is called first. After normal execution, it is written into the cache and the cache expiration time is increased. It is put into the context of the thread in the form of transaction callback execution, and the cache operation is submitted after the database write operation is successfully submitted;

[0027] S403, the logic of the deletion operation: first, the key in the cache is spliced ​​according to the primary key value, and the deletion operation of the database is called first. After normal execution, the key operation in the cache is placed in the transaction context, and the cache deletion operation is submitted after the database transaction is committed;

[0028] S404, the batch write operation uses the batch operation of the ORM framework of the database and the cached batch processing command;

[0029] S405, batch query operation processing: first, convert the parameters into a set of keys, use the batch query command to try to obtain relevant data from the cache, compare the result set returned by the cache with the request parameter list, and if there is data that has not been queried, filter out the results that have been queried according to the Bloom filter, initiate a database batch query for the remaining unqueried results, save the query data ID to the Bloom filter, and save the data to the cache at the same time. When adding the cache, add the expiration time of the key according to the configuration;

[0030] S406, according to the interface for querying data based on the associated foreign key, after receiving the request, first assemble the parameters to obtain the key of the collection type in the cache, then query the cached data to obtain the associated key set, and assemble it into a table-level cache key based on the return, call the batch cache acquisition interface, obtain the data, compare the obtained results with the request parameters to see if they match, filter out the unqueried results, filter out the data that needs to be supplemented by querying the database according to the Bloom filter, execute the cache write operation, complete the missing data, and then return the result to the requester.

[0031] In step S401, if the data does not exist in the Bloom filter, the data in the database is first queried according to the primary key, written into the Bloom filter, and then if the result exists, it is written into the cache and the cache expiration time is added, and the result is returned.

[0032] Another object of the present invention is to provide a templated service cache processing system, the system comprising:

[0033] A data storage module is used for completing data storage in a relational database using structured data by defining a fixed data model as a cache data structure;

[0034] A data conversion module is used to convert the data stored in the cache into a binary format in a serialized manner;

[0035] The interface definition module is used to define the abstract interface of the data structure and standardize the operable data structure; define the abstract interface of data operation, provide public data processing functions, and provide external service usage functions;

[0036] The cache command execution module is used to execute cache commands on the detail problem data used by the template class, encapsulate the detail problems, use a variety of cache operation instructions, arrange them into a fixed process, form a template method, and expose a simple and easy-to-use API for the access party to use; the detail problem data includes cache lifecycle management and miss problem data.

[0037] Combining all the above technical solutions, the beneficial effects of the present invention are as follows: In order to solve the problems existing in the background technology, the present invention extracts a template-based cache management method in the practice process. Based on the self-defined template method, the problems that need to be paid attention to in the use of cache are uniformly solved, and the cache and database operation process implementation that is not related to the business is shielded, so that the business party can focus on the functional logic without having to consider too much about the cache usage regulations.

[0038] The present invention uses a template to encapsulate the specific implementation of the operation steps into a template class, so that a general process that supports multiple data types can be used to improve the code reuse rate and reduce redundancy. At the same time, the template class can be easily expanded to support new data types, and the unified maintenance method in the template class makes the code easier to maintain and expand. Using the components in the present invention can significantly improve the query efficiency of application services, reduce the utilization rate of database resources, and save server hardware resources. A reference solution is provided for the final consistency of data in the cache and the database. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] The accompanying drawings herein are incorporated in and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description, serve to explain the principles of the present disclosure;

[0040] Figure 1 It is a flow chart of a templated service cache processing method provided by an embodiment of the present invention;

[0041] Figure 2is a schematic diagram of a cache data structure provided by an embodiment of the present invention;

[0042] Figure 3 is a query flow chart provided by an embodiment of the present invention;

[0043] Figure 4 It is a write operation flow chart provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0044] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific embodiments of the present invention are described in detail below in conjunction with the accompanying drawings. In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention can be implemented in many other ways different from those described herein, and those skilled in the art can make similar improvements without violating the connotation of the present invention, so the present invention is not limited by the specific implementation disclosed below.

[0045] The innovation of the present invention is that: based on unstructured data modeling, the present invention abstracts the operation process to extract a reusable template, encapsulates the detailed operation process, and provides a simple and easy-to-use API interface.

[0046] The present invention provides a templated cache processing solution, which predefines a data structure that meets the specifications to support operations with the same semantics as the relational database and the cache (such as basic addition, deletion, modification and query), adopts a synchronous double-write (first write to the database, then write to the cache) method to ensure the cache hit rate, and uses the database's transaction and optimistic locking mechanism to ensure the consistency of the synchronous double-write operation.

[0047] In terms of implementation, the general API is encapsulated to abstract common operations such as adding, deleting, modifying and checking. The abstract template class is extracted to provide specific implementation for the encapsulated API, mainly for synchronous reading and writing operations of database operations and caches. In this way, a set of components for unified management of relational databases and caches is established, allowing the access party to obtain synchronous management of data and caches through inheritance and implementation, and the ability to switch to read data in caches or databases as needed.

[0048] Before implementation, the present invention organizes the basic process of cache usage: usually, it is necessary to first define the cache data model (data structure), define the operation method, and then connect with the business system function. In addition, it is also necessary to pay attention to aspects such as cache data life cycle, memory usage management, and cache consistency.

[0049] Embodiment 1, as Figure 1 As shown, the templated service cache processing method provided by the embodiment of the present invention includes:

[0050] S1, for a relational database using structured data, a fixed data model is defined as a cache data structure to complete data storage in the relational database;

[0051] like Figure 2 As shown, to support data storage in a relational database;

[0052] For example, Redis is used to convert the data in a traditional relational database into different data types:

[0053] The data in the table is stored in the row dimension using the redis string type (String), the key is the table name + primary key, and the value is the row data.

[0054] The data relationship between tables is stored using a set type (Set), where the key is the primary table name + foreign key table name + foreign key value, and the value is the primary table id set.

[0055] S2, the data stored in the cache is serialized and converted into binary format;

[0056] When choosing a serialization method, you need to consider issues such as space, time, readability, and scalability.

[0057] Among them, efficient serialization methods can significantly reduce serialization time and ensure that the serialization results occupy as little memory space as possible, but they usually cannot provide good readability and scalability of serialization results such as Kryo serialization.

[0058] Frameworks with high readability (main information can be read directly through the serialization results) and high scalability (modification can be made directly based on the serialization results to achieve the purpose of modifying data), such as the commonly used Json format serialization, perform poorly in terms of performance and storage space occupied by the results.

[0059] Kryo is a high-performance serialization / deserialization tool. Due to its variable-length storage characteristics and the use of bytecode generation mechanism, it has a high running speed and a small size.

[0060] The present invention can select the serialization method according to the actual business scenario. For services with large data volume and very sensitive to read and write speed, it is recommended to use an efficient serialization method such as Kryo, giving up readability and scalability; for scenarios with low requirements for read and write performance and the desire to directly maintain the cache data structure and dynamically expand, it is recommended to use a serialization framework in Json format. Kryo and Json are both widely used serialization methods, and SDKs adapted to different development languages ​​are provided. The present invention can easily complete the serialization operation by directly integrating and using the SDK API.

[0061] S3, defines the abstract interface of data structure and standardizes the operable data structure; defines the abstract interface of data operation, provides public data processing functions, and provides external service usage functions;

[0062] The interface defines the getId() method, which must be implemented by the subclass to provide the ID field. When the abstract template class is used, the primary key of the data is obtained through the interface getId method. The data operation abstract interface is a unified encapsulation of abstract data interface operations. The logic adopts abstract interface-oriented programming rather than instance-oriented programming.

[0063] For example, if row-level data needs to have a primary key column, a persistent entity layer interface is defined here. The interface defines a method for obtaining the primary key, requiring the implementation class to provide the primary key column before caching operations can be performed.

[0064] Methods for obtaining primary keys include:

[0065] Public methods include add, delete, modify and query operations. The parameters of the new and modified methods are entity objects, and the persistent entity object is returned. This object is compatible with the data structure in the database and is actually the data in the cache corresponding to the database data.

[0066] The query method supports returning data results based on the primary key or primary key set. This result may come only from the database, only from the cache, or a combination of the two.

[0067] The delete method parameter is the entity object, which performs synchronous deletion of data in the database and the cache. In addition, adding, modifying, deleting and querying all support batch operations to improve data processing efficiency.

[0068] For data with associated foreign key relationships, the relationship collection will be maintained synchronously when the write operation is performed. This type of data operation also provides an interface for querying the associated data collection based on the relationship.

[0069] Among them, (1) an issue that needs to be focused on is that database operations usually use a transaction mechanism, which means that the data will not be persisted immediately when the operation is executed, but the data will not be actually written to the specified table in the database until the transaction is normally committed. If an exception occurs after the addition, deletion, and modification operations are completed but before the transaction is committed, causing a rollback, the data will not be saved to the database. With synchronous double writing, the write operation to the cache is usually executed immediately and written to the cache component, which will cause inconsistency.

[0070] The present invention selects a solution based on the support of the cache component for transactions and the framework used. Here, Redis with high versatility is taken as an example. Redis supports transaction mechanism, but unlike database transactions, it can only support two options: write and discard.

[0071] Exemplarily, the present invention utilizes the optimistic locking mechanism of the database to use the version number in the scenario with high concurrency to ensure that there will be no overwriting scenario, and based on the unique constraint to ensure that there will be no duplication when inserting data concurrently, with the help of the optimistic locking mechanism provided by the ORM framework such as hibernate, it is very convenient to implement the optimistic lock comparison mechanism of update operations, or it can be implemented by itself, such as using update ... where version=currentVersion to implement optimistic locking. Since when writing database data, the cache write is only submitted but not executed, the cache commit transaction operation is executed only after the database transaction is successfully submitted, so that the database transaction can be used to ensure the consistency of the data in the database and cache.

[0072] When designing a database table, add a version number field of integer type to implement optimistic locking. When updating, compare the version number. If the version number changes, it is considered to have been updated by someone else and the operation is rolled back.

[0073] The above method can solve the consistency problem under normal operation, but there is still a low probability of data inconsistency in some special scenarios under high concurrency. In view of this situation, the present invention newly adds a data comparison timer task to cache the data modified in the recent period and compare it with the database. The data in the database is used as a benchmark to keep the cache consistent with it, clean up the dirty data, and wait for the next reload. The timer task obtains the data in the cache, queries the data in the database based on the primary key information of the data, and deletes the cache if the cached data does not exist in the database. Otherwise, extract the summary based on the object, compare the summary information to see if it is consistent, and update or delete the cached data if it is inconsistent.

[0074] (2) Another issue that needs attention is the problem of cache data occupying memory. According to the business scenario, hot and cold data can usually be distinguished. For hot data, the present invention loads it into the cache to improve the reading speed. Hot data can be divided into time-sensitive data and non-time-sensitive data. For time-sensitive data, the present invention can increase the expiration time to avoid a large amount of cold data occupying memory space. The template has a built-in encapsulation method. The write operation will synchronously maintain the expiration time of the cache key. The value of the expiration time can be passed in by external configuration by providing a configuration acquisition method, which is more flexible to use. Hot data is a logical concept and needs to be manually identified by the application of the present invention component. The design content of automatically identifying hot data based on the frequency of use of statistical data is not involved for the time being; for non-time-sensitive data, the present invention sets a threshold for memory usage to limit its total usage. The cache component itself provides a command to obtain memory usage. According to the current usage obtained, combined with the expected amount of data to be written, it is judged whether it exceeds the specified threshold. If it exceeds, an operation exception prompt is thrown. If this mechanism is not enabled, the cache itself can be used to ensure the availability of the operation. And specify the elimination algorithm to ensure that it can still run normally when the usage limit is reached. The cache component itself provides a cache elimination algorithm, which can clear relatively less used data when the threshold is reached. This may frequently clear data in the cache, reduce the cache hit rate, and reduce the read and write efficiency. Therefore, it is recommended to provide sufficient hardware resources for the cache component.

[0075] The present invention adopts a combination of the above-mentioned abstract interface for defining data structure and the abstract interface for defining data operation to provide a universal interface, allowing the access party to configure the expiration time, specify the elimination strategy, and flexibly respond to various usage scenarios.

[0076] S4, caches the detail problem data used by the template class, encapsulates the detail problems, uses a variety of cache operation instructions, arranges them into a fixed process, forms a template method, and exposes a simple and easy-to-use API for the access party; the detail problem data includes cache lifecycle management and miss problem data.

[0077] The following describes the specific implementation methods, taking various types of operation APIs as examples, including:

[0078] S401, the logic of the query operation: first, splice the key in the cache according to the primary key value. Each cache instance definition requires the implementation of an abstract method to specify the cache name prefix. The splicing method is the functional method of the cache template, which uses the cache prefix and the data primary key to splice to form a unique key. Query the cache data according to the key, and return directly if the data exists. If the data does not exist, query the Bloom filter of the specified type. The cache instance must implement the extraction template. The abstract method defined in the template binds a different Bloom filter to each specific implementation, and the name is spliced ​​according to the cache name. If a value exists, it means that it has been queried and returned directly.

[0079] If the data does not exist in the Bloom filter, first query the data in the database based on the primary key, write it into the Bloom filter, and then if the result exists, write it into the cache and add the cache expiration time. Find the corresponding filter based on the cache naming prefix, and call the filter query API based on the key to query whether the data exists. And return the result. Here, it is necessary to avoid invalid data from being requested to the database every time, so the Bloom filter is introduced. To ensure consistency, the Bloom filter is also saved in the cache;

[0080] S402, logic of adding and modifying operations: first, splice the key in the cache according to the primary key value, first call the database write operation, use the orm framework such as hibernate to perform data writing, and use it in combination with the framework's transaction support (such as Spring declarative transactions, add @Transactional annotations to methods). After normal execution, write to the cache and increase the cache expiration time, put it into the thread context in the form of transaction callback execution, wait for the database write operation to be successfully submitted, and then submit the cache operation. Application frameworks such as Spring themselves provide global transactions, support writing transaction submission post-processing methods, and put them into the thread context. After the database operation transaction is submitted, the post-processing is called. In this way, the execution order of transaction operations can be controlled.

[0081] S403, the logic of the deletion operation: first, the key in the cache is spliced ​​according to the primary key value, and the deletion operation of the database is called first. After normal execution, the key operation in the cache is placed in the transaction context, and the cache deletion operation is submitted after the database transaction is committed.

[0082] S404, batch write operations can use the batch operation of the database ORM framework and the batch processing command of the cache to improve the writing performance. The ORM framework itself and the cache operation API itself provide batch operation methods, but they are difficult to use and have many usage details. The specific details are shielded here through unified encapsulation.

[0083] For example, ORM batch operations can improve write performance by executing commits in batches, and cache batch writes can transform multiple write operations into one instruction to improve write performance.

[0084] S405, the processing logic of batch query operation is slightly more complicated. First, it is converted into a set of keys according to the parameters, and the name is obtained according to the method of obtaining the naming prefix implemented by the subclass, and then it is spliced ​​with the primary key of the data in the table. Use the batch query command to try to obtain relevant data from the cache first, compare the result set returned by the cache with the request parameter list, use the API in the cache client SDK, pass in the spliced ​​Key, and query the returned results. If there is data that has not been queried, the results that have been queried are filtered out according to the Bloom filter, and the remaining unqueried database batch query is initiated. The Bloom filter is used to retrieve whether an element is in a set. Since the write operation is uniformly encapsulated, the new operation will put the key value into the Bloom filter. When querying data, this filter is used to determine whether the data already exists. For data that does not exist in the filter, the key of the query request will be placed in it, which ensures that the queried data will be stored in the filter. How to filter out the results that have been queried according to the Bloom filter, and initiate a database batch query for the remaining unqueried; and save the query data ID to the Bloom filter, and save the data to the cache at the same time, and increase the expiration time of the key added according to the configuration when caching.

[0085] S406, according to the interface of querying data by associated foreign key, after receiving the request, the parameters are assembled first to obtain the key of the collection type in the cache. The assembly logic is encapsulated in the abstract template class, and the cache naming prefix is ​​obtained by associating the foreign key column keyword and splicing the data ID. The foreign key column is identified by the attribute value and keyword pre-defined by the subclass. Then query the cached data to obtain the associated key set, and assemble it into a table-level cache key according to the return. The data queried according to the collection key is the primary key of other tables. The cache naming prefix to be queried can be obtained by parsing the foreign key table corresponding to the field in the previous step, and the encapsulated method of obtaining the associated data key is called to obtain the associated table cache data primary key. Call the batch cache acquisition interface to obtain data, compare the results and request parameters to see if they match. The result set returned by the cache operation API is returned according to the parameter array of the query request, and there is a subscript correspondence. For example, if the data [1,2,3] is queried, [r1,null,r3] is returned, where null represents the query result that does not exist for data 2. Filter out the unqueried results. According to the Bloom filter, filter out the data that needs to be queried in the database. If the filter query returns that it does not exist, it proves that the query has never been executed. In this case, it is necessary to query the database and put the results into the cache. Compare the id in the queried result set with the id in the request. The missing part is the one that was not queried. Perform cache write operations to complete the missing data, and then return the results to the requester. The query process is as follows: Figure 3 , the write operation process is as follows Figure 4 shown.

[0086] Embodiment 2, as another implementation, the templated service cache processing method provided by the embodiment of the present invention includes:

[0087] The present invention is used to improve the reading efficiency by querying cache data when displaying a dynamic flight information list on a flight information management page. When the flight information changes, the cache data is updated synchronously to ensure data consistency.

[0088] The flight information list displays all flight information and flight-related data (no paging). For medium and large airports, a large amount of data needs to be queried (based on 500 flight data, each associated with data from ten other tables, more than 5,000 data items need to be obtained for one query). A single request returns a lot of data and the query takes a long time. A large number of query requests at the same time will increase the pressure on the database and affect the available resources of the business system database connection pool.

[0089] Since the flight list is read and written frequently, the synchronous double write method can improve the cache hit rate. Using the data in the cache for display can effectively reduce the pressure on the system to access the database and improve the system throughput. If the cache eviction solution is to remove the data first and wait for the next load every time there is a change, the data will be frequently deleted and loaded when the data is written frequently, and the hit rate naturally cannot be compared with the synchronous double write of overwriting update.

[0090] Assuming that the flight data displayed in the flight management page list contains ten kinds of related data, create a Set-type cache key composed of the flight ID plus the related table to store the direct correspondence between the flight and various types of data. For the flight data and ten kinds of related data, create a String-type cache with the table as the dimension, and the key is the table name + primary key. Use the pipeline command to batch query the cached data, you can get all the data within 100 milliseconds, and then assemble the results and return them to the display page.

[0091] The daily flight volume of airports with tens of millions of passengers is usually less than 500. The present invention uses 500 flights as a test benchmark to verify the query efficiency. The flight-related data to be queried mainly include about 8,000 security-related data and about 8,000 other types of data in total. The total query data involves more than ten tables.

[0092] When executing queries based on foreign keys, relational databases need to consider the overall data volume of the table, indexes, etc., and the query efficiency is between tens of ms and hundreds of ms. Cache queries such as redis have a QPS of about 100,000 per second, and the query data is one order of magnitude less than the database. The total query response time is in seconds when using the database and in hundreds of milliseconds when using the cache. In terms of implementation, by inheriting the cache template abstract implementation class, defining the keyword prefixes of each type of data, and implementing the cache lifecycle management method, you can get a general database and cache operation capability without paying attention to the specific implementation details. When using it, call the method inherited from the template to manage the flight data.

[0093] When performing a query operation, the cache is queried first. If there is no hit, the Bloom filter is queried. If the Bloom filter is hit, it is returned directly. Otherwise, the database is queried to supplement the cache data and then returned. Querying the cache based on the primary key can quickly obtain the specified data, greatly improving query efficiency.

[0094] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0095] To further illustrate the relevant effects of the embodiments of the present invention, the following experiments are conducted. Spring data cache is used to query and cache flight data and related data. A coarse-grained method can cache a bigKey corresponding to the entire query result set. However, since the data changes frequently, it is difficult to ensure real-time performance of this cache, so it must be divided into fine-grained caches, that is, each flight data corresponds to a separate cache. Since flights are also associated with more than ten other data, it is necessary to maintain many types of caches when querying and displaying flight lists. Since there are associations between cached data, each time new, modified, or deleted data is added, it is necessary to cascade other cached data according to the relationship. At this time, using the API provided by spring data cache to operate will become very cumbersome (each time data is written, it is necessary to determine whether the related data needs to be modified, and then perform the corresponding operation) and prone to problems (the logic needs to be hard-coded, which is easy to miss and write errors), and the traditional cache usage method lacks support for batch operations, and can only be operated one by one through convenience, and performance cannot be guaranteed.

[0096] By using templates to uniformly encapsulate common processing such as associated operations and batch operations, ease of use is greatly improved, maintenance difficulty is reduced, and performance is improved.

[0097] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any modifications, equivalent substitutions and improvements made by any technician familiar with the technical field within the technical scope disclosed by the present invention and within the spirit and principles of the present invention should be covered by the protection scope of the present invention.

Claims

1. A templated service cache processing method, characterized in that: The method provides data structure encapsulation and operation process template definition for cache components that only support non-relational data storage, realizes synchronization with relational database data, and improves query efficiency. Specifically, the method includes the following steps: S1, for a relational database using structured data, a fixed data model is defined as a cache data structure to complete data storage in the relational database; S2, the data stored in the cache is serialized and converted into binary format; S3, defines the abstract interface of data structure and standardizes the operable data structure; defines the abstract interface of data operation, provides public data processing functions, and provides external service usage functions; S4, caches the detailed problem data used by the template class, encapsulates the detailed problems, uses a variety of cache operation instructions, arranges them into a fixed process, forms a template method, and exposes a simple and easy-to-use API for the access party to use; the detailed problem data includes cache lifecycle management and miss problem data; In step S3, define the abstract interface of the data structure, define the abstract interface of the data operation, define the persistent entity layer interface, and define the method of obtaining the primary key; Methods for obtaining primary keys include: Public methods include add, delete, modify and query operations. The parameters of the new and modified methods are entity objects, and the persistent entity objects are returned. The objects are compatible with the data structure in the relational database. The query method supports returning data results based on a primary key or a set of primary keys, wherein the data results include data from only a relational database, or data from only a cache, or data from a combination of a relational database and a cache. The delete method parameter is an entity object, which performs synchronous deletion of data in the relational database and the data in the cache. Adding, modifying, deleting and querying all support batch operations. For data with associated foreign key relationships, the relationship set is maintained synchronously when the write operation is performed. This type of data operation also provides an interface for querying the associated data set based on the relationship. In the method of defining the acquisition of primary keys, the optimistic locking mechanism of the relational database is used to use the version number in high-concurrency scenarios to ensure that no overwriting scenarios occur. Based on the unique constraint, it is ensured that no duplicate additions occur when concurrent data is inserted. When writing database data, the cache write is only submitted but not executed. When the database transaction is successfully submitted, the cache submission transaction operation is executed. For special scenarios under high concurrency, a new data comparison scheduled task is added to compare the cached data modified in the recent period with the database. The data in the relational database is used as the benchmark to keep the cache consistent with the relational database, clean up the dirty data, and wait for the next reload. To solve the problem of cache data occupying memory, we distinguish between hot and cold data according to business scenarios, and load hot data into cache to improve reading speed. Hot data includes time-sensitive data and non-time-sensitive data. We increase the expiration time for time-sensitive data to prevent a large amount of cold data from occupying memory space. We set a memory usage threshold for non-time-sensitive data to limit the total memory usage, and specify an elimination algorithm to ensure normal operation when the usage limit is reached. In step S4, a cache command is issued to the detail problem data used by the template class, and the detail problem is encapsulated, including: S401, the logic of the query operation: first, the key in the cache is spliced ​​according to the primary key value, and the cache data is queried according to the key. If the data exists, it is directly returned; if the data does not exist, the Bloom filter of the specified type is queried. If a value exists, it means that it has been queried and it is directly returned; S402, logic of adding and modifying operations: first, the key in the cache is spliced ​​according to the primary key value, and the write operation of the database is called first. After normal execution, it is written into the cache and the cache expiration time is increased. It is put into the context of the thread in the form of transaction callback execution, and the cache operation is submitted after the database write operation is successfully submitted; S403, the logic of the deletion operation: first, the key in the cache is spliced ​​according to the primary key value, and the deletion operation of the database is called first. After normal execution, the key operation in the cache is placed in the transaction context, and the cache deletion operation is submitted after the database transaction is committed; S404, the batch write operation uses the batch operation of the ORM framework of the database and the cached batch processing command; S405, batch query operation processing: first, convert the parameters into a set of keys, use the batch query command to try to obtain relevant data from the cache, compare the result set returned by the cache with the request parameter list, and if there is data that has not been queried, filter out the results that have been queried according to the Bloom filter, initiate a database batch query for the remaining unqueried results, save the query data ID to the Bloom filter, and save the data to the cache at the same time. When adding the cache, add the expiration time of the key according to the configuration; S406, according to the interface for querying data based on the associated foreign key, after receiving the request, first assemble the parameters to obtain the key of the collection type in the cache, then query the cached data to obtain the associated key set, and assemble it into a table-level cache key based on the return, call the batch cache acquisition interface, obtain the data, compare the obtained results with the request parameters to see if they match, filter out the unqueried results, filter out the data that needs to be supplemented by querying the database according to the Bloom filter, execute the cache write operation, complete the missing data, and then return the result to the requester.

2. The templated service cache processing method according to claim 1, characterized in that: In step S1, data storage in a relational database is completed, including: the data in the table is stored in the dimension of rows using the redis string type, the key is the table name + primary key, and the value is the row data.

3. The templated service cache processing method according to claim 2, characterized in that: Also includes: The data relationship between tables is stored using a collection type, where the key is the primary table name + foreign key table name + foreign key value, and the value is the primary table ID set.

4. The templated service cache processing method according to claim 1, characterized in that: In step S2, the data stored in the cache is serialized, including: for large amounts of data, using Kryo serialization; for low read and write performance requirements, using a Json format serialization framework.

5. The templated service cache processing method according to claim 1, characterized in that: In step S401, if the data does not exist in the Bloom filter, the data in the database is first queried according to the primary key, written into the Bloom filter, and then if the result exists, it is written into the cache and the cache expiration time is added, and the result is returned.

6. A templated business cache processing system, characterized in that: Implementing the templated service cache processing method according to any one of claims 1 to 5, the system comprises: A data storage module is used for completing data storage in a relational database using structured data by defining a fixed data model as a cache data structure; A data conversion module is used to convert the data stored in the cache into a binary format in a serialized manner; The interface definition module is used to define the abstract interface of the data structure and standardize the operable data structure; define the abstract interface of data operation, provide public data processing functions, and provide external service usage functions; The cache command execution module is used to execute cache commands on the detail problem data used by the template class, encapsulate the detail problems, use a variety of cache operation instructions, arrange them into a fixed process, form a template method, and expose a simple and easy-to-use API for the access party to use; the detail problem data includes cache lifecycle management and miss problem data.

Images