Distributed system data access method, device and equipment and storage medium
By using a distributed cache database in a distributed system to save high-frequency access data and update it regularly, the problem of low data access efficiency in distributed systems is solved, and faster data access speed and more flexible cache update strategies are achieved.
Patent Information
- Application Number
- CN202411755563.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-30
- Publication Date
- 2025-03-14
Smart Images
Figure CN119621782A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of data processing technology, and in particular to a distributed system data access method, device, equipment and storage medium. Background Art
[0002] A distributed data processing system is an architecture that distributes data storage and processing to multiple nodes (servers or computers). This system can effectively process large amounts of data and improve the scalability, reliability and performance of the system. At present, data access efficiency in distributed systems is often the bottleneck of system performance. Systems that use distributed data planning usually have relatively large amounts of data and complex design structures. However, due to the complex association of data, when performing data queries, the data that needs to be retrieved may be distributed on different servers. When querying, it is necessary to retrieve it from different nodes, and then integrate and return it in the required format. Data retrieval is the most frequent demand in most application systems, which will seriously reduce the data access speed. Summary of the invention
[0003] In view of this, the purpose of the present invention is to provide a distributed system data access method, device, equipment and storage medium, which can improve the efficiency of distributed system data access. The specific scheme is as follows:
[0004] In a first aspect, the present application discloses a distributed system data access method, comprising:
[0005] Determine target type data in a distributed system, and use a distributed cache database to store the target type data; the target type data is high-frequency access data;
[0006] Create a corresponding scheduled update task for each target type data, and add a data acquisition method for each target type data to the data update service;
[0007] When executing the scheduled update task, calling the corresponding data acquisition method in the data update service to obtain the latest data of the target type data, and updating the distributed cache database according to the latest data;
[0008] The distributed cache database is queried according to the data query instruction. If the query result is empty, the main database of the distributed system is queried according to the query instruction.
[0009] Optionally, the adding the data acquisition method of each target type data to the data update service includes:
[0010] Creating a data update service for the distributed system;
[0011] Obtain the data acquisition methods corresponding to all target type data;
[0012] Add all of the data acquisition methods to the data update service.
[0013] Optionally, the using a distributed cache database to store the target type data includes:
[0014] Utilize a distributed cache database to store the target type data, and add a timestamp to each piece of data stored in the distributed cache database; the timestamp is used to indicate the most recent modification time of the data;
[0015] Accordingly, the distributed system data access method further includes:
[0016] According to the preset validity period for the target type data and the timestamp, invalid data in the distributed cache database is filtered out.
[0017] Optionally, the step of creating a corresponding scheduled update task for each target type data includes:
[0018] A corresponding scheduled update task is created for each target type data, and a scheduled duration corresponding to each target type data is configured so that the scheduled update task is executed when the scheduled update task reaches its own scheduled duration.
[0019] Optionally, querying the distributed cache database according to the data query instruction, and if the query result is empty, querying the main database of the distributed system according to the query instruction includes:
[0020] Obtaining a data query instruction, and querying the cache database according to the data query instruction;
[0021] If the target data corresponding to the data query instruction is queried from the cache database, a query result of the data query instruction is generated based on the target data;
[0022] If the target data corresponding to the data query instruction is not found in the cache database, the distributed storage system is queried according to the data query instruction to obtain the target data corresponding to the data query instruction and generate a query result.
[0023] Optionally, updating the distributed cache database according to the latest data includes:
[0024] Creating a cache task for the latest data; the cache task is used to save the latest data to the distributed cache database;
[0025] Execute the cache task according to the cache task queue, and before executing the current cache task, search for corresponding data from the main database of the distributed system according to the query instruction again to obtain actual data;
[0026] The latest data is compared with the actual data. If the latest data is consistent with the actual data, the distributed cache database is updated based on the latest data. If the latest data is inconsistent with the actual data, the distributed cache database is updated based on the actual data.
[0027] Optionally, the determining of target type data in the distributed system includes:
[0028] Acquire a business platform to be optimized for data access; the business platform adopts the distributed system;
[0029] According to the type of the business platform, query the mapping relationship between the business platform and the high-frequency access data type to determine the target type data corresponding to the platform to be optimized;
[0030] For the parameter type contained in the target type data, a parameter type selection box is generated on the human-computer interaction interface;
[0031] Obtaining the target parameter type required by the user for the target type data through the parameter type selection box;
[0032] Generate a parameter format corresponding to the target type data according to all target parameter types;
[0033] Correspondingly, the calling of the corresponding data acquisition method in the data update service to acquire the latest data of the target type data includes:
[0034] Determine the target data acquisition method and target parameter format corresponding to the target type data of the current scheduled update task;
[0035] According to the target data acquisition method and the target parameter format, the target parameters are extracted from the target file of the distributed system main database to obtain the latest data of the target type.
[0036] In a second aspect, the present application discloses a distributed system data access device, comprising:
[0037] A target type data determination module is used to determine target type data in a distributed system and save the target type data using a distributed cache database; the target type data is high-frequency access data;
[0038] A scheduled update task creation module, used to create a corresponding scheduled update task for each target type of data, and add a data acquisition method for each target type of data to a data update service;
[0039] A data storage module, used for calling the corresponding data acquisition method in the data update service to obtain the latest data of the target type data when executing the scheduled update task, and updating the distributed cache database according to the latest data;
[0040] The data reading module is used to query the distributed cache database according to the data query instruction, and if the query result is empty, query the main database of the distributed system according to the query instruction.
[0041] In a third aspect, the present application discloses an electronic device, comprising:
[0042] Memory, used to store computer programs;
[0043] The processor is used to execute the computer program to implement the aforementioned distributed system data access method.
[0044] In a fourth aspect, the present application discloses a computer-readable storage medium for storing a computer program; wherein the computer program implements the aforementioned distributed system data access method when executed by a processor.
[0045] In this application, the target type data in the distributed system is determined, and the target type data is saved in a distributed cache database; the target type data is high-frequency access data; a corresponding scheduled update task is created for each target type data, and the data acquisition method of each target type data is added to the data update service; when executing the scheduled update task, the corresponding data acquisition method in the data update service is called to obtain the latest data of the target type data, and the distributed cache database is updated according to the latest data; the distributed cache database is queried according to the data query instruction, and if the query result is empty, the main database of the distributed system is queried according to the query instruction. It can be seen that by storing high-frequency access data in a distributed cache database, and querying the distributed cache database first when querying data, the main database of the system is queried when there is no required data in the distributed cache database, the reading pressure of the main database is reduced, and the data access speed is significantly improved; by configuring the scheduled update task to regularly obtain the latest data to update the distributed cache database, the cache can be updated in time, effectively reducing the number of accesses to the system main database and reducing the load of the main database; and the cache update strategy can be flexibly adjusted through the scheduled update task to adapt to the changes in demand in different scenarios; effectively improve the efficiency of data access in the distributed system. BRIEF DESCRIPTION OF THE DRAWINGS
[0046] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.
[0047] Figure 1 A flow chart of a distributed system data access method provided by this application;
[0048] Figure 2 A schematic diagram of a specific distributed system data access method provided by this application;
[0049] Figure 3 A schematic diagram of a specific distributed system data caching method provided in this application;
[0050] Figure 4 A schematic diagram of the structure of a distributed system data access device provided by this application;
[0051] Figure 5 A structural diagram of an electronic device provided for this application. DETAILED DESCRIPTION
[0052] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0053] In the related art, systems that use distributed data planning usually have relatively large amounts of data and complex design structures. However, due to the complex association of data, when performing data queries, the data that needs to be retrieved may be distributed on different servers. When querying, it is necessary to retrieve the data from different nodes, and then integrate and return it in the required format. In this situation, data retrieval is the most frequent requirement in most application systems, which will seriously reduce the data access speed. In order to overcome the above technical problems, this application proposes a distributed system data access method that can improve the efficiency of distributed system data access.
[0054] The present application embodiment discloses a distributed system data access method, see Figure 1 As shown, the method may include the following steps:
[0055] Step S11: determining target type data in a distributed system, and using a distributed cache database to store the target type data; the target type data is high-frequency access data.
[0056] In this embodiment, not all data is stored in the cache, that is, in the distributed cache database. First, the data that needs to be stored in the cache in the distributed system is determined, that is, the above-mentioned target type data. Specifically, the above-mentioned target type data can be high-frequency access data, that is, the frequency of query access is high. Specifically, the above-mentioned target type data can be directly defined by the user. If the user has no definition, the target type data with a higher frequency can be selected according to the access frequency. By pre-retrieval of high-frequency access data and saving it in the distributed cache database, when the data is queried, it can be retrieved from the cache at the first time. The above-mentioned distributed cache database can specifically be Redis, and of course other databases that can implement distributed caching can also be used.
[0057] Step S12: creating a corresponding scheduled update task for each target type data, and adding a data acquisition method for each target type data to the data update service.
[0058] It is understandable that in order to ensure that most queries can be implemented through query cache, the data in the cache needs to be updated in a timely manner. Considering the different characteristics of different target type data, by creating corresponding scheduled update tasks for each target type data, that is, the cycles of different scheduled update tasks can be different, so as to update various target type data in a timely manner. To ensure the smooth acquisition of target type data, data acquisition methods for various target type data are added in advance in the data update service, so that data reading operations can be performed by calling corresponding method functions.
[0059] The step of creating a corresponding scheduled update task for each target type data may include: creating a corresponding scheduled update task for each target type data, that is, using a scheduled task (cron task) mechanism to create a scheduled update task, and configuring the corresponding scheduled duration of each target type data, so that when the scheduled update task reaches its own scheduled duration, the scheduled update task is executed. That is, by setting the scheduled duration for different target type data, the update time of each type of data is controlled, and the frequency of cache data update can be specifically formulated according to the actual needs of data update.
[0060] Wherein, adding the data acquisition method of each target type of data to the data update service may include: creating a data update service for the distributed system; acquiring the data acquisition methods corresponding to all target type data; and adding all the data acquisition methods to the data update service. That is, the data acquisition methods of various types of data need to be added to the data update service in advance. In addition, when a data query instruction is obtained, the corresponding data acquisition method can also be directly called from the data update service to query the cache database.
[0061] In some embodiments, the distributed system data access method may also include: acquiring new scheduled update tasks in real time; implementing a data acquisition method for acquiring new target type data, and adding the data acquisition method for the new target type data to the data update service. That is, it supports users to add new scheduled update tasks at any time according to their needs, and can also modify the original scheduled update tasks, such as modifying the timing duration in the existing scheduled update tasks; it also supports adding new target type data at any time, that is, it supports adding the data acquisition method for new target type data to the data update service at any time, so that the new target type data can be saved to the cache according to the data acquisition method; it improves the flexibility of the distributed system data access configuration.
[0062] Step S13: When executing the scheduled update task, the corresponding data acquisition method in the data update service is called to obtain the latest data of the target type data, and the distributed cache database is updated according to the latest data.
[0063] When a scheduled update task reaches the scheduled duration, it starts to execute. Specifically, it calls the data acquisition method of the target type data for the task in the data update service to obtain the latest data of the target type data. Finally, the distributed cache database is updated according to the latest data to complete a cache update.
[0064] Among them, the use of a distributed cache database to save the target type data may include: using a distributed cache database to save the target type data, and adding a timestamp to each piece of data stored in the distributed cache database; the timestamp is used to characterize the most recent modification time of the data; accordingly, the distributed system data access method may also include: based on the preset effective time for the target type data and the timestamp, filtering out invalid data in the distributed cache database. It is understandable that due to the limited space of the distributed cache database, it is necessary to delete the old data in the cache in a timely manner to provide space for new data. Specifically, it can be determined by configuring effective time for different types of data and combining the most recent modification time of the data to determine which data needs to be deleted.
[0065] In some embodiments, the determination of the target type data in the distributed system includes: obtaining the business platform to be optimized for data access; the business platform adopts the distributed system; according to the type of the business platform, querying the mapping relationship between the business platform and the high-frequency access data type to determine the target type data corresponding to the platform to be optimized; generating a parameter type selection box in the human-computer interaction interface for the parameter type contained in the target type data; obtaining the target parameter type required by the user for the target type data through the parameter type selection box; generating the parameter format corresponding to the target type data according to all the target parameter types. It can be understood that in addition to obtaining the target type data, in order to further improve the utilization of the cache, it is possible to cache by selecting a specific parameter type rather than all the parameters contained in the target type data; the specific target parameter type can be selected by the administrator in the form of a prompt box; of course, the administrator can also directly enter the parameter type by generating an input box; at the same time, it supports the user to select the type of parameter to be cached and saved, thereby improving the flexibility of data access in the distributed system based on the cache database. Correspondingly, the calling of the corresponding data acquisition method in the data update service to obtain the latest data of the target type data may include: determining the target data acquisition method and target parameter format corresponding to the target type data targeted by the current scheduled update task; extracting the target parameters from the target file of the distributed system master database according to the target data acquisition method and the target parameter format to obtain the latest data of the target type data. That is, if the corresponding target parameter format is configured for the target type data, when obtaining the latest data, the files related to the target type data may be determined according to the data acquisition method first, and then all target type parameters may be extracted from the files according to the target parameter format to obtain the latest data of the target type data.
[0066] In some embodiments, before generating the parameter format corresponding to the target type data according to all target parameter types, the method may further include: sorting all parameter types according to the access frequency corresponding to each parameter type contained in the target type data; and selecting a preset number of parameter types before sorting as the target parameter types. That is, the preset number of parameter types with the highest access frequency in a single target type data may be used as the target parameter type by default, thereby caching the parameters with high access frequency in the target type data with high access frequency in the cache database, and the remaining data is stored in the main storage library of the system, ensuring that the data stored in the cache has a high access frequency, so that the cache utilization rate can be improved while saving cache resources.
[0067] Wherein, updating the distributed cache database according to the latest data may include: creating a cache task for the latest data; the cache task is used to save the latest data to the distributed cache database; executing the cache task according to the cache task queue, and before executing the current cache task, re-searching the corresponding data from the main database of the distributed system according to the query instruction to obtain the actual data; comparing the latest data with the actual data, if the latest data is consistent with the actual data, then updating the distributed cache database based on the latest data, if the latest data is inconsistent with the actual data, then updating the distributed cache database based on the actual data. It is understandable that, since a large amount of data may need to be cached at the same time, in order to ensure the consistency of data during the caching process and avoid data loss, a data consistency check is performed before the actual caching is implemented, specifically by obtaining the data again and comparing it with the previously obtained data to ensure the consistency of the data version.
[0068] Step S14: query the distributed cache database according to the data query instruction. If the query result is empty, query the main database of the distributed system according to the query instruction.
[0069] When a data query instruction is received, the distributed cache database is queried first. If a hit is found, the query result is directly returned. If no hit is found, the main database of the distributed system is queried again, which is a traditional node-based query method. This application can be applied to various distributed systems that require high-performance data access.
[0070] In some embodiments, querying the distributed cache database according to the data query instruction, if the query result is empty, querying the main database of the distributed system according to the query instruction, may include: obtaining the data query instruction, querying the cache database according to the data query instruction; if the target data corresponding to the data query instruction is queried from the cache database, generating the query result of the data query instruction based on the target data; if the target data corresponding to the data query instruction is not queried from the cache database, querying the distributed storage system according to the data query instruction to obtain the target data corresponding to the data query instruction, and generating the query result. That is, if the result is queried from the cache, the query instruction is directly responded to according to the result, otherwise, the query continues from the main database of the system and is fed back to the user.
[0071] In some embodiments, after obtaining the target data, the method may further include: responding to the data query instruction based on the query result; if the target data is obtained from the distributed storage system, the target data is saved to the distributed cache database. That is, the data queried from the system main database can be saved to the cache to improve the next query speed, but the target data saving and the response to the data query instruction are performed asynchronously to avoid affecting the query speed.
[0072] Taking the business platform as an e-commerce platform as an example, the target type data includes product details data and user information data; the data access optimization steps for the e-commerce platform are as follows:
[0073] S21: Define commonly used data: On e-commerce platforms, the most frequently accessed data types are product details information and user information. Product details information includes product information and product inventory. Determine the parameter format of product details and user information; for example, the parameter format used for information is: user name, delivery address, user level.
[0074] S22: Use Redis cache: Define and save product details and user personal information in Redis cache, and add updated timestamp when saving. When a user accesses product information, first query from Redis, if the product information exists, quickly return the product information, if the data does not exist, query from the main database.
[0075] S23: Update cache data: Create a service for obtaining the latest information in the background. If it already exists, there is no need to create it; and add a method for obtaining product details and a method for obtaining user information. When the method is executed, the latest data retrieved is written to the Redis cache.
[0076] S24: Scheduled task scheduling: Configure a scheduled task that is automatically executed every hour to query the product information in the main database and update it to the Redis cache. Figure 3 As shown, the timing duration corresponding to configuring product information is one hour, and the timing duration of product inventory is half an hour. Only after the timing is reached can the corresponding method be called in the service to obtain the corresponding data.
[0077] S25: Access optimization: When a user accesses a product, if there is data in the cache, it is returned directly. If it misses, the data is obtained from the database and then updated in Redis. The cache is updated asynchronously before responding to avoid waiting for the user.
[0078] It can be seen that by storing frequently accessed data in the cache, combining it with scheduled task scheduling, and setting a reasonable expiration time and elimination strategy to ensure that the latest data is always kept in the cache, the method of improving data access efficiency on the distributed system and the overall performance of the system can be improved.
[0079] As can be seen from the above, in this embodiment, the target type data in the distributed system is determined, and the target type data is saved by using a distributed cache database; the target type data is high-frequency access data; a corresponding scheduled update task is created for each target type data, and the data acquisition method of each target type data is added to the data update service; when executing the scheduled update task, the corresponding data acquisition method in the data update service is called to obtain the latest data of the target type data, and the distributed cache database is updated according to the latest data; the distributed cache database is queried according to the data query instruction, and if the query result is empty, the main database of the distributed system is queried according to the query instruction. It can be seen that by storing high-frequency access data in a distributed cache database, and querying the distributed cache database first when querying data, the main database of the system is queried when there is no required data in the distributed cache database, the reading pressure of the main database is reduced, and the data access speed is significantly improved; by configuring the scheduled update task to regularly obtain the latest data to update the distributed cache database, the cache can be updated in time, effectively reducing the number of accesses to the system main database and reducing the load of the main database; and the cache update strategy can be flexibly adjusted through the scheduled update task to adapt to the changes in demand in different scenarios; effectively improve the efficiency of data access in the distributed system.
[0080] Correspondingly, the present application embodiment also discloses a distributed system data access device, see Figure 4 As shown, the device comprises:
[0081] The target type data determination module 11 is used to determine the target type data in the distributed system and save the target type data using the distributed cache database; the target type data is high-frequency access data;
[0082] A scheduled update task creation module 12, used to create a corresponding scheduled update task for each target type data, and add a data acquisition method for each target type data to a data update service;
[0083] The data storage module 13 is used to call the corresponding data acquisition method in the data update service to obtain the latest data of the target type data when executing the scheduled update task, and update the distributed cache database according to the latest data;
[0084] The data reading module 14 is used to query the distributed cache database according to the data query instruction, and if the query result is empty, query the main database of the distributed system according to the query instruction.
[0085] As can be seen from the above, in this embodiment, the target type data in the distributed system is determined, and the target type data is saved by using a distributed cache database; the target type data is high-frequency access data; a corresponding scheduled update task is created for each target type data, and the data acquisition method of each target type data is added to the data update service; when executing the scheduled update task, the corresponding data acquisition method in the data update service is called to obtain the latest data of the target type data, and the distributed cache database is updated according to the latest data; the distributed cache database is queried according to the data query instruction, and if the query result is empty, the main database of the distributed system is queried according to the query instruction. It can be seen that by storing high-frequency access data in a distributed cache database, and querying the distributed cache database first when querying data, the main database of the system is queried when there is no required data in the distributed cache database, the reading pressure of the main database is reduced, and the data access speed is significantly improved; by configuring the scheduled update task to regularly obtain the latest data to update the distributed cache database, the cache can be updated in time, effectively reducing the number of accesses to the system main database and reducing the load of the main database; and the cache update strategy can be flexibly adjusted through the scheduled update task to adapt to the changes in demand in different scenarios; effectively improve the efficiency of data access in the distributed system.
[0086] In some specific embodiments, the scheduled update task creation module 12 may specifically include:
[0087] A service creation unit, used for creating a data update service for the distributed system;
[0088] A data method acquisition unit, used to acquire the data acquisition methods corresponding to all target type data;
[0089] A method adding unit is used to add all the data acquisition methods to the data update service.
[0090] In some specific embodiments, the target type data determination module 11 may specifically include:
[0091] A timestamp adding unit, used to save the target type data by using a distributed cache database, and to add a timestamp to each piece of data saved in the distributed cache database; the timestamp is used to indicate the most recent modification time of the data;
[0092] Accordingly, the distributed system data access device may specifically include:
[0093] A screening unit is used to screen out invalid data in the distributed cache database according to a preset validity period for the target type data and the timestamp.
[0094] In some specific embodiments, the scheduled update task creation module 12 may specifically include:
[0095] The scheduled update task creation unit is used to create a corresponding scheduled update task for each target type data, and configure the timing duration corresponding to each target type data, so that when the scheduled update task reaches its own timing duration, the scheduled update task is executed.
[0096] In some specific embodiments, the data reading module 14 may specifically include:
[0097] A cache query unit, used to obtain a data query instruction and query the cache database according to the data query instruction;
[0098] A query result generating unit, configured to generate a query result of the data query instruction based on the target data if the target data corresponding to the data query instruction is queried from the cache database;
[0099] The system query unit is used to query the distributed storage system according to the data query instruction to obtain the target data corresponding to the data query instruction and generate a query result if the target data corresponding to the data query instruction is not found in the cache database.
[0100] In some specific embodiments, the data storage module 13 may specifically include:
[0101] A cache task creation unit, used to create a cache task for the latest data; the cache task is used to save the latest data to the distributed cache database;
[0102] An actual data acquisition unit is used to execute a cache task according to a cache task queue, and before executing a current cache task, search for corresponding data from a main database of the distributed system according to the query instruction again to obtain actual data;
[0103] A consistency comparison unit is used to compare the latest data with the actual data. If the latest data is consistent with the actual data, the distributed cache database is updated based on the latest data; if the latest data is inconsistent with the actual data, the distributed cache database is updated based on the actual data.
[0104] In some specific embodiments, the target type data determination module 11 may specifically include:
[0105] A business platform determination unit, used to obtain a business platform to be optimized for data access; the business platform adopts the distributed system;
[0106] A target type data determination unit, configured to query a mapping relationship between a business platform and a high-frequency access data type according to the type of the business platform, so as to determine the target type data corresponding to the platform to be optimized;
[0107] A selection box generating unit, used for generating a parameter type selection box on a human-computer interaction interface according to the parameter type included in the target type data;
[0108] A target parameter type determination unit, used for obtaining the target parameter type required by the user for the target type data through the parameter type selection box;
[0109] A parameter format generating unit, used for generating a parameter format corresponding to the target type data according to all target parameter types;
[0110] Accordingly, the data storage module 13 may specifically include:
[0111] A method and format determination unit, used to determine a target data acquisition method and a target parameter format corresponding to the target type data targeted by the current scheduled update task;
[0112] The target parameter extraction unit is used to extract the target parameters from the target file of the distributed system main database according to the target data acquisition method and the target parameter format to obtain the latest data of the target type.
[0113] Furthermore, the present application also discloses an electronic device, see Figure 5 As shown, the contents in the drawings should not be considered as any limitation on the scope of use of the present application.
[0114] Figure 5 The present invention provides a schematic diagram of the structure of an electronic device 20 provided in an embodiment of the present application. The electronic device 20 may include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 is used to store a computer program, which is loaded and executed by the processor 21 to implement the relevant steps in the distributed system data access method disclosed in any of the aforementioned embodiments.
[0115] In this embodiment, the power supply 23 is used to provide working voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and the external device, and the communication protocol it follows is any communication protocol that can be applied to the technical solution of the present application, and is not specifically limited here; the input and output interface 25 is used to obtain external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs and is not specifically limited here.
[0116] In addition, the memory 22, as a carrier for resource storage, can be a read-only memory, random access memory, a disk or an optical disk, etc. The resources stored thereon include an operating system 221, a computer program 222, and data 223 including scheduled update tasks, etc. The storage method can be temporary storage or permanent storage.
[0117] The operating system 221 is used to manage and control the hardware devices and computer programs 222 on the electronic device 20, so as to realize the operation and processing of the massive data 223 in the memory 22 by the processor 21, and it can be Windows Server, Netware, Unix, Linux, etc. In addition to including computer programs that can be used to complete the distributed system data access method executed by the electronic device 20 disclosed in any of the aforementioned embodiments, the computer program 222 can further include computer programs that can be used to complete other specific tasks.
[0118] Furthermore, an embodiment of the present application also discloses a computer storage medium, in which computer executable instructions are stored. When the computer executable instructions are loaded and executed by a processor, the steps of the distributed system data access method disclosed in any of the aforementioned embodiments are implemented.
[0119] Furthermore, an embodiment of the present application also discloses a computer program product, in which a computer program is stored. When the computer program is loaded and executed by a processor, the steps of the distributed system data access method disclosed in any of the aforementioned embodiments are implemented.
[0120] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.
[0121] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.
[0122] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.
[0123] The above is a detailed introduction to a distributed system data access method, device, equipment and storage medium provided by the present invention. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea; at the same time, for general technical personnel in this field, according to the idea of the present invention, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.
Claims
1. A distributed system data access method, characterized in that: include: Determine target type data in a distributed system, and save the target type data using a distributed cache database; The target type data is high-frequency access data; Create a corresponding scheduled update task for each target type data, and add a data acquisition method for each target type data to the data update service; When executing the scheduled update task, calling the corresponding data acquisition method in the data update service to obtain the latest data of the target type data, and updating the distributed cache database according to the latest data; The distributed cache database is queried according to the data query instruction. If the query result is empty, the main database of the distributed system is queried according to the query instruction.
2. The distributed system data access method according to claim 1, characterized in that: The method of adding the data acquisition method of each target type data to the data update service includes: Creating a data update service for the distributed system; Obtain the data acquisition methods corresponding to all target type data; Add all of the data acquisition methods to the data update service.
3. The distributed system data access method according to claim 1, characterized in that: The using a distributed cache database to store the target type data includes: Utilize a distributed cache database to store the target type data, and add a timestamp to each piece of data stored in the distributed cache database; the timestamp is used to indicate the most recent modification time of the data; Accordingly, the distributed system data access method further includes: According to the preset validity period for the target type data and the timestamp, invalid data in the distributed cache database is filtered out.
4. The distributed system data access method according to claim 1, characterized in that: The step of creating a corresponding scheduled update task for each target type of data includes: A corresponding scheduled update task is created for each target type data, and a scheduled duration corresponding to each target type data is configured so that the scheduled update task is executed when the scheduled update task reaches its own scheduled duration.
5. The distributed system data access method according to claim 1, characterized in that: The step of querying the distributed cache database according to the data query instruction, and if the query result is empty, querying the main database of the distributed system according to the query instruction, comprises: Obtaining a data query instruction, and querying the cache database according to the data query instruction; If the target data corresponding to the data query instruction is queried from the cache database, a query result of the data query instruction is generated based on the target data; If the target data corresponding to the data query instruction is not found in the cache database, the distributed storage system is queried according to the data query instruction to obtain the target data corresponding to the data query instruction and generate a query result.
6. The distributed system data access method according to claim 1, characterized in that: The updating of the distributed cache database according to the latest data includes: Creating a cache task for the latest data; the cache task is used to save the latest data to the distributed cache database; Execute the cache task according to the cache task queue, and before executing the current cache task, search for corresponding data from the main database of the distributed system according to the query instruction again to obtain actual data; The latest data is compared with the actual data. If the latest data is consistent with the actual data, the distributed cache database is updated based on the latest data. If the latest data is inconsistent with the actual data, the distributed cache database is updated based on the actual data.
7. The distributed system data access method according to any one of claims 1 to 6, characterized in that: The determining of target type data in the distributed system includes: Acquire a business platform to be optimized for data access; the business platform adopts the distributed system; According to the type of the business platform, query the mapping relationship between the business platform and the high-frequency access data type to determine the target type data corresponding to the platform to be optimized; For the parameter type contained in the target type data, a parameter type selection box is generated on the human-computer interaction interface; Obtaining the target parameter type required by the user for the target type data through the parameter type selection box; Generate a parameter format corresponding to the target type data according to all target parameter types; Correspondingly, the calling of the corresponding data acquisition method in the data update service to acquire the latest data of the target type data includes: Determine the target data acquisition method and target parameter format corresponding to the target type data of the current scheduled update task; According to the target data acquisition method and the target parameter format, the target parameters are extracted from the target file of the distributed system main database to obtain the latest data of the target type.
8. A distributed system data access device, characterized in that: include: A target type data determination module is used to determine the target type data in the distributed system and save the target type data using a distributed cache database; The target type data is high-frequency access data; A scheduled update task creation module, used to create a corresponding scheduled update task for each target type of data, and add a data acquisition method for each target type of data to a data update service; A data storage module, used for calling the corresponding data acquisition method in the data update service to obtain the latest data of the target type data when executing the scheduled update task, and updating the distributed cache database according to the latest data; The data reading module is used to query the distributed cache database according to the data query instruction, and if the query result is empty, query the main database of the distributed system according to the query instruction.
9. An electronic device, characterized in that: include: Memory, used to store computer programs; A processor, configured to execute the computer program to implement the distributed system data access method according to any one of claims 1 to 7.
10. A computer-readable storage medium, characterized in that: Used to store computer programs; wherein when the computer programs are executed by a processor, the distributed system data access method as described in any one of claims 1 to 7 is implemented.