A data loading method and device

Abstract

The present invention discloses a data loading method and device, and relates to the field of computer technology. A specific implementation of the method includes: after each query request for data is received, check whether the data requested for query has been loaded into the cache; if it has been loaded into the cache, return the cached data; if it has not been loaded into the cache, check whether there is an equivalent query request for the query request; if so, discard the query request; otherwise, add the business ID included in the query request to the loading cache request queue; query the data corresponding to each business ID in the loading cache request queue according to a preset strategy, and load the queried data into the cache. This implementation can avoid the inquirer from waiting for a long time, save memory, implement request flow limiting, discard excessive requests, avoid rapid consumption of server-side computing resources, and control the rate of querying and obtaining data from the data source, avoid instantaneous high-concurrency access to the data source, and prevent the computing resources of the data source from being rapidly consumed.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a data loading method and device. Background Art

[0002] The memory cache is a layer of buffer between the data queryer and the data source, which can not only reduce the pressure on the data source, but also quickly respond to the data queryer to cope with high concurrent access. There are two existing data loading solutions: one is that the server synchronously loads the data into the memory cache when the data queryer queries it for the first time, and then provides the data loaded into the cache to the data queryer. This solution requires the data queryer to wait for the cache to be loaded before reading the data from the cache; the other is that the server uses background tasks to load all data into the memory cache in advance, so that the data queryer does not need to wait for the cache to be loaded, but this solution is only suitable for small data volumes.

[0003] In the process of implementing the present invention, the inventors found that there are at least the following problems in the prior art:

[0004] The existing solution 1 causes the data queryer to wait for a long time during the process of loading data into the cache on the server side;

[0005] The existing solution 2 loads a large amount of data that may be needed into the memory cache in advance, wasting a lot of memory space;

[0006] In addition, the above-mentioned existing technologies lack a request flow limiting mechanism, which results in the rapid consumption of server computing resources when there are a large number of query requests, making it difficult for the server to continue to provide services; and, there is no limit on the rate of querying data from the data source, which results in the rapid consumption of the computing resources of the data source when there are a large number of requests to query the data source instantaneously, making it difficult for the data source to continue to provide services. Summary of the invention

[0007] In view of this, an embodiment of the present invention provides a data loading method and device, which can avoid long-term waiting for data queryers, and only load the data required by the data queryers, saving memory space. It can also limit the flow of query requests and discard excessive requests to avoid rapid consumption of server computing resources, ensuring that the server continues to provide services, and can control the rate of querying and obtaining data from the data source to avoid instantaneous high-concurrency access to the data source, preventing the computing resources of the data source from being rapidly consumed, and ensuring that the data source continues to provide services.

[0008] To achieve the above objective, according to one aspect of an embodiment of the present invention, a data loading method is provided.

[0009] A data loading method comprises: after each query request for data is received, checking whether the data requested for query has been loaded into a cache, the query request including the business ID of the data requested for query; if the data requested for query has been loaded into the cache, returning the cached data requested for query; if the data requested for query has not been loaded into the cache, checking whether an equivalent query request of the query request already exists, the equivalent query request including the same business ID as the query request; if so, discarding the query request; otherwise, adding the business ID included in the query request to a loading cache request queue; querying the data corresponding to each business ID in the loading cache request queue according to a preset strategy, and loading the queried data into the cache.

[0010] Optionally, the step of checking whether an equivalent query request of the query request already exists includes: checking whether there is a key in the mapping that is equal to the business ID included in the query request, if so, the equivalent query request exists, otherwise the equivalent query request does not exist, wherein the mapping includes a key-value pair, the key of the key-value pair is the business ID, and the value is a preset flag value.

[0011] Optionally, it also includes: when there is no key in the mapping that is equal to the business ID included in the query request, a key-value pair is generated using the business ID included in the query request as the key and the preset flag value as the value, and the generated key-value pair is added to the mapping.

[0012] Optionally, under the condition of checking and determining that the length of the mapping is greater than a first threshold, the operation of adding the business ID included in the query request to the loading cache request queue, as well as the operation of generating a key-value pair using the business ID included in the query request as a key and the preset flag value as a value, and adding the generated key-value pair to the mapping is performed.

[0013] Optionally, after the queried data is loaded into the cache, the key-value pairs corresponding to the business IDs of the queried data in the mapping are cleared.

[0014] Optionally, the step of checking whether an equivalent query request of the query request already exists includes: traversing the loading cache request queue to check whether the loading cache request queue stores the business ID included in the query request; if so, the equivalent query request exists; otherwise, the equivalent query request does not exist.

[0015] Optionally, under the condition that it is checked and determined that the length of the loading cache request queue is greater than a second threshold, the operation of adding the service ID included in the query request to the loading cache request queue is performed.

[0016] Optionally, the step of querying the data corresponding to each business ID in the loading cache request queue according to a preset strategy includes: regularly obtaining business IDs from the loading cache request queue in a specific order according to a configured time interval, wherein the number of business IDs that can be obtained each time is determined by configuration; and querying the data corresponding to each business ID in the obtained business IDs from the data source.

[0017] According to another aspect of an embodiment of the present invention, a data loading device is provided.

[0018] A data loading device comprises: a data cache checking module, which is used to check whether the requested data has been loaded into the cache after each query request for data is received, and the query request includes the business ID of the requested data; a cache data returning module, which is used to return the cached data of the requested data if the requested data has been loaded into the cache; a query request processing module, which is used to check whether an equivalent query request of the query request already exists if the requested data has not been loaded into the cache, and the equivalent query request includes the same business ID as the query request; if so, the query request is discarded; otherwise, the business ID included in the query request is added to a loading cache request queue; a data query and loading module, which is used to query the data corresponding to each business ID in the loading cache request queue according to a preset strategy, and load the queried data into the cache.

[0019] Optionally, the query request processing module includes a first equivalent query request checking submodule, which is used to: check whether there is a key in the mapping that is equal to the business ID included in the query request. If so, the equivalent query request exists, otherwise the equivalent query request does not exist, wherein the mapping includes a key-value pair, the key of the key-value pair is the business ID, and the value is a preset flag value.

[0020] Optionally, the query request processing module is also used to: when there is no key in the mapping that is equal to the business ID included in the query request, generate a key-value pair with the business ID included in the query request as the key and the preset flag value as the value, and add the generated key-value pair to the mapping.

[0021] Optionally, the query request processing module performs the operation of adding the business ID included in the query request to the loading cache request queue, and the operation of generating a key-value pair using the business ID included in the query request as the key and the preset flag value as the value, and adding the generated key-value pair to the mapping, under the condition that the query request checks and determines that the length of the mapping is greater than a first threshold.

[0022] Optionally, a cleaning module is further included, which is used to: after the queried data is loaded into the cache, clean up the key-value pairs corresponding to the business ID of the queried data in the mapping.

[0023] Optionally, the query request processing module includes a second equivalent query request checking submodule, which is used to: traverse the loading cache request queue to check whether the loading cache request queue stores the business ID included in the query request; if so, the equivalent query request exists; otherwise, the equivalent query request does not exist.

[0024] Optionally, the query request processing module performs the operation of adding the service ID included in the query request to the loading cache request queue under the condition that the length of the loading cache request queue is checked and determined to be greater than a second threshold.

[0025] Optionally, the data query and loading module includes a data query sub-module, which is used to: regularly obtain business IDs from the loading cache request queue in a specific order at a configured time interval, wherein the number of business IDs that can be obtained each time is determined by configuration; and query the data corresponding to each business ID in the obtained business IDs from the data source.

[0026] According to yet another aspect of the embodiments of the present invention, an electronic device is provided.

[0027] An electronic device comprises: one or more processors; a memory for storing one or more programs, and when the one or more programs are executed by the one or more processors, the one or more processors implement the data loading method provided by the present invention.

[0028] According to yet another aspect of an embodiment of the present invention, a computer readable medium is provided.

[0029] A computer readable medium stores a computer program, which implements the data loading method provided by the present invention when executed by a processor.

[0030] An embodiment of the above invention has the following advantages or beneficial effects: after each query request for data is received, it is checked whether the data requested for query has been loaded into the cache. If the data requested for query has been loaded into the cache, the cached data requested for query is returned; if the data requested for query has not been loaded into the cache, it is checked whether there is an equivalent query request for the query request. If so, the query request is discarded, otherwise the business ID included in the query request is added to the loading cache request queue, and the data corresponding to each business ID in the loading cache request queue is queried according to the preset strategy, and the queried data is loaded into the cache. It can avoid long-term waiting for data queryers, and only the data required by the data queryer needs to be loaded, saving memory space. By controlling the length of the mapping or the length of the loading cache request queue by the threshold, the query request is limited, and excessive requests are discarded to avoid the rapid consumption of the server computing resources, and ensure that the server continues to provide services. According to the configured time interval, the business ID is regularly obtained from the loading cache request queue in a specific order, and the number of business IDs that can be obtained each time is determined by configuration, so that the rate of querying and obtaining data from the data source can be controlled to avoid instantaneous high concurrent access to the data source, prevent the computing resources of the data source from being rapidly consumed, and ensure that the data source continues to provide services.

[0031] The further effects of the above-mentioned non-conventional optional manner will be described below in conjunction with the specific implementation manner. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] The accompanying drawings are used to better understand the present invention and do not constitute an improper limitation of the present invention.

[0033] Figure 1 is a schematic diagram of main steps of a data loading method according to an embodiment of the present invention;

[0034] Figure 2 is a schematic diagram of a data loading process according to an embodiment of the present invention;

[0035] Figure 3 is a schematic diagram illustrating the principle of data loading according to another embodiment of the present invention;

[0036] Figure 4 is a schematic diagram of a data loading process according to yet another embodiment of the present invention;

[0037] Figure 5 is a schematic diagram of main modules of a data loading device according to an embodiment of the present invention;

[0038] Figure 6 is an exemplary system architecture diagram to which embodiments of the present invention may be applied;

[0039] Figure 7It is a schematic diagram of the structure of a computer system of a terminal device or a server suitable for implementing an embodiment of the present invention. DETAILED DESCRIPTION

[0040] The following is a description of exemplary embodiments of the present invention in conjunction with the accompanying drawings, including various details of the embodiments of the present invention to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present invention. Similarly, for clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0041] Those skilled in the art will appreciate that the embodiments of the present invention may be implemented as a system, device, apparatus, method or computer program product. Therefore, the present disclosure may be implemented in the following forms, namely: complete hardware, complete software (including firmware, resident software, microcode, etc.), or a combination of hardware and software.

[0042] Figure 1 4 is a schematic diagram of main steps of a data loading method according to a first embodiment of the present invention.

[0043] like Figure 1 As shown, the data loading method of this embodiment mainly includes the following steps S101 to S103.

[0044] Step S101: after each query request for data is received, check whether the requested data has been loaded into the cache.

[0045] The query request includes the service ID (identification) of the data requested for query.

[0046] Cache refers to the memory cache on the server side, which is a space allocated in the memory and has the characteristics of fast response to queries. Due to the high cost, not too much data will be stored in the memory cache.

[0047] The embodiments of the present invention do not limit the specific type of cache. The cache may be, for example, Redis (an open source network-supported, memory-based, and persistent log-type, Key-Value database), Memcached (a high-performance distributed memory object caching system), etc.

[0048] The cache has an expiration time (the specific expiration time can be specified in advance), and data has been loaded into the cache means that the data exists in the cache and has not expired.

[0049] Step S102: If the requested query data has been loaded into the cache, the cached requested query data is returned; if the requested query data has not been loaded into the cache, check whether an equivalent query request already exists for the query request, and if so, discard the query request; otherwise, add the business ID included in the query request to the loading cache request queue.

[0050] The equivalent query request of the query request is a request that includes the same service ID as the query request. If two requests include the same service ID, it means that the two requests are essentially the same request. If one of the requests already exists, there is no need to repeatedly process the later received request and it needs to be discarded.

[0051] As an implementation method, the step of checking whether an equivalent query request of the query request already exists may specifically include: checking whether there is a key in the mapping that is equal to the business ID included in the query request; if so, there is an equivalent query request of the query request; otherwise, there is no equivalent query request, wherein the mapping includes a key-value pair, the key of the key-value pair is the business ID, and the value is a preset flag value.

[0052] If there is no key in the mapping that is equal to the business ID included in the query request, a key-value pair is generated using the business ID included in the query request as the key and a preset flag value as the value, and the generated key-value pair is added to the mapping.

[0053] Before adding the business ID included in the query request to the loading cache request queue and adding the generated key-value pair to the mapping, it can be checked whether the length of the mapping is greater than a first threshold. If it is greater than the first threshold, the operation of adding the business ID included in the query request to the loading cache request queue is performed, and a key-value pair is generated using the business ID included in the query request as the key and a preset flag value as the value, and the generated key-value pair is added to the mapping; if it is less than or equal to the first threshold, the query request is discarded.

[0054] As another implementation, the step of checking whether an equivalent query request of the query request already exists may specifically include: traversing the loading cache request queue to check whether the loading cache request queue stores the business ID included in the query request; if so, an equivalent query request of the query request exists; otherwise, the equivalent query request does not exist.

[0055] Before adding the business ID included in the query request to the loading cache request queue, you can first check whether the length of the loading cache request queue is greater than the second threshold. If it is determined that the length of the loading cache request queue is greater than the second threshold, perform the operation of adding the business ID included in the query request to the loading cache request queue; if the length of the loading cache request queue is less than or equal to the second threshold, discard the query request.

[0056] The service ID included in the query request is added to the load cache request queue. Specifically, the service ID included in the query request is added to the tail of the load cache request queue (or the tail of the queue). A queue is a special linear list that only allows deletion operations at the head of the list (i.e., the head of the queue) and insertion operations at the tail of the list (i.e., the tail of the queue).

[0057] The above two implementation modes will be described in detail below. Figures 2 to 4 The corresponding embodiments.

[0058] After the query request is abandoned, a prompt message may be output, such as prompting the data queryer that data is being loaded, please try again later, etc.

[0059] Step S103: query and load the data corresponding to each service ID in the cache request queue according to a preset strategy, and load the queried data into the cache.

[0060] Step S103 specifically includes: regularly obtaining business IDs from the load cache request queue in a specific order according to a configured time interval, wherein the number of business IDs that can be obtained each time is determined by configuration; and querying the data corresponding to each business ID in the obtained business IDs from the data source.

[0061] The specific order is specifically the order in which the business ID is enqueued in the load cache request queue, that is, the business ID is obtained from the head of the queue each time.

[0062] It should be noted that the step numbers of the above step S102 and step S103 do not indicate the order of execution. For each query request for data received, it is checked whether the requested data has been loaded into the cache, and then step S102 is executed, and step S103 is executed periodically, that is, according to the above-configured time interval, the business ID is regularly obtained and the corresponding data is queried from the data source.

[0063] In the above embodiment of using mapping to store key-value pairs with business ID as key, after loading the queried data into the cache, it is also necessary to clean up the key-value pairs corresponding to the business ID of the queried data in the mapping.

[0064] The embodiment of the present invention adopts an asynchronous loading cache solution, striving to use memory resources on the most needed data. It does not require the server to return data to the data query immediately, but also avoids the data queryer from waiting for data for a long time. It can find a balance between providing data to the data queryer in a timely manner and protecting the computing resources of the data source.

[0065] Figure 2 1 is a schematic diagram of a data loading process according to an embodiment of the present invention.

[0066] like Figure 2 As shown, the data loading process of this embodiment includes the following steps S201 to S211.

[0067] Step S201: receiving a query request.

[0068] The query request is used to query data, which includes the business ID of the data requested for query. For example, in the scenario of querying product evaluation information, the query request includes the evaluation ID requested for query, and the evaluation information of a product can be queried based on the evaluation ID.

[0069] Step S202: Check whether the cache hits, if so, execute step S203, otherwise execute step S204.

[0070] Check whether the cache hits, that is, check whether the data requested for the query has been loaded into the cache. If the data has been loaded into the cache, it means that the query request hits the cache, otherwise, it means that the cache does not hit.

[0071] It should be noted that data has been loaded into the cache means that the data exists in the cache and has not expired.

[0072] Step S203: Return the data in the cache corresponding to the received query request.

[0073] The data loaded into the cache is organized into cache objects, each of which includes a business ID and data corresponding to the business ID. When the received query request hits the cache, the data corresponding to the business ID included in the query request is searched in each cache object in the cache to obtain the data corresponding to the received query request.

[0074] Step S204: Determine whether an equivalent query request of the received query request already exists in the mapping (denoted as M), if so, execute step S205, otherwise execute step S206.

[0075] A map consists of key-value pairs, for example in the following form:

[0076] (key1,value1)

[0077] (key2,value2) ......

[0078] In each key-value pair, key is the business ID, and value is a preset flag value, which is set to 1 or other values. The key-value pairs in the mapping indicate that the cache is being loaded for the business IDs corresponding to these keys. These business IDs are also stored in the cache loading request queue. The process of loading the cache includes: querying the data corresponding to each business ID in the cache loading request queue according to the preset strategy, and loading the queried data into the cache.

[0079] To check whether an equivalent query request of the received query request already exists in the mapping M, it is only necessary to check whether there is a key in the mapping M that is equal to the business ID included in the query request. If such a key exists, it means that the equivalent query request exists, otherwise the equivalent query request does not exist.

[0080] The strategy of checking equivalent query requests of a query request in this embodiment does not need to traverse the entire loading cache request queue, has a high efficiency in checking duplicate business IDs, and speeds up the checking of duplicate business IDs.

[0081] Step S205: Abandon the query request and output prompt information to the data inquirer.

[0082] Prompt message, for example: "Loading data, please try again later."

[0083] In this case, the data queryer can initiate a query request again to try to obtain the data.

[0084] Step S206: Determine whether the length of the mapping M is greater than a first threshold, if so, execute the above step S205, otherwise, execute step S207.

[0085] Step S207: using the service ID of the received query request as key and the preset flag value as value, adding the key-value pair of the received query request to the mapping M, and adding the service ID of the received query request to the tail of the loading cache request queue.

[0086] Step S208: The background task periodically obtains n business IDs from the head of the loading cache request queue at a fixed time period t, and sorts out the query statements.

[0087] Among them, the background task is the task used by the server to request query data and can be executed periodically.

[0088] Arranging the query statement refers to generating a query statement for querying the data corresponding to the n business IDs, so as to query the data from the data source in the next step.

[0089] The specific values ​​of t and n are determined by configuration.

[0090] Step S209: query the data corresponding to the n business IDs from the data source.

[0091] That is, the query statements sorted out above are executed to query the data corresponding to the n business IDs from the data source.

[0092] Step S210: Arrange the queried data into cache objects and fill them into the cache.

[0093] Step S211: clear the corresponding key-value pairs in the mapping M according to the service ID that has been added to the cache.

[0094] Each business ID appended to the Map will be added to the cache loading request queue. When a business ID is taken out of the cache loading request queue and the data corresponding to the business ID is loaded into the cache, the key-value pairs corresponding to the business ID in the map are cleaned up to ensure that the business IDs in the cache loading request queue and the map correspond to the same business IDs, and are all business IDs waiting to be loaded into the cache.

[0095] The embodiment of the present invention adopts an asynchronous loading cache strategy, only loading the data required by the data queryer, striving to save memory space and avoid rapid consumption of server-side resources. For query requests that can hit the cache, the corresponding data can be immediately returned to the data queryer, avoiding the data queryer from waiting for a long time.

[0096] In addition, this embodiment also limits the flow of query requests (flow limiting refers to limiting the input flow or output flow of the system to achieve the purpose of protecting the system), discards excessive requests, and further avoids the rapid consumption of server-side computing resources. Specifically, since the number of key-value pairs (key-value pairs) in mapping M changes dynamically. Whenever it is necessary to load the cache for a business ID, a key-value pair will be added to mapping M, and whenever the cache loading is completed, this key-value pair will be cleared out of mapping M. If the rate of adding key-value pairs to mapping M is greater than the rate of clearing key-value pairs from mapping M, the length of mapping M will continue to grow. If the length of mapping M grows infinitely, all the memory on the server will be consumed. At the same time, the excessive length of mapping M also means that the number of query requests is too large, which causes the computing resources on the server to be rapidly consumed. Therefore, this embodiment sets a first threshold value. Before each key-value pair is appended to the mapping M, it is determined whether the length of the mapping M is greater than the first threshold value. When the length of the mapping M is greater than the first threshold value, the key-value pair is no longer appended to the mapping M, but the received query request is discarded, thereby achieving the flow limitation of the query request and reducing the consumption of computing resources on the server.

[0097] Finally, the background task for querying data in this instance periodically obtains n business IDs from the head of the loading cache request queue at a fixed time period t, organizes query statements, and queries the corresponding data. The interval t for the execution of the background task and the number n of business IDs corresponding to the data queried each time are determined by configuration, which controls the rate of obtaining data from the data source to avoid sudden high concurrent access to the data source.

[0098] Figure 3 It is a schematic diagram illustrating the principle of data loading according to another embodiment of the present invention.

[0099] like Figure 3 As shown, the principle of data loading in this embodiment is described as follows:

[0100] Each time the server receives a request for querying data (i.e., a query request for short), it checks whether the cache can be hit. If so, it returns the data in the cache. Figure 3 As shown in step 1.

[0101] If no match is found, check whether there is an equivalent query request to the query request. Figure 3Step 2 shows a mapping M, in which for each key-value pair, the key is the business ID, and the value is a preset flag value, indicating that the system (referring to the server) is loading the cache for this business ID. To check whether the system is loading the cache for a certain business ID, just check whether there is a key in the mapping M that is equal to this business ID. If there is no key equal to the business ID included in the query request, the business ID of this query request is appended to the Map. Specifically, the business ID of this query request is used as the key and a flag value, such as 1, is used as the value to generate a key-value pair and add it to the mapping M to ensure that the business ID appears in the Map, such as Figure 3 As shown in step 2.

[0102] If there is no equivalent query request, after appending the business ID of this query request to the Map, it also needs to be appended to the end of the loading cache request queue (Queue), such as Figure 3 As shown in step 3.

[0103] If the above equivalent query request exists, it means that the system is already loading the cache for this business ID, then the query request will be discarded, and the data queryer will be prompted that the system is loading data and please try again later.

[0104] In the mapping M, the number of key-value pairs changes dynamically. Whenever a cache needs to be loaded for a business ID, a key-value pair will be added to the mapping M. Whenever the cache loading is completed, this key-value pair will be cleared from the mapping M. In order to prevent the length of the mapping M from growing too fast, this solution is to check the length of the mapping M before appending a key-value pair to the mapping M each time. If this length is greater than the threshold V1, no key-value will be appended to the mapping M, and the data queryer will be directly prompted that the data is being loaded and please try again later. When the data queryer initiates a request again, the server may accept its request.

[0105] Cache loading is completed by a background task, which is executed regularly. Every time period t (configurable), it obtains n (configurable) business IDs from the head of the cache loading request queue, organizes query statements, and queries data in the data source, such as Figure 3 As shown in step 4.

[0106] Then, the cache objects are sorted according to the queried data, filled into the memory cache, and the expiration time is specified, such as Figure 3 As shown in step 5.

[0107] After successfully filling the memory cache for a business ID, the corresponding key-value pair in the mapping M needs to be cleared, indicating that the cache loading is complete, such as Figure 3 As shown in step 6.

[0108] In combination with monitoring, the embodiments of the present invention timely adjust t and n, which can not only ensure that the resources of the server are not consumed quickly, but also ensure that the cache is loaded as soon as possible and provide data to the data queryer as soon as possible.

[0109] Figure 4 is a schematic diagram of a data loading process according to yet another embodiment of the present invention.

[0110] The data loading process of this embodiment includes the following steps S401 to S410.

[0111] Step S401: receiving a query request.

[0112] Step S402: Check whether the cache hits, if so, execute step S403, otherwise execute step S404.

[0113] Step S403: Return the data in the cache corresponding to the received query request.

[0114] Step S404: Determine whether an equivalent query request to the received query request already exists in the loading cache request queue; if so, execute step S405; otherwise, execute step S406.

[0115] Step S405: Abandon the query request and output prompt information to the data inquirer.

[0116] Step S406: Determine whether the length of the load cache request queue is greater than a second threshold, if so, execute step S405, otherwise execute step S407.

[0117] Step S407: Add the service ID of the received query request to the tail of the loading cache request queue.

[0118] Step S408: The background task periodically obtains n business IDs from the head of the loading cache request queue at a fixed time period t, and sorts out the query statements.

[0119] Step S409: query the data corresponding to the n business IDs from the data source.

[0120] Step S410: Arrange the queried data into cache objects and fill them into the cache.

[0121] For the above steps S401 to S403 and S405, please refer to the introduction of steps S201 to S203 and S205. For steps S408 to S410, please refer to the introduction of steps S208 to S210.

[0122] The background task of this embodiment loads the cache asynchronously instead of synchronously to avoid long waiting times for data queryers. The rate at which data is obtained from the data source can be controlled to avoid sudden high-concurrency access to the data source. It can also limit the flow of query requests and discard excessive requests to avoid severe consumption of server resources.

[0123] Figure 5 4 is a schematic diagram of main modules of a data loading device according to an embodiment of the present invention.

[0124] like Figure 5 As shown, the data loading device 500 of the embodiment of the present invention mainly includes: a data cache checking module 501, a cache data returning module 502, a query request processing module 503, and a data query and loading module 504.

[0125] The data cache checking module 501 is used to check whether the requested data has been loaded into the cache after receiving each query request for data, wherein the query request includes the service ID of the requested data.

[0126] The cache data returning module 502 is used to return the cached data of the requested query if the data of the requested query has been loaded into the cache.

[0127] The query request processing module 503 is used to check whether there is an equivalent query request for the query request if the data requested by the query is not loaded into the cache, and the equivalent query request includes the same business ID as the query request; if so, the query request is discarded; otherwise, the business ID included in the query request is added to the loading cache request queue.

[0128] In one embodiment:

[0129] The query request processing module 503 may include a first equivalent query request checking submodule, which is used to: check whether there is a key in the mapping that is equal to the business ID included in the query request. If so, there is an equivalent query request for the query request; otherwise, there is no equivalent query request, wherein the mapping includes a key-value pair, the key of the key-value pair is the business ID, and the value is a preset flag value.

[0130] The query request processing module can be specifically used to: if there is no key in the mapping that is equal to the business ID included in the query request, generate a key-value pair using the business ID included in the query request as the key and the preset flag value as the value, and add the generated key-value pair to the mapping.

[0131] Before adding the business ID included in the query request to the loading cache request queue and adding the key-value pair generated above to the mapping, the query request processing module 503 may also check whether the length of the mapping is greater than a first threshold, and, under the condition that it is determined that the length of the mapping is greater than the first threshold, perform the operation of adding the business ID included in the query request to the loading cache request queue, and generate a key-value pair using the business ID included in the query request as the key and a preset flag value as the value, and add the generated key-value pair to the mapping.

[0132] If the length of the mapping is less than or equal to the first threshold, the query request is discarded.

[0133] In another embodiment:

[0134] The query request processing module 503 may include a second equivalent query request checking submodule, which is used to: traverse the loading cache request queue to check whether the loading cache request queue stores the business ID included in the query request. If so, there is an equivalent query request for the query request, otherwise there is no equivalent query request.

[0135] Before adding the business ID included in the query request to the loading cache request queue, the query request processing module 503 may check whether the length of the loading cache request queue is greater than a second threshold, and, under the condition that it is determined that the length of the loading cache request queue is greater than the second threshold, execute the operation of adding the business ID included in the query request to the loading cache request queue.

[0136] If the length of the load cache request queue is less than or equal to the second threshold, the query request is discarded.

[0137] The data query and loading module 504 is used to query and load the data corresponding to each service ID in the cache request queue according to a preset strategy, and load the queried data into the cache.

[0138] The data query and loading module 504 specifically includes a data query submodule, which is used to: regularly obtain business IDs from the loading cache request queue in a specific order according to the configured time interval, wherein the number of business IDs that can be obtained each time is determined by configuration; and query the data corresponding to each business ID in the obtained business IDs from the data source.

[0139] In the above-mentioned embodiment of using mapping to store key-value pairs with business ID as key, the data loading device 500 may also include a cleaning module for: after loading the queried data into the cache, cleaning the key-value pairs corresponding to the business ID of the queried data in the mapping.

[0140] In addition, the specific implementation content of the data loading device in the embodiment of the present invention has been described in detail in the data loading method described above, so the repeated content will not be described again here.

[0141] Figure 6 An exemplary system architecture 600 is shown to which the data loading method or data loading device according to the embodiment of the present invention can be applied.

[0142] like Figure 6 As shown, system architecture 600 may include terminal devices 601, 602, 603, network 604 and server 605. Network 604 is used to provide a medium for communication links between terminal devices 601, 602, 603 and server 605. Network 604 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0143] Users can use terminal devices 601, 602, and 603 to interact with server 605 through network 604 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 601, 602, and 603, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc. (only examples).

[0144] The terminal devices 601 , 602 , and 603 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, and desktop computers, etc.

[0145] The server 605 may be a server that provides various services, such as a backend management server (only an example) that provides support for shopping websites browsed by users using the terminal devices 601, 602, and 603. The backend management server may analyze and process the received data such as product information query requests, and feed back the processing results (such as product information - only an example) to the terminal device.

[0146] It should be noted that the data loading method provided in the embodiment of the present invention is generally executed by the server 605 , and accordingly, the data loading device is generally disposed in the server 605 .

[0147] It should be understood that Figure 6 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0148] Reference below Figure 7 , which shows a schematic diagram of the structure of a computer system 700 of a terminal device or server suitable for implementing an embodiment of the present application. Figure 7The terminal device or server shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0149] like Figure 7 As shown, the computer system 700 includes a central processing unit (CPU) 701, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 702 or a program loaded from a storage part 708 into a random access memory (RAM) 703. In the RAM 703, various programs and data required for the operation of the system 700 are also stored. The CPU 701, the ROM 702, and the RAM 703 are connected to each other via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0150] The following components are connected to the I / O interface 705: an input section 706 including a keyboard, a mouse, etc.; an output section 707 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 708 including a hard disk, etc.; and a communication section 709 including a network interface card such as a LAN card, a modem, etc. The communication section 709 performs communication processing via a network such as the Internet. A drive 710 is also connected to the I / O interface 705 as needed. A removable medium 711, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 710 as needed, so that a computer program read therefrom is installed into the storage section 708 as needed.

[0151] In particular, according to the embodiments disclosed in the present invention, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present invention include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 709, and / or installed from the removable medium 711. When the computer program is executed by the central processing unit (CPU) 701, the above-mentioned functions defined in the system of the present application are executed.

[0152] It should be noted that the computer-readable medium shown in the present invention can be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media can include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present application, a computer-readable signal medium can include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal can take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0153] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the above-mentioned module, program segment or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0154] The modules involved in the embodiments of the present invention may be implemented in software or hardware. The modules described may also be set in a processor, for example, they may be described as: a processor including a data cache check module, a cache data return module, a query request processing module, and a data query and loading module. The names of these modules do not, in some cases, constitute limitations on the modules themselves. For example, the data cache check module may also be described as "a module for checking whether the data requested for query has been loaded into the cache after each query request for data is received."

[0155] As another aspect, the present invention also provides a computer-readable medium, which may be included in the device described in the above embodiment; or it may exist independently and not be assembled into the device. The above computer-readable medium carries one or more programs, and when the above one or more programs are executed by a device, the device includes: after receiving each query request for data, checking whether the data requested for query has been loaded into the cache, and the query request includes the business ID of the data requested for query; if the data requested for query has been loaded into the cache, returning the cached data requested for query; if the data requested for query has not been loaded into the cache, checking whether there is an equivalent query request for the query request, and the equivalent query request includes the same business ID as the query request; if so, discarding the query request; otherwise, adding the business ID included in the query request to the loading cache request queue; querying the data corresponding to each business ID in the loading cache request queue according to the preset strategy, and loading the queried data into the cache.

[0156] According to the technical solution of the embodiment of the present invention, after each query request for data is received, it is checked whether the data requested for query has been loaded into the cache. If the data requested for query has been loaded into the cache, the cached data requested for query is returned; if the data requested for query has not been loaded into the cache, it is checked whether there is an equivalent query request for the query request. If so, the query request is discarded, otherwise the business ID included in the query request is added to the loading cache request queue, and the data corresponding to each business ID in the loading cache request queue is queried according to the preset strategy, and the queried data is loaded into the cache. It can avoid long-term waiting for data queryers, and only the data required by the data queryer needs to be loaded, saving memory space. By controlling the length of the mapping or the length of the loading cache request queue by the threshold, the query request is limited, and excessive requests are discarded to avoid the rapid consumption of the server computing resources, and ensure that the server continues to provide services. According to the configured time interval, the business ID is regularly obtained from the loading cache request queue in a specific order, and the number of business IDs that can be obtained each time is determined by configuration, so that the rate of querying and obtaining data from the data source can be controlled to avoid instantaneous high-concurrency access to the data source, prevent the computing resources of the data source from being rapidly consumed, and ensure that the data source continues to provide services.

[0157] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may occur depending on design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A data loading method, characterized in that: include: After each query request for data is received, checking whether the requested data has been loaded into the cache, wherein the query request includes the service ID of the requested data; If the requested data has been loaded into the cache, returning the cached requested data; If the data requested for query is not loaded into the cache, check whether an equivalent query request of the query request already exists in the mapping or the load cache request queue, and the equivalent query request includes the same business ID as the query request; if so, discard the query request; otherwise, add the business ID included in the query request to the load cache request queue; The data corresponding to each service ID in the load cache request queue is queried according to a preset strategy, and the queried data is loaded into the cache.

2. The method according to claim 1, characterized in that The step of checking whether an equivalent query request of the query request already exists in the mapping comprises: Check whether there is a key in the mapping that is equal to the business ID included in the query request. If so, the equivalent query request exists, otherwise the equivalent query request does not exist, wherein the mapping includes a key-value pair, the key of the key-value pair is the business ID, and the value is a preset flag value.

3. The method according to claim 2, characterized in that Also includes: If there is no key in the mapping that is equal to the service ID included in the query request, A key-value pair is generated with the service ID included in the query request as the key and the preset flag value as the value, and the generated key-value pair is added to the mapping.

4. The method according to claim 3, characterized in that Under the condition of checking and determining that the length of the mapping is greater than a first threshold, the operation of adding the business ID included in the query request to the loading cache request queue is performed, and the operation of generating a key-value pair using the business ID included in the query request as a key and the preset flag value as a value, and adding the generated key-value pair to the mapping is performed.

5. The method according to claim 2, characterized in that: After the queried data is loaded into the cache, the key-value pairs corresponding to the business IDs of the queried data in the mapping are cleared.

6. The method according to claim 1, characterized in that The step of checking whether an equivalent query request of the query request already exists in the loading cache request queue comprises: The load cache request queue is traversed to check whether the load cache request queue stores the service ID included in the query request. If so, the equivalent query request exists; otherwise, the equivalent query request does not exist.

7. The method according to claim 6, characterized in that Under the condition that it is checked and determined that the length of the load cache request queue is greater than the second threshold, the operation of adding the service ID included in the query request to the load cache request queue is performed.

8. The method according to claim 1, characterized in that The step of querying the data corresponding to each service ID in the load cache request queue according to a preset strategy includes: According to the configured time interval, regularly obtain the service ID from the load cache request queue in a specific order, wherein the number of service IDs that can be obtained each time is determined by configuration; The data corresponding to each business ID in the obtained business ID is queried from the data source.

9. A data loading device, characterized in that: include: A data cache checking module, used for checking whether the requested data has been loaded into the cache after receiving each query request for data, wherein the query request includes the business ID of the requested data; A cache data returning module, used for returning the cached data of the requested query if the data of the requested query has been loaded into the cache; A query request processing module, used for checking whether an equivalent query request of the query request already exists in the mapping or the loading cache request queue if the data requested for query is not loaded into the cache, and the equivalent query request includes the same business ID as the query request; if so, discarding the query request; otherwise, adding the business ID included in the query request to the loading cache request queue; The data query and loading module is used to query the data corresponding to each business ID in the loading cache request queue according to a preset strategy, and load the queried data into the cache.

10. The device according to claim 9, characterized in that The query request processing module includes a first equivalent query request checking submodule, which is used to: Check whether there is a key in the mapping that is equal to the business ID included in the query request. If so, the equivalent query request exists, otherwise the equivalent query request does not exist, wherein the mapping includes a key-value pair, the key of the key-value pair is the business ID, and the value is a preset flag value.

11. The device according to claim 10, characterized in that The query request processing module is also used for: When there is no key in the mapping that is equal to the business ID included in the query request, a key-value pair is generated with the business ID included in the query request as the key and the preset flag value as the value, and the generated key-value pair is added to the mapping.

12. The device according to claim 11, characterized in that The query request processing module performs the operation of adding the business ID included in the query request to the loading cache request queue, and the operation of generating a key-value pair using the business ID included in the query request as a key and the preset flag value as a value, and adding the generated key-value pair to the mapping, under the condition that the length of the mapping is checked and determined to be greater than a first threshold.

13. The device according to claim 10, characterized in that Also includes cleanup modules for: After the queried data is loaded into the cache, the key-value pairs corresponding to the business IDs of the queried data in the mapping are cleared.

14. The device according to claim 9, characterized in that The query request processing module includes a second equivalent query request checking submodule, which is used to: The load cache request queue is traversed to check whether the load cache request queue stores the service ID included in the query request. If so, the equivalent query request exists; otherwise, the equivalent query request does not exist.

15. The device according to claim 14, characterized in that The query request processing module performs the operation of adding the service ID included in the query request to the loading cache request queue under the condition that the length of the loading cache request queue is checked and determined to be greater than the second threshold.

16. The device according to claim 9, characterized in that The data query and loading module includes a data query submodule, which is used to: According to the configured time interval, regularly obtain the service ID from the load cache request queue in a specific order, wherein the number of service IDs that can be obtained each time is determined by configuration; The data corresponding to each business ID in the obtained business ID is queried from the data source.

17. An electronic device, characterized in that: include: one or more processors; a memory for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors are enabled to implement the method according to any one of claims 1 to 8.

18. A computer readable medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 8 is implemented.

Images