A data processing method and system

Abstract

The present application discloses a data processing method and system, which uploads the acquired access traffic data to a message queue, obtains the access frequency corresponding to the access traffic data from the message queue, stores the access frequency in an LRUMap structure, traverses the LRUMap structure through preset traversal rules, obtains target access traffic data that meets preset conditions, stores the target access traffic data in a result set list, and the result set list is used to reflect the access traffic data with the highest access frequency within a preset time period, and performs a data preheating operation on the data in the result set list. Through the above, a data preheating operation is performed on the data in the result set list to avoid problems such as user page freezes and database service unavailability caused by cache cold start. In addition, performing a data preheating operation on the data in the result set list can improve the cache hit rate, thereby improving the cache utilization rate.

Description

Technical Field

[0001] The present application relates to the field of Internet technology, and more specifically, to a data processing method and system. Background Art

[0002] The overall process of existing Internet applications is that user requests go from the interface (browser or App interface, etc.) to network forwarding, application services, and then to the storage database (MySQL database, etc.), and then return to the interface to present the content.

[0003] As the number of users continues to increase and application content information becomes more and more complex, our applications need to support higher concurrency, but the number of read and write requests that the database can support per second is limited. In order to break this bottleneck, cache is usually introduced.

[0004] If a cache cold start occurs, when a user request is received, the service is provided directly based on the user request because there is no data in the cache. The cache cold start will fail to hit the cache (the higher the cache hit rate, the higher the cache usage rate), and a back-to-source request will be made to the database.

[0005] Since the performance of the database in processing requests is very limited, if the number of user requests is large at this time, it will cause some user pages to freeze, and even cause the database service to crash, making the entire database service unavailable. Summary of the invention

[0006] In view of this, the present application discloses a data processing method and system, which aims to avoid problems such as user page freeze and database service unavailability caused by cache cold start. In addition, performing data preheating operation on the data in the result set list can improve the cache hit rate, thereby improving the cache utilization rate.

[0007] In order to achieve the above purpose, the disclosed technical solution is as follows:

[0008] The first aspect of the present application discloses a data processing method, the method comprising:

[0009] Obtain access traffic data, and upload the access traffic data to a message queue;

[0010] Obtaining the access frequency corresponding to the access traffic data from the message queue, and storing the access frequency in an LRUMap structure; the LRUMap structure is used to reflect the access frequency arranged in order from high to low;

[0011] Traversing the LRUMap structure according to preset traversal rules to obtain target access traffic data that meets preset conditions;

[0012] The target access flow data is stored in a result set list; the result set list is used to reflect the access flow data corresponding to the access frequency arranged in descending order within a preset time period;

[0013] A data warm-up operation is performed on the data in the result set list.

[0014] Preferably, the obtaining of access traffic data and uploading of the access traffic data to a message queue includes:

[0015] Access traffic data is obtained, and the access traffic data is uploaded to the message queue through a preset upload method.

[0016] Preferably, obtaining the access frequency corresponding to the access traffic data from the message queue and storing the access frequency in an LRUMap structure includes:

[0017] Obtaining the access frequency data source corresponding to the access traffic data from the message queue through a preset data stream processing technology;

[0018] The access frequencies corresponding to the access frequency data source are calculated in parallel through multiple threads, and the access frequencies are stored in the LRUMap structure.

[0019] Preferably, traversing the LRUMap structure according to a preset traversal rule to obtain access traffic data that meets preset conditions includes:

[0020] Traversing the LRUMap structure according to a preset time period;

[0021] When the access frequency corresponding to the access traffic data in the LRUMap structure is greater than a preset number of times, the access traffic data is determined to be target access traffic data that meets a preset condition.

[0022] Preferably, the process of storing the target access flow data in the result set list includes:

[0023] When it is detected that there is no data in the result set list, storing the target access flow data in the result set list;

[0024] When it is detected that there is data in the result set list, the result set list is traversed to determine whether the product ID corresponding to the access flow data in the result set list is consistent with the product ID corresponding to the target access flow data;

[0025] If they are consistent, determining whether the access frequency corresponding to the target access flow data is greater than the access frequency corresponding to the access flow data in the result set list;

[0026] If it is greater, and the capacity of the result set list reaches the preset capacity, the preset access traffic data in the result set list will be replaced by the target access traffic data; the preset access traffic data is the access traffic data with the largest time interval; the time interval is the interval between the moment of entering the result set list and the current moment.

[0027] Preferably, performing a data preheating operation on the data in the result set list includes:

[0028] Asynchronously sending the result set list to the data preheating service through a preset thread;

[0029] The data preheating service is run to store the data in the result set list into the cache, thereby completing data preheating.

[0030] Preferably, before storing the target access traffic data in the result set list, the method further includes:

[0031] Initialize the result set list.

[0032] A second aspect of the present application discloses a data processing system, the system comprising:

[0033] An acquisition unit, used for acquiring access flow data and uploading the access flow data to a message queue;

[0034] A first storage unit is used to obtain the access frequency corresponding to the access traffic data from the message queue, and store the access frequency in an LRUMap structure; the LRUMap structure is used to reflect the access frequency arranged in order from high to low;

[0035] A traversal unit, used to traverse the LRUMap structure according to a preset traversal rule to obtain target access flow data that meets preset conditions;

[0036] A second storage unit is used to store the target access flow data in a result set list; the result set list is used to reflect the access flow data corresponding to the access frequency arranged in order from high to low within a preset time period;

[0037] The execution unit is used to perform a data preheating operation on the data in the result set list.

[0038] Preferably, the acquisition unit is specifically used for:

[0039] Access traffic data is obtained, and the access traffic data is uploaded to the message queue through a preset upload method.

[0040] Preferably, the first storage unit includes:

[0041] An acquisition module, used to acquire the access frequency data source corresponding to the access traffic data from the message queue by using a preset data stream processing technology;

[0042] The first storage module is used to calculate the access frequency corresponding to the access frequency data source in parallel through multiple threads, and store the access frequency in the LRUMap structure.

[0043] It can be known from the above technical scheme that the present application discloses a data processing method and system, which obtains access traffic data, uploads the access traffic data to a message queue, obtains the access frequency corresponding to the access traffic data from the message queue, and stores the access frequency in an LRUMap structure, the LRUMap structure is used to reflect the access frequency arranged in order from high to low, traverses the LRUMap structure through preset traversal rules, obtains the target access traffic data that meets the preset conditions, stores the target access traffic data in a result set list, the result set list is used to reflect the access traffic data corresponding to the access frequency arranged in order from high to low within a preset time period, and performs a data preheating operation on the data in the result set list. Through the above, a data preheating operation is performed on the data in the result set list to avoid problems such as user page freezes and data service unavailability caused by cache cold start. In addition, a data preheating operation is performed on the data in the result set list to improve the cache hit rate, thereby improving the cache utilization rate. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0045] Figure 1 A flowchart of a data processing method disclosed in an embodiment of the present application;

[0046] Figure 2 A schematic diagram of a process for improving cache hit rate disclosed in an embodiment of the present application;

[0047] Figure 3 A schematic diagram of a process for obtaining access frequencies corresponding to access traffic data from a message queue and storing the access frequencies in an LRUMap structure disclosed in an embodiment of the present application;

[0048] Figure 4 A schematic diagram of a process for traversing the LRUMap structure through a preset traversal rule to obtain target access traffic data that meets preset conditions disclosed in an embodiment of the present application;

[0049] Figure 5 A schematic diagram of a process for performing a data preheating operation on data in a result set list disclosed in an embodiment of the present application;

[0050] Figure 6 A schematic diagram of the structure of a data processing system disclosed in an embodiment of the present application. DETAILED DESCRIPTION

[0051] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0052] In this application, the terms "comprises", "comprising" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprising a ..." does not exclude the presence of other identical elements in the process, method, article or device comprising the element.

[0053] From the background technology, we can know that if a cache cold start occurs, the cache cold start will fail to hit the cache, and a back-to-source request will be made to the database. Since the performance of the database in processing requests is very limited, if the number of user requests is large at this time, it will cause some user pages to freeze, and even cause the database service to crash, making the entire database service unavailable.

[0054] In order to solve the above problems, the embodiment of the present application discloses a data processing method and system, which performs a data preheating operation on the data in the result set list to avoid problems such as user page freezes and database service unavailability caused by cache cold start. In addition, performing a data preheating operation on the data in the result set list can improve the cache hit rate, thereby improving the cache utilization rate. The specific implementation method is described by the following embodiment.

[0055] refer to Figure 1 FIG. 1 is a flow chart of a data processing method disclosed in an embodiment of the present application, and the data processing method mainly includes the following steps:

[0056] S101: Obtain access traffic data, and upload the access traffic data to a message queue.

[0057] In S101, access traffic data is obtained, and the access traffic data is uploaded to a message queue through a preset upload method.

[0058] The preset upload method can be the upload method of nginx+lua script, or other upload methods. The preset upload method is determined by the technicians according to the actual situation, and this application does not make specific restrictions. The preset upload method of this application is preferably nginx+lua script.

[0059] nginx is a high-performance Hypertext Transfer Protocol (HTTP) and reverse proxy web server.

[0060] Lua scripts can be easily embedded in nginx for use (for example, when a request arrives at the nginx proxy server, Lua can be used to count HTTP requests, store this information, or call a middleware for further processing.

[0061] The message queue can be a kafka message queue or an MQ message queue. The determination of the message queue is set by the technician according to the actual situation, and this application does not make specific restrictions. The message queue of this application is preferably a kafka message queue.

[0062] To facilitate understanding of obtaining access traffic data and uploading access traffic data to the message queue, an example is given here:

[0063] For example, if you want to obtain product access traffic data, the request URL is: http: / / host / getProductInfo?id=1, and the nginx+lua script uploads id=1 in the nginx request parameter to Kafka.

[0064] S102: Obtain access frequencies corresponding to access traffic data from the message queue, and store the access frequencies in an LRUMap structure; the LRUMap structure is used to reflect access frequencies arranged in descending order.

[0065] The LRUMap structure is a dictionary with limited capacity, similar to a ranking list. The ranking list changes dynamically, such as the top 100 stars with the most likes from fans. The LRUMap structure can realize the dynamic calculation of this ranking list.

[0066] The specific process of obtaining the access frequency corresponding to the access traffic data from the message queue and storing the access frequency in the LRUMap structure is as follows:

[0067] First, the access frequency data source corresponding to the access traffic data is obtained from the message queue through the preset data stream processing technology.

[0068] The preset data stream processing technology may be a storm data stream processing technology or other data stream processing technology. The preset data stream processing technology is determined by the technician according to the actual situation, and this application does not make specific restrictions. The preset data stream processing technology of this application is preferably a storm data stream processing technology.

[0069] The Storm data stream processing technology is used to consume data from the message queue, and the access frequency corresponding to the access traffic data is calculated in parallel through multi-threading.

[0070] By combining multithreading with Storm data stream processing technology, the number of threads can be set according to the actual capabilities of the server, and the number of tasks that can be executed at the same time can be calculated to calculate the access frequency corresponding to the access traffic data.

[0071] The results calculated by Storm are sent to another server instance through a message queue for processing, so that parallel processing can be achieved (calculating while notifying consumers to query and write data, etc.).

[0072] Then, the access frequency corresponding to the access frequency data source is calculated in parallel through multiple threads, and the access frequency is stored in the LRUMap structure.

[0073] Each thread receives data from an LRUMap data structure. The key of the map is the ID of the database where a product is located, and the value is the cumulative frequency of access to the product. When the map capacity reaches the set upper limit, the key with the lowest access frequency will be automatically removed.

[0074] S103: Traverse the LRUMap structure according to preset traversal rules to obtain target access traffic data that meets preset conditions.

[0075] The LRUMap structure is traversed at every preset time period, and the access flow data with an access frequency greater than a preset number of times is calculated as the target access flow data.

[0076] The preset time period can be 30 minutes or 1 hour, etc. The specific preset time period is set by the technician according to the actual situation, and this application does not make any specific limitation.

[0077] The target access traffic data includes the access frequency of the target access traffic and the ID corresponding to the target access traffic.

[0078] The specific process of traversing the LRUMap structure through the preset traversal rules to obtain the target access traffic data that meets the preset conditions is as follows:

[0079] First, the LRUMap structure is traversed according to a preset time period.

[0080] Then, when the access frequency corresponding to the access flow data in the LRUMap structure is greater than a preset number of times, the access flow data is determined to be target access flow data that meets the preset conditions.

[0081] The preset number of times can be 20 times, or 35 times, etc. The specific preset number of times is determined by the technical personnel according to the actual situation, and this application does not make any specific limitation.

[0082] S104: storing the target access flow data in a result set list; the result set list is used to reflect the access flow data corresponding to the access frequency arranged in descending order within a preset time period.

[0083] The target access flow data is stored in the result set list toplist. The result set list toplist is used to store the target access flow data.

[0084] The specific process of storing the target access flow data into the result set list is shown in A1-A7.

[0085] A1: When it is detected that there is no data in the result set list topnList, the target access traffic data is stored in the result set list.

[0086] A2: When it is detected that there is data in the result set list topnList, the result set list is traversed to determine whether the product ID corresponding to the access traffic data in the result set list is consistent with the product ID corresponding to the target access traffic data.

[0087] A3: If the product ID corresponding to the access traffic data in the result set list is consistent with the product ID corresponding to the target access traffic data, determine whether the access frequency corresponding to the target access traffic data is greater than the access frequency corresponding to the access traffic data in the result set list.

[0088] A4: If the access frequency corresponding to the target access traffic data is greater than the access frequency corresponding to the access traffic data in the result set list, and the capacity of the result set list topnList reaches the preset capacity, the preset access traffic data in the result set list will be replaced by the target access traffic data; the preset access traffic data is the access traffic data with the largest time interval; the time interval is the interval between the time of entering the result set list and the current time.

[0089] Among them, the preset capacity is set by technical personnel according to actual conditions, and this application does not make any specific limitations.

[0090] To facilitate understanding of the process of replacing the preset access traffic data in the result set list with the target access traffic data, an example is given here:

[0091] For example, the earliest access traffic data entering the result set list topnList is at 7:00, and the current time when the target access traffic data enters the result set list topnList is 9:00.

[0092] A5: If the access frequency corresponding to the target access traffic data is greater than the access frequency corresponding to the access traffic data in the result set list, and the capacity of the result set list topnList does not reach the preset capacity, the preset access traffic data in the result set list will be replaced by the target access traffic data.

[0093] Add the target access traffic data obtained by traversing the LRUMap result to the end of topnList.

[0094] The end of the result set list topnList is determined by the current time. The location of the target access flow data entering the topnList at the current time (latest time) is the end of the topnList.

[0095] The position of the access traffic data with the largest interval (earliest time) between the time of entering the result set list and the current time is the front end of the result set list topnList.

[0096] To help you understand the end of the result set list topnList, here is an example:

[0097] For example, the current time is 17:10, and the position where the target access traffic data enters the result set list topnList at 17:10 is the end. The time with the largest time interval from the current time 17:10 is 16:00, and the position where the access traffic data enters topnList at 16:00 is the front.

[0098] A6: If the access frequency corresponding to the target access traffic data is less than the access frequency corresponding to the access traffic data in the result set list, and the capacity of the result set list topnList does not reach the preset capacity, the target access traffic data traversed by LRUMap is added to the end of topnList.

[0099] A7: If the access frequency corresponding to the target access traffic data is less than the access frequency corresponding to the access traffic data in the result set list, and the capacity of the result set list topnList reaches the preset capacity, the preset access traffic data in the result set list will be replaced by the target access traffic data; the preset access traffic data is the access traffic data with the largest time interval; the time interval is the interval between the time of entering the result set list and the current time.

[0100] Optionally, before storing the target access traffic data in the result set list, the result set list is initialized.

[0101] S105: Perform a data preheating operation on the data in the result set list.

[0102] Get the hot data ID list (topnList) calculated in the previous step, query it from the database in batches, and store it in Redis to complete the preheating.

[0103] In order to avoid cache failure caused by Redis cold start (the higher the cache hit rate, the higher the cache usage rate), a back-to-source request is made to the database. Since the database's request processing performance is very limited, if the user request volume is large at this time, some user pages will be stuck, and even the database service will crash, resulting in the entire database service being unavailable. In order to avoid the above situation, it is necessary to perform data preheating operations on the data in the result set list.

[0104] The specific process of performing data warm-up operations on the data in the result set list is as follows:

[0105] First, the result set list is asynchronously sent to the data preheating service through a preset thread.

[0106] Among them, asynchrony can delay the final result data to the caller, during which time more additional work can be done, such as querying the database, writing data to the cache, and so on.

[0107] Then, run the data preheating service to store the data in the result set list into the cache to complete the data preheating.

[0108] If the ID corresponding to the target access traffic data in the result set list is obtained from the database, the target access traffic data is stored in the cache to improve the cache hit rate.

[0109] Cache hit rate = number of correct results returned / number of cache requests. Cache hit rate is an important indicator to measure cache effectiveness. The higher the hit rate, the higher the cache usage rate.

[0110] The cache can be a Redis cache, an SRAM cache, etc. The specific cache is determined by the technicians according to the actual situation, and this application does not make specific restrictions. The cache of this application is preferably a Redis cache.

[0111] In order to facilitate understanding and improve cache hit rate, combined Figure 2 Provide explanation. Figure 2 A flow chart showing the process of improving cache hit rate is shown.

[0112] S201: forward the target access traffic data through the network through the browser, and determine whether the target access traffic data hits the cache through the application service. If so, execute S202, if not, execute S203.

[0113] Among them, the target access traffic data is forwarded through the network through a browser or APP.

[0114] S202: Determine that the cache usage rate is improved.

[0115] S203: Perform a cache update operation on the cache through the DB cache.

[0116] In the embodiment of the present application, a data preheating operation is performed on the data in the result set list to avoid problems such as user page freezes and database service unavailability caused by cache cold start. In addition, performing a data preheating operation on the data in the result set list can improve the cache hit rate, thereby improving the cache utilization rate.

[0117] refer to Figure 3 As shown in FIG. 1 , the process of obtaining the access frequency corresponding to the access traffic data from the message queue and storing the access frequency in the LRUMap structure involved in the above S102 mainly includes the following steps:

[0118] S301: Obtaining an access frequency data source corresponding to access traffic data from a message queue through a preset data stream processing technology.

[0119] S302: Calculate the access frequency corresponding to the access frequency data source in parallel through multiple threads, and store the access frequency in the LRUMap structure.

[0120] The execution process and execution principle of S301-S302 are consistent with the execution process and execution principle of the above-mentioned S102, which can be used as a reference and will not be repeated here.

[0121] In an embodiment of the present application, an access frequency data source corresponding to the access traffic data is obtained from a message queue through a preset data stream processing technology, and the access frequency corresponding to the access frequency data source is calculated in parallel through multiple threads, and the access frequency is stored in an LRUMap structure, thereby achieving the purpose of reflecting the access frequency arranged in order from high to low through the LRUMap structure.

[0122] refer to Figure 4 As shown, the process of traversing the LRUMap structure according to the preset traversal rules involved in the above S103 to obtain the target access traffic data that meets the preset conditions mainly includes the following steps:

[0123] S401: Traverse the LRUMap structure according to a preset time period.

[0124] S402: When the access frequency corresponding to the access traffic data in the LRUMap structure is greater than a preset number of times, determining that the access traffic data is target access traffic data that meets a preset condition.

[0125] The execution process and execution principle of S401-S402 are consistent with the execution process and execution principle of the above-mentioned S103, which can be used as a reference and will not be repeated here.

[0126] In an embodiment of the present application, the LRUMap structure is traversed according to a preset time period. When the access frequency corresponding to the access traffic data in the LRUMap structure is greater than a preset number of times, the purpose of determining that the access traffic data is target access traffic data that meets preset conditions is achieved.

[0127] refer to Figure 5 As shown, the process of performing data preheating operation on the data in the result set list involved in the above S105 mainly includes the following steps:

[0128] S501: asynchronously sending the result set list to the data preheating service through a preset thread.

[0129] S502: Run the data preheating service to store the data in the result set list into the cache, thus completing the data preheating.

[0130] The execution process and execution principle of S501-S502 are consistent with the execution process and execution principle in the above S105, which can be used as a reference and will not be repeated here.

[0131] In an embodiment of the present application, the result set list is asynchronously sent to the data preheating service through a preset thread, and the data preheating service is run to store the data in the result set list in the cache. After the data preheating is completed, the cache hit rate can be improved, thereby improving the cache utilization rate.

[0132] Based on the above embodiments Figure 1 A data processing method disclosed in the present application is also disclosed in the embodiment of the present application. Figure 6 As shown, the data processing system includes an acquisition unit 601 , a first storage unit 602 , a traversal unit 603 , a second storage unit 604 and an execution unit 605 .

[0133] The acquisition unit 601 is used to acquire access traffic data and upload the access traffic data to the message queue.

[0134] The first storage unit 602 is used to obtain the access frequency corresponding to the access traffic data from the message queue, and store the access frequency in the LRUMap structure; the LRUMap structure is used to reflect the access frequency arranged in order from high to low.

[0135] The traversal unit 603 is used to traverse the LRUMap structure according to a preset traversal rule to obtain target access traffic data that meets preset conditions.

[0136] The second storage unit 604 is used to store the target access flow data into a result set list; the result set list is used to reflect the access flow data corresponding to the access frequency arranged in descending order within a preset time period.

[0137] The execution unit 605 is used to perform a data warm-up operation on the data in the result set list.

[0138] Furthermore, the acquisition unit 601 is specifically configured to acquire access traffic data and upload the access traffic data to the message queue in a preset upload manner.

[0139] Furthermore, the first storage unit 602 includes:

[0140] The acquisition module is used to obtain the access frequency data source corresponding to the access traffic data from the message queue through a preset data stream processing technology.

[0141] The first storage module is used to calculate the access frequency corresponding to the access frequency data source through multi-threaded parallel computing, and store the access frequency in the LRUMap structure.

[0142] Furthermore, the traversal unit 603 includes:

[0143] The traversal module is used to traverse the LRUMap structure according to a preset time period.

[0144] The determination module is used to determine that the access flow data is target access flow data that meets preset conditions when the access frequency corresponding to the access flow data in the LRUMap structure is greater than a preset number of times.

[0145] Further, the second storage unit includes:

[0146] The first storage module is used to store the target access flow data in the result set list when it is detected that there is no data in the result set list.

[0147] The first judgment module is used to judge whether the commodity ID corresponding to the access flow data in the result set list is consistent with the commodity ID corresponding to the target access flow data when the result set list is monitored to have data.

[0148] The second judgment module is used to judge whether the access frequency corresponding to the target access flow data is greater than the access frequency corresponding to the access flow data in the result set list if they are consistent.

[0149] The replacement module is used to replace the preset access traffic data in the result set list with the target access traffic data if it is greater than and the capacity of the result set list reaches the preset capacity; the preset access traffic data is the access traffic data with the largest time interval; the time interval is the interval between the time of entering the result set list and the current time.

[0150] Furthermore, the execution unit includes:

[0151] The sending module is used to asynchronously send the result set list to the data preheating service through a preset thread.

[0152] The run module is used to run the data preheating service, so that the data in the result set list is stored in the cache to complete the data preheating.

[0153] Furthermore, the data processing system also includes an initialization unit.

[0154] Initialization unit, used to initialize the result set list.

[0155] In the embodiment of the present application, a data preheating operation is performed on the data in the result set list to avoid problems such as user page freezes and database service unavailability caused by cache cold start. In addition, performing a data preheating operation on the data in the result set list can improve the cache hit rate, thereby improving the cache utilization rate.

[0156] For the aforementioned method embodiments, for the sake of simplicity, they are all described as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the order of the actions described, because according to the present application, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application.

[0157] It should be noted that each embodiment in this specification is described in a progressive manner, and each embodiment focuses on the differences from other embodiments, and the same or similar parts between the embodiments can be referred to each other. For system embodiments, since they are basically similar to method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0158] The steps in the methods of the embodiments of the present application can be adjusted in order, combined, and deleted according to actual needs.

[0159] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are merely used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations.

[0160] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

[0161] The above is only a preferred implementation of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A data processing method, characterized in that: The method comprises: Obtain access traffic data, and upload the access traffic data to a message queue; Obtaining the access frequency data source corresponding to the access traffic data from the message queue through a preset data stream processing technology; The access frequencies corresponding to the access frequency data source are calculated in parallel by multiple threads, and the access frequencies are stored in an LRUMap structure, wherein each thread receives data from an LRUMap data structure; the LRUMap structure is used to reflect the access frequencies arranged in order from high to low; Traversing the LRUMap structure according to preset traversal rules to obtain target access traffic data that meets preset conditions; The target access flow data is stored in a result set list; the result set list is used to reflect the access flow data corresponding to the access frequency arranged in descending order within a preset time period; A data warm-up operation is performed on the data in the result set list.

2. The method according to claim 1, characterized in that: The obtaining of access traffic data and uploading of the access traffic data to a message queue includes: Access traffic data is obtained, and the access traffic data is uploaded to the message queue through a preset upload method.

3. The method according to claim 1, characterized in that The traversing the LRUMap structure according to the preset traversal rule to obtain access traffic data that meets the preset conditions includes: Traversing the LRUMap structure according to a preset time period; When the access frequency corresponding to the access traffic data in the LRUMap structure is greater than a preset number of times, the access traffic data is determined to be target access traffic data that meets a preset condition.

4. The method according to claim 3, characterized in that The process of storing the target access flow data in the result set list includes: When it is detected that there is no data in the result set list, storing the target access flow data in the result set list; When it is detected that there is data in the result set list, the result set list is traversed to determine whether the product ID corresponding to the access flow data in the result set list is consistent with the product ID corresponding to the target access flow data; If they are consistent, determining whether the access frequency corresponding to the target access flow data is greater than the access frequency corresponding to the access flow data in the result set list; If it is greater, and the capacity of the result set list reaches the preset capacity, the preset access traffic data in the result set list will be replaced by the target access traffic data; the preset access traffic data is the access traffic data with the largest time interval; the time interval is the interval between the moment of entering the result set list and the current moment.

5. The method according to claim 1, characterized in that The performing of a data preheating operation on the data in the result set list includes: Asynchronously sending the result set list to the data preheating service through a preset thread; The data preheating service is run to store the data in the result set list into the cache, thereby completing data preheating.

6. The method according to claim 1, characterized in that Before storing the target access traffic data in the result set list, the method further includes: Initialize the result set list.

7. A data processing system, characterized in that: The system comprises: An acquisition unit, used for acquiring access flow data and uploading the access flow data to a message queue; A first storage unit is used to obtain the access frequency corresponding to the access traffic data from the message queue, and store the access frequency in an LRUMap structure; the LRUMap structure is used to reflect the access frequency arranged in order from high to low; A traversal unit, used to traverse the LRUMap structure according to a preset traversal rule to obtain target access flow data that meets preset conditions; A second storage unit is used to store the target access flow data in a result set list; the result set list is used to reflect the access flow data corresponding to the access frequency arranged in order from high to low within a preset time period; An execution unit, configured to perform a data preheating operation on the data in the result set list; The first storage unit comprises: An acquisition module, used to acquire the access frequency data source corresponding to the access traffic data from the message queue by using a preset data stream processing technology; The first storage module is used to calculate the access frequency corresponding to the access frequency data source in parallel through multiple threads, and store the access frequency in an LRUMap structure, wherein each thread receives data from an LRUMap data structure.

8. The system according to claim 7, characterized in that The acquisition unit is specifically used for: Access traffic data is obtained, and the access traffic data is uploaded to the message queue through a preset upload method.

Images