Double-cache concurrency control method and device, computing equipment and storage medium

Abstract

The invention discloses a double-cache concurrency control method, relates to the technical field of cloud computing, and particularly relates to a double-cache concurrency control method and device, computing equipment and a storage medium. A blocking queue is adopted as a cache queue in a system, and externally cached requests are taken into the system in time; therefore, two-stage cache operation is realized for dense access of a third-party system, the pressure on message queues of external cache can be reduced, and concurrent requests can be processed more flexibly; the same type of operation data is compressed for one time, so that the operation times of system service on the data can be reduced, and the processing capability of an application service single node is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of cloud computing, and particularly to a dual-cache concurrent control method, device, computing device, and storage medium. Background Art

[0002] Cloud computing is a service model that centralizes computing resources and services through a network to achieve on-demand allocation and elastic expansion. It abstracts computing resources, storage resources, network resources, etc. into a "cloud" through the network and provides them for users to use.

[0003] In the specific industry of online travel agencies (OTAs), the system needs to interface with the ticketing systems of different hotels, scenic spots, and airlines, and also needs to exchange products with the OTA systems of competitors. Therefore, the changes in the status and quantity of third-party products are very frequent and extremely important. Therefore, the OTA system platform must not miss the notifications of product changes on the third-party platform. When the data of the docked third-party platform reaches a certain level, the notifications of product information changes will become extremely intensive. To ensure that each notification is processed, a certain processing strategy is required.

[0004] Currently, most of the ways to handle this problem are to increase the number of application server cluster nodes to solve the single-machine capacity problem. There are also ways to use MQ for caching and asynchronous mechanisms to perform peak shaving. However, due to too many external requests, the cache of MQ may become full and data loss may occur. In addition, there are also attempts to increase the processing capacity of a single node.

[0005] For example, the Chinese patent with the publication number CN118295786A, "Concurrent Request Configuration Throttling Method", discloses a concurrent request configuration throttling method based on "distributing queue processing of the high-concurrency requests according to a preset first threshold group" to achieve configurable throttling control, reduce the risk of system downtime, avoid duplicate development, and improve the overall development efficiency.

[0006] Another example is the Chinese patent with the publication number CN117785450A, "A Strategy Scheduling Method and Device for Concurrent Control", which discloses a method of "strategy scheduling for concurrent control under the coordination of nodes with management authority" to ensure the integrity of resources without locking the resources, and can shorten the resource scheduling duration in a distributed system and improve concurrent performance during multiple concurrent scheduling operations.

[0007] However, based on the above-mentioned prior art, when a request storm of tens of thousands per second comes, if only one request data is processed each time, the server needs to process tens of thousands of request data instantly, and the service node resources are easily exhausted, resulting in the service crashing and stopping. Summary of the Invention

[0008] The object of the present invention is to avoid the deficiencies in the prior art and provide a technology that can reduce the number of data processing threads and the number of data processing operations, thereby increasing the ability of a single-node server to process service data and improving the utilization rate of the server.

[0009] The object of the present invention is achieved through the following technical solutions:

[0010] Therefore, according to one aspect disclosed by the present invention, a dual-cache concurrent control method is provided, including the following steps:

[0011] S1: Retrieve request data from the message queue through an MQ consumption thread and place it into the blocking queue;

[0012] S2: Generate a corresponding data set for the request data in the blocking queue through a resident thread;

[0013] S3: Obtain the corresponding keywords of each request data in the data set and clean out the key data corresponding to each request data;

[0014] S4: Merge and classify the key data according to the keywords to form corresponding compression tasks.

[0015] Specifically, step S2 further includes:

[0016] S21: Monitor the blocking queue in real time through a resident thread to determine whether there is request data in the blocking queue. If so, proceed to step S22;

[0017] S22: Determine whether the data set is empty. If so, create a corresponding data set; if not, store the request data into the data set.

[0018] More specifically, the data set created in step S22 has a maximum capacity value; when the data set is not empty, determine whether the number of request data in the data set is greater than or equal to the maximum capacity value. If so, submit the data set and create a corresponding data set; if not, store the request data into the data set.

[0019] More specifically, step S22 further includes: determining whether the interval of the submission time is greater than or equal to a preset time interval threshold. If so, directly submit the data set.

[0020] In addition, before step S1, it further includes:

[0021] S01: Obtain the request data of the third-party system and calculate the corresponding hash value through a message digest algorithm;

[0022] S02: Determine whether the requested data exists in the message queue or the blocking queue according to the hash value. If not, place the requested data into the message queue.

[0023] Specifically, after step S4, the following steps are further included:

[0024] S5: Execute the compression task. If the operation is successful, clear the data set corresponding to the compression task. If the operation fails, generate corresponding record logs according to the corresponding operation data and exception information.

[0025] Another specifically, step S1 further includes the following steps: After the MQ consumption thread retrieves the requested data, perform validity verification on the requested data to determine whether the requested data is valid data.

[0026] According to another aspect disclosed by the present invention, there is provided a dual-cache concurrency control device, adopting a dual-cache concurrency control method as above, including: a data acquisition module for acquiring requested data of a third-party system; a hash value verification module for calculating a corresponding hash value through a message digest algorithm and determining whether the requested data exists in the cache server according to the hash value; a data verification module for determining whether the requested data is valid data; a queue module for retrieving the requested data from the message queue through the MQ consumption thread and placing it into the blocking queue; a data set generation module for monitoring the blocking queue in real time through a resident thread and generating a corresponding data set for the requested data in the blocking queue; a compression task generation module for obtaining the corresponding keywords of each requested data in the data set and cleaning out the key data corresponding to each requested data; merging and classifying the key data according to the keywords to form a corresponding compression task.

[0027] According to still another aspect disclosed by the present invention, there is provided a computing device, including a memory, a processor, and computer instructions stored on the memory and executable on the processor. When the processor executes the instructions, the steps of a method for generating an electronic music score as above are implemented.

[0028] According to another aspect disclosed by the present invention, there is provided a computer-readable storage medium storing computer instructions, and when the instructions are executed by a processor, the steps of a method for generating an electronic music score as above are implemented.

[0029] The beneficial effects of the present invention: A dual-cache concurrency control method, which uses a blocking queue as the cache queue within the system, timely retrieves requests from the external cache into the system, thereby realizing dual-level cache operation for intensive access to the third-party system. It can not only reduce the pressure on the message queue of the external cache, but also more flexibly process concurrent requests; by compressing and processing the same type of operation data once, it can reduce the number of operations of the system service on the data, thereby increasing the processing capacity of a single node of the application service. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] The exemplary embodiments disclosed in the present invention can be better understood by combining the accompanying drawings, in which:

[0031] Figure 1 Shown is a schematic flowchart of a dual - buffer concurrent control method according to Embodiment 1 of the present invention's disclosure Figure 1 ;

[0032] Figure 2 Shown is a schematic flowchart of a dual - buffer concurrent control method according to Embodiment 1 of the present invention's disclosure Figure 2 ;

[0033] Figure 3 Shown is a schematic flowchart of a resident thread in a dual - buffer concurrent control method according to Embodiment 1 of the present invention for processing data in a blocking queue;

[0034] Figure 4 Shown is a general schematic flowchart of a dual - buffer concurrent control method according to Embodiment 1 of the present invention;

[0035] Figure 5 Shown is a schematic diagram of program modules of a dual - buffer concurrent control device according to Embodiment 1 of the present invention;

[0036] Figure 6 Shown is a schematic diagram of the hardware structure of a computing device according to Embodiment 1 of the present invention's disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0037] The following will describe the specific embodiments of the present invention. It should be noted that in the process of the specific description of these embodiments, for the sake of concise description, this specification cannot describe all the features of the actual embodiments in detail. It should be understood that in the actual implementation process of any embodiment, just as in the process of any engineering project or design project, in order to achieve the specific goals of the developer and to meet system - related or business - related restrictions, various specific decisions are often made, and these will also change from one embodiment to another. In addition, it can also be understood that although the efforts made in this development process may be complex and lengthy, for those of ordinary skill in the art related to the content disclosed in the present invention, some design, manufacturing, or production changes based on the technical content disclosed in the present invention are just conventional technical means and should not be understood as the content of the present invention being insufficient.

[0038] Unless otherwise defined, technical terms or scientific terms used in the claims and the specification shall have the ordinary meanings understood by those of ordinary skill in the technical field to which the present invention pertains. The terms "first", "second" and similar terms used in the specification and claims of this patent application for invention do not denote any order, quantity or importance, but are merely used to distinguish different components. The terms such as "a" or "an" do not denote a limitation of quantity, but mean that there is at least one. The terms such as "comprising" or "including" mean that the elements or items appearing before "comprising" or "including" cover the elements or items listed after "comprising" or "including" and their equivalent elements, and do not exclude other elements or items. The terms such as "connected" or "coupled" are not limited to physical or mechanical connections, nor are they limited to direct or indirect connections.

[0039] Embodiment 1

[0040] Please refer to Figure 1 and Figure 4 This embodiment provides a dual-buffer concurrent control method, which includes the following steps S01 to S02, and steps S1 to S5;

[0041] S01: Obtain the request data of the third-party system, and calculate the corresponding hash value through the message digest algorithm.

[0042] S02: Determine whether the request data exists in the message queue (MQ) or the blocking queue according to the hash value. If not, place the request data into the message queue (MQ).

[0043] S1: Use the MQ consumption thread to take out the request data from the message queue (MQ) and place it into the blocking queue (Queue).

[0044] Among them, MQ (Message Queue, message queue) is a "first in, first out" data structure in the basic data structure. It means that the data (message) to be transmitted is placed in the queue, and the queue mechanism is used to implement message passing - the producer generates the message and puts the message into the queue, and then the consumer processes it. MQ has a peak shaving effect and can evenly disperse the data to different nodes of the cluster. For the consumers of MQ, the system starts a separate thread to monitor MQ and read the data in MQ.

[0045] The message queue (MQ) is stored in the system external cache, and the blocking queue is stored in the system internal cache; by using the blocking queue as the cache queue inside the system, the requests in the external cache are fetched into the system in time, so as to realize the dual-level cache operation for the intensive access of the third-party system, which can not only reduce the pressure on the message queue of the external cache, but also process concurrent requests more flexibly.

[0046] Specifically, step S1 further includes the following steps:

[0047] After the MQ consumption thread retrieves the request data, it performs a validity check on the request data to determine whether the request data is valid. If the request data is invalid, it is directly discarded. If the request data is valid, the data request is classified, and then the data is placed in a blocking queue (Queue), and step S2 is entered.

[0048] S2: Through a resident thread (Monitor), generate a corresponding data set (List) from the request data in the blocking queue (Queue).

[0049] S3: Obtain the data set (List) submitted by the resident thread (Monitor), identify the keywords corresponding to each request data in the data set (List), and clean out the key data corresponding to each request data.

[0050] S4: Merge and classify the key data according to the keywords to form a corresponding compression task.

[0051] Using the keyword as the unique data for the business, merge and classify the cleaned key data, and compress the tasks that originally required multiple business operations into one operation to achieve the same purpose.

[0052] S5: Execute the compression task. If the operation is successful, clear the data set corresponding to the compression task; if the operation fails, generate corresponding record logs according to the corresponding operation data and exception information.

[0053] By merging and classifying, the same type of operation data is compressed and processed once, which can reduce the number of operations of the system service on the data, thereby increasing the processing capacity of a single node of the application service.

[0054] Specifically, please refer to Figure 2 and Figure 3 , step S2 further includes:

[0055] S21: Through a resident thread (Monitor), perform real-time monitoring on the blocking queue (Queue) to determine whether there is request data in the blocking queue (Queue). If so, step S22 is entered.

[0056] Among them, when the application system starts, a resident thread (Monitor) is started in advance. The resident thread (Monitor) continuously monitors the blocking queue (Queue). When there is data in the blocking queue (Queue), the data is retrieved from the blocking queue (Queue). If there is no data all the time, the resident thread (Monitor) waits in a suspended animation state. When the resident thread (Monitor) monitors that there is data in the blocking queue (Queue), it is immediately awakened and retrieves the data to step S22 to check whether the array or the linked set List storing the data is empty.

[0057] S22: Determine whether the data set (List) is empty. If it is, create the corresponding data set (List); if not, store the request data into the data set (List).

[0058] Among them, the data set generated in step S22 is provided with a maximum capacity value (MaxRequestLimit), that is, the maximum number of compressed request data, and a time interval threshold (MaxInterval); these data are defined as constants that can be dynamically modified so as to be dynamically adjusted according to the actual situation during operation.

[0059] When the data set (List) is not empty, determine whether the number of request data in the data set (List) is greater than or equal to the maximum capacity value (MaxRequestLimit). If it is, submit the data set (List), create a corresponding new data set (List), and set the maximum capacity value (MaxRequestLimit) of the newly created data set (List); if not, store the request data into the data set (List). Example: 1. When the set is empty, at most MaxRequestLimit request data can be retrieved and stored into the data set (List); 2. When there are N request data in the set, at most MaxRequestLimit - N request data can be retrieved and placed into the set data set (List).

[0060] More specifically, step S22 further includes:

[0061] Determine whether the interval of the submission time is greater than or equal to the preset time interval threshold (MaxInterval). If it is, directly submit the data set (List).

[0062] When third-party requests are not particularly intensive (such as late at night or in the morning), the data fetched by the resident thread (Monitor) each time may not be particularly much, and there may be no data within N seconds (N seconds refers to the interval from the last data acquisition to the current data acquisition). When N is greater than or equal to the given time interval threshold (MaxInterval), the set will be submitted for processing regardless of the amount of data in the data set (List).

[0063] Please continue to refer to Figure 5 which shows a dual-cache concurrent control device. In this embodiment, a dual-cache concurrent control device may include or be divided into one or more program modules. One or more program modules are stored in a storage medium and executed by one or more processors to complete the present invention and implement the above-mentioned dual-cache concurrent control method. The program modules referred to in the present invention refer to a series of computer program instruction segments that can complete specific functions, and are more suitable for describing the execution process of a dual-cache concurrent control device in a storage medium than the program itself. The following description will specifically introduce the functions of each program module in this embodiment:

[0064] The data acquisition module is used to obtain the request data of the third-party system.

[0065] The hash value verification module is used to calculate the corresponding hash value through the message digest algorithm, and judge whether the request data exists in the message queue (MQ) or the blocking queue according to the hash value.

[0066] The data verification module is used to judge whether the request data is valid data.

[0067] The first-level cache module is used to store the data in the message queue (MQ).

[0068] The queue module is used to fetch the request data from the message queue (MQ) through the MQ consumption thread and place it into the blocking queue.

[0069] The second-level cache module is used to store the data in the blocking queue.

[0070] The data set generation module is used to monitor the blocking queue in real time through the resident thread (Monitor) and generate a corresponding data set (List) for the request data in the blocking queue.

[0071] The compression task generation module is used to obtain the corresponding keywords of each request data in the data set (List), and clean out the key data corresponding to each request data; merge and classify the key data according to the keywords to form the corresponding compression task.

[0072] The execution module is used to execute the compression task and output the operation result.

[0073] A log generation module, which is used to generate corresponding record logs according to the corresponding operation data and exception information when the compression task operation fails.

[0074] This embodiment also provides a computing device, such as a smart phone, a tablet computer, a notebook computer, a desktop computer, a rack server, a blade server, a tower server, or a cabinet server (including an independent server or a server cluster composed of multiple servers) that can execute programs. The computing device 20 of this embodiment at least includes, but is not limited to: a memory 21 and a processor 22 that can be communicatively connected to each other through a system bus, as Figure 3 shown. It should be noted that Figure 6 only the computing device 20 with components 21-22 is shown, but it should be understood that it is not required to implement all the shown components, and more or fewer components can be alternatively implemented.

[0075] In this embodiment, the memory 21 (i.e., the readable storage medium) includes flash memory, a hard disk, a multimedia card, a card-type memory (e.g., SD or DX memory, etc.), a random access memory (RAM), a static random access memory (SRAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a programmable read-only memory (PROM), a magnetic memory, a magnetic disk, an optical disk, etc. In some embodiments, the memory 21 may be an internal storage unit of the computing device 20, such as the hard disk or memory of the computing device 20. In other embodiments, the memory 21 may also be an external storage device of the computing device 20, such as a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) card, a Flash Card, etc., equipped on the computing device 20. Of course, the memory 21 may also include both the internal storage unit and the external storage device of the computing device 20. In this embodiment, the memory 21 is generally used to store the operating system and various application software installed in the computing device 20, such as the program code of a dual-buffer concurrency control device in Embodiment 1. In addition, the memory 21 may also be used to temporarily store various data that have been output or will be output.

[0076] In some embodiments, the processor 22 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chips. The processor 22 is generally used to control the overall operation of the computing device 20. In this embodiment, the processor 22 is used to run the program code stored in the memory 21 or process data, such as running a dual-buffer concurrency control device to implement a dual-buffer concurrency control method in Embodiment 1.

[0077] This embodiment also provides a computer-readable storage medium, such as a flash memory, a hard disk, a multimedia card, a card-type memory (e.g., SD or DX memory, etc.), a random access memory (RAM), a static random access memory (SRAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a programmable read-only memory (PROM), a magnetic memory, a magnetic disk, an optical disk, a server, an App application store, etc., on which a computer program is stored, and when the program is executed by a processor, corresponding functions are implemented. The computer-readable storage medium of this embodiment is used to store a double-buffer concurrent control device, and when executed by a processor, it implements a double-buffer concurrent control method of Embodiment 1.

[0078] In summary, according to the exemplary embodiment, a double-buffer concurrent control method, device, computing device, and storage medium of the present invention use a blocking queue as the cache queue within the system, fetch the requests of the external cache into the system in a timely manner, and process each request in a more convenient and flexible way; it cleverly uses the system resident thread to monitor the blocking queue, within a reasonable given time range, classify a certain number of requests for the purpose of operating on multiple data at a time, and then process multiple request data at a time, so as to reduce the number of times of repeated processing of each data, thereby reducing the system's data processing ability and increasing the ability of the application node to process concurrency.

[0079] The serial numbers of the above embodiments of the present invention are only for description and do not represent the superiority or inferiority of the embodiments.

[0080] Any process or method description shown in the flowchart or described in other ways herein can be understood as representing a module, segment, or part of the code of an executable instruction including one or more steps for implementing a specific logical function or process, and the scope of the preferred embodiment of the present invention includes additional implementations, where the functions can be executed in a substantially simultaneous manner or in the reverse order according to the involved functions, rather than in the order shown or discussed, and this should be understood by those skilled in the technical field to which the embodiments of the present invention belong.

[0081] Those of ordinary skill in the art of this technology can understand that all or part of the steps carried by the method of the above embodiments can be completed by instructing relevant hardware through a program, and the program can be stored in a computer-readable medium. When the program is executed, it includes one or a combination of the steps of the method embodiment.

[0082] In the description of this specification, the descriptions referring to terms such as "one embodiment", "some embodiments", "examples", "specific examples", or "some examples" etc. mean that the specific features, structures, materials, or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described can be combined in a suitable manner in any one or more embodiments or examples.

[0083] Through the description of the above embodiments, those skilled in the art can clearly understand that the above embodiment methods can be implemented by means of software plus a necessary general hardware platform. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method.

[0084] The above are only the preferred embodiments of the present invention, and do not limit the patent scope of the present invention accordingly. Any equivalent structure or equivalent process transformation made by using the content of the specification and drawings of the present invention, or directly or indirectly applied in other related technical fields, shall be equally included in the patent protection scope of the present invention.

Claims

1. A double-buffer concurrency control method, characterized in that: The following steps are involved: S1: Take the request data from the message queue through the MQ consumer thread and put it into the blocking queue; S2: Generate a corresponding data set from the request data in the blocking queue through the resident thread; S3: Obtain the keywords corresponding to each request data in the data set, and clean out the key data corresponding to each request data; S4: The key data are merged and classified according to the keywords to form corresponding compression tasks.

2. A double buffer concurrency control method according to claim 1, characterized in that: The step S2 further comprises: S21: Monitor the blocking queue in real time through a resident thread to determine whether there is request data in the blocking queue. If so, proceed to step S22; S22: Determine whether the data set is empty, if so, create a corresponding data set; if not, store the requested data into the data set.

3. A double buffer concurrency control method according to claim 2, characterized in that: The data set created in step S22 is provided with a maximum capacity value; When the data set is not empty, determine whether the number of requested data in the data set is greater than or equal to the maximum capacity value. If so, submit the data set and create a corresponding data set; if not, store the requested data into the data set.

4. A double buffer concurrency control method according to claim 3, characterized in that: The step S22 further includes: It is determined whether the submission time interval is greater than or equal to a preset time interval threshold. If so, the data set is directly submitted.

5. A double buffer concurrency control method according to any one of claims 1 to 4, characterized in that: Before step S1, the following steps are also included: S01: Obtain request data from a third-party system and calculate the corresponding hash value using an information digest algorithm; S02: judging whether the request data exists in the message queue or the blocking queue according to the hash value, and if not, placing the request data into the message queue.

6. A double buffer concurrency control method according to claim 5, characterized in that: After step S4, The following steps are involved: S5: Execute the compression task. If the operation is successful, clear the data set corresponding to the compression task; if the operation fails, generate a corresponding record log according to the corresponding operation data and exception information.

7. A double buffer concurrency control method according to claim 5, characterized in that: The step S1 further comprises the following steps: After the MQ consumer thread takes out the request data, it performs validity check on the request data to determine whether the request data is valid data.

8. A double-buffer concurrency control device, using a double-buffer concurrency control method according to any one of claims 1 to 7, characterized in that: include: Data collection module, used to obtain request data from third-party systems; A hash value verification module, used to calculate a corresponding hash value through an information digest algorithm, and determine whether the requested data exists in the cache server according to the hash value; A data verification module, used to determine whether the requested data is valid data; The queue module is used to take out the request data from the message queue through the MQ consumer thread and put it into the blocking queue; A data set generation module, used to monitor the blocking queue in real time through a resident thread, and generate a corresponding data set from the request data in the blocking queue; The compression task generation module is used to obtain the corresponding keywords of each request data in the data set, and clean out the key data corresponding to each request data; merge and classify the key data according to the keywords to form a corresponding compression task.

9. A computing device comprising a memory, a processor, and computer instructions stored in the memory and executable on the processor, characterized in that: When the processor executes the instructions, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium storing computer instructions, characterized in that: When the instruction is executed by a processor, the steps of the method described in any one of claims 1 to 7 are implemented.

Images