Batch query request processing method, device, equipment and readable storage medium
By configuring the priority of query requests and using timed tasks to process batch query requests, the problems of low flexibility and low resource utilization in the existing technology are solved, and more efficient batch query processing is achieved to meet the high security of the financial industry and real-time requirements.
Patent Information
- Application Number
- CN202011514804.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2020-12-18
- Publication Date
- 2024-11-08
- Estimated Expiration
- 2040-12-18
AI Technical Summary
The prior art cannot configure the priority of query requests when processing batch query requests, resulting in low flexibility and low resource utilization, which cannot meet the high security and real-time requirements of the financial industry.
By receiving batch query requests, the priority of each query request is configured based on the preset dimension, batch to be looked up tables are generated, and the timing task is enabled to obtain pending query requests according to priority, perform deduplication processing and query, and use preset threads to query to get query results.
It improves the flexibility and resource utilization of batch query requests, achieves more efficient processing capabilities, and meets the security and real-time requirements of the financial industry.
Smart Images

Figure CN112632363B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of financial technology (Fintech), and in particular to a method, device, equipment and computer-readable storage medium for processing batch query requests. Background Art
[0002] With the development of computer technology, more and more technologies are being applied in the financial field. The traditional financial industry is gradually transforming into financial technology (Fintech). However, due to the security and real-time requirements of the financial industry, higher requirements are also placed on technology.
[0003] At present, the processing method for batch query requests is generally to convert batch query requests into single queries, and the priority of each query request in the batch query request cannot be configured. The query can only be performed in sequence based on time. Therefore, the flexibility of batch query is low and the resource utilization rate is not high enough. Summary of the invention
[0004] The present invention provides a method, device, equipment and computer-readable storage medium for processing batch query requests, aiming to improve the flexibility of processing batch queries and maximize resource utilization.
[0005] To achieve the above object, the present invention provides a method for processing batch query requests, the method comprising:
[0006] Receive batch query requests, configure the priority of each query request based on a preset dimension, and generate a batch to-be-queried table based on the priority of the query request;
[0007] Enable the scheduled task and obtain the pending query requests from the batch pending query table according to the priority;
[0008] After deduplication processing is performed on the thread processing the batch query request, the corresponding request file is downloaded based on the identity identifier of the query request to be processed, and the request content of each request file is read;
[0009] A query is performed through the preset thread through the request content of the query request to be processed to obtain a query result.
[0010] Optionally, obtaining the pending query requests from the batch pending query table according to the priority level includes:
[0011] Obtaining a single pending query request from the batch pending query table in descending order of priority;
[0012] The step of obtaining a single pending query request from the batch pending query table in descending order of priority may further include:
[0013] Transferring a single pending query request into an online queue table, and determining a queue depth of the online queue table and / or a processing success rate of the pending query request;
[0014] If the queue depth is greater than a depth threshold, and / or the processing success rate is less than a success rate threshold, the processing of the corresponding batch query request is suspended, and the next batch query request is received.
[0015] Optionally, the obtaining the query request to be processed from the batch to-be-queried table according to the priority level further includes:
[0016] After deduplication processing is performed on the thread processing the batch query request, the corresponding request file is downloaded based on the identity identifier of the query request to be processed, and the request content of each request file is read.
[0017] Optionally, reading the request content of each of the request files includes:
[0018] monitoring whether the application is stopped during the process of reading the request content;
[0019] If the application is detected to be stopped, the request content that has been read is backed up and / or the interruption position read when it stops is saved, and the next batch of request files is connected.
[0020] Optionally, reading the request content of each of the request files includes:
[0021] Determining the file quantity of the requested file;
[0022] If the number of files is greater than a first preset number, reading the request content of the requested file by an iterator;
[0023] If the number of files is less than the second preset number, the next request file is accessed until the number of accessed request files reaches a third preset number, and the request content of each request file is read.
[0024] Optionally, querying the content of the request file through a preset thread to obtain a query result includes:
[0025] The read content of the request file is submitted to a batch thread pool, and the asynchronous threads in the batch thread pool respectively perform online queries on each pending query request and return the query result to the caller.
[0026] The reading performs a query through the request content of the query request to be processed through a preset thread to obtain a query result, and the method further includes:
[0027] The parameters of the request content are verified, and after successful verification, the query request to be processed is connected to the preset thread based on a preset flow and / or quantity control mechanism.
[0028] Optionally, the connecting the pending query request to the preset thread based on a preset flow and / or quantity control mechanism includes:
[0029] Determine the data packet size of the query request to be processed and the cache free area size of each preset thread;
[0030] Determine a target preset thread for accessing the pending query request based on the data packet size and the cache free area size, and access the pending query request to the target preset thread; and / or
[0031] Determine the number of received requests of the pending query requests received in each preset thread;
[0032] A target preset thread is determined based on the connected quantity and the processing capacity of the corresponding thread, and the query request to be processed is connected to the target preset thread.
[0033] In addition, to achieve the above-mentioned purpose, the present invention also provides a batch query request processing device, the device comprising:
[0034] A receiving module, used to receive batch query requests, configure the priority of each query request based on a preset dimension, and generate a batch to-be-queried table based on the priority of the query request;
[0035] A reading module, configured to perform deduplication processing on the thread processing the batch query request, download the corresponding request file based on the identity identifier of the query request to be processed, and read the request content of each request file;
[0036] An acquisition module is used to enable a scheduled task and acquire pending query requests from the batch pending query table according to the priority;
[0037] A query module is used to query the request content of the pending query request through a preset thread to obtain the query result
[0038] In addition, to achieve the above-mentioned purpose, the present invention also provides a batch query request processing device, which includes a processor, a memory, and a batch query request processing program stored in the memory. When the batch query request processing program is executed by the processor, the steps of the batch query request processing method as described above are implemented.
[0039] In addition, to achieve the above objectives, the present invention also provides a computer-readable storage medium, on which a batch query request processing program is stored. When the batch query request processing program is executed by a processor, the steps of the batch query request processing method described above are implemented.
[0040] Compared with the prior art, the present invention provides a method, device, equipment and computer-readable storage medium for processing batch query requests, receiving batch query requests, configuring the priority of each query request based on a preset dimension, and generating a batch to-be-queried table based on the priority of the query request; enabling a scheduled task, obtaining the query request to be processed from the batch to-be-queried table according to the priority; deduplicating the thread processing the batch query request, downloading the corresponding request file based on the identity of the query request to be processed, and reading the request content of each request file; querying through the request content of the query request to be processed through a preset thread to obtain a query result. In this way, the priority of each query request in the batch query request is set based on the preset dimension, and then the request to be processed is obtained based on the priority, and the query is performed in sequence to obtain the query result, which improves the flexibility of processing batch queries and maximizes resource utilization. BRIEF DESCRIPTION OF THE DRAWINGS
[0041] Figure 1 It is a schematic diagram of the hardware structure of a batch query request processing device involved in each embodiment of the present invention;
[0042] Figure 2 It is a flowchart of a first embodiment of a method for processing batch query requests of the present invention;
[0043] Figure 3 It is a schematic diagram of a scenario of a first embodiment of a method for processing batch query requests of the present invention;
[0044] Figure 4 It is a flow chart of a second embodiment of a method for processing batch query requests of the present invention;
[0045] Figure 5 It is a functional module diagram of the first embodiment of the processing device for batch query requests of the present invention.
[0046] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION
[0047] It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not used to limit the present invention.
[0048] The batch query request processing device mainly involved in the embodiments of the present invention refers to a network connection device that can achieve network connection. The batch query request processing device can be a server, a cloud platform, etc.
[0049] Reference Figure 1 , Figure 1 1 is a schematic diagram of the hardware structure of the batch query request processing device involved in each embodiment of the present invention. In the embodiment of the present invention, the batch query request processing device may include a processor 1001 (such as a central processing unit, CPU), a communication bus 1002, an input port 1003, an output port 1004, and a memory 1005. Among them, the communication bus 1002 is used to realize the connection and communication between these components; the input port 1003 is used for data input; the output port 1004 is used for data output, and the memory 1005 can be a high-speed RAM memory or a stable memory (non-volatile memory), such as a disk memory. The memory 1005 can optionally be a storage device independent of the aforementioned processor 1001. Those skilled in the art can understand that Figure 1 The hardware structure shown in the figure does not constitute a limitation of the present invention, and may include more or less components than those shown in the figure, or combine certain components, or arrange the components differently.
[0050] Continue to refer to Figure 1 , Figure 1 The memory 1005 as a computer-readable storage medium may include an operating system, a network communication module, an application module, and a batch query request processing program. Figure 1 In the embodiment, the network communication module is mainly used to connect to the server and perform data communication with the server; and the processor 1001 can call the batch query request processing program stored in the memory 1005 and execute the batch query request processing method provided in the embodiment of the present invention.
[0051] An embodiment of the present invention provides a method for processing batch query requests.
[0052] Reference Figure 2 , Figure 2 It is a flowchart of a first embodiment of a method for processing batch query requests of the present invention.
[0053] In this embodiment, the method for processing batch query requests is applied to a device for processing batch query requests, and the method includes:
[0054] Step S101, receiving batch query requests, configuring the priority of each query request based on a preset dimension, and generating a batch to-be-queried table based on the priority of the query request;
[0055] In this embodiment, the batch query request includes a large number of query requests. In this embodiment, a unified parameter input format file is predefined, and batch query requests that conform to the parameter input format file are accessed. It can be understood that the formats of batch query requests uploaded by different callers may be inconsistent. Therefore, after receiving the batch query request uploaded by the caller, it is necessary to first parse the batch query file to obtain relevant parameters, and then reorganize the relevant parameters into a batch query request corresponding to the parameter input format file based on the parameter input format file to obtain a batch query request whose format conforms to the parameter input format, thereby facilitating the access of batch query requests and subsequent parsing and conversion.
[0056] In this embodiment, the priority of the query request is configured based on a preset dimension. The preset dimension includes at least one of a business line, a product, and a subsystem. For example, the priority of product 1 is set to the lowest priority, the priority of product 2 is set to the third priority, the priority of product 3 is set to the second priority, and the priority of the product with a priority increase request is set to the highest priority. It can be understood that priority setting can also be performed based on a combination of multiple dimensions in the preset dimensions.
[0057] In this embodiment, the basic principle of setting the priority is: while guaranteeing the minimum query resources, when a product query is idle, the corresponding resources are occupied by other products.
[0058] For example, if the total number of resources is 10, product 1 is allocated 1 resource, product 2 is allocated 2 resources, and product 3 is allocated 3 resources. When there is no request for the highest priority (4 resources), the resources for processing 4 files are all allocated to product 3; if there are 6 files in the queue of product 3, product 3 occupies 6 resources, and the remaining 1 resource can be allocated to product 2 first; if there are 2 files in the queue of product 3, product 3 occupies 2, and the remaining 5 resources are allocated to product 2, and so on.
[0059] Reference Figure 3 , Figure 3 It is a scene diagram of the first embodiment of the processing method of batch query requests of the present invention. After the batch query request is received, the received batch query request is distributed as a warehouse request, the priority is configured based on the product dimension, and query requests of various priorities are obtained to form a batch priority queue. In addition, considering that when the number of query requests is too large, a service for improving the priority request can also be set based on the business scenario or customer needs, that is, the query request with the lowest priority, the second priority or the third priority is improved to the highest priority.
[0060] Furthermore, the priority configuration is dynamically loaded and supports manual real-time adjustment, so that when the query is blocked, the priority configuration can be adjusted in time to ensure the stability of the business system.
[0061] After the priority of each query request in the batch query request is configured, a single query request to be processed is obtained from the batch query table in order of priority from high to low; it can be understood that for multiple single query requests with the same priority, they can be sorted in time order, user name order, etc., or the sorting can be randomly determined, thereby obtaining query requests arranged in a certain order.
[0062] Further, a single pending query request is transferred to an online queue table, and the queue depth of the online queue table and / or the processing success rate of the pending query request is determined; the queue depth refers to the number of pending query requests in the online queue table, and the queue depth is determined by the processing capacity of the processing device for batch query requests. The success rate refers to the ratio of the number of pending query requests that can be normally processed by the system to the total number of pending query requests.
[0063] If the queue depth is greater than the depth threshold, and / or the processing success rate is less than the success rate threshold, the processing of the corresponding batch query request is suspended, and the next batch query request is received. It can be understood that when a batch query request is converted into a single query request, in the online queue table, the depth threshold has been exceeded, or the success rate is lower than the success rate threshold, the batch query request of this batch is directly suspended, and the next batch of batch query requests is connected to prevent excessive query requests from blocking the query queue, or increasing the downstream processing pressure, causing the downstream system to crash.
[0064] Specifically, the batch query request is a batch credit query request, and the success rate is the ratio of the number of credit query requests that have queried the credit source to the total number of credit query requests in the batch credit query request. In actual applications, the situation of each credit source in the flow is regularly counted according to the category and query channel. When the success rate is lower than the success rate threshold within 5 minutes, the corresponding credit source route is closed, and the traffic is switched to the available backup credit source route. If there is no backup credit source route, the relevant batch query will also be suspended, and the query requests in the batch query request will continue to be queried after the credit source route is restored.
[0065] After the credit source route is closed, the application will start the detection task according to the preset detection strategy: if the total number of query requests per minute is greater than 10, the query success rate exceeds 90%, and lasts for more than 5 minutes, then 50% of the traffic of the credit source route will be restored. If it lasts for more than 10 minutes, then 100% of the traffic of the credit source route will be restored. Since the success rate has exceeded the success rate threshold, the batch query requests of this batch will also be restored.
[0066] Step S102, enabling the scheduled task, and obtaining the pending query requests from the batch pending query table according to the priority;
[0067] The status of the query request in the batch query table is marked as pending. The pre-set timer task processes the pending query requests in order of priority from the highest priority to the lowest priority according to the priority of each pending query request, and marks the status of the obtained pending query request as being processed.
[0068] Step S103: after deduplication processing is performed on the thread processing the batch query request, the corresponding request file is downloaded based on the identity identifier of the query request to be processed, and the request content of each request file is read;
[0069] For distributed systems, in order to ensure the normal query request processing flow, a batch of batch query requests is only downloaded by one thread at the same time, so deduplication is required to avoid multiple threads accessing the batch processing requests of the batch, resulting in resource waste. The corresponding request file is downloaded according to the identity of the query request to be processed, and the request content of each request file is read and downloaded locally.
[0070] After downloading the requested content to the local computer, batch file selection is performed to obtain files of different priorities in batches according to the priority level. Figure 3 The lowest priority corresponds to 10% of the requested files, the third priority corresponds to 20% of the requested files, the second priority corresponds to 30% of the requested files, and the highest priority corresponds to 40% of the requested files, so that the corresponding threads can query them in sequence after obtaining the files.
[0071] Specifically, determining the file quantity of the requested file;
[0072] If the number of files is greater than the first preset number, the request content of the request file is read through the iterator method; if the number of files is less than the second preset number, the next request file is connected until the number of connected request files reaches the third preset number, and the request content of each request file is read. Among them, the first preset number, the second preset number, and the third preset number are the thresholds for the number of single batch processing, which are specifically set according to actual conditions. The setting of the threshold for the number of single batch processing is to prevent the request file from being too large (the number of request files is greater than the first preset number), which may easily lead to memory overflow when reading files; the iterator method (non-full file reading) should also be used when reading. When the number of request files is greater than the first preset number, only one batch of batch query requests is processed each time. When the number of request files is less than the second preset number, because the third preset number has not been reached, multiple batches of batch query requests can be processed, thereby realizing breakpoint continuation based on processing volume, ensuring that the system load will not be exceeded, and achieving efficient processing of query requests.
[0073] Furthermore, it is also necessary to set corresponding breakpoint resumption when the application stops. Specifically, monitor whether the application stops during the process of reading the request content; if the application stops, back up the read request content and / or save the interruption position read when it stops, and access the next batch of request files.
[0074] Specifically, during the process of reading the request file, if the application stops, the interrupt position of the request file is saved; when the application is restarted, it accesses and processes the next batch of batch query requests, or jumps to the interrupt position of the last request file that has been recorded to filter the processed request file (if it is not an interrupt request file, the application starts processing from the first line of the file).
[0075] When the application stops, it needs to be stopped gracefully, that is, first back up information such as the interruption position in the request file to ensure that the application information is not lost due to the application stopping; if the application is accidentally interrupted (such as power outage), after the application is restarted, you can manually capture the "interruption position" printed in the log, update the relevant records, and then resume the processing of the corresponding batch query request.
[0076] Step S104: query the request content of the query request to be processed through a preset thread to obtain a query result.
[0077] Specifically, the read content of the request file is submitted to a batch thread pool, and the asynchronous threads in the batch thread pool respectively perform online queries on each pending query request and return the query result to the caller.
[0078] The batch thread pool includes at least one asynchronous thread, so that multi-thread processing can be implemented based on the batch thread pool, a single request is stored in an online flow table, and enters a priority-based queue table. Multiple asynchronous threads simultaneously perform online queries on each pending query request, each obtains a corresponding query result, and generates a report based on the query result, and then returns the report including the query result to the corresponding caller.
[0079] This embodiment uses the above scheme to receive batch query requests, configure the priority of each query request based on a preset dimension, and generate a batch query table based on the priority of the query request; enable a scheduled task, obtain the query request to be processed from the batch query table according to the priority; query the request content of the query request to be processed through a preset thread to obtain the query result. In this way, the priority of each query request in the batch query request is set based on the preset dimension, and then the request to be processed is obtained based on the priority, and the query is performed in sequence to obtain the query result, which improves the flexibility of processing batch queries and maximizes resource utilization.
[0080] based on Figure 2The second embodiment of the present invention provides a method for processing batch query requests. Specifically, referring to Figure 4 , Figure 4 FIG. 2 is a flow chart of a second embodiment of a method for processing batch query requests of the present invention. Figure 4 As shown, the step S104 also includes:
[0081] Step S104-1: Verify the parameters of the request content, and if the verification is successful, connect the query request to be processed to the preset thread based on a preset flow and / or quantity control mechanism.
[0082] In this embodiment, parameter verification is implemented based on annotation expressions, for example:
[0083] @SerializedName("onlylocalquery")
[0084] @Length(max=1,message=“onlylocalquery maximum length is 1”)
[0085] @pattern(regexp="[0-1]{1}",message="onlylocalquery value is 0 or 1")
[0086] Protected String onlyLocalQuery="0"
[0087] In addition, in order to prevent too many query requests from being received by each thread, causing abnormalities in the data source system and request timeouts, it is necessary to control the flow of the query requests received.
[0088] In this embodiment, the control mechanism includes a flow control mechanism and a quantity control mechanism.
[0089] The step of connecting the pending query request to the preset thread based on the flow control mechanism includes:
[0090] Determine the data packet size of the pending query request and the cache free area size of each preset thread; determine the target preset thread to access the pending query request based on the data packet size and the cache free area size, and connect the pending query request to the target preset thread.
[0091] Understandably, when communicating with the client, the sending rate of query requests is not necessarily equal to the receiving rate. If the sending rate is too fast, batch query requests cannot be processed. At this time, the unprocessed data can only be stored in the cache of each preset thread (out-of-order data packets will also be stored in the cache). If the cache is full and query requests continue to be received, the corresponding thread may discard the received data packets. A large number of packet losses will greatly waste network resources, so it is necessary to control the access traffic.
[0092] First, the data packet size of the query request to be processed is obtained, and the cache free area size of each preset thread is obtained, and the preset thread whose cache free area size is larger than the data packet size is determined as a candidate preset thread, and then the target preset thread is selected based on the first target preset thread determination rule. The first target preset thread determination rule can be random allocation, allocation by name, allocation in chronological order, priority allocation of preset threads with larger cache free area size, etc. In this way, it can be ensured that the query requests to be processed accessed in each target preset thread will not exceed the traffic load.
[0093] The step of connecting the pending query request to the preset thread based on the quantity control mechanism includes:
[0094] Determine the number of connected requests of pending query requests that have been connected in each preset thread; determine the target preset thread based on the connected number and the processing capacity of the corresponding thread, and the pending query request is connected to the target preset thread. Specifically, determine the candidate preset thread based on the number of connected requests in each thread and the maximum processing capacity, and then determine the target preset thread based on the second target preset thread. The first target preset thread determination rule can be random allocation, allocation by name, allocation in chronological order, priority allocation of preset threads with larger cache free area size, etc. In this way, it can be ensured that the pending query requests connected in each target preset thread will not exceed the traffic load.
[0095] It can be understood that the flow control mechanism and the quantity control mechanism can be used simultaneously or separately.
[0096] In addition, the pending query request can be connected to the preset thread based on the concurrency control mechanism, that is, the number of preset threads executed concurrently is controlled to determine whether to connect the pending query request, and strict restrictions can be placed on TPS (transactions per second) to perform flow control.
[0097] This embodiment verifies the parameters of the request content through the above scheme. After the verification is successful, the pending query request is connected to the preset thread based on the preset flow and / or quantity control mechanism, which can prevent query congestion and ensure the normal operation of each business system.
[0098] In addition, this embodiment also provides a processing device for batch query requests. Figure 5 , Figure 5 This is a functional module diagram of a first embodiment of a device for processing batch query requests according to the present invention.
[0099] In this embodiment, the processing device for the batch query request is a virtual device, which is stored in Figure 1 The memory 1005 of the batch query request processing device shown is used to implement all functions of the batch query request processing program: used to receive batch query requests, configure the priority of each query request based on a preset dimension, and generate a batch to-be-queried table based on the priority of the query request; used to enable a scheduled task, and obtain the query requests to be processed from the batch to-be-queried table according to the priority; used to query the request content of the query request to be processed through a preset thread to obtain a query result.
[0100] Specifically, the batch query request processing device includes:
[0101] The receiving module 10 is used to receive batch query requests, configure the priority of each query request based on a preset dimension, and generate a batch to-be-queried table based on the priority of the query request;
[0102] An acquisition module 20 is used to enable a scheduled task and acquire pending query requests from the batch pending query table according to the priority;
[0103] A reading module 30, configured to perform deduplication processing on the thread processing the batch query request, download the corresponding request file based on the identity identifier of the query request to be processed, and read the request content of each request file;
[0104] The query module 40 is used to query the request content of the query request to be processed through a preset thread to obtain a query result.
[0105] Furthermore, the receiving module is also used for:
[0106] Obtaining a single pending query request from the batch pending query table in descending order of priority;
[0107] Transferring a single pending query request into an online queue table, and determining a queue depth of the online queue table and / or a processing success rate of the pending query request;
[0108] If the queue depth is greater than a depth threshold, and / or the processing success rate is less than a success rate threshold, the processing of the corresponding batch query request is suspended, and the next batch query request is received.
[0109] Furthermore, the acquisition module is also used for:
[0110] After deduplication processing is performed on the thread processing the batch query request, the corresponding request file is downloaded based on the identity identifier of the query request to be processed, and the request content of each request file is read.
[0111] Furthermore, the acquisition module is also used for:
[0112] monitoring whether the application is stopped during the process of reading the request content;
[0113] If the application is detected to be stopped, the request content that has been read is backed up and / or the interruption position read when it stops is saved, and the next batch of request files is connected.
[0114] Furthermore, the acquisition module is also used for:
[0115] Determining the file quantity of the requested file;
[0116] If the number of files is greater than a first preset number, reading the request content of the requested file by an iterator;
[0117] If the number of files is less than the second preset number, the next request file is accessed until the number of accessed request files reaches a third preset number, and the request content of each request file is read.
[0118] Furthermore, the query module is also used for:
[0119] The read content of the request file is submitted to a batch thread pool, and the asynchronous threads in the batch thread pool respectively perform online queries on each pending query request and return the query result to the caller.
[0120] Furthermore, the reading module is also used for:
[0121] The read content of the request file is submitted to a batch thread pool, and the asynchronous threads in the batch thread pool respectively perform online queries on each pending query request and return the query result to the caller.
[0122] Furthermore, the reading module is also used for:
[0123] Determine the data packet size of the query request to be processed and the cache free area size of each preset thread;
[0124] Determine a target preset thread for accessing the pending query request based on the data packet size and the cache free area size, and access the pending query request to the target preset thread; and / or
[0125] Determine the number of received requests of the pending query requests received in each preset thread;
[0126] A target preset thread is determined based on the connected quantity and the processing capacity of the corresponding thread, and the query request to be processed is connected to the target preset thread.
[0127] In addition, an embodiment of the present invention further provides a computer-readable storage medium, on which a batch query request processing program is stored. When the batch query request processing program is executed by a processor, the steps of the batch query request processing method described above are implemented.
[0128] Compared with the prior art, the present invention proposes a method, device, equipment and computer-readable storage medium for processing batch query requests, which receives batch query requests, configures the priority of each query request based on a preset dimension, and generates a batch to-be-queried table based on the priority of the query request; enables a scheduled task, obtains the query request to be processed from the batch to-be-queried table according to the priority; deduplicates the thread processing the batch query request, downloads the corresponding request file based on the identity of the query request to be processed, and reads the request content of each request file; queries the request content of the query request to be processed through a preset thread to obtain a query result. In this way, the priority of each query request in the batch query request is set based on the preset dimension, and then the request to be processed is obtained based on the priority, and queries are performed in sequence to obtain query results, which improves the flexibility of processing batch queries and maximizes resource utilization.
[0129] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or system including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or system. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or system including the element.
[0130] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.
[0131] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a computer-readable storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes a number of instructions for a terminal device to execute the methods described in each embodiment of the present invention.
[0132] The above description is only a preferred embodiment of the present invention, and does not limit the patent scope of the present invention. Any equivalent structure or process transformation made by using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.
Claims
1. A method for processing batch query requests, characterized in that: The method comprises: receiving batch query requests, configuring the priority of each query request based on a preset dimension, and generating a batch to-be-queried table based on the priority of the query request; the preset dimension includes at least one of a business line, a product, and a subsystem; the batch to-be-queried table refers to a table recording all the query requests of this time; Enable the scheduled task, and obtain a single pending query request from the batch pending query table in descending order of priority; Transferring a single pending query request to an online queue table, determining the queue depth of the online queue table and / or the processing success rate of the pending query request; the online queue table is a sequential list of each pending query request when performing online query; the queue depth is determined by the processing capacity of the processing device for batch query requests; the processing success rate refers to the ratio of the number of pending query requests that can be normally processed by the system to the total number of pending query requests; After deduplication processing is performed on the thread processing the batch query request, the corresponding request file is downloaded based on the identity identifier of the query request to be processed, and the request content of each request file is read; the deduplication processing is to adjust a batch of batch query requests at the same time from being processed by multiple threads to being processed by one thread; If the queue depth is less than or equal to the depth threshold, and / or the processing success rate is greater than or equal to the success rate threshold, the read content of the request file is submitted to the batch thread pool, and the asynchronous threads in the batch thread pool respectively perform online queries on each pending query request and return the query result to the caller; If the queue depth is greater than the depth threshold, and / or the processing success rate is less than the success rate threshold, suspending the processing of the corresponding batch query request and receiving the next batch query request; suspending the processing of the corresponding batch query request includes: If there is a backup credit source route, close the credit source route corresponding to the pending query request whose processing success rate is less than the success rate threshold, and switch the traffic to the available backup credit source route; If there is no backup credit source route, the relevant batch query will be suspended, and the query requests in the batch query request will continue to be queried after the credit source route is restored.
2. The method according to claim 1, characterized in that: The reading of the request content of each of the request files includes: monitoring whether the application is stopped during the process of reading the request content; If the application is detected to be stopped, the request content that has been read is backed up and / or the interruption position read when it stops is saved, and the next batch of request files is connected.
3. The method according to claim 1, characterized in that The reading of the request content of each of the request files includes: Determining the file quantity of the requested file; If the number of files is greater than a first preset number, reading the request content of the requested file by an iterator; If the number of files is less than the second preset number, the next request file is accessed until the number of accessed request files reaches a third preset number, and the request content of each request file is read.
4. The method according to claim 1, characterized in that: Before submitting the read content of the request file to the batch thread pool, and having the asynchronous threads in the batch thread pool respectively perform online queries on each pending query request, and returning the query result to the caller, the method further includes: The parameters of the request content are verified, and after successful verification, the query request to be processed is connected to a preset thread based on a preset flow and / or quantity control mechanism.
5. The method according to claim 4, characterized in that The step of connecting the pending query request to a preset thread based on a preset flow and / or quantity control mechanism includes: Determine the data packet size of the query request to be processed and the cache free area size of each preset thread; Determine a target preset thread for accessing the pending query request based on the data packet size and the cache free area size, and access the pending query request to the target preset thread; and / or Determine the number of received requests of the pending query requests received in each preset thread; A target preset thread is determined based on the connected quantity and the processing capacity of the corresponding thread, and the query request to be processed is connected to the target preset thread.
6. A batch query request processing device, characterized in that: The device comprises: A receiving module, configured to receive batch query requests, configure the priority of each query request based on a preset dimension, and generate a batch to-be-queried table based on the priority of the query request; the preset dimension includes at least one of a business line, a product, and a subsystem; the batch to-be-queried table refers to a table recording all the query requests this time; The acquisition module is used to enable the timed task, obtain a single pending query request from the batch pending query table in order of priority from high to low; transfer the single pending query request to the online queue table, and determine the queue depth of the online queue table and / or the processing success rate of the pending query request; the online queue table is a sequential list of each pending query request when conducting an online query; the queue depth is determined by the processing capacity of the batch query request processing device; the processing success rate refers to the ratio of the number of pending query requests that can be normally processed by the system to the total number of pending query requests; A reading module, configured to perform deduplication processing on the thread processing the batch query request, download the corresponding request file based on the identity identifier of the query request to be processed, and read the request content of each request file; the deduplication processing is to adjust the batch query requests of a batch at the same time from being processed by multiple threads to being processed by one thread; A query module is used to submit the read content of the request file to a batch thread pool if the queue depth is less than or equal to a depth threshold, and / or the processing success rate is greater than or equal to a success rate threshold, and the asynchronous threads in the batch thread pool respectively perform online queries on each pending query request and return the query results to the caller; if the queue depth is greater than a depth threshold, and / or the processing success rate is less than a success rate threshold, suspend the processing of the corresponding batch query request and receive the next batch query request; the suspension of the processing of the corresponding batch query request includes: if there is a backup credit source route, closing the credit source route corresponding to the pending query request with a processing success rate less than the success rate threshold, and switching the traffic to the available backup credit source route; if there is no backup credit source route, suspending the relevant batch query, and continuing to query the query requests in the batch query requests after the credit source route is restored.
7. A batch query request processing device, characterized in that: The batch query request processing device includes a processor, a memory, and a batch query request processing program stored in the memory. When the batch query request processing program is executed by the processor, the steps of the batch query request processing method according to any one of claims 1 to 5 are implemented.
8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a batch query request processing program, and when the batch query request processing program is executed by a processor, the steps of the batch query request processing method according to any one of claims 1 to 5 are implemented.
Citation Information
Patent Citations
Asynchronous batch processing method and system
CN106126354A
An information service scheduling method and system for cross-mechanism batch service requests
CN109840144A
Data query method, device, computer equipment and storage medium
CN110489447A
Processing Batch Database Workload While Avoiding Overload
US20100083264A1
