Multi-thread HTTP (Hyper Text Transport Protocol) request counting method, system, equipment and medium
By creating a communication pipeline between data processing threads and work processing threads in the firewall, and using signals to pass data, the problem of low performance of HTTP request counting module in the multi-threaded model is solved, efficient HTTP request processing is achieved, and firewall equipment is improved performance.
Patent Information
- Application Number
- CN202411410269.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-10
- Publication Date
- 2025-01-14
AI Technical Summary
In the multi-threaded model, the performance of the HTTP request counting module is low, mainly due to the problem of global resource mutual exclusion, the use of thread locks affects processing efficiency, resulting in the data forwarding performance of firewall equipment not meeting the standards.
By creating data processing threads and work processing threads when the main program is started, and creating communication pipelines between work processing threads and data processing threads, data is passed using signals and pipelines, the use of thread locks is avoided, and data merging and state are achieved Modified non-blocking mode.
It improves the processing efficiency of HTTP requests, reduces the frequency of data merging, solves the problem of resource competition, improves the performance of the device, avoids the use of thread locks, and greatly improves the request processing performance of the multi-threaded model.
Smart Images
Figure CN119311413A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of HTTP request counting, and in particular to a multi-threaded HTTP request counting method, system, device and medium. Background Art
[0002] HTTP request counting is a very common functional module in network firewalls. The data obtained can be used for data visualization of the firewall, and can also provide a basis for the firewall's protection module to determine whether the protected WEB server is under DDOS attack. When used for data visualization, the performance requirements for the HTTP request counting module are very low because the timeliness requirement is not high. However, for the functional requirements of determining DDOS attacks, which have very high timeliness, an efficient HTTP request counting implementation solution is particularly important.
[0003] In a multi-threaded program model, HTTP requests from the same IP may be processed by different threads. Therefore, if you want to count the number of visits to the WEB server from the same IP within a period of time, you need to calculate the sum of the number of HTTP requests from the same IP processed by all threads within a period of time.
[0004] In common design schemes, global variables are generally used to count the same IP. After the counting starts, the worker thread adds one to the global variable value when it processes an HTTP request from the IP. After the counting time ends, the value of the global variable is the total number of visits to the WEB server by the IP within the counting time range. In a multi-threaded model, threads are parallel to each other, so when a thread modifies the global counting variable, it needs to lock the variable to ensure the atomicity of the global variable and avoid multiple threads modifying the variable at the same time, thereby affecting the accuracy of the count. However, when using thread locks, after a thread obtains the thread lock, the remaining threads can only wait for the thread lock to be released before operating, which greatly affects the processing performance of the multi-threaded model.
[0005] In the main program, define a global map (all threads can read and modify the data) with the source IP as the key and a custom data structure containing request count-related data (including count value, count time range, count start time, count end time, count status, thread lock for controlling count modification, and other variables) as the value.
[0006] After starting the worker thread, each time the worker thread processes an HTTP request, it obtains the corresponding count information in the global map through the source IP of the request, and locks the relevant thread lock. After the lock is successful, the obtained count information is modified (the count value is increased by one), and the thread lock is unlocked after the modification is completed.
[0007] If multiple threads need to modify the count data corresponding to the same source IP at the same time, all threads that need to modify the data will perform the locking operation. The thread that can successfully lock the data will be the first to modify it. The other threads will be blocked and wait. After the modification is completed and unlocked, the other threads will perform the locking operation again. Only one thread can successfully lock the data each time. This process is repeated until all threads have completed the modification. Finally, accurate count data can be obtained.
[0008] In program design, operations such as locking a thread lock and blocking and waiting for the thread lock to be unlocked require a lot of time, and this time is much longer than the time to modify the count data. Therefore, this solution using thread locks will seriously affect the request processing efficiency of the multi-threaded model, and ultimately lead to substandard data forwarding performance of the firewall device. Summary of the invention
[0009] Aiming at the problems of low performance of HTTP request counting module of existing firewall and low efficiency of HTTP request counting caused by mutual exclusion of global resources in multi-thread model, the present invention proposes a multi-thread HTTP request counting method, system, device and medium. Firstly, a data processing thread and a work processing thread are created when a main program is started. Secondly, a communication pipeline is created when the work processing thread is started. When the acquired HTTP request counting time reaches the counting time set externally, a counting merging signal and relevant data corresponding to the current IP are sent to the current data processing thread through the communication pipeline, and a data acquisition signal and current IP information are generated. Finally, all work thread data are acquired and merged, the state information of the work thread data is modified, and the counting data inside the work processing thread is modified. The resource competition problem is solved, the use of thread locks is avoided, and there is no problem of multiple threads blocking and waiting for one thread to execute a task, thereby greatly improving the processing efficiency of HTTP requests.
[0010] The specific implementation contents of the present invention are as follows:
[0011] A multi-threaded HTTP request counting method comprises the following steps: firstly, creating a data processing thread and a work processing thread when a main program is started; secondly, creating a communication pipeline between the current data processing thread and the current work processing thread when the work processing thread is started; when the HTTP request counting time obtained by the work processing thread reaches the counting time set externally, sending a counting merging signal and relevant data corresponding to the current IP to the current data processing thread through the communication pipeline, and generating a data acquisition signal and current IP information; finally, according to the data acquisition signal and the current IP information, acquiring and merging all the work thread data, modifying the state information of the work thread data, obtaining the modified data information, and sending the modified data information to the work processing thread to modify the counting data inside the work processing thread.
[0012] In order to better implement the present invention, further, the multi-threaded HTTP request counting method specifically includes the following steps:
[0013] Step S1: Create a data processing thread and a work processing thread when the main program starts;
[0014] Step S2: When the work processing thread is started, a communication pipe is created between the current data processing thread and the current work processing thread. When the HTTP request counting time obtained by the work processing thread reaches the counting time set externally, the counting merging signal and the relevant data corresponding to the current IP are sent to the current data processing thread through the communication pipe, and a data acquisition signal and current IP information are generated;
[0015] Step S3: According to the data acquisition signal and the current IP information, all working thread data are acquired and merged, the state information of the working thread data is modified, the modified data information is obtained, and the modified data information is sent to the working processing thread to modify the internal data of the working processing thread.
[0016] In order to better implement the present invention, further, the step S1 specifically includes the following steps:
[0017] Step S11: When the main program is started, a data processing thread and a work processing thread are created;
[0018] Step S12: When the work processing thread is started, a communication channel is created to communicate with the data processing thread, and a count map is created with a custom data structure variable containing request count related data as the value, using the source IP of the current work processing thread as the key.
[0019] In order to better implement the present invention, further, the communication pipeline includes a first communication pipeline and a second communication pipeline;
[0020] The first communication channel is used to send messages from the work processing thread to the data processing thread;
[0021] The second communication channel is used to send data from the data processing thread to the work processing thread.
[0022] In order to better implement the present invention, further, the step S2 specifically includes the following steps:
[0023] Step S21: the work processing thread cyclically receives external HTTP request data and obtains the source IP of the HTTP request data;
[0024] Step S22: using the obtained source IP as a key, querying the counting data of the counting map; if no counting data is found, creating a custom counting structure variable, and adding the obtained HTTP request data to the counting map, and executing step S23; if counting data is found, obtaining the current system time, and determining whether the current system time has reached the end counting time; if the current system time has reached the end counting time, executing step S24; if the current system time has not reached the end counting time, executing step S25;
[0025] Step S23: Create a first message, send the first message to the data processing thread through the communication channel, generate a counting data creation signal SIG_CREATE_DATA, and execute step S25;
[0026] Step S24: create a second message, send the second message to the data processing thread through the communication channel, generate a count data merge signal SIG_COUNT_DATA, and execute step S25;
[0027] Step S25: Check whether there are any pending messages in the message receiving pipeline of the working processing thread. If there are no pending messages, execute step S21; if there is a counting data acquisition signal SIG_GET_DATA, query the counting map of the current working thread according to the source IP data in the message, and form the queried data content and the return counting data signal SIG_GET_DATA_RSP and the source IP data into a message and send it to the data processing thread, and execute step S21; if there is a counting data creation signal SIG_CREATE_DATA, obtain the data information in the message, and determine whether it exists in the counting map. If it exists, no processing is performed, and if it does not exist, it is added to the counting map of the current thread, and execute step S21; if there is a counting data merging and processing completion signal SIG_COUNT_DATA_FINISH message, the data in the message is updated to the counting map of the current thread, and execute step S26;
[0028] Step S26: Obtain and use the data that has been counted, and set the count status of the data that has been used to be to be recycled;
[0029] Step S27: traverse the count map to check whether there is count data in the count map that is to be recycled. If there is, delete the data. After the traversal is completed, execute step S21.
[0030] In order to better implement the present invention, further, after creating the custom counting structure variable described in step S22, and adding the obtained HTTP request data to the counting map, it also includes setting the count value to 1, setting the start counting time to the current system time, setting the counting time range to the externally configured statistical duration, setting the end counting time to the sum of the start counting time and the counting time range, and setting the counting state to counting.
[0031] In order to better implement the present invention, further, the step S3 specifically includes the following steps:
[0032] Step S31: Create an event processing queue, and create a data summary map with the source IP as the key and the counting result list as the value;
[0033] Step S32: cyclically check the communication pipeline for receiving count data, determine whether there is count data to be processed in the communication pipeline, and if there is a create signal message and a count data merge signal message to be processed, add the create signal message and the count data merge signal message to the event processing queue; if there is a return count data signal message to be processed, obtain the source IP of the return count data signal message, query the data summary map, and add the data in the count data signal message to the list of queried values;
[0034] Step S33: traverse the event processing queue, if the event processing queue has a count data creation signal message, send the count data creation signal message to each working processing thread; if the event processing queue has a count data merge signal message, obtain the source IP in the count data merge signal message, and add a data with the source IP as the key and the empty list as the value in the data summary map, and form a count data acquisition message with the source IP and the count data creation signal SIG_GET_DATA, and send it to each working thread;
[0035] Step S34: traverse all the data in the data summary map, check whether the list of all thread counting information is obtained in the value, if so, merge all the counting data, change the counting status to counting completed, and merge the merged data with the source IP and counting data to form a message with the processing completion signal, and send it to each working thread.
[0036] Based on the multi-threaded HTTP request counting method proposed above, in order to better implement the present invention, further, a multi-threaded HTTP request counting system is proposed, including an initialization unit, a creation unit, and a merging and modification unit;
[0037] The initialization unit is used to create a data processing thread and a work processing thread when the main program is started;
[0038] The creation unit is used to create a communication pipeline between the current data processing thread and the current work processing thread when the work processing thread is started, and when the HTTP request counting time obtained by the work processing thread reaches the counting time set externally, send the counting merging signal and the relevant data corresponding to the current IP to the current data processing thread through the communication pipeline, and generate a data acquisition signal and current IP information;
[0039] The merging and modifying unit is used to obtain and merge all working thread data according to the data acquisition signal and current IP information, modify the status information of the working thread data, obtain the modified data information, and send the modified data information to the working processing thread to modify the counting data inside the working processing thread.
[0040] Based on the multi-threaded HTTP request counting method proposed above, in order to better implement the present invention, further, an electronic device is proposed, including a memory and a processor; a computer program is stored on the memory; when the computer program is executed on the processor, the multi-threaded HTTP request counting method described above is implemented.
[0041] Based on the multi-threaded HTTP request counting method proposed above, in order to better implement the present invention, further, a computer-readable storage medium is proposed, on which computer instructions are stored; when the computer instructions are executed on the above-mentioned electronic device, the multi-threaded HTTP request counting method is implemented.
[0042] The present invention has the following beneficial effects:
[0043] (1) The present invention allocates independent memory resources to each thread. Each thread only reads and modifies its own memory resources, and sends all data that need to be merged to the same thread for data merging. This reduces the frequency of data merging, eliminates the use of thread locks, and improves program performance.
[0044] (2) When allocating memory resources to working threads, the present invention uses the memory resources of the main program, and the remaining working threads can also access the memory data of the current working thread. There is no need to use a management thread to obtain all working thread data, and other working thread data can be directly obtained in the working thread.
[0045] (3) Each working thread of the present invention creates its own counting data when it is started, and only the current thread can read and modify it; when data merging is required, all related data are sent to the same thread for data merging, which solves the problem of resource competition and avoids the use of thread locks. At the same time, the communication between the working thread and the data processing thread uses a non-blocking mode, and there will be no problem of multiple threads blocking and waiting for one thread to execute a task, which greatly improves the processing efficiency of HTTP requests and thus improves the performance of the device. BRIEF DESCRIPTION OF THE DRAWINGS
[0046] Figure 1 A schematic diagram of the HTTP request counting control flow provided by the present invention. DETAILED DESCRIPTION
[0047] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. It should be understood that the described embodiments are only part of the embodiments of the present invention, not all of the embodiments, and therefore should not be regarded as limiting the scope of protection. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technical personnel in this field without making creative work are within the scope of protection of the present invention.
[0048] In the description of the present invention, it should be noted that, unless otherwise clearly specified and limited, the terms "disposed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be an indirect connection through an intermediate medium, or it can be the internal communication of two elements. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.
[0049] The complete English corresponding to the professional terms and English abbreviations involved in this embodiment are as follows:
[0050] The full name of WEB in English is World Wide Web.
[0051] DDOS stands for Distributed Denial of Service.
[0052] HTTP stands for HyperText Transfer Protocol.
[0053] The full name of IP address is Internet Protocol Address, which is a digital address used to identify and locate devices on the Internet.
[0054] Map, the full name of which is Mapping, is a common data structure in programming languages, used to store key-value pairs. It provides a way to associate keys with values, and the corresponding values can be quickly found through the keys.
[0055] Embodiment 1:
[0056] This embodiment proposes a multi-threaded HTTP request counting method, which first creates a data processing thread and a work processing thread when the main program is started; secondly, when the work processing thread is started, a communication pipe is created between the current data processing thread and the current work processing thread, and when the HTTP request counting time obtained by the work processing thread reaches the counting time set externally, the counting merging signal and the relevant data corresponding to the current IP are sent to the current data processing thread through the communication pipe, and a data acquisition signal and current IP information are generated; finally, according to the data acquisition signal and the current IP information, all work thread data are acquired and merged, the state information of the work thread data is modified, the modified data information is obtained, and the modified data information is sent to the work processing thread, and the counting data inside the work processing thread is modified.
[0057] Working principle: This embodiment first creates a data processing thread and a work processing thread when the main program starts; secondly, a communication pipeline is created when the work processing thread starts. When the acquired HTTP request counting time reaches the externally set counting time, the counting merging signal and the relevant data corresponding to the current IP are sent to the current data processing thread through the communication pipeline, and a data acquisition signal and current IP information are generated; finally, all work thread data are acquired and merged, the status information of the work thread data is modified, and the counting data inside the work processing thread is modified; the resource competition problem is solved, the use of thread locks is avoided, and there will be no problem of multiple threads blocking waiting for a thread to execute tasks, which greatly improves the processing efficiency of HTTP requests.
[0058] Embodiment 2:
[0059] This embodiment is described in detail in the form of steps based on the above embodiment 1. The multi-threaded HTTP request counting method specifically includes the following steps.
[0060] Step S1: Create a data processing thread and a work processing thread when the main program starts.
[0061] The step S1 specifically includes the following steps:
[0062] Step S11: When the main program is started, a data processing thread and a work processing thread are created;
[0063] Step S12: When the work processing thread is started, a communication channel is created to communicate with the data processing thread, and a count map is created with a custom data structure variable containing request count related data as the value, using the source IP of the current work processing thread as the key.
[0064] Further, the communication pipeline includes a first communication pipeline and a second communication pipeline;
[0065] The first communication channel is used to send messages from the work processing thread to the data processing thread;
[0066] The second communication channel is used to send data from the data processing thread to the work processing thread.
[0067] Step S2: When the work processing thread is started, a communication pipe is created between the current data processing thread and the current work processing thread. When the HTTP request counting time obtained by the work processing thread reaches the externally set counting time, the counting merging signal and the relevant data corresponding to the current IP are sent to the current data processing thread through the communication pipe, and a data acquisition signal and current IP information are generated.
[0068] Furthermore, the step S2 specifically includes the following steps:
[0069] Step S21: The work processing thread cyclically receives external HTTP request data and obtains the source IP of the HTTP request data.
[0070] Step S22: Use the obtained source IP as the key to query the counting data of the counting map. If no counting data is found, create a custom counting structure variable, add the obtained HTTP request data to the counting map, and execute step S23; if the counting data is found, obtain the current system time, and determine whether the current system time has reached the end counting time. If the current system time has reached the end counting time, execute step S24; if the current system time has not reached the end counting time, execute step S25.
[0071] Furthermore, after creating the custom counting structure variable in step S22, and before adding the obtained HTTP request data to the counting map, it also includes setting the count value to 1, setting the start counting time to the current system time, setting the counting time range to the externally configured statistical duration, setting the end counting time to the sum of the start counting time and the counting time range, and setting the counting state to counting.
[0072] Step S23: Create a first message, send the first message to the data processing thread through the communication channel, generate a counting data creation signal SIG_CREATE_DATA, and execute step S25.
[0073] Step S24: create a second message, send the second message to the data processing thread through the communication channel, generate a count data merge signal SIG_COUNT_DATA, and execute step S25.
[0074] Step S25: Check whether there are any pending messages in the message receiving pipeline of the working processing thread. If there are no pending messages, execute step S21; if there is a counting data acquisition signal SIG_GET_DATA, query the counting map of the current working thread according to the source IP data in the message, and form the queried data content and the return counting data signal SIG_GET_DATA_RSP and the source IP data into a message and send it to the data processing thread, and execute step S21; if there is a counting data creation signal SIG_CREATE_DATA, obtain the data information in the message, and determine whether it exists in the counting map. If it exists, no processing is performed. If it does not exist, it is added to the counting map of the current thread, and step S21 is executed; if there is a counting data merge processing completion signal SIG_COUNT_DATA_FINISH message, the data in the message is updated to the counting map of the current thread, and step S26 is executed.
[0075] Step S26: Obtain and use the data that has been counted, and set the count status of the data that has been used to be to be recycled.
[0076] Step S27: traverse the count map to check whether there is count data in the count map that is to be recycled. If there is, delete the data. After the traversal is completed, execute step S21.
[0077] Step S3: According to the data acquisition signal and the current IP information, all working thread data are acquired and merged, the state information of the working thread data is modified, the modified data information is obtained, and the modified data information is sent to the working processing thread to modify the internal data of the working processing thread.
[0078] Furthermore, the step S3 specifically includes the following steps:
[0079] Step S31: Create an event processing queue, and create a data summary map with the source IP as the key and the counting result list as the value;
[0080] Step S32: cyclically check the communication pipeline for receiving count data, determine whether there is count data to be processed in the communication pipeline, and if there is a create signal message and a count data merge signal message to be processed, add the create signal message and the count data merge signal message to the event processing queue; if there is a return count data signal message to be processed, obtain the source IP of the return count data signal message, query the data summary map, and add the data in the count data signal message to the list of queried values;
[0081] Step S33: traverse the event processing queue, if the event processing queue has a count data creation signal message, send the count data creation signal message to each working processing thread; if the event processing queue has a count data merge signal message, obtain the source IP in the count data merge signal message, and add a data with the source IP as the key and the empty list as the value in the data summary map, and form a count data acquisition message with the source IP and the count data creation signal SIG_GET_DATA, and send it to each working thread;
[0082] Step S34: traverse all the data in the data summary map, check whether the list of all thread counting information is obtained in the value, if so, merge all the counting data, change the counting status to counting completed, and merge the merged data with the source IP and counting data to form a message with the processing completion signal, and send it to each working thread.
[0083] The other parts of this embodiment are the same as those of the above-mentioned embodiment 1, and thus will not be described in detail.
[0084] Embodiment 3:
[0085] This embodiment is based on any one of the above embodiments 1 to 2. Figure 1 As shown, a specific embodiment is described in detail.
[0086] like Figure 1 As shown in the program design process:
[0087] During program design, a variety of data processing signals are customized for different data processing communications, including count data creation signal SIG_CREATE_DATA, count data merging signal SIG_COUNT_DATA, count data merging processing completion signal SIG_COUNT_DATA_FINISH, count data acquisition signal SIG_GET_DATA, and count data return signal SIG_GET_DATA_RSP.
[0088] When designing a program, you can customize the structure of HTTP counting information, including the counting value count_value, the counting time range time_range, the counting start time start_time, the counting end time end_time, the counting status count_state, etc. The counting status includes counting, counting completed, and waiting for collection.
[0089] When designing the program, customize the message structure between the working thread and the data processing thread, and design the cache mechanism of the communication pipeline to cache multiple message data; the message structure includes two parts: processing signals and processing data content.
[0090] In the program workflow:
[0091] 1. When the main program starts, it creates a data processing thread;
[0092] 2. The main program creates multiple worker threads. When each worker thread starts, two pipelines are created to communicate with the data processing thread. One pipeline is used for the worker thread to send messages to the data processing thread, and the other pipeline is used for the data processing thread to send data to the worker thread.
[0093] 3. When each worker thread is started, a count map is created for the current worker thread only, with the source IP as the key and a custom data structure variable containing request count-related data as the value.
[0094] In the worker thread:
[0095] 4. The worker thread starts to receive external HTTP request data in a loop. After receiving an HTTP request, the worker thread obtains the source IP of the request;
[0096] 5. Use the acquired source IP as the key to query the data in the counting map of the working thread. If no relevant counting data is found in the map, create a custom counting structure variable, set the counting value count_value to 1, set the start counting time start_time to the current system time, set the counting time range time_range to the externally configured statistical duration, set the end counting time to the value of start_time plus time_range, and set the counting state count_state to counting; add the data to the counting map and execute step 6; if relevant counting data is found, obtain the current system time and determine whether the end counting time has been reached. If the end counting time has been reached, execute step 7; if the end counting time has not been reached, add one to the count value count_value in the acquired counting data and execute step 8;
[0097] 6. After adding a piece of data to the count map, create a message, where the signal is the count data creation signal SIG_CREATE_DATA, and the data content is the data of the custom structure variable created in step 5 and the source IP data. Send the created message to the data processing thread through the pipeline and execute step 8;
[0098] 7. Create a message, where the signal is the count data merge signal SIG_COUNT_DATA, and the data content is the count-related data and source IP data obtained in step 5. Send the created message to the data processing thread through the pipeline and execute step 8;
[0099] 8. Check whether there are any pending messages in the message receiving pipeline of the working thread. If there are no pending messages, execute step 4. If there is a message of obtaining count data SIG_GET_DATA, query the map of the current working thread according to the source IP data in the message, and form a message with the queried data content, the return count data signal SIG_GET_DATA_RSP and the source IP data to send to the data processing thread, and then execute step 4. If no data content is found, all are 0. If there is a count data creation signal SIG_CREATE_DATA message, obtain the data information in the message and determine whether it exists in the count map. If it exists, do not process it. If it does not exist, add it to the count map of the current thread, and then execute step 4. If there is a count data merge processing completion signal SIG_COUNT_DATA_FINISH message, update the data in the message to the count map of the current thread, and execute step 9.
[0100] 9. Other functional modules of the current working thread obtain the data that has been counted and use it. After the use is completed, the counting state is set to be recycled, and step 10 is executed;
[0101] 10. Traverse the count map and check whether there is any count data to be recycled in the map. If there is, delete the data to release memory space. After the traversal is completed, execute step 4.
[0102] In the data management thread:
[0103] 11. In the data management thread, create an event processing queue, create a data summary map with the source IP as the key and the custom counting result list as the value, and execute step 12;
[0104] 12. Loop through each pipeline that receives worker thread information to determine whether there is any pending information. If there is a pending count data creation signal SIG_CREATE_DATA message and a pending count data merge signal SIG_COUNT_DATA message, add the message to the event processing queue. If there is a pending return count data signal SIG_GET_DATA_RSP message, obtain the source IP in the message, query the relevant data in the data summary map, and add the data in the message to the list of values queried from the map. After checking each pipeline that receives worker threads, execute step 13.
[0105] 13. Traverse the event processing queue. If there is a count data creation signal SIG_CREATE_DATA message, send the message to each worker thread. If there is a count data merge signal SIG_COUNT_DATA message, get the source IP in the message and add a data with the key of source IP and the value of empty list in the summary map. Use the source IP and SIG_GET_DATA to form a count data acquisition message and send it to each worker thread. After the event processing queue is traversed, execute step 14.
[0106] 14. Traverse and summarize all the data in the map, check whether the value has a list of all thread count information, if so, merge all the count data, change the count status to count completed, and then merge the merged data with the source IP and count data to form a message with the processing completion signal (SIG_COUNT_DATA_FINISH), and send it to each working thread; after the map traversal is completed, continue to step 12.
[0107] Working principle: In order to solve the low performance problem of the HTTP request counting module in the existing firewall, this embodiment designs a lock-free HTTP request counting statistics solution from the perspectives of real-time, security, and robustness, which solves the problem of low HTTP request counting efficiency caused by the global resource mutual exclusion problem in the multi-threaded model.
[0108] In the design scheme of this embodiment, a data processing thread is established, and a communication channel is created between the data processing thread and the work processing thread. The data processing signals are customized and used for different data processing needs. The work thread places all data inside its own thread for access and modification only inside its own thread. When the current work thread needs to obtain and process data from other work threads, the current work thread sends the data to be processed to the data processing thread through the signal and the communication channel with the data processing thread; after receiving the signal and data, the data processing thread performs relevant processing according to the signal type. For example, when a request signal for HTTP access count merging is received, the data processing thread sends a signal to obtain relevant data to other work threads.
[0109] After receiving the data acquisition signal, other worker threads will read the relevant data from the data inside the thread and send it to the data processing thread through the pipeline; after receiving the data from all worker threads, the data processing thread will merge and count the count data. After the statistics are completed, the processed data will be sent to all worker threads through the pipeline. When the worker thread receives the processed data, it will update the data inside its own thread according to the received data.
[0110] In this embodiment, the processing of shared resource data between multiple threads is converted into thread internal resource data processing, which reduces the frequency of data merging, avoids resource competition problems between multiple threads, and improves program performance.
[0111] Each worker thread creates its own count data when it starts, and only the current thread can read and modify it. When data merging is required, all relevant data are sent to the same thread for data merging, which solves the problem of resource competition and avoids the use of thread locks. At the same time, the communication between the worker thread and the data processing thread uses a non-blocking mode, so there will be no problem of multiple threads blocking and waiting for a thread to execute tasks, which greatly improves the processing efficiency of HTTP requests and thus improves the performance of the device. Independent memory resources are allocated to each thread, each thread only reads and modifies its own memory resources, and all data that needs to be merged are sent to the same thread for data merging. The frequency of data merging is reduced, the use of thread locks is cancelled, and program performance is improved. When the worker thread allocates memory resources, the memory resources of the main program can be used, so that the remaining worker threads can also access the memory data of the current worker thread. There is no need to use a management thread to obtain all worker thread data, and other worker thread data can be obtained directly in the worker thread, but the data security of this alternative solution is not as high as the current solution.
[0112] The other parts of this embodiment are the same as any one of the above-mentioned embodiments 1-2, so they will not be repeated here.
[0113] Embodiment 4:
[0114] This embodiment, based on any one of the above-mentioned embodiments 1 to 3, proposes a multi-threaded HTTP request counting system, and proposes a multi-threaded HTTP request counting system, including an initialization unit, a creation unit, and a merging and modifying unit;
[0115] The initialization unit is used to create a data processing thread and a work processing thread when the main program is started;
[0116] The creation unit is used to create a communication pipeline between the current data processing thread and the current work processing thread when the work processing thread is started, and when the HTTP request counting time obtained by the work processing thread reaches the counting time set externally, send the counting merging signal and the relevant data corresponding to the current IP to the current data processing thread through the communication pipeline, and generate a data acquisition signal and current IP information;
[0117] The merging and modifying unit is used to obtain and merge all working thread data according to the data acquisition signal and current IP information, modify the status information of the working thread data, obtain the modified data information, and send the modified data information to the working processing thread to modify the counting data inside the working processing thread.
[0118] This embodiment also proposes an electronic device, including a memory and a processor; the memory stores a computer program; when the computer program is executed on the processor, the multi-threaded HTTP request counting method is implemented.
[0119] This embodiment also provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer instructions; when the computer instructions are executed on the above-mentioned electronic device, the above-mentioned multi-threaded HTTP request counting method is implemented.
[0120] The other parts of this embodiment are the same as any one of the above-mentioned embodiments 1 to 3, so they will not be repeated here.
[0121] The above description is only a preferred embodiment of the present invention and does not limit the present invention in any form. Any simple modification or equivalent change made to the above embodiment based on the technical essence of the present invention shall fall within the protection scope of the present invention.
Claims
1. A multi-threaded HTTP request counting method, characterized in that: First, a data processing thread and a work processing thread are created when the main program starts; secondly, when the work processing thread starts, a communication pipeline is created between the current data processing thread and the current work processing thread. When the HTTP request counting time obtained by the work processing thread reaches the counting time set externally, the counting merging signal and the relevant data corresponding to the current IP are sent to the current data processing thread through the communication pipeline, and a data acquisition signal and current IP information are generated; finally, according to the generated data acquisition signal and current IP information, all work thread data are acquired and merged, the state information of the work thread data is modified, the modified data information is obtained, and the modified data information is sent to the work processing thread to modify the counting data inside the work processing thread.
2. A multi-threaded HTTP request counting method according to claim 1, characterized in that: The multi-threaded HTTP request counting method specifically comprises the following steps: Step S1: Create a data processing thread and a work processing thread when the main program starts; Step S2: When the work processing thread is started, a communication pipe is created between the current data processing thread and the current work processing thread. When the HTTP request counting time obtained by the work processing thread reaches the counting time set externally, the counting merging signal and the relevant data corresponding to the current IP are sent to the current data processing thread through the communication pipe, and a data acquisition signal and current IP information are generated; Step S3: According to the data acquisition signal and the current IP information, all working thread data are acquired and merged, the state information of the working thread data is modified, the modified data information is obtained, and the modified data information is sent to the working processing thread to modify the internal data of the working processing thread.
3. A multi-threaded HTTP request counting method according to claim 2, characterized in that: The step S1 specifically includes the following steps: Step S11: When the main program is started, a data processing thread and a work processing thread are created; Step S12: When the work processing thread is started, a communication channel is created to communicate with the data processing thread, and a count map is created with a custom data structure variable containing request count related data as the value, using the source IP of the current work processing thread as the key.
4. A multi-threaded HTTP request counting method according to claim 3, characterized in that: The communication pipe comprises a first communication pipe and a second communication pipe; The first communication channel is used to send messages from the work processing thread to the data processing thread; The second communication channel is used to send data from the data processing thread to the work processing thread.
5. A multi-threaded HTTP request counting method according to claim 3, characterized in that: The step S2 specifically includes the following steps: Step S21: the work processing thread cyclically receives external HTTP request data and obtains the source IP of the HTTP request data; Step S22: using the obtained source IP as a key, querying the counting data of the counting map; if no counting data is found, creating a custom counting structure variable, and adding the obtained HTTP request data to the counting map, and executing step S23; if counting data is found, obtaining the current system time, and determining whether the current system time has reached the end counting time; if the current system time has reached the end counting time, executing step S24; if the current system time has not reached the end counting time, executing step S25; Step S23: Create a first message, send the first message to the data processing thread through the communication channel, generate a counting data creation signal SIG_CREATE_DATA, and execute step S25; Step S24: create a second message, send the second message to the data processing thread through the communication channel, generate a count data merge signal SIG_COUNT_DATA, and execute step S25; Step S25: Check whether there are any pending messages in the message receiving pipeline of the working processing thread. If there are no pending messages, execute step S21; if there is a counting data acquisition signal SIG_GET_DATA, query the counting map of the current working thread according to the source IP data in the message, and form the queried data content and the return counting data signal SIG_GET_DATA_RSP and the source IP data into a message and send it to the data processing thread, and execute step S21; if there is a counting data creation signal SIG_CREATE_DATA, obtain the data information in the message, and determine whether it exists in the counting map. If it exists, no processing is performed, and if it does not exist, it is added to the counting map of the current thread, and execute step S21; if there is a counting data merging and processing completion signal SIG_COUNT_DATA_FINISH message, the data in the message is updated to the counting map of the current thread, and execute step S26; Step S26: Obtain and use the data that has been counted, and set the count status of the data that has been used to be to be recycled; Step S27: traverse the count map to check whether there is count data in the count map that is to be recycled. If there is, delete the data. After the traversal is completed, execute step S21.
6. A multi-threaded HTTP request counting method according to claim 5, characterized in that: After creating the custom counting structure variable as described in step S22, the obtained HTTP request data is added to the counting map, and the count value is set to 1, the start counting time is set to the current system time, the counting time range is set to the externally configured statistical duration, the end counting time is set to the sum of the start counting time and the counting time range, and the counting state is set to counting.
7. A multi-threaded HTTP request counting method according to claim 5, characterized in that: The step S3 specifically comprises the following steps: Step S31: Create an event processing queue, and create a data summary map with the source IP as the key and the counting result list as the value; Step S32: cyclically check the communication pipeline for receiving count data, determine whether there is count data to be processed in the communication pipeline, and if there is a create signal message and a count data merge signal message to be processed, add the create signal message and the count data merge signal message to the event processing queue; if there is a return count data signal message to be processed, obtain the source IP of the return count data signal message, query the data summary map, and add the data in the count data signal message to the list of queried values; Step S33: traverse the event processing queue, if the event processing queue has a count data creation signal message, send the count data creation signal message to each working processing thread; if the event processing queue has a count data merge signal message, obtain the source IP in the count data merge signal message, and add a data with the source IP as the key and the empty list as the value in the data summary map, and form a count data acquisition message with the source IP and the count data creation signal SIG_GET_DATA, and send it to each working thread; Step S34: traverse all the data in the data summary map, check whether the list of all thread counting information is obtained in the value, if so, merge all the counting data, change the counting status to counting completed, and merge the merged data with the source IP and counting data to form a message with the processing completion signal, and send it to each working thread.
8. A multi-threaded HTTP request counting system, characterized in that: Includes initialization unit, creation unit, and merging and modifying unit; The initialization unit is used to create a data processing thread and a work processing thread when the main program is started; The creation unit is used to create a communication pipeline between the current data processing thread and the current work processing thread when the work processing thread is started, and when the HTTP request counting time obtained by the work processing thread reaches the counting time set externally, send the counting merging signal and the relevant data corresponding to the current IP to the current data processing thread through the communication pipeline, and generate a data acquisition signal and current IP information; The merging and modifying unit is used to obtain and merge all working thread data according to the data acquisition signal and current IP information, modify the status information of the working thread data, obtain the modified data information, and send the modified data information to the working processing thread to modify the counting data inside the working processing thread.
9. An electronic device, characterized in that: It comprises a memory and a processor; a computer program is stored in the memory; when the computer program is executed on the processor, the multi-threaded HTTP request counting method as described in any one of claims 1-7 is implemented.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions; when the computer instructions are executed on the electronic device as described in claim 9, the multi-threaded HTTP request counting method as described in any one of claims 1-7 is implemented.