A method and device for automatic reconnection based on asynchronous data transmission
By using the sliding time window to record the data transmission volume and response time in the asynchronous sending scenario, calculate the average request response time and the proportion of unresponsive data, and automatically disconnect and re-establish the connection in the connection pool, solving the network fluctuations in the asynchronous sending scenario. The high data transmission failure rate and poor system robustness have achieved a more efficient detection and reconnection mechanism.
Patent Information
- Application Number
- CN202311486416.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-11-08
- Publication Date
- 2024-11-12
- Estimated Expiration
- 2043-11-08
AI Technical Summary
The existing network connection technology lacks an automatic reconnection mechanism in asynchronous transmission scenarios, resulting in high data transmission failure rate caused by network fluctuations, poor system robustness, and continuous monitoring of each connection is required, increasing the system load.
Based on the results of business data processing, the data transmission amount and response time are recorded through the sliding time window, the average request response time and the proportion of unresponsive data are calculated, and the connection in the connection pool is automatically disconnected and re-established to realize automatic reconnection of the connection pool.
It reduces the data transmission failure rate caused by network fluctuations, enhances the robustness of the system, reduces the system load, improves the detection efficiency, and avoids manual intervention.
Smart Images

Figure CN118101747B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of data intelligent algorithms, and in particular to the field of a connection pool automatic reconnection method and device based on asynchronous data transmission. Background Art
[0002] Network connections will jump through multiple layers of nodes. If there are some network fluctuations at a node in the middle, data loss or large data delay may occur. Existing network connection technologies are usually synchronous transmission or asynchronous transmission. Synchronous transmission means that during the data transmission process, the data transmission of the sender and the receiver is synchronous. The sender and the receiver must reach an agreement before transmitting data. During the transmission process, the receiver will continuously send requests to the sender, and the sender needs to transmit data according to the received requests. Synchronous transmission requires the sender and the receiver to maintain synchronized speed and time to ensure that the data is transmitted at the appropriate speed and accurately received. Asynchronous transmission means that during the data transmission process, the data transmission of the sender and the receiver is asynchronous. The sender and the receiver do not need to maintain synchronization. During the transmission process, after the sender transmits the data to the receiver, it can continue to perform other operations without waiting for the receiver's response. The receiver will process the data after receiving it and does not need to send a request to the sender immediately.
[0003] If some network fluctuations occur during the data transmission process, it is possible that some of the sent data will never receive the processing results, or the results will be received after the timeout period. These will be considered as data transmission failures, resulting in multiple retries or too many failed responses to users. Multiple retries will cause a decrease in data processing capabilities, and continuous failed responses to users will reduce users' trust in the system.
[0004] The existing technologies for the above problems are usually aimed at synchronous transmission scenarios, using the heartbeat mechanism to ensure the validity of the connection, and judging whether the connection is still valid based on the time taken by the heartbeat result. If there is jitter in the network, it is easy to misjudge and lead to unnecessary connection reconstruction; and the sender cannot send other data during this time. However, for asynchronous data transmission scenarios, some research on automatic reconnection is missing. Summary of the invention
[0005] Based on this, the purpose of the present invention is to provide an automatic reconnection method and device based on asynchronous data transmission, and to expand an automatic connection reconnection mechanism based on the business data processing results on the basis of the existing connection pool. According to the business data, it is determined whether the connection needs to be reestablished without manual intervention. Thereby reducing the data transmission failure rate caused by network fluctuations, enhancing the robustness of the system, and eliminating the need to continuously monitor each connection, reducing the system load and improving the detection efficiency.
[0006] The present invention provides a connection pool automatic reconnection method based on asynchronous data transmission, which comprises:
[0007] Obtain the data to be sent from the sender, connect to a channel in the connection pool and complete the transmission of the data to be sent, and record the current time as the sending time of the data to be sent;
[0008] After the data transmission is completed, a sliding time window is used to record the data transmission volume per unit time, and the data identifier and transmission time corresponding to each piece of transmitted data are stored in a cache data table;
[0009] Obtain response data from the receiving end and record the response time, find the sending time of the corresponding sending data in the cache data table according to the data identifier of the response data, and obtain the request response time according to the difference between the sending time and the response time;
[0010] Get the total response volume of data to be sent and the corresponding total response time in the previous cycle, calculate the average request response time, and if the average request response time is greater than the request response time threshold in the cycle, disconnect all connections in the connection pool and destroy all connections, where the cycle is the preset data statistics cycle;
[0011] When the number of available connections in the connection pool is less than the number of allocated connections, the connection pool connection is reestablished to complete the reconnection.
[0012] Further, if the average request response time is less than the request response time threshold within the period, the following detection is performed;
[0013] Skip one cycle from the current time, obtain the total amount of sent data in the second-to-last cycle and the total amount of data for which no response was received in the second-to-last cycle, calculate the proportion of unresponse data, and if the proportion of unresponse data is greater than the unresponse threshold in the pre-cycle, disconnect all connections in the connection pool and destroy all connections.
[0014] Further, the cache data table is searched for data corresponding to a time obtained by subtracting a data timeout period from the current time, and the corresponding cache data is removed.
[0015] Furthermore, if data transmission fails during the process of connecting to a channel in the connection pool and completing the transmission of the data to be sent, the currently used connection is disconnected, the corresponding connection in the connection pool is destroyed, and a new connection is established to send the data to be sent.
[0016] On the other hand, the present application provides a connection pool automatic reconnection device based on asynchronous data transmission, comprising:
[0017] The sending unit is used to obtain the data to be sent from the sending end, connect to a channel in the connection pool and complete the transmission of the data to be sent, and record the current time as the sending time of the data to be sent;
[0018] Data cache unit: used to record the amount of data sent per unit time using a sliding time window after the data to be sent is completed, and store the data identifier and sending time corresponding to each sent data in a cache data table;
[0019] Request response time calculation unit: used to obtain the response data of the receiving end and record the response time, find the sending time of the corresponding sending data in the cache data table according to the data identifier of the response data, and obtain the request response time according to the difference between the sending time and the response time;
[0020] Connection abnormality determination unit: used to obtain the total response amount of data to be sent in the previous cycle and the corresponding total response time, calculate the average request response time, and if the average request response time is greater than the request response time threshold in the cycle, disconnect all connections in the connection pool and destroy all connections, where the cycle is a preset data statistics cycle;
[0021] Connection reconnection unit: used to reestablish the connection in the connection pool and complete the reconnection when the number of available connections in the connection pool is less than the number of allocated connections.
[0022] Furthermore, the connection abnormality determination unit further includes:
[0023] Unresponsive exception determination subunit: used to skip one cycle from the current time, obtain the total amount of sent data in the second-to-last cycle and the total amount of data for which no response was received in the second-to-last cycle, calculate the proportion of unresponsive data, and if the proportion of unresponsive data is greater than the unresponsive threshold in the pre-cycle, disconnect all connections in the connection pool and destroy all connections.
[0024] Furthermore, the connection pool automatic reconnection device based on asynchronous data transmission also includes:
[0025] Data cleaning unit: used to search the cache data table for data corresponding to a time obtained by subtracting a data timeout period from the current time, and remove the corresponding cache data.
[0026] Furthermore, the sending unit further includes:
[0027] Send abnormal reconnection subunit: used to disconnect the currently used connection, destroy the corresponding connection in the connection pool, and re-establish the connection to send the data to be sent if data transmission fails during the process of connecting to a channel in the connection pool and completing the transmission of the data to be sent.
[0028] On the other hand, the present application also provides an electronic device, including:
[0029] at least one memory and at least one processor;
[0030] The memory is used to store one or more programs;
[0031] When the one or more programs are executed by the at least one processor, the at least one processor implements any one of the steps of the above-mentioned method for automatic connection pool reconnection based on data asynchrony.
[0032] On the other hand, the present application also provides a computer-readable storage medium, which stores a computer program, characterized in that when the computer program is executed by a processor, the steps of a connection pool automatic reconnection method based on data asynchrony are implemented as described in any one of the above.
[0033] The present invention provides a connection pool automatic reconnection method based on asynchronous data transmission, which expands the connection automatic reconnection mechanism based on the business data processing results on the basis of the existing connection pool. The average request response time of the response data in a preset period is calculated based on the request response time of all response data of the business data transmission and response data and the total amount of response data; the average request response time is compared with the request response time threshold in the preset period to determine the smoothness of the received response data in the process of establishing a network connection, and further determine whether the status of the connection channel on the connection pool meets the asynchronous transmission requirements. If not, the connection is disconnected and destroyed, and the connection pool re-establishes a new connection. Furthermore, in order to avoid the situation where data loss cannot be discovered due to network anomalies, the proportion of unresponsive data within a cycle is calculated. If the proportion of unresponsive data is greater than the preset unresponsive threshold within the cycle, the connection is disconnected and destroyed, and the connection pool re-establishes a new connection. This method can determine whether there is a network connection anomaly or a high network delay in the connection in the connection pool without human intervention, and whether the connection needs to be re-established, thereby reducing the data transmission failure rate caused by network fluctuations and enhancing the robustness of the system. There is no need to continuously monitor each connection, which reduces the system load and improves detection efficiency.
[0034] For better understanding and implementation, the present invention is described in detail below with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS
[0035] Figure 1 A flow chart of a connection pool automatic reconnection method based on asynchronous data transmission provided by the present invention;
[0036] Figure 2 For execution Figure 1 A structural block diagram of a connection pool automatic reconnection device based on asynchronous data transmission of the automatic reconnection method;
[0037] Figure 3 The present invention is a flowchart of a connection pool automatic reconnection method based on asynchronous data transmission in another embodiment of the present invention. DETAILED DESCRIPTION
[0038] Asynchronous sending is different from synchronous sending. When synchronous sending sends data over a connection, you must wait for a response before sending the next piece of data. When the synchronous method fails to send data, the result is clearly known. When there is no response or other problems, the synchronous method can determine to reconnect immediately. However, after asynchronous sending sends data over a connection, the next piece of data can be sent without waiting for a response. The result of asynchronous sending data is uncertain, which leads to the need to find a better way to automatically reconnect the connection pool for asynchronous data sending.
[0039] Taking into account the above-mentioned shortcomings of the synchronous reconnection method in the asynchronous sending scenario, the inventor has proposed a connection pool automatic reconnection method and device based on asynchronous data sending after research. By presetting a statistical period, the data identification and sending time of each asynchronous data sent by the sender within this period are recorded, and the corresponding request response time of each responded data is calculated based on the response time and the corresponding sending time of the corresponding response data generated by each asynchronous data sending. The average request response time of the responded data is compared with the request response time threshold within a preset period. If the average request response time is greater than the request response time threshold within the preset period, all connections in the connection pool are disconnected and all connections are destroyed, and then reconnected.
[0040] Please combine Figure 1 and Figure 2 , Figure 1 A flowchart of a connection pool automatic reconnection method based on asynchronous data transmission provided by the present invention. Figure 2 For execution Figure 1 The automatic reconnection method described above is a structural block diagram of a connection pool automatic reconnection device based on asynchronous data transmission, wherein the connection pool automatic reconnection method based on asynchronous data transmission specifically includes the following steps:
[0041] S10: Acquire the data to be sent from the sender, connect to a channel in the connection pool and complete the transmission of the data to be sent, and record the current time as the sending time of the data to be sent. Step S10 is performed by the sending unit 10.
[0042] Connection pooling is a technology that creates and manages a pool of connections that are ready to be used by any thread that needs them. Connection pooling can greatly improve application performance while reducing resource usage. Connection pooling reduces the number and time of connection creation, and the connections in the pool are all ready and available; it limits the size of the pool, controls resource usage, and avoids high-load and time-consuming operations of frequently creating and destroying connections.
[0043] In the asynchronous transmission scenario, after sending a set of data, the sender will continue to send the next set of data, record the data identifier and sending time of each data transmission, and use it to match the corresponding transmission data when receiving the response data and calculate the request response time. The request response time refers to the time it takes from sending the data to receiving the response. Its value directly reflects the speed of the network connection.
[0044] S20: After the data transmission to be sent is completed, a sliding time window is used to record the data transmission volume per unit time, and the data identifier and transmission time corresponding to each piece of sent data are stored in a cache data table. Step S20 is performed by the data cache unit 20.
[0045] The sliding time window refers to a time window that slides at fixed intervals within a period of time, and is used to record the data traffic within a period of time. In this embodiment, 1 second is taken as a unit time, that is, the amount of data sent every second is counted, and the statistical information includes the total number of submissions, the total number of responses, and the total request response time within one second.
[0046] The cache data table is used by users to cache the data identifiers and sending times that have been sent but have not received a response. That is, it is used to store data information that has not received a response. An attribute of the sent data is used as the data identifier. In the response data, there must be a corresponding data identifier to confirm that the response data belongs to the sent data. In this way, the corresponding sent data can be found according to the response data.
[0047] S30: Obtain the response data of the receiving end and record the response time, find the corresponding sending time in the cache data table according to the data identifier of the response data, and obtain the request response time according to the difference between the sending time and the response time. Step S30 is performed by the request response duration calculation unit 30.
[0048] Calculate the request response time of each response data and the corresponding send data, and add up all request response times within a certain time range to get the total request response time within the time range.
[0049] S40: Obtain the total response amount of the data to be sent in the previous cycle and the corresponding total response time, calculate the average request response time, and if the average request response time is greater than the request response time threshold in the cycle, disconnect all connections in the connection pool and destroy all connections, wherein the cycle is a preset data statistics cycle. Step S40 is performed by the connection abnormality determination unit 40.
[0050] In this embodiment, 10 seconds is used as a statistical cycle, and the total amount of response data received for sending data within the first ten seconds is obtained. The total amount of response data received within the cycle is obtained by adding the amount of response data received in each sliding time window, and the total response time within the cycle is obtained by adding the response time of each window, so that:
[0051] Average request response time = total response time within a cycle / total number of response data received within a cycle
[0052] The average request response time reflects the speed of the network connection in the response data during the network transmission process, and to a certain extent reflects the network connection status and fluctuations.
[0053] S50: When the number of available connections in the connection pool is less than the number of allocated connections, the connection in the connection pool is reestablished to complete the reconnection.
[0054] In the case of asynchronous transmission, the specific situation of the connection is uncertain, and there is a half connection. At this time, the data will be lost or delayed after being sent. The newly established connection has just been successfully established, and there is basically no half connection at this time. And the newly established connection can confirm whether the current network is normal. Step S50 is performed by the connection reconnection unit 50.
[0055] See also Figure 3 In another preferred embodiment, if the average request response time is less than the request response time threshold within the period, it also includes:
[0056] S41: Skip one cycle from the current time, obtain the total amount of sent data in the second-to-last cycle and the total amount of data for which no response was received in the second-to-last cycle, calculate the proportion of unresponse data, and if the proportion of unresponse data is greater than the unresponse threshold in the pre-cycle, disconnect all connections in the connection pool and destroy all connections.
[0057] In this embodiment, the asynchronous connection has a timeout period. The current cycle data has not timed out yet, so it is not possible to determine how much sent data has not received a response. However, the previous cycle has timed out, so it is possible to determine how much data has not received a response. Therefore, the data sent in the second to last cycle is used as the data statistics source.
[0058] Considering the abnormal network connection, that is, some connections in the connection pool are abnormal and have been in an unresponsive state, abnormal connection and network jitter and packet loss may cause data loss. At this time, no response data is returned. Without response data, the request response time of this data will not be calculated. At this time, it is not enough to judge whether the connection is abnormal only by the request response time. Therefore, the proportion of unresponsive data is introduced to further judge and further ensure the normal connection status of the system.
[0059] In another preferred embodiment, the present application further includes step S60: searching the cache data table for data corresponding to a time obtained by subtracting a data timeout period from the current time, and removing the corresponding cache data.
[0060] For some data that has exceeded the preset data timeout period but has not received response data, it is identified as response failure data. Since this part of data has not received a submission response, the corresponding data in the cache data table will not be cleared. If it is not processed, then this part of data will occupy more and more memory space as the system runs, and the large amount of data will also affect the efficiency of the search. Therefore, this embodiment defines an independent data cleaning thread, and there is a timer task in the thread. The cache data table is checked once every 1 second to find the data corresponding to the time obtained by subtracting the data timeout period from the current time when the sending time is earlier than the current time, and the cache data found is removed from the cache data table, thereby reducing the data in the cache data table that has not responded for a long time and reducing the redundancy of the cache data table.
[0061] The present invention provides a connection pool automatic reconnection method based on asynchronous data transmission. On the basis of the existing connection pool, an automatic connection reconnection mechanism based on the processing result of business data is expanded. According to the request response time of all response data in a preset period of the sending of business data and the response data and the total amount of response data, the average request response time of the response data in this period is calculated; the average request response time is compared with the preset request response time threshold in the period to judge the fluency of the received response data in the process of establishing a network connection, and further determine whether the status of the connection channel on the connection pool meets the asynchronous transmission requirements. If not, the connection is disconnected and destroyed, and the connection pool re-establishes a new connection. Further, in order to avoid the situation that data loss cannot be discovered due to network abnormalities, the proportion of unresponsive data in a period is also calculated. If the proportion of unresponsive data is greater than the preset unresponsive threshold in the period, the connection is disconnected and destroyed, and the connection pool re-establishes a new connection. This method can automatically determine whether the connection in the connection pool has a network connection abnormality or a high network delay without manual intervention, and whether it is necessary to re-establish the connection. This reduces the data transmission failure rate caused by network fluctuations, enhances the robustness of the system, and eliminates the need to continuously monitor each connection, reducing the system load and improving detection efficiency.
[0062] An embodiment of the present application also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the method for automatically reconnecting a connection pool based on data asynchrony described in any one of the above embodiments is implemented.
[0063] The present invention may take the form of a computer program product implemented on one or more storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing program code. Computer-readable storage media include permanent and non-permanent, removable and non-removable media, and information storage can be achieved by any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include but are not limited to: phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission medium that can be used to store information that can be accessed by a computing device.
[0064] The above-mentioned embodiments only express several implementation methods of the present invention, and the description is relatively specific and detailed, but it cannot be understood as limiting the scope of the invention patent. It should be pointed out that for ordinary technicians in this field, several modifications and improvements can be made without departing from the concept of the present invention, and the present invention is also intended to include these modifications and modifications.
Claims
1. A connection pool automatic reconnection method based on data asynchrony, characterized in that: include: Obtain the data to be sent from the sender, connect to a channel in the connection pool and complete the transmission of the data to be sent, and record the current time as the sending time of the data to be sent; After the data transmission is completed, a sliding time window is used to record the data transmission volume per unit time, and the data identifier and transmission time corresponding to each piece of transmitted data are stored in a cache data table; Obtain response data from the receiving end and record the response time, find the sending time of the corresponding sending data in the cache data table according to the data identifier of the response data, and obtain the request response time according to the difference between the sending time and the response time; Get the total response volume of data to be sent and the corresponding total response time in the previous cycle, calculate the average request response time, and if the average request response time is greater than the request response time threshold in the cycle, disconnect all connections in the connection pool and destroy all connections, where the cycle is the preset data statistics cycle; When the number of available connections in the connection pool is less than the number of allocated connections, the connection pool connection is reestablished to complete the reconnection.
2. According to the method of automatic reconnection of connection pool based on data asynchrony in claim 1, it is characterized in that: If the average request response time is less than the request response time threshold within the period, the following detection is performed; Skip one cycle from the current time, obtain the total amount of sent data in the second-to-last cycle and the total amount of data for which no response was received in the second-to-last cycle, calculate the proportion of unresponse data, and if the proportion of unresponse data is greater than the unresponse threshold in the pre-cycle, disconnect all connections in the connection pool and destroy all connections.
3. According to the method of automatic reconnection of connection pool based on data asynchrony in claim 2, it is characterized in that: Also includes: Search the cache data table for data corresponding to a time obtained by subtracting a data timeout period from a current time, and remove the corresponding cache data.
4. The method for automatic reconnection of a connection pool based on data asynchrony according to claim 3, characterized in that: If data transmission fails during the process of connecting to a channel in the connection pool and completing the transmission of the data to be sent, the currently used connection is disconnected, the corresponding connection in the connection pool is destroyed, and a new connection is established to send the data to be sent.
5. A connection pool automatic reconnection device based on asynchronous data transmission, characterized in that: include: The sending unit is used to obtain the data to be sent from the sending end, connect to a channel in the connection pool and complete the transmission of the data to be sent, and record the current time as the sending time of the data to be sent; Data cache unit: used to record the amount of data sent per unit time using a sliding time window after the data to be sent is completed, and store the data identifier and sending time corresponding to each sent data in a cache data table; Request response time calculation unit: used to obtain the response data of the receiving end and record the response time, find the sending time of the corresponding sending data in the cache data table according to the data identifier of the response data, and obtain the request response time according to the difference between the sending time and the response time; Connection abnormality determination unit: used to obtain the total response amount of data to be sent in the previous cycle and the corresponding total response time, calculate the average request response time, and if the average request response time is greater than the request response time threshold in the cycle, disconnect all connections in the connection pool and destroy all connections, where the cycle is a preset data statistics cycle; Connection reconnection unit: used to reestablish the connection in the connection pool and complete the reconnection when the number of available connections in the connection pool is less than the number of allocated connections.
6. The connection pool automatic reconnection device based on asynchronous data transmission according to claim 5, characterized in that: The connection abnormality determination unit further includes: Unresponsive exception determination subunit: used to skip one cycle from the current time, obtain the total amount of sent data in the second-to-last cycle and the total amount of data for which no response was received in the second-to-last cycle, calculate the proportion of unresponsive data, and if the proportion of unresponsive data is greater than the unresponsive threshold in the pre-cycle, disconnect all connections in the connection pool and destroy all connections.
7. The automatic reconnection device for connection pool based on asynchronous data transmission according to claim 6, characterized in that Also includes: Data cleaning unit: used to search the cache data table for data corresponding to a time obtained by subtracting a data timeout period from the current time, and remove the corresponding cache data.
8. The connection pool automatic reconnection device based on asynchronous data transmission according to claim 7, characterized in that: The sending unit also includes: Send abnormal reconnection subunit: used to disconnect the currently used connection, destroy the corresponding connection in the connection pool, and re-establish the connection to send the data to be sent if data transmission fails during the process of connecting to a channel in the connection pool and completing the transmission of the data to be sent.
9. An electronic device, characterized in that: include: at least one memory and at least one processor; The memory is used to store one or more programs; When the one or more programs are executed by the at least one processor, the at least one processor implements the steps of a connection pool automatic reconnection method based on data asynchrony as described in any one of claims 1 to 4.
10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of a connection pool automatic reconnection method based on data asynchrony as described in any one of claims 1 to 4 are implemented.
