Data interaction method and system based on complex message queue and TCP protocol

Abstract

The invention provides a data interaction method and system based on a complex message queue and a TCP protocol, and the method comprises the steps: building a queue based on the type of to-be-sent message data, and carrying out the combination and queue recombination of different message data types; establishing a TCP transmission connection based on the message data to be sent, sending the message data after the TCP connection is established, and then waiting for a response; triggering an automatic retransmission mechanism according to the current state of the message data if the message data fails to be sent or the message data is successfully sent but no response is received within the agreed time, establishing a new TCP transmission connection, resending the message data and destroying the old TCP transmission connection; after the message data is successfully sent and a response is received, the corresponding TCP connection is destroyed, and memory resources are released; and recording all states of the message data in the whole transmission process. According to the invention, the problems of network congestion, delay increase, packet loss and low transmission performance during processing of complex data flow and high-concurrency environment data interaction can be effectively solved.

Description

Technical Field

[0001] The present invention belongs to the technical field of data interaction transmission, and in particular relates to a data interaction method and system based on a complex message queue and a TCP protocol. Background Art

[0002] In existing data interaction systems, packet loss is a common problem, especially in unstable network environments. In the prior art, the following methods are usually used to solve the packet loss problem in data transmission: retransmission mechanism, when the sender does not receive a confirmation response within a specified time, it resends the data packet until a confirmation response is received; this method is suitable for small-scale data transmission, but it will cause network congestion and increased delays in large-scale data transmission. Verification mechanism, add a check code to the data packet, the receiver verifies the received data, and requests retransmission if the verification fails; this method can improve the accuracy of the data, but cannot completely solve the packet loss problem. Reliable transmission protocols, such as the TCP protocol, establish a connection through a three-way handshake to ensure the reliability of data transmission; however, the performance of the TCP protocol degrades in a high-concurrency environment and cannot meet real-time requirements.

[0003] Therefore, existing traditional solutions have limited effectiveness in processing complex data streams and data interactions in high-concurrency environments, and there are problems such as network congestion, increased latency, packet loss, and low transmission performance. Summary of the invention

[0004] In view of the shortcomings of the prior art, the present invention proposes a data interaction method based on complex message queues and TCP protocol, which can realize efficient and reliable data transmission in complex data streams and high concurrency environments. The present invention is specifically implemented through the following technical solutions: On the one hand, the present invention provides a data interaction method based on a complex message queue and a TCP protocol, which specifically comprises the following steps: Step S1: Establish a queue based on the type of message data to be sent, combine different message data types and reorganize the queue to accurately control the sending timing of the message; Step S2: Establish a TCP transmission connection based on the message data to be sent, send the message data after the TCP connection is established, and then wait for a response; Step S3: If the message data fails to be sent or the message data is sent successfully but no response is received within the agreed time, an automatic retransmission mechanism is triggered according to the current state of the message data, a new TCP transmission connection is created to resend the message data and the old TCP transmission connection is destroyed; Step S4: If the message data is sent successfully and a response is received, the message transmission is successful, and the corresponding TCP connection is destroyed to release memory resources; Step S5: Based on steps S1 to S4, record all states of the message data during the transmission process.

[0005] Furthermore, step S1 also includes the following steps: Step S101: based on the type of message data, dynamically create and expand the corresponding message queue, inject the same type of message data into the same message queue, and check the queue length and status to monitor the queue situation and prevent message congestion; Step S102: Based on the message data in the same message queue, dynamically adjust the message sending order and priority to facilitate controlling the message sending timing.

[0006] Furthermore, in step S2, the establishment of the TCP transmission connection is specifically: a TCP transmission loop is established for each message data, and each message immediately destroys its corresponding TCP connection after the transmission is completed, releasing memory resources, avoiding resource waste and security risks caused by maintaining the connection for a long time.

[0007] Furthermore, in step S5, all states of the message data in the transmission process are recorded, including sending, sending, sent, waiting for response and received response, wherein: the received response also includes received positive response and received negative response; the positive response indicates that the message data is successfully received by the other party and the message data is correct and can be parsed; the negative response indicates that the message data is successfully received by the other party, but the type of the message data or the message data format does not meet the predetermined requirements and cannot be parsed. In this case, the corresponding message data is discarded and an error log is recorded.

[0008] On the one hand, the present invention also provides a data interaction system based on a complex message queue and a TCP protocol, the system comprising a message queue management module, a TCP transmission module and a message log management module, wherein: The message queue management module acts on the message entry, exit and queue monitoring to ensure the reliable transmission of messages in the queue; The TCP transmission module is used to establish and destroy TCP connections, so that each message is processed independently to ensure the reliability of the data transmission channel; The message log management module is used to record the status and processing process of each message in real time, update the status, archive and store, and regularly clean up the data, ensuring the integrity and accuracy of the log information and the traceability of historical data.

[0009] Furthermore, in the message queue management module, the message entry, exit and queue monitoring specifically include: Whenever a new message arrives, it is added to the message queue, and the queue status is monitored by regularly checking the queue length and status to prevent message congestion. Messages of the same type are injected into the same queue and strong timing control is maintained, so that messages of different types do not affect each other after entering the queue; Automatically expand the queue according to the type of message. When the system obtains different types of message processing requirements, it can dynamically create and expand the corresponding message queue to ensure efficient processing of different types of messages; By combining message types and reorganizing queues, the system can accurately control the timing of message sending, flexibly adjust the order and priority of message sending according to specific application scenarios and needs, and improve the efficiency and reliability of data transmission.

[0010] Furthermore, in the TCP transmission module, establishing and destroying a TCP connection specifically includes: Before sending a message, the TCP transmission module first establishes a TCP connection to ensure the reliability of the data transmission channel; A loop is established for each message, and the corresponding TCP connection is destroyed immediately after each message is transmitted, releasing system resources and avoiding resource waste and security risks caused by maintaining the connection for a long time; When a message fails to be sent or no response is received, the system triggers a retransmission mechanism based on the current status to ensure that the message is eventually transmitted successfully.

[0011] Furthermore, in the message log management module, the status and processing of each message are recorded in real time, the status is updated, archived and stored, and the data is cleaned up regularly, specifically including: The status information of each message during the queue entry and transmission process will be recorded in real time to ensure that every step of message processing can be accurately tracked and recorded; When the system receives a message, it immediately records its status information and updates the log in real time when the status information changes, ensuring the integrity and accuracy of the log information; When a message fails to be sent or no response is received, the system will decide whether to resend the message based on the current status through the retransmission strategy. If the message is to be resent, the automatic retransmission mechanism or manual retransmission will be activated to ensure that the message can be successfully transmitted in the end; Archive messages regularly and clean up expired data to keep the system running efficiently. Archived messages serve as historical records for subsequent data analysis and auditing.

[0012] Compared with the prior art, the present invention has the following advantages: 1. The present invention utilizes a message queue mechanism, establishes queues according to message data types and combines and reorganizes them, accurately controls the sending timing of message data, flexibly adjusts the sending order and priority, and improves the efficiency of data transmission. At the same time, it combines the response mechanism and the retransmission mechanism to ensure that data can be transmitted successfully during the data transmission process through response and retransmission, effectively avoiding the loss of data packets.

[0013] 2. The present invention uses the TCP transmission module to establish an independent TCP connection for each message, and immediately destroys the connection after completing data transmission and response processing. This design enables the system to frequently establish and destroy TCP connections, adapt to high-frequency data interaction needs, and avoid resource waste and security risks caused by maintaining connections for a long time.

[0014] Other features and advantages of the present invention will be described in the following description, and partly become apparent from the description, or understood by practicing the present invention. The purpose and other advantages of the present invention can be realized and obtained by the structures pointed out in the description, claims and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0016] Figure 1 A schematic flow chart of a data interaction method based on a complex message queue and TCP protocol is shown.

[0017] Figure 2 Another flow chart of a data interaction method based on a complex message queue and TCP protocol is shown.

[0018] Figure 3 The data interaction system architecture diagram based on complex message queues and TCP protocol is shown. DETAILED DESCRIPTION

[0019] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0020] In one embodiment, see Figure 1 and Figure 2 , provides a data interaction method based on complex message queue and TCP protocol, which specifically includes the following steps: Step S1: Establish a queue based on the type of message data to be sent, combine different message data types and reorganize the queue to accurately control the sending timing of the message; Furthermore, step S1 also includes the following steps: Step S101: based on the type of message data, dynamically create and expand the corresponding message queue, inject the same type of message data into the same message queue, and check the queue length and status to monitor the queue situation and prevent message congestion; Step S102: Based on the message data in the same message queue, dynamically adjust the message sending order and priority to facilitate controlling the message sending timing.

[0021] Step S2: Establish a TCP transmission connection based on the message data to be sent, send the message data after the TCP connection is established, and then wait for a response; Furthermore, in step S2, the establishment of the TCP transmission connection is specifically: a TCP transmission loop is established for each message data, and each message immediately destroys its corresponding TCP connection after the transmission is completed, releasing memory resources, avoiding resource waste and security risks caused by maintaining the connection for a long time.

[0022] Step S3: If the message data fails to be sent or the message data is sent successfully but no response is received within the agreed time, an automatic retransmission mechanism is triggered according to the current state of the message data, a new TCP transmission connection is created to resend the message data and the old TCP transmission connection is destroyed; Step S4: If the message data is sent successfully and a response is received, the message transmission is successful, and the corresponding TCP connection is destroyed to release memory resources; Step S5: Based on steps S1 to S4, record all states of the message data during the transmission process.

[0023] Furthermore, the record includes all states of the message data in the transmission process, including sending, sending, sent, waiting for response and received response, wherein: the received response also includes a positive response received and a negative response received; the positive response indicates that the message data is successfully received by the other party and the message data is correct and can be parsed; the negative response indicates that the message data is successfully received by the other party, but the type or format of the message data does not meet the predetermined requirements and cannot be parsed. In this case, the corresponding message data is discarded and an error log is recorded.

[0024] In one embodiment, see Figure 3, provides a data interaction system based on complex message queue and TCP protocol, the system includes a message queue management module, a TCP transmission module and a message log management module; The message queue management module acts on the message entry, exit and queue monitoring to ensure the reliable transmission of messages in the queue; Furthermore, in the message queue management module, the message entry, exit and queue monitoring specifically include: Whenever a new message arrives, it is added to the message queue, and the queue status is monitored by regularly checking the queue length and status to prevent message congestion. Messages of the same type are injected into the same queue and strong timing control is maintained, so that messages of different types do not affect each other after entering the queue; Automatically expand the queue according to the type of message. When the system obtains different types of message processing requirements, it can dynamically create and expand the corresponding message queue to ensure efficient processing of different types of messages; By combining message types and reorganizing queues, the system can accurately control the timing of message sending, flexibly adjust the order and priority of message sending according to specific application scenarios and needs, and improve the efficiency and reliability of data transmission.

[0025] The TCP transmission module is used to establish and destroy TCP connections, so that each message is processed independently to ensure the reliability of the data transmission channel; Furthermore, in the TCP transmission module, establishing and destroying a TCP connection specifically includes: Before sending a message, the TCP transmission module first establishes a TCP connection to ensure the reliability of the data transmission channel; A loop is established for each message, and the corresponding TCP connection is destroyed immediately after each message is transmitted, releasing system resources and avoiding resource waste and security risks caused by maintaining the connection for a long time; When a message fails to be sent or no response is received, the system triggers a retransmission mechanism based on the current status to ensure that the message is eventually transmitted successfully.

[0026] The message log management module is used to record the status and processing process of each message in real time, update the status, archive and store, and regularly clean up the data, ensuring the integrity and accuracy of the log information and the traceability of historical data.

[0027] Furthermore, in the message log management module, the status and processing of each message are recorded in real time, the status is updated, archived and stored, and the data is cleaned up regularly, including: The status information of each message during the queue entry and transmission process will be recorded in real time to ensure that every step of message processing can be accurately tracked and recorded; When the system receives a message, it immediately records its status information and updates the log in real time when the status information changes, ensuring the integrity and accuracy of the log information; When a message fails to be sent or no response is received, the system will decide whether to resend the message based on the current status through the retransmission strategy. If the message is to be resent, the automatic retransmission mechanism or manual retransmission will be activated to ensure that the message can be successfully transmitted in the end; Archive messages regularly and clean up expired data to keep the system running efficiently. Archived messages serve as historical records for subsequent data analysis and auditing.

[0028] Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent substitutions for some of the technical features therein; and these modifications or substitutions do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A data interaction method based on complex message queue and TCP protocol, characterized in that: The method comprises: Establish queues based on the type of message data to be sent, combine different message data types and reorganize queues to accurately control the sending sequence of messages; Establish a TCP transmission connection based on the message data to be sent, send the message data after the TCP connection is established, and then wait for a response; If the message data fails to be sent or the message data is sent successfully but no response is received within the agreed time, the automatic retransmission mechanism is triggered according to the current status of the message data, a new TCP transport connection is created to resend the message data and the old TCP transport connection is destroyed; If the message data is sent successfully and a response is received, the message transmission is successful, and the corresponding TCP connection is destroyed to release memory resources; Record all states of message data during the entire transmission process.

2. A data interaction method based on complex message queue and TCP protocol as claimed in claim 1, characterized in that: The method of establishing a queue based on the type of the message data to be sent, combining different message data types and reorganizing the queues, further includes: Based on the type of message data, dynamically create and expand the corresponding message queue, inject the same type of message data into the same message queue, and check the queue length and status to monitor the queue status and prevent message congestion; Based on the message data in the same message queue, the message sending order and priority are dynamically adjusted to facilitate the control of the message sending timing.

3. A data interaction method based on complex message queue and TCP protocol as claimed in claim 2, characterized in that: The dynamic adjustment of the message sending order and priority is specifically to ensure that high-priority messages can be sent first by combining and reorganizing queues, while low-priority messages can be processed later.

4. A data interaction method based on complex message queue and TCP protocol as claimed in claim 1, characterized in that: The TCP transmission connection is established based on the message data to be sent, specifically: A TCP transmission loop is established for each message data. After each message is transmitted, its corresponding TCP connection is destroyed immediately to release memory resources.

5. A data interaction method based on complex message queue and TCP protocol as claimed in claim 1, characterized in that: The recorded message data has all states in the transmission process, including sending, sending, sent, waiting for response and received response states, wherein the received response state also includes received positive response and received negative response.

6. A data interaction method based on complex message queue and TCP protocol as claimed in claim 5, characterized in that: The positive response means that the corresponding message data is successfully received by the other party, and the message data is correct and can be parsed; The negative response means that the corresponding message is successfully received by the other party, but the type or format of the message data does not meet the predetermined requirements and cannot be parsed. In this case, the corresponding message data needs to be discarded and an error log is recorded.

7. A data interaction system based on complex message queue and TCP protocol, used for implementing a data interaction method based on complex message queue and TCP protocol as claimed in any one of claims 1 to 6, characterized in that: It includes a message queue management module, a TCP transmission module and a message log management module, among which: The message queue management module acts on the message entry, exit and queue monitoring to ensure the reliable transmission of messages in the queue; The TCP transmission module is used to establish and destroy TCP connections, so that each message is processed independently to ensure the reliability of the data transmission channel; The message log management module is used to record the status and processing process of each message in real time, update the status, archive and store, and regularly clean up the data, ensuring the integrity and accuracy of the log information and the traceability of historical data.

8. A data interaction system based on complex message queue and TCP protocol as claimed in claim 7, characterized in that: In the message queue management module, the message entry, exit and queue monitoring specifically include: Whenever a new message is sent, it is added to the message queue, and the queue status is monitored by regularly checking the queue length and status to prevent message congestion; Automatically expand the queue according to the type of message. When the system obtains different types of message processing requirements, it can dynamically create and expand the corresponding message queue to ensure efficient processing of different types of messages; By combining message types and reorganizing queues, the system can accurately control the timing of message sending, flexibly adjust the order and priority of message sending according to specific application scenarios and needs, and improve the efficiency and reliability of data transmission.

9. A data interaction system based on complex message queue and TCP protocol as claimed in claim 7, characterized in that: In the TCP transmission module, establishing and destroying a TCP connection specifically includes: Before sending a message, the TCP transmission module first establishes a TCP connection to ensure the reliability of the data transmission channel; A loop is established for each message, and the corresponding TCP connection is destroyed immediately after each message is transmitted, releasing system resources and avoiding resource waste and security risks caused by maintaining the connection for a long time; When a message fails to be sent or no response is received, the system triggers a retransmission mechanism based on the current status to ensure that the message is eventually transmitted successfully.

10. A data interaction system based on complex message queue and TCP protocol as claimed in claim 7, characterized in that: In the message log management module, the status and processing of each message are recorded in real time, updated, archived and stored, and data is cleaned up regularly, including: The status information of each message during the queue entry and transmission process will be recorded in real time to ensure that every step of message processing can be accurately tracked and recorded; When the system receives a message, it immediately records its status information and updates the log in real time when the status information changes, ensuring the integrity and accuracy of the log information; When a message fails to be sent or no response is received, the system will decide whether to resend the message based on the current status through the retransmission strategy. If the message is to be resent, the automatic retransmission mechanism or manual retransmission will be activated to ensure that the message can be successfully transmitted in the end; Archive messages regularly and clean up expired data to keep the system running efficiently. Archived messages serve as historical records for subsequent data analysis and auditing.

Images