Data transmission method, data distribution system, device, storage medium and program product

Abstract

The embodiment of the invention provides a data transmission method, a data distribution system, equipment, a storage medium and a program product. In the data transmission method, when a sending end sends a log data packet, a log serial number of the log data packet can be sent together. After a receiving end receives a log data packet, whether the log data packet is a continuous data packet or not can be judged based on the log serial number of the log data packet and the log serial number of the received previous log data packet. Wherein the log sequence number of each log data packet is composed of a stream sequence identifier of the stream transmission connection to which the log data packet belongs and a local sequence identifier of the log data packet in the stream transmission connection. The flow sequence identification can be used for accurately sequencing the flow transmission connection so as to cope with the situation that the flow transmission connection is restarted; according to the technical scheme, the stream sequence identifier and the local sequence identifier can be used for accurately sequencing the log data packets in the stream transmission connection, so that whether the received data packets are continuous or not can be accurately judged even if the stream transmission connection is restarted, and the risks of packet loss and disorder of the data packets are reduced.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer, in particular to a data transmission method, a data distribution system, a device, a storage medium and a program product. BACKGROUND

[0002] With the deep development of the big data era, the collection, transmission, storage and consumption of time series data have become increasingly important. Such time series data is arranged in time order and covers, for example, stock price fluctuation data in the financial service industry, monitoring data of industrial equipment sensors in the industrial automation industry, and state parameters of Internet of Things devices in the intelligent transportation industry. In the above-mentioned various industries, the accuracy and reliability of time series data are the cornerstone of the stable operation of each industry. Therefore, how to ensure the integrity and consistency of time series data in the processing process is particularly crucial for the above-mentioned many industry scenarios.

[0003] The existing method for processing time series data cannot strongly guarantee the integrity and consistency of data. Therefore, a new solution needs to be proposed. SUMMARY

[0004] Aspects of the present application provide a data transmission method, a data distribution system, a device, a storage medium and a program product to improve the data integrity and consistency in the data transmission process.

[0005] The embodiment of the present application provides a data transmission method, which is suitable for a receiving end and includes the following steps: acquiring a log data packet sent through a streaming connection and a first log sequence number corresponding to the log data packet, the first log sequence number including a first stream sequence identifier of the streaming connection and a first local sequence identifier of the log data packet in the streaming connection; acquiring a second log sequence number of a previous log data packet that has been received from receiving progress information of the streaming connection, the second log sequence number including a second stream sequence identifier of a streaming connection to which the previous log data packet belongs and a second local sequence identifier of the previous log data packet in the streaming connection; judging whether the log data packet is a continuous data packet according to the first log sequence number and the second log sequence number; and if yes, storing the log data packet.

[0006] Optionally, judging whether the log data packet is a continuous data packet according to the first log sequence number and the second log sequence number includes: judging whether the first stream sequence identifier is equal to the second stream sequence identifier; if yes, judging whether the first local sequence identifier and the second local sequence identifier are continuous; and if yes, determining that the log data packet is a continuous data packet relative to the previous log data packet.

[0007] Optionally, it further includes: if the value of the first stream sequence identifier is greater than the value of the second stream sequence identifier, then determining whether the value of the first local sequence identifier is equal to the preset starting value of the local sequence identifier; if yes, then determining that the log data packet is a continuous data packet relative to the previous log data packet.

[0008] Optionally, after storing the log data packet, the method further includes: sending the log data packet to a replica storage terminal for archiving, so as to store the log data packet in multiple replicas; updating the archiving progress information corresponding to the streaming connection according to the first log sequence number of the log data packet; the archiving progress information corresponding to the streaming connection is used to restore the archiving progress after the streaming connection is disconnected and restarted.

[0009] Optionally, the log data packets are sent to a replica storage terminal for archiving to perform multi-replica storage of the log data packets, including: adding the log data packets to an asynchronous archiving queue; and using the archiving process, under the triggering of a set archiving event, sending the log data packets in the asynchronous archiving queue to the replica storage terminal for persistent storage.

[0010] Optionally, sending the log data packets to be archived in the asynchronous archiving queue to the replica storage terminal for persistent storage includes: in the replica storage terminal, determining the first file object that has been opened corresponding to the first streaming connection; obtaining the remaining writable data volume of the first file object; determining whether the data volume of the log data packets to be archived is greater than or equal to the remaining writable data volume of the first file object; if yes, then writing the log data packets to be archived into the first file object; if no, then opening a second file object in the replica storage terminal and writing the log data to be archived into the second file object.

[0011] Optionally, obtaining the remaining writable data amount of the first file object includes: obtaining the offset of the first file object from the metadata information of the streaming connection; determining the remaining writable data amount of the first file object based on the offset of the first file object; wherein the offset of the first file object is updated to the metadata information by the archiving process after each write operation on the first file object.

[0012] Optionally, before obtaining the log data packets sent by the sending end through the streaming connection and the first log sequence number corresponding to the log data packets, the method further includes: responding to the connection establishment request corresponding to the streaming connection initiated by the sending end, determining whether the streaming connection is a streaming connection that has been restarted after being disconnected; if so, obtaining the receiving progress information and archiving progress information of the streaming connection based on the identification identifier of the streaming connection; obtaining the target log data that was received but not archived before the streaming connection was disconnected from the sending end based on the receiving progress information and the archiving progress information; sending the target log data to the replica storage end for archiving, and updating the archiving progress information of the streaming connection; and returning a lease establishment success message to the sending end so that the sending end can send log data packets through the streaming connection.

[0013] Optionally, determining whether the streaming connection is a streaming connection that has been disconnected and then restarted includes: determining the metadata information of the streaming connection based on the identification identifier of the streaming connection; determining whether the metadata information of the streaming connection contains the identifier of the receiver that the streaming connection has registered; if so, determining that the streaming connection is a streaming connection that has been disconnected and then restarted.

[0014] This application also provides a data transmission method applicable to a sending end, comprising: acquiring a log data packet to be transmitted; and sending the log data packet and a first log sequence number corresponding to the log data packet to a receiving end via a streaming connection, so that the receiving end receives the log data packet when it determines that the log data packet is a consecutive data packet based on the first log sequence number and a second log sequence number of a previously received log data packet; the first log sequence number includes: a first stream sequence identifier of the streaming connection and a first local sequence identifier of the log data packet in the streaming connection; the second log sequence number includes: a second stream sequence identifier of the streaming connection to which the previous log data packet belongs and a second local sequence identifier of the previous log data packet in the streaming connection to which it belongs.

[0015] Optionally, after sending the log data packet and the first log sequence number corresponding to the log data packet to the receiving end via the streaming connection, the method further includes: updating the sending progress information of the streaming connection according to the first stream sequence identifier and the first local sequence identifier; repeatedly performing the operation of sending the log data packet and the first log sequence number to the receiving end; obtaining the receiving progress information of the streaming connection returned by the receiving end; if the receiving progress information is the same as the sending progress information, then determining that the log data packet was successfully sent, and stopping the repeated sending of the log data packet and the first log sequence number.

[0016] This application embodiment also provides a data distribution system, including: at least one sending end and at least one receiving end; wherein, the first sending end among the at least one sending end is configured to: obtain a log data packet to be transmitted based on the time-series data output by the data production equipment; and send the log data packet and a first log sequence number corresponding to the log data packet to the first receiving end among the at least one receiving end via a streaming connection; the first log sequence number includes: a first stream sequence identifier of the streaming connection and a first local sequence identifier of the log data packet in the streaming connection; wherein, the first receiving end is configured to: obtain the log data packet and the first log sequence number; obtain a second log sequence number of a previously received log data packet from the reception progress information of the streaming connection, the second log sequence number including: a second stream sequence identifier of the streaming connection to which the previous log data packet belongs and a second local sequence identifier of the previous log data packet in the streaming connection to which it belongs; determine whether the log data packet is a continuous data packet based on the first log sequence number and the second log sequence number; if so, store the log data packet and distribute the log data packet to the data consumer end corresponding to the streaming connection.

[0017] This application also provides an electronic device, including: a memory and a processor; the memory is used to store one or more computer instructions; the processor is used to execute the one or more computer instructions to perform the steps in the method provided in this application.

[0018] This application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, can implement the steps in the method provided in this application.

[0019] This application also provides a computer program product, including: a computer program / instructions, which, when executed by a processor, can implement the steps in the method provided in this application.

[0020] In the data transmission method provided in this application embodiment, when the sending end sends log data packets to the receiving end, it can also send the first log sequence number of the log data packets. After receiving the log data packets, the receiving end can determine whether the log data packets are consecutive based on the first log sequence number of the log data packets and the second log sequence number of the previously received log data packets. The log sequence number of each log data packet is composed of the stream sequence identifier of the streaming connection to which the log data packet belongs and the local sequence identifier of the log data packet within the streaming connection. The stream sequence identifier can be used to accurately order the streaming connection to cope with the situation of streaming connection restart; the stream sequence identifier and the local sequence identifier can be used to accurately order the log data packets within the streaming connection. Therefore, even if the streaming connection restarts, the receiving end can accurately determine whether the received data packets are consecutive based on the sequence identifier and the local sequence identifier, reducing the risk of packet loss and out-of-order data packets, thereby improving data integrity and consistency during data transmission. Attached Figure Description

[0021] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0022] Figure 1 A schematic diagram of multiple partitions of a message topic in a streaming message processing system;

[0023] Figure 2 A schematic diagram of the structure of a data distribution system provided in an exemplary embodiment of this application;

[0024] Figure 3 A schematic diagram of the data transmission process of a data distribution system provided in an exemplary embodiment of this application;

[0025] Figure 4 A flowchart illustrating the data transmission method provided in an exemplary embodiment of this application when executed at the sending end;

[0026] Figure 5 A flowchart illustrating the data transmission method provided in an exemplary embodiment of this application when executed at the receiving end;

[0027] Figure 6 A schematic diagram of the structure of an electronic device provided for an exemplary embodiment of this application. Detailed Implementation

[0028] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0029] The terminology used in the embodiments of this invention is for the purpose of describing particular embodiments only and is not intended to limit the invention. The singular forms “a,” “the,” and “the” used in the embodiments of this invention and the appended claims are also intended to include the plural forms, unless the context clearly indicates otherwise. “Multiple” generally includes at least two, but does not exclude the inclusion of at least one.

[0030] It should be understood that the term "and / or" used in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, the character " / " in this article generally indicates that the preceding and following related objects have an "or" relationship.

[0031] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a product or system comprising a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a product or system. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the product or system that includes said element.

[0032] For streaming data processing frameworks, dealing with time-series data in streaming environments presents various challenges in the transmission and storage stages. For example, data in streaming data processing frameworks may be corrupted or out of order during transmission, or data packets may be lost or retransmitted due to network failures. These challenges make it difficult for streaming data processing frameworks to meet the requirements for ensuring the integrity and consistency of time-series data.

[0033] In streaming data processing scenarios, semantics such as At-Least Once, At-MostOnce, and Exactly Once are used to describe the guarantee levels of data integrity and the number of data processing operations.

[0034] "At least once" means that each piece of data is processed at least once. If data processing fails, the data processing system will initiate a retry. This retry operation ensures that data is not lost, but there is a possibility of duplicate processing, thus data consistency cannot be guaranteed.

[0035] "At most once" means that each piece of data will be processed at most once. If data is lost during transmission, the data processing system will not retry. This operation allows for data loss and therefore cannot guarantee data integrity.

[0036] The "exactly once" semantic means that each piece of data is guaranteed to be processed exactly once, avoiding both data loss and data duplication. This semantic provides the highest level of data processing assurance, ensuring the accuracy and consistency of the system.

[0037] Of the three semantics mentioned above, the "exactly once" semantic offers the highest level of guarantee for data consistency and integrity. However, in distributed storage systems, the large number and wide distribution of components, along with issues such as node failures, network latency, and network partitioning, increase the complexity of implementing the "exactly once" semantic. Furthermore, unstable network environments further exacerbate the uncertainty during data transmission, making ensuring data integrity and consistency a challenge.

[0038] In an existing streaming message processing system, message integrity and consistency are maintained through multiple replicas and commit semantics during message transmission. In this system, producers (or production nodes) are responsible for sending messages to designated topics, while consumers subscribe to specific topics of interest to retrieve and process messages within those topics. A message topic is an abstract container for messages; it does not directly store messages but rather uses an internal partitioning structure to actually store the message data.

[0039] In this context, multiple replicas refer to dividing each message topic into multiple partitions, each of which can have one or more replicas distributed across different servers. For example... Figure 1As shown, Topic 1 can be divided into Partition 0, Partition 1, and Partition 2. Each partition has one leader node and multiple follower nodes. Within each partition, the leader node is responsible for handling all read and write requests and synchronizing data changes to the other follower nodes. Follower nodes acquire the data synchronized by the leader node and store it as a copy. Followers do not directly serve client requests but instead perform failover when the leader node fails. That is, when the server hosting the leader node fails, a follower node can switch to become the leader node and provide services to clients based on the data copy stored by the leader node, thus ensuring data availability.

[0040] The commit mechanism refers to the process where data written to the streaming data processing system is typically first written to a log and then distributed to consumers. Each message in the log has an offset, which represents the message's position within the log. In each partition, the leader and follower nodes maintain a high-watermark (HW), indicating the highest safe-to-read offset within a partition. Only messages with offsets less than or equal to the high-watermark are considered replicated and visible to consumers. Consumers can pull messages from the log based on the HW maintained by the leader node. After pulling and processing messages, the consumer returns a consumption progress, represented by the offsets of messages successfully processed. These offsets can be committed to the `consumer_offsets` topic. The streaming message processing system can then persistently store these offsets in the `consumer_offsets` topic and replicate them to all leader and follower nodes. Therefore, when the consumer restarts after a failure, it can resume consuming message data from the last committed offset to ensure that no messages are lost.

[0041] Furthermore, in this streaming data processing system, data reliability can be guaranteed through a replication protocol. In any partition, after the leader node receives a message from the production device, it writes the message to its local log and waits for follower nodes to confirm that the message has been replicated. Once a majority of replicas in the ISR (In-Sync Replicas, i.e., the set of follower nodes synchronized with the leader node) confirm that the message has been replicated, the message is considered successfully replicated. This replication protocol ensures that in the event of a leader node failure, a sufficient number of follower nodes have replicated the message, allowing for the election of a new leader node to continue serving clients without data loss. However, during the process of follower nodes replicating messages from the leader node, inconsistencies may occur between the messages in the leader node and the replicators due to factors such as network latency or replication time. That is, the consumer can read the latest message from the leader node, but when the leader node fails and a follower node is elected as the new leader node, the consumer may not be able to obtain the latest message from the new leader node.

[0042] Furthermore, within any partition, write requests (sending messages from production devices) and read requests (retrieving messages and committing offsets from consumers) require interaction with the partition's leader node. In high-concurrency write and read scenarios, a large number of network requests and data processing burdens are concentrated on the leader node. The leader node needs to handle a large number of I / O (Input / Output) operations, network communication, and related metadata management and coordination. In this situation, the leader node becomes the performance bottleneck of the streaming message processing system, limiting the overall throughput and response speed of the system.

[0043] To address the aforementioned technical problems, a solution is provided in some embodiments of this application. The technical solutions provided by each embodiment of this application are described in detail below with reference to the accompanying drawings.

[0044] Figure 2 This is a schematic diagram of the structure of a data distribution system provided in an exemplary embodiment of this application, as shown below. Figure 2 As shown, the data distribution system 100 mainly includes: at least one sending end 10, at least one receiving end 20, and a replica storage end 30. The at least one sending end 10 may include one or more sending ends, such as... Figure 2 The first transmitting end 101 and the second transmitting end 102 are shown. At least one receiving end 20 may include one or more receiving ends, such as... Figure 2 The first receiver 201 and the second receiver 202 are shown. It should be understood that... Figure 2 The number of transmitters and receivers shown is for illustrative purposes only and does not constitute a limitation on the number of transmitters and receivers.

[0045] Each sending end is used to collect data output from the data production equipment, write the collected data into a log, and send the log to the receiving end. The data production equipment can be any device or process used to output time-series data in different application scenarios. For example, in industrial automation, data production equipment can be sensors used to monitor the status data of industrial equipment; in intelligent transportation, it can be IoT devices used to collect traffic data, such as cameras, weather detectors, and ultrasonic sensors; in the power industry, it can be sensors used to collect operating parameters of power grid generating equipment (such as power, voltage, and current) and substation monitoring data; and in the healthcare field, it can be medical equipment used to collect vital sign monitoring data of patients (heart rate, blood pressure, and blood oxygen saturation), and so on.

[0046] In some alternative embodiments, any transmitting end can be implemented as a proxy component for data collection, which can be deployed on the same device as the data production equipment. For example... Figure 3 As shown, data production equipment can write data to local shared memory or disk using an SDK (Software Development Kit). The sending end can read data from the shared memory or disk via data dumping and send the read data to the receiving end.

[0047] The receiving end can be a physical or virtual device deployed with a data distribution service. The data distribution service is a middleware service used for publishing, subscribing to, and managing data, enabling applications to easily share data without needing to understand the underlying network details or directly handle complex communication protocols. The data consumer can subscribe to data from the receiving end, and the receiving end can distribute the subscribed data to the data consumer based on the data distribution service. The data distribution service system can be deployed in a distributed manner on a cluster of computing nodes, and the receiving end can be any computing node in that cluster. A computing node refers to the basic unit that undertakes data processing and computing tasks in a distributed computing environment or cluster architecture. Each computing node typically has independent computing resources, such as a central processing unit (CPU), memory, and storage space, and can execute the computing tasks or parts of the tasks assigned to it. In different application scenarios, computing nodes can be implemented as physical servers, virtual machines, or cloud servers; this embodiment does not impose any limitations.

[0048] In some embodiments, the data distribution service may be a Data Distribution Service for Real-Time Systems (DDS). The DDS service is used to support high-performance, real-time, and reliable distributed data sharing in real-time systems and is suitable for systems that need to exchange data under strict time constraints.

[0049] In the data distribution system 100, a data transmission connection can be established between any one sender and one or more receivers, thereby enabling data to flow from the sender to the receiver. The following will use the first sender 101 and the first receiver 201 as examples to illustrate the data transmission method between any group of senders and receivers.

[0050] In the data distribution system 100, the first sending end 101 is mainly used to: obtain the log data packet to be transmitted based on the time-series data output by the data production equipment, and send the log data packet and the first log sequence number corresponding to the log data packet to the first receiving end 201 through a streaming connection. The streaming connection is a network connection established between the first sending end 101 and the first receiving end 201 for the directed transmission of an ordered log data stream. A transmission connection is a logical or physical connection established in a communication network, enabling data to be transmitted from one end to the other. For example, the transmission connection can be a TCP (Transmission Control Protocol) connection established between the first sending end 101 and the first receiving end 201. In practice, multiple streaming connections can be established between the first sending end 101 and the first receiving end 201 to transmit multiple log data streams. Each embodiment of this application uses any streaming connection as an example for illustrative purposes.

[0051] The Log Sequence Number (LSN) is used to sequence the log data packets acquired by the first sender 101 and uniquely identify each log data packet. Each log data packet corresponds to a log file, which may include one or more log records. In this embodiment, a multi-segment sequence identifier can be used to mark the log data packets to ensure accurate sequencing and consistency during transmission. The multi-segment sequence identifier may include at least a stream sequence identifier and a local sequence identifier.

[0052] Here, the stream sequence identifier refers to the sequence identifier of the streaming connection to which the log data packet belongs. The stream sequence identifier increments after any streaming connection is disconnected and restarted. In some optional embodiments, the stream sequence identifier can be determined based on the startup time of the streaming connection. The time-based increment attribute makes the stream sequence identifier of the streaming connection correspondingly have an incrementing attribute. In other optional embodiments, the stream sequence identifier can be determined based on an auto-incrementing ID (Identity Document), and after the streaming connection is disconnected and restarted, the stream sequence identifier increments by a specified step size relative to before the disconnection. For example, this specified step size can be 1.

[0053] The local sequence identifier refers to the sequence identifier of a log packet within its respective streaming connection. During the lifetime of a single streaming connection, the value of the local sequence identifier can continuously increment from 0, increasing by 1 for each log packet sent. This continuously incrementing local sequence identifier ensures the continuity of log packets within the streaming connection.

[0054] In subsequent embodiments, for ease of description and differentiation, the stream sequence identifier is marked as LogID_major, and the local sequence identifier is marked as LogID_minor.

[0055] Taking any log data packet sent by the first sender 101 as an example, the first log sequence number of the log data packet includes: the first stream sequence identifier of the streaming connection and the first local sequence identifier of the log data packet in the streaming connection. It should be understood that the use of "first" here to limit the object being described is only for the convenience of description and to distinguish it from similar objects in the following description, and does not impose any restrictions on the order or number of the objects being described. The same applies to the "second" that appears later.

[0056] In the data distribution system 100, the first receiving end 201 is mainly used to: obtain the log data packets of the streaming connection sent by the first sending end 101 and the first log sequence number corresponding to the log data packets, and obtain the second log sequence number of the previously received log data packets from the receiving progress information of the streaming connection. The receiving progress information of the streaming connection refers to the completion progress of the first receiving end 201 in performing a receiving operation on the log data packets in the streaming connection. This receiving progress information can be described using the sequence identifiers of the log data packets that have been received by the first receiving end 201.

[0057] In this embodiment, the first receiving end 201 can maintain the sequence identifier of the log data packets that have completed the receiving operation as a cursor to query the receiving progress information based on the maintained cursor. The previously received log data packet refers to the most recently received log data packet that the first receiving end 201 has completed the receiving operation. The second log sequence number of the previous log data packet may include: the second stream sequence identifier of the streaming connection to which the previous log data packet belongs and the second local sequence identifier of the previous log data packet within the streaming connection. Where no streaming connection switch has occurred, the streaming connection to which the previous log data packet belongs is the same streaming connection currently transmitting log data packets. Where a streaming connection switch has occurred, the streaming connection to which the previous log data packet belongs is different from the streaming connection currently transmitting log data packets. The first stream sequence identifier and the second stream sequence identifier can be used to determine whether a streaming connection switch has occurred, and can be used to determine whether the streaming connections are continuous when no switching has occurred. Therefore, the first receiving end 201 can determine whether the log data packets are continuous data packets based on the first log sequence number and the second log sequence number. If the log data packet is a continuous data packet, the first receiving end 201 can store the log data packet and distribute it to the data consumer end corresponding to the streaming connection. Conversely, if the log data packet is a non-continuous data packet, the first receiving end 201 can directly discard the log data packet.

[0058] After storing the log data packet, the first receiving end 201 can update the reception progress information of the streaming connection based on the first log sequence number of the log data packet. The updated progress information of the streaming connection can be used to determine whether the next log data packet is a consecutive data packet when it is received; this will not be elaborated further.

[0059] In this implementation, the sending end can send the first log sequence number of the log data packet along with the log data packet to the receiving end. After receiving the log data packet, the receiving end can determine whether the log data packets are consecutive based on the first log sequence number of the log data packet and the second log sequence number of the previously received log data packet. The log sequence number of each log data packet consists of the stream sequence identifier of the streaming connection to which the log data packet belongs and the local sequence identifier of the log data packet within the streaming connection. The stream sequence identifier can be used to accurately order the streaming connection to handle the situation of streaming connection restart; the stream sequence identifier and the local sequence identifier can be used to accurately order the log data packets within the streaming connection. Therefore, even if the streaming connection restarts, the receiving end can accurately determine whether the received data packets are consecutive based on the sequence identifier and the local sequence identifier, reducing the risk of packet loss and out-of-order data packets, thereby improving data integrity and consistency during data transmission.

[0060] In some optional embodiments, to ensure that log data packets are successfully sent, either sender may repeatedly send log data packets to the receiver until it receives an acknowledgment message from the receiver based on the log data packets.

[0061] Continuing with the example of the first sender 101 and the first receiver 201, the following description is provided. Optionally, the first sender 101 may maintain transmission progress information for the streaming connection. This transmission progress information refers to the completion progress of the first sender 101 in performing transmission operations on log data packets within the streaming connection. This transmission progress information can be described using the sequence identifiers of log data packets whose transmission operations have been completed by the first receiver 201. In this embodiment, the first sender 101 may maintain the sequence identifiers of the log data packets whose transmission operations have been completed as a cursor, so as to query the transmission progress information based on the maintained cursor.

[0062] Optionally, after sending the log data packet and the corresponding first log sequence number to the first receiving end 201, the first sending end 101 can update the transmission progress information of the streaming connection based on the first stream sequence identifier and the first local sequence identifier. When the first sending end 101 obtains the transmission progress information of the streaming connection returned by the first receiving end 201, it can determine whether the transmission progress information is the same as the transmission progress information. If the transmission progress information is the same as the transmission progress information, the first sending end 101 can determine that the log data packet was successfully sent and can stop repeatedly sending the log data packet. Optionally, while sending the log data packet, any sending end can start a thread to wait for the transmission progress information from the receiving end. That is, the log data packet sending operation and the waiting for the receiving confirmation message operation can be performed asynchronously to reduce the impact on the log data packet sending operation. It should be noted that when the number of log data packets that have been sent but have not returned a receiving confirmation message exceeds a certain upper limit threshold, the sending of log data packets can be paused until the number of log data packets that have been sent but have not received a receiving confirmation message is less than a certain lower limit threshold.

[0063] Based on this implementation, the first sending end 101 can determine whether the log data packet has been successfully sent by verifying the sending progress information maintained by itself and the receiving progress information fed back by the first receiving end 201, thereby reducing the risk of data loss during transmission and ensuring data consistency on different ends.

[0064] In some exemplary embodiments, when the first receiving end 201 determines whether a log data packet is a continuous data packet based on the first log sequence number of the log data packet and the second log sequence number of the previous log data packet, it can determine whether the first stream sequence identifier in the first log sequence number is equal to the second stream sequence identifier in the second log sequence number. Optionally, if the value of the first stream sequence identifier is equal to the value of the second stream sequence identifier, the first receiving end 201 can determine that the streaming connection has not been disconnected. To ensure that the received log data packets are continuous data packets, the first receiving end 201 can further determine whether the value of the first local sequence identifier in the first log sequence number and the value of the second local sequence identifier in the second log sequence number are continuous. As can be seen from the foregoing embodiments, the value of the local sequence identifier increases continuously during the lifetime of the same streaming connection. Based on this, the first receiving end 201 can determine whether the value of the first local sequence identifier conforms to a continuously increasing pattern relative to the value of the second local sequence identifier. If it conforms to a continuously increasing pattern, it can be determined that the values ​​of the first local sequence identifier and the second local sequence identifier are continuous. If the values ​​of the first local sequence identifier and the second local sequence identifier are consecutive, then the log data packet can be determined to be a consecutive data packet relative to the previous log data packet.

[0065] Based on this implementation, when it is determined from the stream sequence identifier that the stream transport connection has not been disconnected and reconnected, the continuity of log data packets is determined based on the local sequence identifier. This allows for strict verification of the continuity of data packets within a single stream transport connection, ensuring the integrity and consistency of data transmission.

[0066] Based on the above embodiments, optionally, if the value of the first stream sequence identifier is greater than the value of the second stream sequence identifier, the first receiving end 201 can determine that the streaming connection is a streaming connection that has been restarted after being disconnected. In this case, to ensure that the received log data packets are consecutive data packets, the first receiving end 201 can determine whether the value of the first local sequence identifier is equal to the preset starting value of the local sequence identifier. The preset starting value of the local sequence identifier is usually the value of the initial sequence identifier set for the first data packet in any streaming connection. If the first local sequence identifier is equal to the initial sequence identifier set for the first data packet in any streaming connection, it can be determined that the log data packet is a consecutive data packet relative to the previous log data packet. The initial sequence identifier set for the first data packet in any streaming connection can be 0 or 1, and this embodiment does not impose any restrictions. That is, in the case where the streaming connection is a streaming connection that has been restarted after being disconnected, if the log data packet is the first data packet after the streaming connection restarts, it can be determined that the log data packet is a consecutive data packet relative to the previously received log data packet, and the previous log data packet is the last data packet received before the streaming connection was disconnected.

[0067] Based on this implementation, when the stream transmission connection is disconnected and reconnected according to the stream sequence identifier, the continuity of log data packets is determined based on the local sequence identifier and the preset starting value of the local sequence identifier. This allows for strict verification of the continuity of data packets between two adjacent connection operations of the stream transmission connection, ensuring the integrity and consistency of data transmission.

[0068] In some exemplary embodiments, log data can be stored using a multi-copy approach. That is, after receiving log data, any receiving end stores the log data as a single copy locally, and can also send the log data to a replica storage end 30 with multi-copy storage capability for multi-copy storage to achieve disaster recovery backup.

[0069] Optionally, the replica storage terminal 30 can be implemented as an object storage service (OSS) receiver capable of multi-replica storage. OSS employs a Three Availability Zones (3AZ) design strategy to enhance service high availability and disaster recovery capabilities based on the three availability zones. Within a geographical region, the OSS service distributes its data across three physically isolated but interconnected availability zones. Each availability zone contains at least one data center, and there is sufficient distance between them to ensure that data in other availability zones is unaffected in the event of a catastrophic event in a single availability zone. Through the 3AZ design strategy, the OSS service can achieve data redundancy backup and automatically store user data replicas in multiple locations, thereby ensuring that even if one or two availability zones fail, the service can continue to provide data read and write functions from the remaining availability zones, maintaining service continuity. During the process of any receiver sending log data to the replica storage terminal 30, the integrity and consistency of the log data can be ensured through multi-segment LSNs. The following description will continue to use the first receiver 201 as an example.

[0070] Optionally, after storing the log data packet, the first receiving end 201 can send the log data packet to the replica storage end 30 for archiving, and can update the archiving progress information corresponding to the streaming connection according to the sequence identifier of the data packet. The archiving progress information corresponding to the streaming connection is used to restore the archiving progress after the streaming connection is disconnected and restarted. This archiving progress information can be stored in the metadata information of the streaming connection.

[0071] In the above and following embodiments of this application, the metadata information of the streaming connection can be stored in... Figure 1 The metadata management component 40 shown is deployed in a distributed manner and can synchronize the metadata information of the streaming connection through a consistency protocol, so that the metadata information of the streaming connection can be accessed by the sender, receiver, and replica storage. The consistency protocol can be Raft (Reliable, Replicated, Redundant, Fault-Tolerant), ZAB (Zookeeper Atomic Broadcast), etc., and this embodiment does not impose any restrictions.

[0072] Based on this implementation, a multi-availability zone storage strategy can be used to send log data to a replica storage terminal for multi-replica storage, thereby improving the availability and disaster recovery capabilities of the log data. Furthermore, the archiving progress of the log data can be recorded, facilitating the retrieval of data from the sender after the streaming connection is lost, thus ensuring data integrity.

[0073] Optionally, the first receiving end 201 can use asynchronous archiving to send log data packets to the replica storage end 30 for archiving. Continuing with an example of any log data packet received by the first receiving end 201, optionally, the first receiving end 201 can add the log data packet to an asynchronous archiving queue, and using the archiving process, under the triggering of a set archiving event, send the log data packets in the asynchronous archiving queue to the replica storage end 30 for persistent storage. Optionally, the archiving event may include: an event where the amount of log data packets accumulated in the asynchronous archiving queue exceeds a set data volume threshold. The data volume threshold can be 2M, 5M, 8M, or other values, specifically determined based on the bandwidth and throughput pressure of the replica storage end. Alternatively, the archiving event may include: an event where the archiving operation waits for a certain period of time. The certain period of time can be 2 minutes, 3 minutes, 5 minutes, or other durations; this embodiment does not impose any restrictions on this.

[0074] Optionally, when the amount of log data accumulated in the asynchronous archiving queue of the first receiving end 201 is large, the first sending end 101 can be notified to switch to another receiving end to alleviate the archiving pressure on the first receiving end 201. In this implementation, the first receiving end 201 uses asynchronous archiving to send log data to the replica storage end. On the one hand, this separates the log data archiving operation from the log data receiving operation of the first receiving end 210, preventing the blocking of data receiving operations; on the other hand, using asynchronous archiving makes it easier to control the frequency of writing data to the replica storage end, reducing the QPS (Queries Per Second) of the replica storage end.

[0075] In some exemplary embodiments, when the first receiving end 201 sends the log data packets to be archived in the asynchronous archiving queue to the replica storage end 30 for persistent storage, it can write the data in the same log data packet into the same file object of the replica storage end 30 to satisfy the semantic requirement of "one and only once". The following will provide an exemplary description.

[0076] Optionally, the asynchronous archiving process can determine the opened first file object in the replica storage 30. The opened first file object is in a writable state and can receive external write operations. Optionally, the first file object can be a file object opened by the first receiving end 201 in the replica storage 30 after establishing a streaming connection. That is, the first file object is used to store log data transmitted from the streaming connection established by the first receiving end 201, and not used to store data from other streaming connections, so that the same file object is only used to store log data from the same streaming connection.

[0077] Optionally, any file object in the replica storage 30 can be written using an append method. In this method, log data does not overwrite the existing data in the file object, but is appended after the last byte of the data already written to the file object. Based on this, to ensure that log data packets are not split into different file objects, the asynchronous archiving process can obtain the remaining writable data volume of the first file object and determine whether the data volume of the log data packet to be archived is greater than or equal to the remaining writable data volume of the first file object. If the data volume of the log data packet to be archived is greater than or equal to the remaining writable data volume of the first file object, then the log data packet to be archived is written to the first file object. Otherwise, a second file object is opened in the replica storage 30, and the log data to be archived is written to the second file object. Therefore, log data packets to be archived can be written to the same file object without splitting the log data packets to be archived, thus ensuring that the same log data packet is not written to different file objects.

[0078] Based on this implementation method, the log data packets corresponding to an LSN can be archived on a file object. Accordingly, there is one and only one copy of an LSN in the same file object, which satisfies the guarantee level of "one and only once" for data integrity and data processing.

[0079] In some optional embodiments, the last byte of data written to a file object can be recorded using the file object's offset. Specifically, in the replica storage 30, the offset of any file object corresponding to the streaming connection can be stored in the streaming connection's metadata information. After data is written to any file object, the offset of that file object can be dynamically updated in the streaming connection's metadata information.

[0080] Continuing with the example of the first file object, the offset of the first file object can be updated in the metadata information of the streaming connection by the archiving process after each write operation on the first file object. When the asynchronous archiving process obtains the remaining writable data amount of the first file object, it can obtain the offset of the first file object from the metadata information of the streaming connection, and determine the remaining writable data amount of the first file object based on the offset. The remaining writable data amount can be determined based on the difference between the total capacity of the first file object and its offset.

[0081] In this implementation, based on the offset of the first dynamically recorded file object, the asynchronous archiving process can quickly determine whether to switch file objects during the archiving of log data, thereby improving data archiving efficiency.

[0082] It should be noted that, in the event of a restart after a streaming connection is disconnected, the data distribution system 100 can still re-execute the transmission and archiving operations of streaming connections that were received but not yet archived before the connection was disconnected, using multi-segment log sequence numbers. The following will continue to use the first sender 101 and the first receiver 201 as examples for illustrative purposes. Before transmitting data from the streaming connection, the first sender 101 and the first receiver 201 can establish a connection corresponding to the streaming connection. Optionally, the first sender 101 can initiate a connection establishment request corresponding to the streaming connection to the first receiver 201 to register the streaming connection on the first receiver 201.

[0083] In some cases, the first sending end 101 and the second receiving end 202 establish a connection corresponding to the streaming connection. During the continuous transmission of log data packets from the first sending end 101 to the second receiving end 202, the second receiving end 202 experiences a crash or accumulates a large amount of log data awaiting archiving. In these situations, the first sending end 101 can disconnect from the second receiving end 202 and switch the streaming connection to another receiving end. When switching receiving ends, the first sending end 101 can select one receiving end from multiple receiving ends as the switching target. In this embodiment, it is assumed that the first receiving end 201 is the target receiving end selected by the first sending end 101. Optionally, when selecting the target receiving end, the first sending end 101 can use a consistent hashing algorithm to calculate the hash value of the log data in the streaming connection, and determine the target receiving end from multiple receiving ends based on the hash range to which the calculated hash value belongs and the hash ranges corresponding to each of the multiple receiving ends. After determining the first receiving end 201 as the target receiving end, the first sending end 101 may initiate a connection establishment request corresponding to the streaming connection to the first receiving end 201.

[0084] In other cases, the first sending end 101 and the first receiving end 201 establish a connection corresponding to the streaming connection. During the continuous transmission of log data packets from the first sending end 101 to the second receiving end 202, the second receiving end 202 restarts, and the connection corresponding to the streaming connection is broken. After the second receiving end 202 restarts, the first sending end 101 can initiate a connection establishment request to the first receiving end 202 to re-establish the connection corresponding to the streaming connection with the second receiving end 202.

[0085] The first receiving end 201 can respond to the connection establishment request corresponding to the streaming connection initiated by the first sending end 101 and determine whether the streaming connection is a streaming connection that has been disconnected and restarted.

[0086] Optionally, the connection establishment request may include an identification identifier corresponding to the transport connection. When determining whether the stream transport connection is a stream transport connection that has been disconnected and then restarted, the first receiving end 201 may determine the metadata information of the stream transport connection based on the identification identifier of the stream transport connection, and determine whether the metadata information of the stream transport connection contains the identifier of the receiving end that has been registered for the stream transport connection. If it does, then the stream transport connection can be determined to be a stream transport connection that has been disconnected and then restarted.

[0087] The identifier for any streaming connection can be a Universally Unique Identifier (UUID), which is typically generated from the sender's IP address, port number, and a custom tag. Each time a streaming connection registers with a receiver, its corresponding metadata needs to be updated. This metadata records the unique identifier of the streaming connection, the stream sequence identifier (LogID_major) at the time of registration, and the identifier of the receiver for this registration, for subsequent lookup.

[0088] If it is determined that the streaming connection was not disconnected and then restarted, the first receiving end 201 can obtain the receiving progress information and archiving progress information of the streaming connection based on the identification identifier of the streaming connection. The receiving progress information and archiving progress information can be stored in the metadata information of the streaming connection, which is stored on the metadata management component 40. The first receiving end 201 can obtain the target log data that was received but not archived before the streaming connection was disconnected from the first sending end 101 based on the receiving progress information and archiving progress information. For example, if the receiving progress information is: LogID_major = 10, LogID_minor = 102, and the archiving progress information is: LogID_major = 10, LogID_minor = 90, then it can be determined that in the streaming connection where LogID_major = 10, log data packets with LogID_minor in the range (90, 102) were not archived, and these data packets are described as target log data.

[0089] In this implementation, the first receiving end 201 can send a lazy load request to the first sending end 101 based on the receiving progress information and the archiving progress information, so that the first sending end 101 retransmits the target log data. After receiving the target log data, the first receiving end 201 can send the target log data to the replica storage end 30 for archiving and update the archiving progress information of the streaming connection. After completing the archiving of the target log data, it can return a lease establishment success message to the first sending end 101, so that the first sending end 101 can send the corresponding log data packet through the streaming connection.

[0090] Based on this implementation, after the streaming connection is disconnected and restarted, the data that was received but not archived before the disconnection is re-received and archived, which can ensure that the data received by the receiving end can be fully synchronized to the replica storage end, thereby ensuring that the data of the receiving end and the replica storage end 30 have a high degree of consistency.

[0091] The following will combine Figure 3 Taking the complete data distribution link between the first transmitting end 101 and the first receiving end 201 as an example, the data distribution system 100 will be further explained.

[0092] like Figure 3 As shown, the data production device can write data to the device's local shared memory via the SDK. The first sending end 101 can execute as follows: Figure 3 The steps shown are as follows:

[0093] 1. The first transmitter 101 can read data from shared memory via data dumping, and can also... Figure 3The ordered storage method shown saves or caches the read data as log data packets.

[0094] 2. The first sending end 101 can select the first receiving end 201 as the destination receiving end through a consistent hashing algorithm, and send the obtained log data packets to the first receiving end 201.

[0095] 3. The first sending end 101 may send a connection establishment request corresponding to the streaming connection to the first receiving end 201. The connection establishment request may include the UUID of the streaming connection and the stream sequence identifier.

[0096] Specifically, such as Figure 3 As shown, after receiving a connection establishment request, the first sending end 101 and the first receiving end 201 can perform the following steps:

[0097] 4. The first receiving end 201 can initialize the metadata information of the stream transmission link in the metadata management component 40 according to the UUID and the stream sequence identifier, and return a connection success message to the first sending end 101.

[0098] 5. The first sending end 101 can send log data packets to the first receiving end, such as... Figure 3 The log data packets shown are 1 to 4. These log data packets carry a stream sequence identifier and a local sequence identifier, serving as transmission progress information.

[0099] 6. After receiving the log data packet, the first receiving end 201 may return an acknowledgment message, which may include reception progress information. For example, after receiving log data packet 2, the first receiving end 201 may return the stream sequence identifier and local sequence identifier of log data packet 2 as reception progress information.

[0100] 7. After receiving the log data packet, the first receiving end 201 can use the storage component to persistently store the log data locally, and can also write the log data packet to the replica storage end 30 in an append-only manner to perform multi-replica persistent storage of the log data packet.

[0101] 8. After the first receiving end 201 persists the log data packet, it can update the metadata information of the streaming connection in the metadata management component 40 according to the stream sequence identifier and local sequence identifier of the log data packet.

[0102] Steps 5 through 8 above can be repeated to continuously transmit log data packets.

[0103] It is also worth noting that in the above and following embodiments of this application, in the log data transmission link formed by the sending end, receiving end, and replica storage end, a full-link CRC (Cyclic Redundancy Check) can be further performed to detect whether errors have occurred in the log data throughout the complete transmission path. During the full-link CRC check, the sending end can generate a CRC checksum based on the log data packet and retain the checksum during the transmission of the log data packet to the receiving end. After receiving the log data packet, the receiving end can regenerate a CRC checksum based on the received log data packet and determine whether the regenerated CRC checksum is consistent with the received CRC checksum. If they are inconsistent, it is determined that a transmission error has occurred in the log data packet. If they are consistent, it is determined that no error has occurred in the log data packet during transmission. Based on this implementation method, the integrity of data transmission and the consistency of data across multiple ends are further improved.

[0104] In addition to the data distribution system provided in the foregoing embodiments, this application also provides a data transmission method, which will be described exemplarily below with reference to the accompanying drawings.

[0105] Figure 4 A flowchart illustrating the execution of a data transmission method at the sending end, as provided in an exemplary embodiment of this application, is shown below. Figure 4 As shown, the method includes:

[0106] Step 401: Obtain the log data packet to be transmitted.

[0107] Step 401: Send the log data packet and the first log sequence number corresponding to the log data packet to the receiving end through the streaming connection, so that the receiving end receives the log data packet when it determines that the log data packet is a continuous data packet based on the first log sequence number and the second log sequence number of the previously received log data packet; the first log sequence number includes: the first stream sequence identifier of the streaming connection and the first local sequence identifier of the log data packet in the streaming connection; the second log sequence number includes: the second stream sequence identifier of the streaming connection to which the previous log data packet belongs and the second local sequence identifier of the previous log data packet in the streaming connection to which it belongs.

[0108] In some exemplary embodiments, after sending the log data packet and the first log sequence number corresponding to the log data packet to the receiving end via the streaming connection, the method further includes: updating the sending progress information of the streaming connection according to the first stream sequence identifier and the first local sequence identifier; repeatedly performing the operation of sending the log data packet and the first log sequence number to the receiving end; obtaining the receiving progress information of the streaming connection returned by the receiving end; if the receiving progress information is the same as the sending progress information, then determining that the log data packet was successfully sent, and stopping the repeated sending of the log data packet and the first log sequence number.

[0109] In this implementation, the sending end can send the first log sequence number of the log data packet along with the log data packet to the receiving end. After receiving the log data packet, the receiving end can determine whether the log data packets are consecutive based on the first log sequence number of the log data packet and the second log sequence number of the previously received log data packet. The log sequence number of each log data packet consists of the stream sequence identifier of the streaming connection to which the log data packet belongs and the local sequence identifier of the log data packet within the streaming connection. The stream sequence identifier can be used to accurately order the streaming connection to handle the situation of streaming connection restart; the stream sequence identifier and the local sequence identifier can be used to accurately order the log data packets within the streaming connection. Therefore, even if the streaming connection restarts, the receiving end can accurately determine whether the received data packets are consecutive based on the sequence identifier and the local sequence identifier, reducing the risk of packet loss and out-of-order data packets, thereby improving data integrity and consistency during data transmission.

[0110] Figure 5 This is a flowchart illustrating the execution of a data transmission method on the receiving end side, as provided in an exemplary embodiment of this application. The receiving end can be any of the receiving ends in a distributed deployment. Figure 5 As shown, the method includes:

[0111] Step 501: Obtain the log data packet sent through the streaming connection and the first log sequence number corresponding to the log data packet. The first log sequence number includes: the first stream sequence identifier of the streaming connection and the first local sequence identifier of the log data packet in the streaming connection.

[0112] Step 502: Obtain the second log sequence number of the previously received log data packet from the reception progress information of the streaming connection. The second log sequence number includes: the second stream sequence identifier of the streaming connection to which the previous log data packet belongs and the second local sequence identifier of the previous log data packet in the streaming connection to which it belongs.

[0113] Step 503: Determine whether the log data packets are consecutive data packets based on the first log sequence number and the second log sequence number.

[0114] Step 504: If yes, then store the log data packet.

[0115] In some exemplary embodiments, one way to determine whether the log data packet is a continuous data packet based on the first log sequence number and the second log sequence number may include: determining whether the first stream sequence identifier is equal to the second stream sequence identifier; if they are equal, determining whether the first local sequence identifier and the second local sequence identifier are continuous; if they are continuous, determining that the log data packet is a continuous data packet relative to the previous log data packet.

[0116] In some exemplary embodiments, the method further includes: if the value of the first stream sequence identifier is greater than the value of the second stream sequence identifier, then determining whether the value of the first local sequence identifier is equal to a preset starting value of the local sequence identifier; if yes, then determining that the log data packet is a continuous data packet relative to the previous log data packet.

[0117] In some exemplary embodiments, after storing the log data packet, the method further includes: sending the log data packet to a replica storage terminal for archiving; updating the archiving progress information corresponding to the streaming connection according to the first log sequence number of the log data packet; the archiving progress information corresponding to the streaming connection is used to restore the archiving progress after the streaming connection is disconnected and restarted.

[0118] In some exemplary embodiments, one method of sending the log data packets to a replica storage terminal for archiving may include: adding the log data packets to an asynchronous archiving queue; and using the archiving process, upon triggering a set archiving event, sending the log data packets in the asynchronous archiving queue to the replica storage terminal for persistent storage.

[0119] In some exemplary embodiments, one method of sending log data packets to be archived in the asynchronous archiving queue to the replica storage for persistent storage may include: in the replica storage, determining an open first file object corresponding to the first streaming connection; obtaining the remaining writable data volume of the first file object; determining whether the data volume of the log data packets to be archived is greater than or equal to the remaining writable data volume of the first file object; if yes, then writing the log data packets to be archived into the first file object; if no, then opening a second file object in the replica storage and writing the log data to be archived into the second file object.

[0120] In some exemplary embodiments, one way to obtain the remaining writable data of the first file object may include: obtaining the offset of the first file object from the metadata information of the streaming connection; determining the remaining writable data of the first file object based on the offset of the first file object; wherein the offset of the first file object is updated to the metadata information by the archiving process after each write operation on the first file object.

[0121] In some exemplary embodiments, before obtaining the log data packets sent by the sending end through the streaming connection and the first log sequence number corresponding to the log data packets, the method further includes: responding to the connection establishment request for the streaming connection initiated by the sending end, determining whether the streaming connection is a streaming connection that has been restarted after being disconnected; if so, obtaining the receiving progress information and archiving progress information of the streaming connection based on the identification identifier of the streaming connection; obtaining the target log data that was received but not archived before the streaming connection was disconnected from the sending end based on the receiving progress information and the archiving progress information; sending the target log data to the replica storage end for archiving, and updating the archiving progress information of the streaming connection; and returning a lease establishment success message to the sending end so that the sending end can send log data packets through the streaming connection.

[0122] In some exemplary embodiments, one way to determine whether the streaming connection is a streaming connection that has been disconnected and restarted may include: determining the metadata information of the streaming connection based on the identification identifier of the streaming connection; determining whether the metadata information of the streaming connection contains the identifier of the receiver that the streaming connection has registered; if so, determining that the streaming connection is a streaming connection that has been disconnected and restarted.

[0123] In this implementation, the sending end can send the first log sequence number of the log data packet along with the log data packet to the receiving end. After receiving the log data packet, the receiving end can determine whether the log data packets are consecutive based on the first log sequence number of the log data packet and the second log sequence number of the previously received log data packet. The log sequence number of each log data packet consists of the stream sequence identifier of the streaming connection to which the log data packet belongs and the local sequence identifier of the log data packet within the streaming connection. The stream sequence identifier can be used to accurately order the streaming connection to handle the situation of streaming connection restart; the stream sequence identifier and the local sequence identifier can be used to accurately order the log data packets within the streaming connection. Therefore, even if the streaming connection restarts, the receiving end can accurately determine whether the received data packets are consecutive based on the sequence identifier and the local sequence identifier, reducing the risk of packet loss and out-of-order data packets, thereby improving data integrity and consistency during data transmission.

[0124] It should be noted that the execution subject of each step of the method provided in the above embodiments can be the same device, or the method can be executed by different devices. For example, the execution subject of steps 501 to 504 can be device A; or the execution subject of steps 501 and 502 can be device A, and the execution subject of step 503 can be device B; and so on.

[0125] Furthermore, in some of the processes described in the above embodiments and accompanying drawings, multiple operations appear in a specific order. However, it should be clearly understood that these operations may not be executed in the order they appear herein, or they may be executed in parallel. The operation numbers, such as 501, 502, etc., are merely used to distinguish different operations and do not represent any execution order. Additionally, these processes may include more or fewer operations, and these operations may be executed sequentially or in parallel. It should be noted that the descriptions such as "first" and "second" in this document are used to distinguish different messages, devices, modules, etc., and do not represent a chronological order, nor do they limit "first" and "second" to different types.

[0126] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation portals are provided for users to choose to authorize or refuse.

[0127] Figure 6This application illustrates a schematic diagram of the structure of an electronic device provided in an exemplary embodiment, as shown below. Figure 6 As shown, the electronic device includes: a memory 601, a processor 602, and a communication component 603.

[0128] The memory 601 is used to store computer programs and can be configured to store various other data to support operation on the electronic device. Examples of this data include instructions for any application or method used to operate on the electronic device.

[0129] In some exemplary embodiments, Figure 6 The illustrated electronic device can serve as a transmitter in the data distribution system described in the foregoing embodiments. In this embodiment, processor 602, coupled to memory 601, is used to execute a computer program in memory 601 for: acquiring a log data packet to be transmitted; controlling communication component 603 to send the log data packet and a first log sequence number corresponding to the log data packet to a receiving end via a streaming connection, so that the receiving end receives the log data packet when it determines that the log data packet is a consecutive data packet based on the first log sequence number and a second log sequence number of a previously received log data packet; the first log sequence number includes: a first stream sequence identifier of the streaming connection and a first local sequence identifier of the log data packet in the streaming connection; the second log sequence number includes: a second stream sequence identifier of the streaming connection to which the previous log data packet belongs and a second local sequence identifier of the previous log data packet in its respective streaming connection.

[0130] Optionally, after the processor 602 controls the communication component 603 to send the log data packet and the first log sequence number corresponding to the log data packet to the receiving end via the streaming connection, it is further configured to: update the sending progress information of the streaming connection according to the first stream sequence identifier and the first local sequence identifier; repeatedly execute the operation of sending the log data packet and the first log sequence number to the receiving end; obtain the receiving progress information of the streaming connection returned by the receiving end; if the receiving progress information is the same as the sending progress information, determine that the log data packet was successfully sent, and stop repeatedly sending the log data packet and the first log sequence number.

[0131] In some exemplary embodiments, Figure 6The illustrated electronic device can serve as a receiving end in the data distribution system described in the foregoing embodiments. The processor 602, coupled to the memory 601, is used to execute a computer program in the memory 601 to: acquire log data packets sent via a streaming connection and a first log sequence number corresponding to the log data packets, the first log sequence number including: a first stream sequence identifier of the streaming connection and a first local sequence identifier of the log data packets in the streaming connection; acquire a second log sequence number of a previously received log data packet from the reception progress information of the streaming connection, the second log sequence number including: a second stream sequence identifier of the streaming connection to which the previous log data packet belongs and a second local sequence identifier of the previous log data packet in its respective streaming connection; determine, based on the first log sequence number and the second log sequence number, whether the log data packets are consecutive data packets; if so, store the log data packets.

[0132] Optionally, when the processor 602 determines whether the log data packet is a continuous data packet based on the first log sequence number and the second log sequence number, it is specifically configured to: determine whether the first stream sequence identifier is equal to the second stream sequence identifier; if they are equal, determine whether the first local sequence identifier and the second local sequence identifier are continuous; if they are continuous, determine that the log data packet is a continuous data packet relative to the previous log data packet.

[0133] Optionally, the processor 602 is further configured to: if the value of the first stream sequence identifier is greater than the value of the second stream sequence identifier, determine whether the value of the first local sequence identifier is equal to the preset starting value of the local sequence identifier; if yes, determine that the log data packet is a continuous data packet relative to the previous log data packet.

[0134] Optionally, after storing the log data packet, the processor 602 is further configured to: send the log data packet to a replica storage end for archiving; update the archiving progress information corresponding to the streaming connection according to the first log sequence number of the log data packet; the archiving progress information corresponding to the streaming connection is used to restore the archiving progress after the streaming connection is disconnected and restarted.

[0135] Optionally, when the processor 602 sends the log data packet to the replica storage for archiving, it specifically performs the following: adding the log data packet to an asynchronous archiving queue; and using the archiving process, under the triggering of a set archiving event, sending the log data packet in the asynchronous archiving queue to the replica storage for persistent storage.

[0136] Optionally, when the processor 602 sends the log data packets to be archived in the asynchronous archiving queue to the replica storage terminal for persistent storage, it specifically performs the following steps: in the replica storage terminal, it determines the first file object that has been opened corresponding to the first streaming connection; obtains the remaining writable data volume of the first file object; determines whether the data volume of the log data packets to be archived is greater than or equal to the remaining writable data volume of the first file object; if yes, it writes the log data packets to be archived into the first file object; if no, it opens a second file object in the replica storage terminal and writes the log data to be archived into the second file object.

[0137] Optionally, when the processor 602 obtains the remaining writable data of the first file object, it is specifically configured to: obtain the offset of the first file object from the metadata information of the streaming connection; determine the remaining writable data of the first file object based on the offset of the first file object; wherein the offset of the first file object is updated to the metadata information by the archiving process after each write operation on the first file object.

[0138] Optionally, before acquiring the log data packets sent by the sending end through the streaming connection and the first log sequence number corresponding to the log data packets, the processor 602 is further configured to: respond to the connection establishment request for the streaming connection initiated by the sending end, determine whether the streaming connection is a streaming connection that has been restarted after being disconnected; if so, acquire the receiving progress information and archiving progress information of the streaming connection according to the identification identifier of the streaming connection; acquire the target log data that was received but not archived before the streaming connection was disconnected from the sending end according to the receiving progress information and the archiving progress information; send the target log data to the replica storage end for archiving, and update the archiving progress information of the streaming connection; return a lease establishment success message to the sending end, so that the sending end can send log data packets through the streaming connection.

[0139] Optionally, when determining whether the streaming connection is a streaming connection that has been disconnected and then restarted, the processor 602 is specifically configured to: determine the metadata information of the streaming connection based on the identification identifier of the streaming connection; determine whether the metadata information of the streaming connection contains the identifier of the receiver that the streaming connection has registered; if so, determine that the streaming connection is a streaming connection that has been disconnected and then restarted.

[0140] Furthermore, such as Figure 6 As shown, the electronic device also includes other components such as a power supply component 604, a display component 605, and an audio component 606. Figure 6 The diagram only shows some components and does not mean that the electronic device includes only these components.Figure 6 The components shown. Figure 6 In this embodiment, the components within the dashed boxes are optional, not mandatory, and their specific requirements depend on the product form of the electronic device. The electronic device in this embodiment can be a terminal device such as a desktop computer, laptop computer, smartphone, or IoT device, or a receiving device such as a conventional electronic device, cloud electronic device, or electronic device array. If the electronic device in this embodiment is a terminal device such as a desktop computer, laptop computer, or smartphone, it may include... Figure 6 The components within the dashed box; if the electronic device in this embodiment is a conventional electronic device, a cloud electronic device, or an electronic device array, etc., as a receiving end device, then it may not include... Figure 6 The component within the dashed box.

[0141] The memory 601 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random-access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0142] The communication component 603 is configured to facilitate wired or wireless communication between the device containing the communication component and other devices. The device containing the communication component can access wireless networks based on communication standards, such as Wi-Fi, 2G (e.g., Global System for Mobile Communications (GSM)), 3G (e.g., Wideband Code Division Multiple Access (WCDMA), 4G (e.g., Long Term Evolution (LTE)), 4G+ (e.g., LTE-Advanced (LTE-A)), or 5G (5th Generation Mobile Communication Technology), or combinations thereof. In one exemplary embodiment, the communication component receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, the communication component may be implemented based on Near Field Communication (NFC), Radio Frequency Identification (RFID), Infrared Data Association (IrDA), Ultra Wide Band (UWB), Bluetooth (BT), and other technologies.

[0143] The power supply component 604 is used to provide power to various components of the device in which it resides. The power supply component may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the device in which it resides.

[0144] The display components include a screen, which may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen can be implemented as a touchscreen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors can sense not only the boundaries of the touch or swipe action but also the duration and pressure associated with the touch or swipe operation.

[0145] The audio component can be configured to output and / or input audio signals. For example, the audio component includes a microphone (MIC) configured to receive external audio signals when the device containing the audio component is in an operating mode, such as call mode, recording mode, or voice recognition mode. The received audio signals can be further stored in memory or transmitted via a communication component. In some embodiments, the audio component also includes a speaker for outputting audio signals.

[0146] In this implementation, the sending end can send the first log sequence number of the log data packet along with the log data packet to the receiving end. After receiving the log data packet, the receiving end can determine whether the log data packets are consecutive based on the first log sequence number of the log data packet and the second log sequence number of the previously received log data packet. The log sequence number of each log data packet consists of the stream sequence identifier of the streaming connection to which the log data packet belongs and the local sequence identifier of the log data packet within the streaming connection. The stream sequence identifier can be used to accurately order the streaming connection to handle the situation of streaming connection restart; the stream sequence identifier and the local sequence identifier can be used to accurately order the log data packets within the streaming connection. Therefore, even if the streaming connection restarts, the receiving end can accurately determine whether the received data packets are consecutive based on the sequence identifier and the local sequence identifier, reducing the risk of packet loss and out-of-order data packets, thereby improving data integrity and consistency during data transmission.

[0147] Accordingly, embodiments of this application also provide a computer-readable storage medium storing a computer program, which, when executed, can implement the steps that can be performed by an electronic device in the above method embodiments.

[0148] This application also provides a computer program product, including: a computer program / instructions, which, when executed by a processor, can implement the steps in the method provided in this application.

[0149] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM (Compact Disc Read-Only Memory), optical storage, etc.) containing computer-usable program code.

[0150] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0151] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0152] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0153] In a typical configuration, a computing device includes one or more processors (Central Processing Unit, CPU), input / output interfaces, network interfaces, and memory.

[0154] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0155] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, Digital Video Disc (DVD) or other optical storage, magnetic tape, disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0156] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0157] The above description is merely an embodiment of this application and is not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A data transmission method, applicable to a receiving end, characterized in that, include: Obtain the log data packet sent through the streaming connection and the first log sequence number corresponding to the log data packet, wherein the first log sequence number includes: the first stream sequence identifier of the streaming connection and the first local sequence identifier of the log data packet in the streaming connection; From the reception progress information of the streaming connection, obtain the second log sequence number of the previously received log data packet. The second log sequence number includes: the second stream sequence identifier of the streaming connection to which the previous log data packet belongs and the second local sequence identifier of the previous log data packet in the streaming connection to which it belongs. Based on the first log sequence number and the second log sequence number, determine whether the log data packets are consecutive data packets; If so, then store the log data packet.

2. The method according to claim 1, characterized in that, Determining whether the log data packets are consecutive based on the first log sequence number and the second log sequence number includes: Determine whether the first stream sequence identifier is equal to the second stream sequence identifier; If they are equal, then determine whether the first local sequence identifier and the second local sequence identifier are consecutive; If they are consecutive, then the log data packets are determined to be consecutive data packets relative to the previous log data packet.

3. The method according to claim 2, characterized in that, Also includes: If the value of the first stream sequence identifier is greater than the value of the second stream sequence identifier, then determine whether the value of the first local sequence identifier is equal to the preset starting value of the local sequence identifier; If so, then the log data packet is determined to be a consecutive data packet relative to the previous log data packet.

4. The method according to claim 1, characterized in that, After storing the log data packet, the following is also included: The log data packets are sent to a replica storage terminal for archiving, thereby storing the log data packets in multiple replicas; The archiving progress information corresponding to the streaming connection is updated based on the first log sequence number of the log data packet; the archiving progress information corresponding to the streaming connection is used to restore the archiving progress after the streaming connection is disconnected and restarted.

5. The method according to claim 4, characterized in that, Sending the log data packets to a replica storage terminal for archiving, thereby performing multi-replica storage of the log data packets, includes: Add the log data packet to the asynchronous archiving queue; Using the archiving process, when a set archiving event is triggered, the log data packets in the asynchronous archiving queue are sent to the replica storage for persistent storage.

6. The method according to claim 5, characterized in that, Sending the log data packets to be archived in the asynchronous archiving queue to the replica storage for persistent storage includes: In the copy storage terminal, the first file object that has been opened corresponding to the first streaming connection is determined; Get the remaining writable data amount of the first file object; Determine whether the amount of data in the log data packet to be archived is greater than or equal to the amount of remaining writable data in the first file object; If so, the log data packet to be archived is written to the first file object; If not, then open a second file object in the replica storage and write the log data to be archived into the second file object.

7. The method according to claim 6, characterized in that, Obtaining the remaining writable data amount of the first file object includes: Obtain the offset of the first file object from the metadata information of the streaming connection; Based on the offset of the first file object, determine the remaining writable data amount of the first file object; The offset of the first file object is updated in the metadata information by the archiving process after each write operation on the first file object.

8. The method according to any one of claims 1-7, characterized in that, Before obtaining the log data packets sent by the sending end through the streaming connection and the first log sequence number corresponding to the log data packets, the method further includes: In response to the connection establishment request for the streaming connection initiated by the sending end, determine whether the streaming connection is a streaming connection that has been disconnected and then restarted. If so, then based on the identification identifier of the streaming connection, obtain the receiving progress information and archiving progress information of the streaming connection; Based on the receiving progress information and archiving progress information, obtain the target log data that was received but not archived before the streaming connection was disconnected from the sending end; The target log data is sent to the replica storage for archiving, and the archiving progress information of the streaming connection is updated. A lease establishment success message is returned to the sending end, so that the sending end can send log data packets through the streaming connection.

9. The method according to claim 8, characterized in that, Determining whether the streaming connection is a streaming connection that has been disconnected and then restarted includes: The metadata information of the streaming connection is determined based on the identification identifier of the streaming connection; Determine whether the metadata information of the streaming connection contains the identifier of the receiver that the streaming connection has registered; If so, the streaming connection is determined to be a streaming connection that was disconnected and then restarted.

10. A data transmission method, applicable to a sending end, characterized in that, include: Obtain the log data packet to be transmitted; The log data packet and the first log sequence number corresponding to the log data packet are sent to the receiving end through a streaming connection, so that the receiving end can receive the log data packet when it determines that the log data packet is a continuous data packet based on the first log sequence number and the second log sequence number of the previously received log data packet. The first log sequence number includes: a first stream sequence identifier of the streaming connection and a first local sequence identifier of the log data packet in the streaming connection; the second log sequence number includes: a second stream sequence identifier of the streaming connection to which the previous log data packet belongs and a second local sequence identifier of the previous log data packet in the streaming connection to which it belongs.

11. The method according to claim 10, characterized in that, After sending the log data packet and the first log sequence number corresponding to the log data packet to the receiving end via a streaming connection, the method further includes: Update the transmission progress information of the streaming connection based on the first stream sequence identifier and the first local sequence identifier; Repeat the operation of sending the log data packet and the first log sequence number to the receiving end; Obtain the reception progress information of the streaming connection returned by the receiving end; If the receiving progress information is the same as the sending progress information, then it is determined that the log data packet was sent successfully, and the operation of repeatedly sending the log data packet and the first log sequence number is stopped.

12. A data distribution system, characterized in that, include: At least one transmitter and at least one receiver; The first sending end among the at least one sending end is configured to: obtain a log data packet to be transmitted based on the time-series data output by the data production equipment; and send the log data packet and the first log sequence number corresponding to the log data packet to the first receiving end among the at least one receiving end via a streaming connection; the first log sequence number includes: a first stream sequence identifier of the streaming connection and a first local sequence identifier of the log data packet in the streaming connection; The first receiving end is configured to: acquire the log data packet and the first log sequence number; acquire the second log sequence number of the previously received log data packet from the reception progress information of the streaming connection, wherein the second log sequence number includes: the second stream sequence identifier of the streaming connection to which the previous log data packet belongs and the second local sequence identifier of the previous log data packet in the streaming connection to which it belongs; determine whether the log data packet is a consecutive data packet based on the first log sequence number and the second log sequence number; if so, store the log data packet and distribute the log data packet to the data consumer end corresponding to the streaming connection.

13. An electronic device, characterized in that, include: Memory and processor; The memory is used to store one or more computer instructions; The processor is configured to execute one or more computer instructions for performing the steps of the method according to any one of claims 1-12.

14. A computer-readable storage medium storing a computer program, characterized in that, When a computer program is executed by a processor, it can implement the data transmission method according to any one of claims 1-12.

15. A computer program product, characterized in that, include: A computer program / instruction that, when executed by a processor, enables the data transmission method according to any one of claims 1-12.

Images