Data transmission method and apparatus

Abstract

This application provides a data transmission method and apparatus, relating to the field of data transmission technology. The method includes: defining a transmission protocol stack, comprising: a network transport layer protocol abstraction layer, specific interfaces based on different languages, and a network application layer protocol abstraction layer; defining various application layer protocol messages on the network application layer protocol abstraction layer; developing Topic logic for the application layer protocol messages, the Topic logic including a pull method and a publish method; developing an abstract data file used for the Topic and abstract data file logic; establishing a mapping relationship between the abstract data file and actual files in the file system; sending data from the file system to the peer via the Topic logic, the abstract data file logic, and the transmission protocol stack; and / or writing received data into the file system via the transmission protocol stack, the Topic logic, and the abstract data file logic. The embodiments of this application provide a reliable and convenient means of data transmission.

Description

Technical Field

[0001] This application relates to the field of data transmission technology, specifically to a data transmission method, a data transmission device, an electronic device, and a corresponding storage medium. Background Technology

[0002] The process industry generates a wide variety of data that needs to be transmitted and exchanged across various systems. However, unlike data transmission in traditional internet scenarios, the process industry often lacks large-scale server clusters, resulting in applications typically being deployed as monolithic applications. In traditional internet server cluster scenarios, there are sufficient server resources to utilize various open-source message queue middleware, providing high-performance and highly reliable data transmission. However, using such middleware in the process industry would lead to a significant waste of server resources, and some servers might even be unable to run such middleware. Therefore, a new data transmission method needs to be redesigned. Summary of the Invention

[0003] The purpose of this application is to provide a data transmission method and apparatus that has high performance and high reliability, does not rely on other middleware, supports lightweight message queues deployed on a single machine, provides data persistence, breakpoint resumption, consumer group consumption, compatibility with multiple protocols such as TCP / UDP / Unix Socket, and supports clients to publish and subscribe using the Kafka protocol, so as to at least solve some of the problems in the background art.

[0004] To achieve the above objectives, this application provides a data transmission method, comprising: defining a transmission protocol stack, the transmission protocol stack including: a network transport layer protocol abstraction layer, specific interfaces based on different languages, and a network application layer protocol abstraction layer; defining various application layer protocol messages on the network application layer protocol abstraction layer; developing Topic logic for the application layer protocol messages, the Topic logic including a pull method and a publish method; an abstract data file used to develop the Topic and abstract data file logic; the abstract data file having a mapping relationship with actual files in the file system; sending data from the file system to the peer via the Topic logic, the abstract data file logic, and the transmission protocol stack, and / or writing received data into the file system via the transmission protocol stack, the Topic logic, and the abstract data file logic.

[0005] Optionally, the network transport layer protocol abstraction layer includes the following abstract interfaces: Socket type definition interface, message sending interface, and message processing callback interface; specific interfaces based on different languages ​​include: TCP Socket definition interface, UDP Socket definition interface, and Unix Socket definition interface; the network application layer protocol abstraction layer includes: application layer protocol type definition interface, message sending interface, and message processing callback interface.

[0006] Optionally, various application layer protocol messages include: production messages, consumption messages, viewing consumption offsets, viewing queue metadata information, finding the coordinator, and synchronizing protocol versions.

[0007] Optionally, the pull method in the Topic logic includes: starting a scheduled task from an asynchronous thread, the scheduled task being used to retrieve target data from a message queue; the publish method in the Topic logic includes: determining the latest abstract data file from all abstract data files used to store data in the current Topic, and calling the abstract data file's method to append the target data to the latest abstract data file.

[0008] Optionally, the abstract data file and abstract data file logic used in developing the Topic include: encapsulating the index file, time index file, data file and lock file associated with the Topic into an abstract data file; the abstract data file logic includes: the method for creating the abstract data file, the method for operating the abstract data file and the method for locating the abstract data file.

[0009] Optionally, the method for locating the abstract data file includes: determining the abstract data file containing the consumption progress offset based on all abstract data file names in the target data directory; finding the data position closest to the target index based on the index file in the determined abstract data file; and after finding the data position closest to the target index, traversing each data item to find the target data corresponding to the consumption progress offset.

[0010] Optionally, the messages processed in the transport protocol stack adopt the following data structure: they are stored in binary data and include a message header and a message payload. The message header includes: offset, actual length, verification information, protocol version, compatibility reservation, timestamp, key length, and the lengths of the key and message payload.

[0011] Optionally, writing the received data to the file system via the transport protocol stack, Topic logic, and abstract data file logic includes: parsing the received data into application layer protocol messages via the transport protocol stack; and writing the obtained application layer protocol messages to the file system via the Topic logic and abstract data file logic using a memory mapping mechanism.

[0012] Optionally, the obtained application layer protocol message is written to the file system via the Topic logic and the abstract data file logic using a memory mapping mechanism, including: finding the latest data file to be written in the abstract data file; creating a memory-mapped file cache for memory mapping based on the file descriptor of the found data file; obtaining the starting byte to be written for the current data based on the byte offset of the latest data in the found data file plus the length of the latest data; writing data after the position of the starting byte, and updating the metadata information of the Topic and the abstract data file.

[0013] Optionally, after finding the latest data file to be written in the abstract data file, the method further includes: checking whether the data file to be written is full; if it is full, creating a new data file and updating the metadata information of the abstract data file.

[0014] Optionally, before defining the transport protocol stack, the method further includes: setting the data directory path; setting the number of entries that can be stored in each data file; selecting a specific implementation of the network abstraction layer based on the network protocol required in the specific scenario; and calling the send message and / or subscribe message functions of this queue after starting the business system.

[0015] This application also provides a data transmission apparatus, comprising: a transmission protocol stack, the transmission protocol stack including: a network transport layer protocol abstraction layer, specific interfaces based on different languages, and a network application layer protocol abstraction layer; multiple application layer protocol messages are defined on the network application layer protocol abstraction layer; message Topic logic, the Topic logic including a pull method and a publish method; an abstract data file used by the Topic and abstract data file logic, the abstract data file having a mapping relationship with actual files in the file system; and a processing module, used to send data from the file system to the peer end via the Topic logic, the abstract data file logic, and the transmission protocol stack, and / or write received data to the file system via the transmission protocol stack, the Topic logic, and the abstract data file logic.

[0016] This application also provides an electronic device, including: at least one processor; and a memory connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the at least one processor implements the aforementioned data transmission method by executing the instructions stored in the memory.

[0017] This application also provides a machine-readable storage medium storing instructions that, when executed by a processor, configure the processor to perform the aforementioned data transfer method.

[0018] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the aforementioned data transmission method.

[0019] The above technical solution has the following beneficial effects:

[0020] Compared with existing technologies, this application solves the problems of complex deployment, high configuration requirements for deployment servers, and cluster deployment requirements caused by the need for middleware such as Zookeeper to be used in previous message queues such as Kafka.

[0021] It also references Kafka's underlying implementation mechanism, solving the problems of low concurrency in data consumption, difficulty in supporting multiple consumers, and inability to start consumption from arbitrary offsets in previous local message queues.

[0022] The implemented network abstraction layer can adapt to complex network scenarios in process industries and also supports multiple network protocols.

[0023] Other features and advantages of the embodiments of this application will be described in detail in the following detailed description section. Attached Figure Description

[0024] The accompanying drawings are provided to further illustrate the embodiments of this application and form part of the specification. They are used together with the following detailed description to explain the embodiments of this application, but do not constitute a limitation on the embodiments of this application. In the drawings:

[0025] Figure 1 The schematic diagram illustrates the steps of the data transmission method according to an embodiment of this application;

[0026] Figure 2 The diagram illustrates, schematically, the classes used in the implementation of some data according to the embodiments of this application;

[0027] Figure 3 A schematic diagram of the structure of the data transmission device according to an embodiment of this application is shown.

[0028] Figure 4 The diagram schematically illustrates the internal structure of an electronic device according to an embodiment of this application. Detailed Implementation

[0029] The specific embodiments of this application will be described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are for illustration and explanation only and are not intended to limit the embodiments of this application.

[0030] Figure 1The schematic diagram illustrates the steps of a data transmission method according to an embodiment of this application. For example... Figure 1 As shown, a data transmission method includes:

[0031] S01. Define a transport protocol stack, which includes: a network transport layer protocol abstraction layer, specific interfaces based on different languages, and a network application layer protocol abstraction layer.

[0032] S02. Define multiple application layer protocol messages on the network application layer protocol abstraction layer;

[0033] S03. Develop Topic logic for the application layer protocol message, wherein the Topic logic includes a pull method and a publish method;

[0034] S04. The abstract data file used to develop the Topic and the logic of the abstract data file; the abstract data file has a mapping relationship with the actual files in the file system;

[0035] S05. Send the data from the file system to the other end via the Topic logic, the abstract data file logic, and the transmission protocol stack, and / or write the received data to the file system via the transmission protocol stack, the Topic logic, and the abstract data file logic.

[0036] Through the above implementation methods, the functional architecture of the transmission protocol stack, Topic logic, and abstract data file logic has been redesigned to improve the reliability of message transmission in the process industry and reduce the hardware resources required for middleware. This provides a more convenient data transmission method for scenarios where industrial internet server resources are scarce.

[0037] In the technical solution of this embodiment, the following steps are required: develop abstract interfaces at each level, develop a general network transmission framework, connect the abstract interfaces by calling them, and finally realize a data transmission method that supports multiple network protocols and is compatible with the Kafka protocol.

[0038] In some embodiments of this application, the network transport layer protocol abstraction layer includes the following abstract interfaces: Socket type definition interface, message sending interface, and message processing callback interface; specific interfaces based on different languages ​​include: TCP Socket definition interface, UDP Socket definition interface, and Unix Socket definition interface; the network application layer protocol abstraction layer includes: application layer protocol type definition interface, message sending interface, and message processing callback interface. Specifically, 1) Defining the network transport layer protocol abstraction layer requires implementing at least the following abstract interfaces: a) Socket type definition interface; b) Message sending interface; c) Message processing callback interface. 2) Implement the Socket type definition interface based on the three socket types TCP / UDP / Unix Socket. Initialize the epoll object and start listening on the port according to the system methods provided by different languages. 3) Defining the network application layer protocol abstraction layer requires implementing at least the following abstract interfaces: a) Application layer protocol type definition interface; b) Message sending interface; c) Message processing callback interface. Message sending and receiving can be achieved through the aforementioned steps.

[0039] In some embodiments of this application, various application layer protocol messages include: production messages, consumption messages, viewing consumption offsets, viewing queue metadata information, finding the coordinator, and synchronizing the protocol version. Based on the network application layer protocol abstraction layer in the previous embodiment, to facilitate seamless migration from Kafka to this message queue, at least the following application layer protocol messages must be defined: a) production messages; b) consumption messages; c) viewing consumption offsets; d) viewing queue metadata information; e) finding the coordinator; f) synchronizing the protocol version. Once these functions are implemented, data sending and receiving can be completed.

[0040] In some embodiments of this application, the pull method in the Topic logic includes: starting a scheduled task from an asynchronous thread, the scheduled task being used to retrieve target data from the message queue; the publish method in the Topic logic includes: determining the latest abstract data file from all abstract data files used to store data in the current Topic, and calling the method of the abstract data file to append the target data to the latest abstract data file. Based on the data transmission and reception provided in the previous embodiment, the next step is to implement a message queue supporting file slicing based on MMAP. The development of the Topic logic can adopt the following steps: a) The Topic needs to provide two methods, "publish" and "pull," to support message production and message consumption. b) The implementation logic of the "pull" method is to start an asynchronous thread and start a scheduled task in the thread to periodically retrieve data to be pulled from the message queue. c) The implementation logic of the "publish" method is to obtain all abstract data files used to store data in the current Topic, obtain the latest abstract data file, and call the method of the abstract data file to append the data to the end of the abstract data file.

[0041] In some embodiments of this application, the abstract data file and abstract data file logic used to develop a Topic include: encapsulating the index file, time index file, data file, and lock file associated with the Topic into an abstract data file; the abstract data file logic includes: a method for creating the abstract data file, a method for operating the abstract data file, and a method for locating the abstract data file. This embodiment provides implementation steps for developing the abstract data file logic, including: a) When a file does not exist, it needs to be created automatically. The abstract data file contains the following real files: index file, time index file, data file, and lock file. b) The following methods need to be provided: determining whether the current data file is full, appending data to the end of the file, retrieving the list of data to be consumed, scanning all existing data files under the data path, locking the abstract data file, and unlocking the abstract data file. c) When consuming data from the abstract data file, since each consumer has a different consumption progress, it needs to support quickly finding the abstract data file where the data corresponding to the consumption progress offset is located.

[0042] In some embodiments of this application, the method for locating the abstract data file, i.e., the aforementioned data corresponding to the consumption progress offset, includes: determining the abstract data file containing the consumption progress offset based on all abstract data filenames in the target data directory; finding the data position closest to the target index based on the index file in the determined abstract data file; and after finding the data position closest to the target index, traversing each data entry to find the target data corresponding to the consumption progress offset. Specifically, it includes the following three steps: a) Viewing all abstract data filenames in the current data directory. Since the filename represents the smallest offset in the current abstract data file, binary search can be used in the filename list to quickly find the abstract data file containing the target offset with a time complexity of O(logn). b) After finding the abstract data file, it is necessary to find the specific location of the target data in the file. If each data entry is traversed, the performance will drop sharply when the file size is large. Therefore, an index file should be used to quickly find the data position closest to the target index using binary search. c) After finding the data closest to the index, traversing each data entry to find the target data.

[0043] In some embodiments of this application, the messages processed in the transport protocol stack adopt the following data structure: they are stored as binary data and include a message header and a message payload. The message header includes: offset, actual length, verification information, protocol version, compatibility reservation, timestamp, key length, and the lengths of the key and message payload. To achieve the above-mentioned fast lookups, a special message data structure needs to be defined according to the following steps: a) First, the entire message data structure is stored in binary, thus ensuring the most efficient storage. b) Store the fields in the following order using the following structure: 8 bytes for offset, representing the offset of this data in the entire Topic; 4 bytes for the actual length of this message; 4 bytes for the CRC checksum, used to verify whether the current data has been modified; 1 byte for the current protocol version; 1 byte reserved for compatibility with Kafka data structures; 8 bytes for the timestamp when the current data was stored; 4 bytes for the length of the current message key; a byte array with the length of the first 4 bytes for storing the key; 4 bytes for the length of the actual data; a byte array with the length of the first 4 bytes for storing the actual data.

[0044] In some embodiments of this application, writing received data to the file system via the transport protocol stack, Topic logic, and abstract data file logic includes: parsing the received data into application layer protocol messages via the transport protocol stack; and writing the obtained application layer protocol messages to the file system via the Topic logic and abstract data file logic using a memory mapping mechanism. After implementing the functions of the aforementioned embodiments, all preliminary work for the local message queue can be completed. When actually writing data, the MMAP mechanism is used for writing, which allows for fast writing of stored data.

[0045] In some embodiments of this application, the obtained application layer protocol messages are written to the file system via the Topic logic and the abstract data file logic using a memory mapping mechanism. This includes: finding the latest data file to be written in the abstract data file; creating a memory-mapped file cache for memory mapping based on the file descriptor of the found data file; obtaining the starting byte for writing the current data based on the byte offset of the latest data in the found data file plus the length of the latest data; writing data from the position of the starting byte, and updating the metadata information of the Topic and the abstract data file. Furthermore, after finding the latest data file to be written in the abstract data file, the method further includes: checking whether the data file to be written is full; if it is full, creating a new data file and updating the metadata information of the abstract data file. Specifically, the aforementioned method for quickly writing stored data can be implemented using the following steps: a) Locate the latest data file to be written in the abstract data file; b) Check if the data file to be written is full. If it is, create a new data file and update the metadata information of the abstract data file; c) Create a memory-mapped file cache used by MMAP based on the file descriptor of the data file; d) Obtain the starting byte for writing the current data by adding the byte offset of the latest data in the current data file to the length of the latest data; e) Write all the data from step three starting from the position of the starting byte. Update the metadata information of the Topic and the abstract data file.

[0046] In some embodiments of this application, before defining the transport protocol stack, the method further includes: setting a data directory path; setting the number of entries that can be stored in each data file; selecting a specific implementation of the network abstraction layer based on the network protocol required in the specific scenario; and calling the send message and / or subscribe message functions of this queue after starting the business system. This embodiment provides the pre-deployment steps of this method. Since the target scenario is a process industry with low server performance and a small number of servers, only one server is needed for deployment. The specific implementation steps are as follows: a) Set the data directory path, such as / home / test / data-root; b) Set the number of entries that can be stored in each data file, such as 10000; c) Select a specific implementation of the network abstraction layer based on the network protocol required in the specific scenario, such as the ProtoBuffer protocol under the TCP protocol; d) After starting the business system, the send message and subscribe message functions of this queue can be called.

[0047] Figure 2 This diagram schematically illustrates the classes used in implementation of some data according to embodiments of this application. For example... Figure 2 As shown, it illustrates the data structure diagram of the class used by structures or objects such as Topic, abstract data file, and data entry during programming implementation. It defines the construction and behavior of the aforementioned structures or objects, including attributes and methods.

[0048] The above implementation methods greatly improve the reliability of message transmission in the process industry and reduce the hardware resources required for middleware, providing a more convenient data transmission method for scenarios where industrial internet server resources are scarce.

[0049] Based on the same inventive concept, this application also provides a data transmission device. Figure 3 A schematic diagram illustrating the structure of a data transmission apparatus according to an embodiment of this application is shown. Figure 3 As shown, the device includes: a transport protocol stack, which includes: a network transport layer protocol abstraction layer, specific interfaces based on different languages, and a network application layer protocol abstraction layer; various application layer protocol messages are defined on the network application layer protocol abstraction layer; message Topic logic, which includes a pull method and a publish method; an abstract data file used by the Topic and abstract data file logic, wherein the abstract data file has a mapping relationship with the actual files in the file system; and a processing module, used to send data from the file system to the peer end via the Topic logic, the abstract data file logic, and the transport protocol stack, and / or write received data to the file system via the transport protocol stack, the Topic logic, and the abstract data file logic.

[0050] In some optional implementations, a transport protocol stack is defined, comprising: a network transport layer protocol abstraction layer, specific interfaces based on different languages, and a network application layer protocol abstraction layer; various application layer protocol messages are defined on the network application layer protocol abstraction layer; Topic logic is developed for the application layer protocol messages, the Topic logic including a pull method and a publish method; an abstract data file used to develop the Topic and abstract data file logic are defined; the abstract data file has a mapping relationship with actual files in the file system; data from the file system is sent to the peer via the Topic logic, the abstract data file logic, and the transport protocol stack, and / or received data is written to the file system via the transport protocol stack, the Topic logic, and the abstract data file logic.

[0051] In some optional implementations, the network transport layer protocol abstraction layer includes the following abstract interfaces: Socket type definition interface, message sending interface, and message processing callback interface; specific interfaces based on different languages ​​include: TCPSocket definition interface, UDP Socket definition interface, and Unix Socket definition interface; the network application layer protocol abstraction layer includes: application layer protocol type definition interface, message sending interface, and message processing callback interface.

[0052] In some alternative implementations, various application layer protocol messages include: production messages, consumption messages, viewing consumption offsets, viewing queue metadata information, finding the coordinator, and synchronizing protocol versions.

[0053] In some optional implementations, the pull method in the Topic logic includes: starting a scheduled task from an asynchronous thread, the scheduled task being used to retrieve target data from a message queue; the publish method in the Topic logic includes: determining the latest abstract data file from all abstract data files used to store data in the current Topic, and calling the abstract data file's method to append the target data to the latest abstract data file.

[0054] In some alternative implementations, the abstract data file and abstract data file logic used to develop a Topic include: encapsulating the index file, time index file, data file, and lock file associated with the Topic into an abstract data file; the abstract data file logic includes: a method for creating the abstract data file, a method for operating the abstract data file, and a method for locating the abstract data file.

[0055] In some optional implementations, the method for locating the abstract data file includes: determining the abstract data file containing the consumption progress offset based on all abstract data file names in the target data directory; finding the data position closest to the target index based on the index file in the determined abstract data file; and after finding the data position closest to the target index, traversing each data item to find the target data corresponding to the consumption progress offset.

[0056] In some optional implementations, the messages processed in the transport protocol stack adopt the following data structure: they are stored as binary data and include a message header and a message payload. The message header includes: offset, actual length, verification information, protocol version, compatibility reservation, timestamp, length of key, and length of key and message payload.

[0057] In some optional implementations, writing the received data to the file system via the transport protocol stack, Topic logic, and abstract data file logic includes: parsing the received data into application layer protocol messages via the transport protocol stack; and writing the obtained application layer protocol messages to the file system via the Topic logic and abstract data file logic using a memory mapping mechanism.

[0058] In some optional implementations, the obtained application layer protocol messages are written to the file system via the Topic logic and the abstract data file logic using a memory mapping mechanism. This includes: finding the latest data file to be written in the abstract data file; creating a memory-mapped file cache for memory mapping based on the file descriptor of the found data file; obtaining the starting byte to be written for the current data based on the byte offset of the latest data in the found data file plus the length of the latest data; writing data after the position of the starting byte, and updating the metadata information of the Topic and the abstract data file.

[0059] In some alternative implementations, after finding the latest data file to be written in the abstract data file, the method further includes: checking whether the data file to be written is full; if it is full, creating a new data file and updating the metadata information of the abstract data file.

[0060] In some optional implementations, before defining the transport protocol stack, the apparatus further includes: setting a data directory path; setting the number of entries that can be stored in each data file; selecting a specific implementation of the network abstraction layer based on the network protocol required in the specific scenario; and calling the send message and / or subscribe message functions of this queue after starting the business system.

[0061] The specific limitations of each functional module in the aforementioned data transmission device can be found in the limitations of the data transmission method described above, and will not be repeated here. Each module in the aforementioned system can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the corresponding operations of each module. This also significantly improves the reliability of message transmission in process industries and reduces the hardware resources required for middleware, providing a more convenient data transmission method for scenarios where industrial internet server resources are scarce.

[0062] In some embodiments of this application, an electronic device is also provided, comprising: at least one processor; and a memory connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the at least one processor executes the aforementioned data transmission method. Its internal structure diagram can be as follows: Figure 4 As shown. Figure 4 This schematic diagram illustrates the internal structure of an electronic device according to an embodiment of the present application. The electronic device includes a processor A01, a network interface A02, a memory (not shown), and a database (not shown) connected via a system bus. The processor A01 provides computing and control capabilities. The memory includes internal memory A03 and a non-volatile storage medium A04. The non-volatile storage medium A04 stores an operating system B01, a computer program B02, and a database (not shown). The internal memory A03 provides an environment for the operation of the operating system B01 and the computer program B02 stored in the non-volatile storage medium A04. The network interface A02 is used for communication with external terminals via a network connection. When the computer program B02 is executed by the processor A01, it implements a data transmission method.

[0063] Those skilled in the art will understand that Figure 4 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the electronic device to which the present application is applied. The specific electronic device may include more or fewer components than shown in the figure, or combine certain components, or have different component arrangements.

[0064] In one embodiment provided in this application, a machine-readable storage medium is provided, on which instructions are stored, which, when executed by a processor, cause the processor to be configured to perform the aforementioned data transfer method.

[0065] In one embodiment provided in this application, a computer program product is provided, including a computer program that, when executed by a processor, implements the aforementioned data transmission method.

[0066] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0067] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0068] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0069] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0070] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0071] Memory may include non-persistent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0072] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0073] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0074] The above are merely embodiments of this application and are not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A data transmission method, characterized in that, The method includes: Define a transport protocol stack, which includes: a network transport layer protocol abstraction layer, specific interfaces based on different languages, and a network application layer protocol abstraction layer; Multiple application layer protocol messages are defined on the network application layer protocol abstraction layer; Develop Topic logic for the application layer protocol message, the Topic logic including a pull method and a publish method; The abstract data file and its logic used in developing a Topic include: encapsulating the index file, time index file, data file, and lock file associated with the Topic into an abstract data file; the abstract data file logic includes: a method for creating the abstract data file, a method for operating the abstract data file, and a method for locating the abstract data file; the abstract data file has a mapping relationship with the actual files in the file system; the method for locating the abstract data file includes: determining the abstract data file containing the consumption progress offset based on all abstract data file names in the target data directory; finding the data position closest to the target index based on the index file in the determined abstract data file; and after finding the data position closest to the target index, traversing each data item to find the target data corresponding to the consumption progress offset. Data from the file system is sent to the other end via the Topic logic, the abstract data file logic, and the transport protocol stack, and / or received data is written to the file system via the transport protocol stack, the Topic logic, and the abstract data file logic.

2. The method according to claim 1, characterized in that, The network transport layer protocol abstraction layer includes the following abstract interfaces: Socket type definition interface, message sending interface, and message processing callback interface; The specific interfaces based on different languages ​​include: TCP Socket definition interface, UDP Socket definition interface, and Unix Socket definition interface; The network application layer protocol abstraction layer includes: application layer protocol type definition interface, message sending interface, and message processing callback interface.

3. The method according to claim 1, characterized in that, Various application layer protocol messages include: production messages, consumption messages, viewing consumption offsets, viewing queue metadata information, finding the coordinator, and synchronizing protocol versions.

4. The method according to claim 1, characterized in that, The fetching method in the Topic logic includes: starting a scheduled task from an asynchronous thread, wherein the scheduled task is used to retrieve target data from the message queue; The publishing method in the Topic logic includes: determining the latest abstract data file from all abstract data files used to store data in the current Topic, and calling the method of the abstract data file to append the target data to the latest abstract data file.

5. The method according to claim 1, characterized in that, The messages processed in the transport protocol stack use the following data structure: The data is stored in binary format and includes a message header and a message payload. The message header includes: offset, actual length, verification information, protocol version, compatibility reservation, timestamp, key length, and the lengths of the key and message payload.

6. The method according to claim 1, characterized in that, The received data is written to the file system via the aforementioned transport protocol stack, Topic logic, and abstract data file logic, including: The received data is parsed into application layer protocol messages via the transport protocol stack. The obtained application layer protocol messages are written to the file system using a memory mapping mechanism through the Topic logic and the abstract data file logic.

7. The method according to claim 6, characterized in that, The obtained application layer protocol messages are processed by the Topic logic and abstract data file logic, and then written to the file system using a memory mapping mechanism, including: Locate the latest data file to be written in the abstract data file; Based on the file descriptor of the found data file, create a cache of memory-mapped files used for memory mapping; The starting byte to be written for the current data is obtained by adding the byte offset of the latest data in the found data file to the length of the latest data; Data is written from the position of the first byte to be written, updating the metadata information of the Topic and abstract data file.

8. The method according to claim 7, characterized in that, After finding the latest data file to be written in the abstract data file, the method further includes: checking whether the data file to be written is full; if it is full, creating a new data file and updating the metadata information of the abstract data file.

9. The method according to claim 1, characterized in that, Before defining the transport protocol stack, the method further includes: Set the data directory path; Set the number of entries that can be stored in each data file; Choose the specific implementation of the network abstraction layer based on the network protocol required in the specific scenario; After the business system starts, it calls the message sending and / or message subscription functions of this queue.

10. A data transmission device, characterized in that, The device includes: The transport protocol stack includes: a network transport layer protocol abstraction layer, specific interfaces based on different languages, and a network application layer protocol abstraction layer; various application layer protocol messages are defined on the network application layer protocol abstraction layer. The Topic logic of the message, which includes a fetch method and a publish method; The abstract data file and its logic used by the Topic include: encapsulating the index file, time index file, data file, and lock file associated with the Topic into an abstract data file; the abstract data file logic includes: the creation method of the abstract data file, the operation method of the abstract data file, and the location method of the abstract data file; the abstract data file has a mapping relationship with the actual files in the file system; the location method of the abstract data file includes: determining the abstract data file where the consumption progress offset is located based on all abstract data file names in the target data directory; finding the data position closest to the target index based on the index file in the determined abstract data file; after finding the data position closest to the target index, traversing each data item to find the target data corresponding to the consumption progress offset; The processing module is used to send data from the file system to the other end via the Topic logic, the abstract data file logic, and the transport protocol stack, and / or write received data to the file system via the transport protocol stack, the Topic logic, and the abstract data file logic.

11. An electronic device, characterized in that, include: At least one processor; A memory connected to the at least one processor; The memory stores instructions executable by the at least one processor, which implements the data transmission method according to any one of claims 1 to 9 by executing the instructions stored in the memory.

12. A computer-readable storage medium having a computer program / instructions stored thereon, characterized in that, When the computer program / instruction is executed by the processor, it implements the data transmission method as described in any one of claims 1 to 9.

13. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instruction is executed by the processor, it implements the data transmission method as described in any one of claims 1 to 9.

Images