Transmitting Device, Receiving Device, Transmitting / Receiving System Having Heterogeneous Protocol Data Conversion Function, and Data Processing Method Thereof

Abstract

According to an embodiment, a transmission and reception system capable of detecting network conditions in real time to dynamically adjust transmission bandwidth, performing automatic conversion between various protocols, and ensuring data loss recovery and integrity during transmission, a transmitting device and a receiving device constituting the system, and a method for processing data thereof are provided.

Description

Technology Field

[0001] The present invention relates to a transmitting and receiving system having a heterogeneous protocol data conversion function, a transmitting device and a receiving device constituting the system, and a method for processing data thereof. Background Technology

[0002] Recently, data exchange between systems has been increasing explosively in various industrial sectors, including cloud computing, the Internet of Things (IoT), big data, and smart factories.

[0003] However, there is a problem in that it is difficult to achieve stable and real-time data integration because each system uses different communication protocols and data formats, and network quality also varies depending on the environment. For example, some systems frequently use HTTP-based REST APIs, while others use messaging systems based on MQTT, Kafka, or AMQP. Consequently, data exchange between systems requires the development of separate gateways or manual conversion logic, which causes increased maintenance costs and latency.

[0004] Furthermore, when network congestion, packet loss, or transmission errors occur during the real-time transmission of large volumes of data, data loss or delays result, and these issues degrade data integrity and reliability. Prior art literature

[0005] Registered Patent Publication No. 10-2669617, Real-time automatic video protocol conversion device and method. The problem to be solved

[0006] According to an embodiment, a transmission and reception system capable of detecting network conditions in real time to dynamically adjust transmission bandwidth, performing automatic conversion between various protocols, and ensuring data loss recovery and integrity during transmission, a transmitting device and a receiving device constituting the system, and a method for processing data thereof are provided.

[0007] The problems to be solved by the present invention are not limited to those mentioned above, and other problems not mentioned will be clearly understood by those skilled in the art to which the present invention pertains from the description below. means of solving the problem

[0008] A data processing method by a transmitting device having a heterogeneous protocol conversion function according to the first aspect comprises: a step of automatically inferring a schema through sample analysis of data formats of data to be transmitted and performing schema evolution management through compatibility verification when a change in the schema occurs; a step of converting a transmitting protocol to a set target protocol by parsing and converting the header of a transmitting message using previously stored header structure information and user-defined header information; a step of predicting real-time network bandwidth of a transmitting environment through periodic measurement of network bandwidth and analysis of packet transmission and reception delays; a step of automatically selecting and applying a compression algorithm according to data type and applied protocol; a step of dividing transmission target data exceeding the predicted bandwidth into a predetermined size and including data division information in the header of each divided data; a step of transmitting the data after the division and packaging are completed to a receiving device, storing the transmitted data until an acknowledgment (ack) for the transmission data is received, and ensuring integrity by processing data transmission and reception and acknowledgment transmission and reception in an asynchronous callback manner.

[0009] Here, the step of predicting the network bandwidth may execute a Kalman filter-based bandwidth estimation algorithm based on network congestion, round-trip latency (RTT), and packet loss rate.

[0010] The step of including the above data partitioning information involves partitioning the data exceeding the predicted bandwidth using a sliding window method, and may include a sequence number (seqID), a total block number (total block), and a data hash value as the data partitioning information in the header of each partitioned data.

[0011] The step of automatically selecting and applying the above compression algorithm may select the Zstandard compression algorithm when the data type is real-time data processing and streaming processing, and select the Brotli compression algorithm when the data type is the HTTP / 2 protocol.

[0012] The step of performing the above schema evolution management may apply type deduction to data in JSON and CSV formats, and apply entropy-based boundary detection to data in binary formats.

[0013] The step of ensuring the integrity above may store the transmitted data in a file or an in-memory DB until the response is received.

[0014] A data processing method by a receiving device having a heterogeneous protocol conversion function according to a second perspective comprises: receiving encrypted, split, and compressed data from a transmitting device, processing the transmission of a reception completion response for the received data in an asynchronous callback manner, and controlling the transmission of the response automatically or manually according to a set reception window size; extracting a compression algorithm from the header information of the received data and applying a decompression method suitable for the data through a built-in compression policy engine; parsing the header of the decompressed split message to verify data splitting information, performing a request for retransmission of the message if the message is missing, and automatically reassembling and recovering the original data in order upon completion of reception of all split messages and integrity verification; parsing the reassembled original data using previously stored protocol-specific header structure information, and converting and processing this data into a final protocol structure according to the receiving device environment; and automatically inferring a data schema through sample analysis by data format of the final received data after the protocol conversion is completed, and performing schema evolution management through compatibility verification when a change in the schema occurs.

[0015] Here, the step of automatically reassembling and recovering can recover data by checking the header information of the split message to perform an automatic retransmission request for the undelivered message and comparing the header information of the retransmitted message with the existing message header information.

[0016] The protocol-specific header structure information utilized in the step of converting and processing into the above-mentioned final protocol structure may include at least one of HTTP, MQTT, REST API, Kafka, ActiveMQ, and RabbitMQ.

[0017] The step of performing the above schema evolution management may apply type deduction to data in JSON and CSV formats, and structure merging to unstructured logs.

[0018] A transmitting device having a heterogeneous protocol conversion function according to a third aspect includes: a memory storing a computer program including at least one instruction; a processor that performs a data processing method of the transmitting device by executing the instruction; and a communication unit that performs communication under the control of the processor.

[0019] A receiving device having a heterogeneous protocol conversion function according to the fourth aspect includes: a memory in which a computer program including at least one instruction is stored; a processor that performs a data processing method of the receiving device by executing the instruction; and a communication unit that performs communication under the control of the processor.

[0020] According to the fifth aspect, a computer program stored on a computer-readable recording medium includes instructions for a processor to perform a data processing method of the transmitting device or a data processing method of the receiving device.

[0021] According to the sixth aspect, a computer-readable recording medium storing a computer program includes instructions for a processor to perform a data processing method of the transmitting device or a data processing method of the receiving device, said computer program. Effects of the invention

[0022] According to the examples, the following effects are present.

[0023] First, flexible linkage between heterogeneous systems and maintenance efficiency are increased.

[0024] By automatically converting data between systems using different communication protocols, the costs associated with manually writing separate conversion logic or building gateways can be drastically reduced. In particular, by providing automatic schema inference and schema evolution management functions through sample analysis by data format, operational convenience is maximized by enabling flexible responses without system interruption even if the data structure of linked systems changes.

[0025] Second, transmission efficiency optimized for real-time network conditions is ensured.

[0026] Through a Kalman Filter-based real-time bandwidth prediction algorithm, constantly changing network congestion and latency can be identified in advance, enabling the dynamic optimization of data transmission paths and bandwidth. Furthermore, by applying adaptive compression algorithms (such as Zstandard and Brotli) optimized for data types and protocols, data transmission volume is minimized and transmission speeds are significantly improved.

[0027] Third, the stability and reliability of large-capacity data transmission are guaranteed.

[0028] Large volumes of data exceeding bandwidth are precisely partitioned using a sliding window method, and complete reassembly and integrity verification are performed at the receiving end through the sequence ID (seqID) and data hash value included in each data header. This ensures transmission reliability by activating automatic retransmission requests and recovery mechanisms even in the event of data loss due to network instability, while the asynchronous callback method minimizes system load and maintains real-time performance even when processing a large volume of response messages.

[0029] Fourth, it provides scalability and versatility for various industrial sites.

[0030] It embeds header structure information for various heterogeneous protocols widely used in the industry, such as HTTP, MQTT, Kafka, and ActiveMQ, enabling immediate application in all industrial sectors requiring data integration, including smart factories, IoT, and cloud computing. In particular, through a flexible conversion method utilizing user-defined header information, it offers high versatility not only for standard protocols but also for special-purpose data transmission environments. Brief explanation of the drawing

[0031] FIG. 1 is a configuration diagram of a transmission and reception system having a heterogeneous protocol data conversion function according to an embodiment of the present invention. Figure 2 is a configuration diagram showing the functional division of a computer program loaded by the processor of the transmitting device shown in Figure 1. Figure 3 is a configuration diagram showing the functional division of a computer program loaded by the processor of the receiving device shown in Figure 1. FIG. 4 is a flowchart showing an example of a data transmission and reception process by a transmission and reception system according to an embodiment of the present invention. Specific details for implementing the invention

[0032] The advantages and features of the present invention and the methods for achieving them will become clear by referring to the embodiments described below in conjunction with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below but may be implemented in various different forms. These embodiments are provided merely to ensure that the disclosure of the present invention is complete and to fully inform those skilled in the art of the scope of the invention, and the present invention is defined only by the scope of the claims.

[0033] The terms used in this specification will be briefly explained, and the invention will be described in detail.

[0034] The terms used in this invention have been selected based on currently widely used general terms, taking into account their functions within the invention; however, these terms may vary depending on the intent of those skilled in the art, case law, the emergence of new technologies, etc. Additionally, in specific cases, terms have been arbitrarily selected by the applicant, and in such cases, their meanings will be described in detail in the relevant description of the invention. Therefore, the terms used in this invention should be defined not merely by their names, but based on their meanings and the overall content of the invention.

[0035] When a part of a specification is described as 'comprising' a certain component, this means that, unless specifically stated otherwise, it does not exclude other components but may include additional components.

[0036] Additionally, the term "part" as used in the specification refers to software or hardware components, such as FPGAs or ASICs, and the "part" performs certain roles. However, the meaning of "part" is not limited to software or hardware. The "part" may be configured to reside in an addressable storage medium or configured to run one or more processors. Thus, by example, the "part" includes components such as software components, object-oriented software components, class components, and task components, as well as processes, functions, attributes, procedures, subroutines, segments of program code, drivers, firmware, microcode, circuits, data, databases, data structures, tables, arrays, and variables. The functions provided within the components and "parts" may be combined into a smaller number of components and "parts" or further separated into additional components and "parts."

[0037] Below, embodiments of the present invention are described in detail with reference to the attached drawings so that those skilled in the art can easily implement the invention. Additionally, parts of the drawings that are irrelevant to the description are omitted to clearly explain the invention.

[0038] FIG. 1 is a configuration diagram of a transmission and reception system having a heterogeneous protocol data conversion function according to an embodiment of the present invention, FIG. 2 is a configuration diagram functionally separating a computer program loaded by a processor of a transmission device shown in FIG. 1, and FIG. 3 is a configuration diagram functionally separating a computer program loaded by a processor of a reception device shown in FIG. 1.

[0039] Referring to FIGS. 1 to 3, according to an embodiment, a transmission and reception system (100) includes a transmission device (110) and a reception device (120) that control the entire process from the source of data to the destination.

[0040] The transmitting device (110) includes a processor (112) that processes operations, a memory (111) in which an operating system and an executable program are loaded, and a communication unit (113) responsible for an interface with a physical network layer. A computer program (not shown) containing at least one instruction is stored in the memory (111), and the processor (112) can perform a transmitting side data processing method by executing the instruction stored in the memory (111). The communication unit (113) performs communication with a receiving device (120) under the control of the processor (112) while performing the transmitting side data processing method.

[0041] When a processor (112) functionally distinguishes a computer program (not shown) loaded by executing instructions stored in memory (111), it may include a schema evolution manager (210), a protocol conversion engine (220), a network state predictor (230), an adaptive data compressor (240), a smart message splitter (250), and a reliable transmission controller (260).

[0042] A schema evolution manager (210) running inside a transmission device (110) performs format-specific sample analysis to determine the structure of the source data. Specifically, for text-based formats such as JSON or CSV, a type deduction technique can be applied to determine the data type by analyzing the patterns of the key and value of each field. If binary format data without a fixed structure is input, the beginning and end of meaningful data records can be identified through entropy-based boundary detection, which analyzes the statistical distribution of the data. Additionally, if structural changes are detected during operation, such as the addition or deletion of fields, schema evolution management is performed to dynamically reflect these changes after compatibility verification, thereby ensuring uninterrupted system connectivity.

[0043] The protocol conversion engine (220) maps the data identified by the schema evolution manager (210) to the actual transmission protocol. This engine has standard header structure information for various protocols such as HTTP, MQTT, REST API, Kafka, ActiveMQ, and RabbitMQ stored in it, and converts and parses the headers of the original message into a form suitable for the destination protocol based on header settings defined by the user.

[0044] To maximize network transmission efficiency, the network state predictor (230) periodically measures network bandwidth through the communication unit (113) of the transmitting device (110). At this time, instead of simple measurement, a Kalman Filter algorithm is executed using time-series data such as network congestion, round-trip delay time (RTT), and packet loss rate as input values ​​to predict real-time network bandwidth with minimized error.

[0045] The adaptive data compressor (240) selects an optimal compression algorithm based on the bandwidth and the nature of the data predicted by the network state predictor (230). For example, when processing log streaming where real-time performance is extremely important, Zstandard (Zstd), which enables high-speed compression, can be applied, and in a web-based HTTP / 2 protocol environment, the Brotli algorithm, which shows a high compression rate, can be selected to optimize transmission efficiency.

[0046] If the compressed data exceeds the predicted bandwidth, the smart message splitter (250) operates. The smart message splitter (250) can divide the data into message fragments of a predetermined size by applying a sliding window method to reliably transmit the data. The header of each fragment may include a sequence number (seqID) for recovering the original data, a total block number indicating the total number of messages, and a data hash value for checking whether the data has been tampered with, as data splitting information.

[0047] A reliable transmission controller (260) that handles the final stage of transmission processes data transmission and reception using an asynchronous callback method. This is intended to increase system resource efficiency by eliminating the processor's waiting time associated with processing a large volume of response messages. Additionally, until a final acknowledgment (ack) is received from the receiving device (120), the transmitted data is stored in the form of an in-memory DB within the memory (111) of the transmitting device or as a file within storage connected to the memory (111) of the transmitting device (110), thereby ensuring integrity so that immediate retransmission is possible in the event of a transmission error.

[0048] Corresponding to the transmitting device (110), the receiving device (120) is also equipped with a processor (122), a memory (121), and a communication unit (123), so that the transmitting device (110) and the receiving device (120) form a symmetric data processing structure. A computer program containing at least one instruction is stored in the memory (121), and the processor (122) can perform a receiving-side data processing method by executing the instruction stored in the memory (121). The communication unit (123) performs communication with the transmitting device (110) under the control of the processor (122) while performing the receiving-side data processing method.

[0049] When a processor (122) functionally classifies a computer program (not shown) loaded by executing instructions stored in memory (121), it may include an asynchronous reception manager (310), an adaptive decompression unit (320), a data recovery and reassembly engine (330), a protocol conversion processor (340), and a reception data schema monitor (350).

[0050] When data arrives at the receiving device (120), the asynchronous receiving manager (310) detects it. The asynchronous receiving manager (310) immediately or asynchronously sends an acknowledgment (ack) for the received message, bundled according to a set receiving window size. The window size can be controlled manually or automatically depending on the network delay state to perform flow control.

[0051] The adaptive decompression unit (320) extracts compression algorithm information from the header of the received message and then restores the state before compression by operating an appropriate decompression logic according to the built-in compression policy engine.

[0052] Subsequently, the data recovery and reassembly engine (330) parses the header of each divided message and compares the sequence number and the total number of blocks. If it is determined that a message with a specific sequence number is missing, the data recovery and reassembly engine (330) can send an automatic retransmission request for the message to the sender. When all blocks are successfully received, the data recovery and reassembly engine (330) can verify integrity by comparing data hash values ​​and can recover a large amount of original data by combining them in order.

[0053] The reassembled original data is transmitted to a protocol conversion processor (340). The protocol conversion processor (340) utilizes previously stored protocol-specific header structure information to convert the format of the received original data into a final protocol structure suitable for the receiving device environment (e.g., a receiving application or DB server) and completes the processing. In this process, various protocol templates such as the aforementioned HTTP and MQTT may be referenced.

[0054] The receiving data schema monitor (350) samples the final processed data in real time to infer the schema once again. This monitors whether the schema evolution that occurred at the transmitting side is accurately reflected in the final result at the receiving side, and if field type inconsistencies or omissions occur, it reflects this in the schema evolution management, thereby completing the reliability of data linkage of the entire transmitting and receiving system (100). Even at this stage, type inference for text-based data and structure merging technology for unstructured logs are applied to flexibly respond to complex data changes.

[0055] FIG. 4 is a flowchart showing an example of a data transmission and reception process by a transmission and reception system (110) according to an embodiment of the present invention.

[0056] Hereinafter, with reference to FIGS. 1 to 4, a data processing method and a data transmission and reception process performed in a transmission and reception system (110) according to an embodiment will be described.

[0057] Schema analysis and evolution management phase (S410)

[0058] First, when source data to be transmitted is generated, the processor (112) of the transmitting device (110) activates the schema evolution manager (210) to analyze the structural characteristics of the data. The schema evolution manager (210) can analyze the data structure by extracting the first n samples of the incoming data and scanning them in real time. Specifically, for text-based formats such as JSON or CSV, a type deduction technique can be applied to analyze the patterns of the key and value of each field and match the data type (String, Integer, Float, etc.). If binary format data without a fixed structure is input, the beginning and end of meaningful data records can be identified through entropy-based boundary detection that analyzes the statistical distribution of the data.

[0059] Subsequently, if the analyzed result does not match the previously stored schema or if a change in the data structure (such as adding or deleting fields) is detected, this is defined as an 'evolution' event, and the continuity of the linkage can be ensured by dynamically adding new fields or changing the types of existing fields to immediately update the latest schema template and transformation template. If the type of the sampled data causes a serious compatibility error with the existing schema (e.g., missing required fields), the processor (112) may not immediately transmit the data and may generate an error log, and may perform exception handling such as retrying the schema analysis of the data or sending a notification to the administrator (S410).

[0060] Target protocol header conversion step (S420)

[0061] Next, the protocol conversion engine (220) determines the format for sending the data analyzed in step S410 to an actual network. The processor (112) refers to the header structure information of various protocols (HTTP, MQTT, Kafka, etc.) stored in memory (111), dynamically converts and parses the message header to match the target protocol defined in advance by the user, and completes the preparation for transmission.

[0062] The above protocol conversion engine (220) can wrap original data according to user-defined settings to match the protocol specifications of the target system. For example, if internal system data is in the REST API format and needs to be transmitted to Kafka, the engine can generate a message header containing the Topic name, partition information, etc. required for the Kafka protocol and insert the original data into the payload. At this time, if a user-defined header that deviates from the standard specifications of the configured target protocol is entered, or if the target protocol engine is not activated, the processor (112) can automatically switch to the default protocol or assign a conversion failure flag to quarantine the message (S420).

[0063] Network state prediction and adaptive compression step (S430)

[0064] In parallel with data conversion, the network state predictor (230) monitors the channel status with the receiving device (120) in real time through the communication unit (113). The network state predictor (230) does not stop at simply measuring the current speed, but inputs indicators such as round-trip delay time (RTT), packet loss rate, and congestion into a Kalman Filter algorithm to calculate the rate of change, thereby deriving the available bandwidth with minimized prediction error.

[0065] The predicted bandwidth information is transmitted to the adaptive data compressor (240). The adaptive data compressor (240) can reduce the bulk size of the data by selectively applying the Zstandard (Zstd) or Brotli algorithm according to the real-time requirements of the data. If the predicted bandwidth drops sharply below a threshold, the adaptive data compressor (240) can switch from the Zstd algorithm, which is faster than the compression rate, to an algorithm that maximizes the compression rate, or increase the compression level. Conversely, if the bandwidth is sufficient, it can maintain a high-speed compression mode to reduce processor resource consumption. Additionally, if a network timeout occurs or the error of the prediction model exceeds a threshold, the processor (112) can fix the predicted bandwidth value to the lowest safe value and activate a back-off algorithm that temporarily increases the data transmission cycle until the network state stabilizes (S430).

[0066] Smart message splitting and packaging step (S440)

[0067] If the above compressed data still exceeds the predicted bandwidth or the maximum transmission unit (MTU) that can be transmitted, the smart message splitter (250) is activated. The smart message splitter (250) can divide the compressed message into message fragments of a predetermined size by applying a sliding window method to reliably transmit the data. At this time, the header of each fragment can be precisely filled with splitting information, such as a sequence number (seqID) for recovering the original data, a total block number indicating the total number of messages, a data hash value, and an index indicating which number of the message it is.

[0068] If a memory shortage is detected during message splitting, the smart message splitter (250) can write the split piece directly to the storage file system (Spill-to-disk) instead of the memory (111) to prevent system crash and readjust the transmission order (S440).

[0069] Asynchronous transmission and reliability control step (S450)

[0070] The packaged message blocks are transmitted to the receiving device (120) via the communication unit (113) by the reliable transmission controller (260). At this time, the transmitting device (110) may adopt an asynchronous callback method to maximize the resource efficiency of the processor (112). The transmitted data may be temporarily stored in an in-memory DB of memory (111) or in a separate file storage until an acknowledgment (ack) is returned, and the reliable transmission controller (260) may start a timer for each seqID simultaneously with transmitting the data.

[0071] When an ack response arrives from the receiving device (120), the corresponding seqID can be deleted from the storage to release the resource. On the other hand, if an ack does not arrive within a set timeout for a specific seqID or if a nack (retransmission request) is received from the receiving side, the reliability transmission controller (260) can ensure integrity by immediately retrieving the stored data and performing retransmission. If the number of retransmissions exceeds a threshold, the connection can be initialized and an attempt can be made to transmit via an alternative path (S450).

[0072] Receiving side data reception and reassembly step (S460)

[0073] When a message begins to arrive through the communication unit (123) of the receiving device (120), the asynchronous receiving manager (310) receives it. The asynchronous receiving manager (310) can control the network flow by determining the time for sending an efficient acknowledgment (ACK) based on the set receiving window size and asynchronously sending it back to the transmitting device (110). Subsequently, the received data is restored to its pre-compression state through the adaptive decompression unit (320) and then transmitted to the data recovery and reassembly engine (330).

[0074] The above data recovery and reassembly engine (330) checks the seqID of the incoming fragmented messages to sort them sequentially into a buffer, and verifies the continuity of the data by comparing the header information of each message fragment. If a packet with a specific sequence number (seqID) is lost or some messages arrive out of order, the asynchronous receiving manager (310) can immediately request the transmission device (110) to retransmit the corresponding block. Once all blocks (total block) are successfully collected, the data hash values ​​are finally compared to verify whether tampering has occurred, and if integrity is deemed to be secured, the data can be combined in order to completely reassemble it into a large amount of original data. However, if data tampering is detected where the final hash value of the reassembled data does not match, the entire data set can be discarded and an exception processing request for retransmission can be performed (S460).

[0075] Final protocol conversion and schema monitoring step (S470)

[0076] The reassembled original data can be converted by a protocol conversion processor (340) into a final protocol structure that can be immediately processed in the internal environment of the receiving side (application, database, etc.). That is, the protocol conversion processor (340) converts the reassembled original data to match the endpoint specifications of the receiving side application, and this can be performed by referring to a previously stored protocol template.

[0077] After the conversion is complete, the receiving data schema monitor (350) can sample the final converted data stream in real time to re-infer the schema and compare it with the final schema evolution information of the sending side. It verifies whether the structural changes that occurred at the schema evolution manager (210) of the sending side have safely reached the receiving side, and, especially in cases where structure merging is required such as with unstructured log data, multiple records can be normalized into a single integrated structure and loaded into the database.

[0078] At this time, if a loading failure occurs due to a connection error or schema mismatch of the receiving target system during the final conversion process, the receiving device (120) can prevent data loss by moving the data to an error queue and sending a status message to the administrator (S470). Through a series of step-by-step processing as described above, the transmitting and receiving system (100) can perfectly guarantee data integrity and realize optimal linkage performance despite the uncertainty of the network environment.

[0079] Meanwhile, a computer program stored on a computer-readable recording medium according to an embodiment includes at least one instruction for a processor of a computing device to perform each step included in the data processing method of a transmitting device and / or a receiving device according to the above-described embodiments.

[0080] Additionally, according to an embodiment, a computer-readable recording medium storing a computer program includes at least one instruction in the computer program for causing a processor of a computing device to perform each step included in the data processing method of a transmitting device and / or a receiving device according to the above embodiments.

[0081] Combinations of each step of each flowchart attached to the present invention may be performed by computer program instructions. Since these computer program instructions may be loaded into the processor of a general-purpose computer, a computer for special purposes, or other programmable data processing equipment, the instructions performed through the processor of the computer or other programmable data processing equipment create means for performing the functions described in each step of the flowchart. Since these computer program instructions may also be stored in a computer-available or computer-readable recording medium that can be oriented toward the computer or other programmable data processing equipment to implement the function in a specific manner, the instructions stored in the computer-available or computer-readable recording medium may also produce a manufactured item containing instruction means for performing the function described in each step of the flowchart. Since computer program instructions can be loaded onto a computer or other programmable data processing equipment, instructions that execute a computer or other programmable data processing equipment by performing a series of operation steps on the computer or other programmable data processing equipment to create a process executed by the computer can also provide steps for executing the functions described in each step of the flowchart.

[0082] Additionally, each step may represent a module, segment, or part of code containing one or more executable instructions for executing a specified logical function(s). Also, it should be noted that in some alternative embodiments, the functions mentioned in the steps may occur out of order. For example, two steps described in succession may actually be performed substantially simultaneously, or the steps may sometimes be performed in reverse order according to the corresponding function.

[0083] The above description is merely an illustrative explanation of the technical concept of the present invention, and those skilled in the art to which the present invention pertains will be able to make various modifications and variations within the scope of the essential quality of the present invention. Accordingly, the embodiments disclosed in the present invention are intended to explain, not limit, the technical concept of the present invention, and the scope of the technical concept of the present invention is not limited by such embodiments. The scope of protection of the present invention shall be interpreted by the claims below, and all technical concepts within the equivalent scope shall be interpreted as being included within the scope of rights of the present invention. Explanation of the symbols

[0084] 100: Transmitting and receiving system 110: Transmitter 111: Memory of the transmitting device 112: Processor of the transmitting device 113: Communication section of the transmitting device 120: Receiving device 121: Memory of the receiving device 122: Processor of the receiving device 123: Communication section of the receiving device 210: Schema Evolution Manager 220: Protocol conversion engine 230: Network State Prediction System 240: Adaptive Data Compressor 250: Smart Message Splitter 260: Reliability Transmission Controller 310: Asynchronous Receive Manager 320: Adaptive Decompression System 330: Data Recovery and Reassembly Engine 340: Protocol conversion handler 350: Incoming Data Schema Monitor

Claims

Claim 1 A data processing method by a transmitting device having a heterogeneous protocol conversion function, comprising: a step of automatically inferring a schema through sample analysis of data formats of data to be transmitted and performing schema evolution management through compatibility verification when a change in the schema occurs; a step of converting a transmission protocol to a set target protocol by parsing and converting the header of a transmission message using previously stored protocol-specific header structure information and user-defined header information; a step of predicting real-time network bandwidth of a transmission environment through periodic measurement of network bandwidth and analysis of packet transmission and reception delays; a step of automatically selecting and applying a compression algorithm according to data type and applied protocol; a step of dividing transmission target data exceeding the predicted bandwidth into a predetermined size and including data division information in the header of each divided data; a step of transmitting the data after the division and packaging are completed to a receiving device, storing the transmission data until an acknowledgment (ack) for the transmission data is received, and ensuring integrity by processing data transmission and reception and acknowledgment transmission and reception in an asynchronous callback manner. Claim 2 In claim 1, the step of predicting network bandwidth is a data processing method that executes a Kalman filter-based bandwidth estimation algorithm based on network congestion, round-trip delay time (RTT), and packet loss rate. Claim 3 A data processing method according to claim 1, wherein the step of including the data partitioning information comprises partitioning data exceeding the predicted bandwidth using a sliding window method, and including a sequence number (seqID), a total block, and a data hash value as the data partitioning information in the header of each partitioned data. Claim 4 In claim 1, the step of automatically selecting and applying the compression algorithm is a data processing method in which the Zstandard compression algorithm is selected when the data type is real-time data processing and streaming processing, and the Brotli compression algorithm is selected when the data type is HTTP / 2 protocol. Claim 5 A data processing method according to claim 1, wherein the step of performing schema evolution management applies type deduction to data in JSON and CSV formats and applies entropy-based boundary detection to data in binary formats. Claim 6 In claim 1, the step of ensuring integrity is a data processing method that stores the transmitted data in a file or an in-memory DB until the response is received. Claim 7 A data processing method by a receiving device having a heterogeneous protocol conversion function, comprising: receiving encrypted, split, and compressed data from a transmitting device, processing the transmission of a reception acknowledgment (ack) for the received data in an asynchronous callback manner, and controlling the transmission of the acknowledgment automatically or manually according to a set reception window size; extracting a compression algorithm from the header information of the received data and applying a decompression method suitable for the data through a built-in compression policy engine; parsing the header of the decompressed split message to verify data splitting information, performing a request for retransmission of the message if the message is missing, and automatically reassembling and recovering the original data in order upon completion of reception of all split messages and integrity verification; parsing the reassembled original data using previously stored protocol-specific header structure information, and converting and processing this data into a final protocol structure according to the receiving device environment; and automatically inferring a data schema through sample analysis of data formats of the final received data after the protocol conversion is completed, and performing schema evolution management through compatibility verification when a change in the schema occurs; comprising a data processing method. Claim 8 In claim 7, the step of automatically reassembling and recovering is a data processing method that checks the header information of the split message to perform an automatic retransmission request for the undelivered message, and recovers data by comparing the header information of the retransmitted message with the existing message header information. Claim 9 A data processing method according to claim 7, wherein the protocol-specific header structure information utilized in the step of converting and processing into the final protocol structure includes at least one of HTTP, MQTT, REST API, Kafka, ActiveMQ, and RabbitMQ. Claim 10 In claim 7, the step of performing the schema evolution management is a data processing method that applies type deduction to data in JSON and CSV formats and applies structure merging to unstructured logs. Claim 11 A transmitting device having a heterogeneous protocol conversion function comprises: a memory storing a computer program including at least one instruction; a processor that performs a data processing method by executing said instruction; and a communication unit that performs communication under the control of said processor, wherein the data processing method comprises: a step of automatically inferring a schema through sample analysis of data formats of data to be transmitted and performing schema evolution management through compatibility verification when a change in said schema occurs; a step of converting a transmission protocol to a set target protocol by parsing and converting the header of a transmission message using previously stored protocol-specific header structure information and user-defined header information; a step of predicting real-time network bandwidth of a transmission environment through periodic measurement of network bandwidth and analysis of packet transmission and reception delays; a step of automatically selecting and applying a compression algorithm according to data type and applied protocol; a step of dividing transmission target data exceeding the predicted bandwidth into a predetermined size and including data division information in the header of each divided data. A transmitting device comprising the step of transmitting the data after the above-mentioned splitting and packaging is completed to a receiving device, storing the transmitted data until an acknowledgment (ack) for the transmitted data is received, and ensuring integrity by processing the transmission and reception of data and the transmission and reception of acknowledgments in an asynchronous callback manner. Claim 12 A receiving device having a heterogeneous protocol conversion function, comprising: a memory storing a computer program including at least one instruction; a processor that performs a data processing method by executing said instruction; and a communication unit that performs communication under the control of said processor, wherein the data processing method comprises: receiving encrypted, divided, and compressed data from a transmitting device, processing the transmission of a reception acknowledgment (ack) for the received data in an asynchronous callback manner, and controlling the transmission of the acknowledgment automatically or manually according to a set reception window size; extracting a compression algorithm from the header information of the received data and applying a decompression method suitable for the data through a built-in compression policy engine; parsing the header of the decompressed divided message to verify data division information, performing a retransmission request for the message if the message is missing, and automatically reassembling and recovering the original data in order upon completion of reception of all divided messages and integrity verification; parsing the reassembled original data using previously stored protocol-specific header structure information, and converting and processing this data into a final protocol structure according to the receiving device environment. A receiving device comprising: a step of automatically inferring a data schema through sample analysis by data format of the final received data after the protocol conversion is completed, and performing schema evolution management through compatibility verification when a change in the schema occurs. Claim 13 A computer program stored on a computer-readable recording medium, wherein the computer program comprises instructions for causing the processor to perform a data processing method of any one of claims 1 to 10 by having the processor execute at least one instruction. Claim 14 A computer-readable recording medium storing a computer program, wherein the computer program includes instructions for the processor to perform a data processing method of any one of claims 1 to 10 by the processor executing at least one instruction. Claim 15 A transmission and reception system comprising a transmitting device and a receiving device having a heterogeneous protocol conversion function, wherein a data processing method performed by a processor of the transmitting device comprises: a step of automatically inferring a schema through sample analysis by data format of data to be transmitted and performing schema evolution management through compatibility verification when a change in the schema occurs; a step of converting a transmission protocol to a set target protocol by parsing and converting the header of a transmission message using previously stored protocol-specific header structure information and user-defined header information; a step of predicting real-time network bandwidth of a transmission environment through periodic measurement of network bandwidth and analysis of packet transmission and reception delays; a step of automatically selecting and applying a compression algorithm according to data type and applied protocol; a step of dividing transmission target data exceeding the predicted bandwidth into a predetermined size and including data division information in the header of each divided data; a step of transmitting the data after the division and packaging are completed to a receiving device, storing the transmission data until an acknowledgment (ack) for the transmission data is received, and ensuring integrity by processing data transmission and reception and acknowledgment transmission and reception in an asynchronous callback manner; and wherein the data processing method performed by a processor of the receiving device comprises A step of receiving encrypted, split, and compressed data from a transmitting device, processing the transmission of a reception acknowledgment (ACK) for the received data in an asynchronous callback manner, and controlling the transmission of the acknowledgment automatically or manually according to a set reception window size; a step of extracting a compression algorithm from the header information of the received data and applying a decompression method suitable for the data through a built-in compression policy engine; a step of parsing the header of the decompressed split message to verify data splitting information, performing a request for retransmission of the message if the message is missing, and automatically reassembling and recovering the original data in order upon completion of reception of all split messages and integrity verification;A transmitting and receiving system comprising: a step of parsing reassembled original data using previously stored protocol-specific header structure information and converting and processing this data into a final protocol structure according to the receiving device environment; and a step of automatically inferring a data schema through sample analysis by data format of the final received data after the protocol conversion is completed, and performing schema evolution management through compatibility verification when a change in the schema occurs.

Images