Edge gateway system and data processing method

Abstract

The invention provides an edge gateway system and a data processing method.The edge gateway system comprises a protocol adaptation module, a data transmission module and a cloud processing module, and analyzing and converting the original data packet into a target data packet with a uniform data format based on the identified protocol type, thereby realizing adaptation of the edge gateway system to multiple protocols. And the data transmission module transmits the target data packet to the cloud processing module through a standardized private protocol based on the data priority and the data attribute corresponding to the target data packet, so that the data acquisition throughput is improved, and the transmission delay is reduced. And finally, the cloud processing module receives the target data packet through a standardized private protocol. And based on the data attribute corresponding to the target data packet, the data in the target data packet is cached to the corresponding data queue, so that the received data is quickly processed.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to an edge gateway system and a data processing method. Background Art

[0002] With the rapid development of visual networking technology, its business applications are constantly expanding and deepening, covering a wide range of fields such as intelligent security, telemedicine, and intelligent transportation. These scenarios have extremely stringent requirements for data real-time, accuracy, and integrity. Furthermore, the data sources involved in visual networking services are extremely rich and diverse, including various types of video acquisition devices and sensor devices. The data types they generate are also diverse, covering video streams, images, text data, and other forms. As a result, the demand for real-time collection of this multi-source data has also increased dramatically.

[0003] Currently, in actual data collection, different devices and data types often use different protocols for transmission. This complexity forces data collection systems to handle each protocol individually, making them complex and inefficient. Furthermore, because different devices and data types often use different protocols for transmission, different devices and data types may use different methods to send data to the cloud, which can lead to high data transmission latency and low throughput. Summary of the Invention

[0004] The present invention provides an edge gateway system and a data processing method to solve the defects in the prior art that different devices and different types of data are often transmitted using different protocols, resulting in low data collection efficiency, high data transmission delay and low throughput.

[0005] The present invention provides an edge gateway system, comprising a protocol adaptation module, a data transmission module and a cloud processing module, wherein: The protocol adapter module is used to perform protocol identification on the collected original data packets and parse and convert the original data packets into target data packets in a unified data format based on the identified protocol type; The data transmission module is configured to transmit the target data packet to the cloud processing module through a standardized private protocol based on the data priority and data attributes corresponding to the target data packet; The cloud processing module is used to receive the target data packet through the standardized private protocol; and based on the data attributes corresponding to the target data packet, cache the data in the target data packet to a corresponding data queue.

[0006] In some embodiments, the data transmission module is further configured to: For a plurality of target data packets having the same target address and data priority, marking data type fields corresponding to the plurality of target data packets based on the data attributes; Based on the data type field, multiple target data packets are simultaneously transmitted to the cloud processing module through a standardized private protocol.

[0007] In some embodiments, the data transmission module is further configured to: When the data type field is video data, a fragment transmission strategy is enabled and the corresponding data packet is transmitted to the cloud processing module via the UDP protocol; When the data type field is graphic data, a compression transmission strategy is enabled and the corresponding data packet is transmitted to the cloud processing module through the TCP protocol.

[0008] In some embodiments, the cloud processing module is further configured to: Parsing the data length field corresponding to the current target data packet through the standardized private protocol to determine the byte length of the current target data packet; In the case that the parsing of the current target data packet fails, the data of the byte length of the current target data packet is skipped, and the data length field corresponding to the next target data packet is parsed continuously through the standardized private protocol.

[0009] In some embodiments, the cloud processing module is further configured to: Based on the data attributes corresponding to each of the data queues, the data in each of the data queues is processed in parallel by a stream processing engine.

[0010] In some embodiments, the protocol adaptation module is further configured to: Extracting protocol feature data of the original data packet; Determining the protocol fingerprint information of the original data packet according to the protocol characteristic data; Determine whether there is target protocol fingerprint information matching the protocol fingerprint information of the original data packet in a preset protocol fingerprint library; the preset protocol fingerprint library includes protocol fingerprint information corresponding to multiple protocol types; If target protocol fingerprint information matching the protocol fingerprint information of the original data packet exists in the preset protocol fingerprint library, determining that the protocol type corresponding to the target protocol fingerprint information is the protocol type of the original data packet; When there is no target protocol fingerprint information matching the protocol fingerprint information of the original data packet in the preset protocol fingerprint library, the protocol feature data of the original data packet is output to the front-end display interface, and the protocol type of the original data packet fed back by the user based on the front-end display interface is received.

[0011] In some embodiments, the protocol adaptation module is further configured to: Based on the identified protocol type, the corresponding parsing plug-in is called, and the protocol configuration parameters corresponding to the identified protocol type are applied to convert the original data packet into a target data packet in a unified data format; the protocol configuration parameters are configured by the user based on the front-end display interface.

[0012] In some embodiments, the standardized private protocol includes: a protocol header and a data packet; wherein the protocol header includes a protocol identifier, a protocol version, a data type, a priority, a sequence number, and a timestamp; and the data packet includes a data format, a data length, and data content.

[0013] The present invention provides a data processing method, which is applied to the edge gateway system described above. The edge gateway system includes a protocol adapter module, a data transmission module, and a cloud processing module. The method includes: The protocol adaptation module performs protocol identification on the collected original data packets, and parses the original data packets into target data packets in a unified data format based on the identified protocol type; The data transmission module transmits the target data packet to the cloud processing module through a standardized private protocol based on the data priority and data attributes corresponding to the target data packet; The cloud processing module receives the target data packet through the standardized private protocol; and based on the data attributes corresponding to the target data packet, caches the data in the target data packet to a corresponding data queue.

[0014] In some embodiments, the data transmission module transmits the target data packet to the cloud processing module through a standardized private protocol based on the data priority and data attributes corresponding to the target data packet, including: The data transmission module marks, for a plurality of target data packets having the same target address and data priority, data type fields corresponding to the plurality of target data packets based on the data attributes; The data transmission module simultaneously transmits the plurality of target data packets to the cloud processing module through a standardized private protocol based on the data type field.

[0015] The edge gateway system and data processing method provided by the present invention include a protocol adapter module, a data transmission module and a cloud processing module. The protocol adapter module performs protocol identification on the collected original data packets, and parses the original data packets into target data packets in a unified data format based on the identified protocol type, thereby realizing the adaptation of the edge gateway system to multiple protocols. The data transmission module transmits the target data packet to the cloud processing module through a standardized private protocol based on the data priority and data attributes corresponding to the target data packet, thereby improving data collection throughput and reducing transmission delay. Finally, the cloud processing module receives the target data packet through a standardized private protocol; and according to the data attributes corresponding to the target data packet, caches the data in the target data packet into the corresponding data queue, thereby realizing rapid processing of the received data. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the present invention or the prior art, a brief introduction is given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0017] Figure 1 It is a structural diagram of the edge gateway system provided by the present invention.

[0018] Figure 2 It is a flow chart of the data processing method provided by the present invention.

[0019] Figure 3 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0020] To make the objectives, technical solutions, and advantages of the present invention more clear, the technical solutions of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the embodiments described are only some of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0021] Figure 1 This is a schematic diagram of the structure of the edge gateway system provided by the present invention. Figure 1 As shown, the edge gateway system includes a protocol adaptation module 10 , a data transmission module 20 and a cloud processing module 30 .

[0022] In this embodiment, the protocol adaptation module is used to perform protocol identification on the collected original data packets, and parse and convert the original data packets into target data packets in a unified data format based on the identified protocol type.

[0023] Raw data packets refer to unprocessed data packets obtained from various business systems, network devices, sensors, and other sources. This data may come from different business scenarios, such as network traffic data, transaction records, and device logs. In actual applications, data transmission is based on specific standardized data formats, transmission rules, and interaction processes.

[0024] In this embodiment, after receiving raw data packets collected from various data sources, data characteristics (such as packet header information and specific field formats) are analyzed to determine the protocol used to transmit these raw data packets. After identifying the protocol type of these raw data packets, the relevant data is parsed from the raw data packets based on the protocol type. The raw data from different sources and protocols is then converted into target data packets in a unified, standardized data format.

[0025] The data transmission module is used to transmit the target data packet to the cloud processing module through a standardized private protocol based on the data priority and data attributes corresponding to the target data packet.

[0026] It should be understood that proprietary protocols refer to custom communication protocols, rather than publicly available standards such as HTTP or TCP / IP. Standardization refers to the implementation of unified specifications within proprietary protocols. By developing and applying a set of common rules, formats, and standards, data, systems, or processes can be interoperable and consistent across different environments and systems.

[0027] In other words, although this embodiment defines a proprietary protocol based on its own needs, it incorporates standardized elements into its design and implementation. This protocol retains the flexibility and ability to meet specific needs of proprietary protocols while also achieving a certain degree of standardization to improve interoperability and compatibility.

[0028] Here, the standardized private protocol includes: a protocol header and a data packet.

[0029] The protocol header contains the protocol identifier (Tag), protocol version (Version), data type (DataType), priority (Priority), sequence number (Sequence), and timestamp (Timestamp). The Tag field is an 8-bit fixed string that serves as the protocol identifier; the Version field is an 8-bit field that identifies the protocol version (e.g., v1.1, v2.1); the DataType field is a 4-bit field that identifies the data format (e.g., 0000 for video, 0001 for graphics, 0010 for control signaling); the Priority field is a 4-bit field that indicates the data transmission priority (0 for highest, 15 for lowest); the Sequence field is a 32-bit field that is primarily used for fragmented transmission of large files, such as video, to prevent data reordering; and the Timestamp field is a 64-bit field that represents a millisecond-level timestamp.

[0030] A data packet contains data format, data length, and data content. Data format refers to the encoding format of the transmitted data; data length refers to the byte length of the data content; and data content is a variable length that stores specific data content (such as a compressed data file).

[0031] Here, data priority is used to measure the real-time requirements and importance of data. High-priority data is typically processed and transmitted first to ensure efficient system operation and smooth business operations. In this embodiment, the data priority of each target data packet can be determined based on the service type of the data source corresponding to the original data packet to which it belongs. For example, alarm information from security monitoring is transmitted first to ensure that it reaches the cloud for timely processing.

[0032] Here, data attributes refer to data types and data sources. In this embodiment, by understanding the attributes of data, data transmission can be better performed.

[0033] The cloud processing module is used to receive the target data packet through the standardized private protocol; and based on the data attributes corresponding to the target data packet, cache the data in the target data packet to a corresponding data queue.

[0034] It should be understood that the target data packets are all data encapsulated through a standardized private protocol, so the cloud processing module uniformly receives the target data packets through a standardized private protocol.

[0035] In addition, due to the diversity of data sources, which may contain multiple types of data (such as video, graphics, and signaling), the cloud processing module also dynamically routes the target data packets to different message queues based on the data attributes corresponding to the target data packets to ensure that the data enters the subsequent processing process (such as real-time analysis and long-term storage) according to business needs.

[0036] The edge gateway system provided by the embodiment of the present invention includes a protocol adapter module, a data transmission module and a cloud processing module. The protocol adapter module performs protocol identification on the collected original data packets, and parses the original data packets into target data packets in a unified data format based on the identified protocol type, thereby realizing the adaptation of the edge gateway system to multiple protocols. The data transmission module transmits the target data packet to the cloud processing module through a standardized private protocol based on the data priority and data attributes corresponding to the target data packet, thereby improving the data collection throughput and reducing the transmission delay. Finally, the cloud processing module receives the target data packet through a standardized private protocol; and according to the data attributes corresponding to the target data packet, caches the data in the target data packet into the corresponding data queue, thereby realizing rapid processing of the received data.

[0037] In some embodiments, the protocol adaptation module is further configured to: Extracting protocol feature data of the original data packet; Determining the protocol fingerprint information of the original data packet according to the protocol characteristic data; Determine whether there is target protocol fingerprint information matching the protocol fingerprint information of the original data packet in a preset protocol fingerprint library; the preset protocol fingerprint library includes protocol fingerprint information corresponding to multiple protocol types; If target protocol fingerprint information matching the protocol fingerprint information of the original data packet exists in the preset protocol fingerprint library, determining that the protocol type corresponding to the target protocol fingerprint information is the protocol type of the original data packet; When there is no target protocol fingerprint information matching the protocol fingerprint information of the original data packet in the preset protocol fingerprint library, the protocol feature data of the original data packet is output to the front-end display interface, and the protocol type of the original data packet fed back by the user based on the front-end display interface is received.

[0038] Here, protocol feature data refers to feature data that can characterize the protocol type, including but not limited to: protocol header information (such as HTTP GET / POST, DNS query type), port number (such as HTTP-80, HTTPS-443, SSH-22), payload characteristics (such as keywords for specific protocols, binary feature codes), etc.

[0039] In this embodiment, after extracting the protocol feature data, the protocol fingerprint information (i.e., the unique identifier fingerprint of the protocol) is calculated using a specific algorithm. This algorithm considers a combination of multiple features to generate a fingerprint that accurately represents the protocol characteristics of the data packet. For example, the protocol feature data is converted into a fixed-length protocol fingerprint information using a hash function or a machine learning embedding algorithm.

[0040] It should be understood that the preset protocol fingerprint library includes pre-stored protocol fingerprint information corresponding to multiple protocol types. For example, the user first determines the protocol feature data of each protocol based on its characteristics, then uses a specific algorithm to convert the protocol feature data of each protocol into the corresponding protocol fingerprint information. Finally, the type identifier corresponding to each protocol type and the corresponding protocol fingerprint information are associated and stored in the preset protocol fingerprint library.

[0041] Next, the calculated protocol fingerprint information is compared one by one with the protocol fingerprint information in the protocol feature fingerprint library. If they are exactly the same or the similarity is greater than a preset threshold, the comparison is considered successful and the protocol type corresponding to the corresponding target protocol fingerprint information is marked as the protocol type of the original data packet.

[0042] Furthermore, if the corresponding target protocol fingerprint information is not matched, the protocol feature data corresponding to the original data packet is output in the front-end display interface to notify the relevant user to manually assist in determining the protocol type of the original data packet.

[0043] The edge gateway system provided by an embodiment of the present invention improves the efficiency of protocol identification and avoids the redundant calculation problem when directly matching protocol feature data by extracting the protocol feature data of the original data packet and converting the protocol feature data into protocol fingerprint information before identifying the protocol type.

[0044] In some embodiments, the protocol adaptation module is further configured to: Based on the identified protocol type, the corresponding parsing plug-in is called, and the protocol configuration parameters corresponding to the identified protocol type are applied to convert the original data packet into a target data packet in a unified data format; the protocol configuration parameters are configured by the user based on the front-end display interface.

[0045] In this embodiment, the protocol adaptation module adopts a protocol plug-in repository and a dynamic configuration mechanism, allowing users to dynamically load new protocol support or adjust parameters of existing protocols without modifying the core code.

[0046] Specifically, the protocol plug-in repository stores and manages parsing plug-ins of various protocol types, supports on-demand loading, and avoids hard-coding all protocol logic.

[0047] It should be understood that parsing plug-ins are used to decode, verify, and structure packets of specific protocols so that they can be correctly identified, parsed, and converted by the system. Each protocol (such as HTTP, MQT, etc.) corresponds to a separate parsing plug-in (dynamic library, script, or microservice).

[0048] Here, a front-end display interface is provided, through which users can dynamically manage protocol configuration parameters (such as protocol parameters and parsing rules) through a visual interface (such as a web management console). The protocol configuration parameters configured by users through the front-end interface will be dynamically passed to the parsing plug-in to guide how the parsing plug-in works. In this embodiment, after the protocol type of the original data packet is identified, the parsing plug-in corresponding to this protocol type is dynamically called from the plug-in repository, and the parsing plug-in reads the configuration parameters corresponding to this protocol type (such as QoS level, subject filtering rules) and performs parsing.

[0049] For example, for the original data packet of the ONVIF protocol, the protocol adaptation module can dynamically call the ONVIF parsing plug-in from the plug-in repository. The ONVIF parsing plug-in reads the configuration parameters corresponding to this protocol type to parse its data encoding format, resolution and other information, and converts the data into a format suitable for subsequent transmission.

[0050] For example, for the original data packet of the MQTT protocol, the protocol adapter module can dynamically call the MQTT parsing plug-in from the plug-in repository. The MQTT parsing plug-in reads the configuration parameters corresponding to this protocol type to parse the message content and extract key data.

[0051] For example, for the original data packet of the HTTP protocol, the protocol adaptation module can dynamically call the HTTP parsing plug-in from the plug-in repository, and the HTTP parsing plug-in reads the configuration parameters corresponding to this protocol type for parsing.

[0052] In an edge gateway system provided by an embodiment of the present invention, the protocol adapter module dynamically manages protocol configuration parameters through a front-end display interface, flexibly adjusting the behavior of parsing plug-ins without rewriting or redeploying plug-in code. This enables the system to quickly adapt to different protocol requirements and business scenarios, improving system flexibility and maintainability.

[0053] In some embodiments, the data transmission module is further configured to: For a plurality of target data packets having the same target address and data priority, marking data type fields corresponding to the plurality of target data packets based on the data attributes; Based on the data type field, multiple target data packets are simultaneously transmitted to the cloud processing module through a standardized private protocol.

[0054] Here, the destination address refers to the final destination of the data packet, usually an IP address or device identifier.

[0055] In this embodiment, when multiple target data packets share the same destination address and data priority, they are treated as a group and processed and transmitted uniformly. Prior to transmission, the data transmission module labels the data type field of each target data packet based on the data attributes. Finally, a standardized private protocol is used to carry and transmit mixed data types such as video, graphics, and signaling, reducing network overhead. During transmission, the labeling of the data type field enables the simultaneous transmission of multiple data types based on the same protocol.

[0056] The edge gateway system provided by an embodiment of the present invention simultaneously transmits multiple target data packets with the same target address and data priority through a standardized private protocol according to the identifier of the data type field corresponding to the target data packet, thereby reducing the number of network interactions and improving transmission efficiency while ensuring business consistency.

[0057] In some embodiments, the data transmission module is further configured to: When the data type field is video data, a fragment transmission strategy is enabled and the corresponding data packet is transmitted to the cloud processing module via the UDP protocol; When the data type field is graphic data, a compression transmission strategy is enabled and the corresponding data packet is transmitted to the cloud processing module through the TCP protocol.

[0058] Specifically, video data is typically large in size. To improve transmission efficiency and reliability, it can be segmented into multiple smaller data segments (or packets) for transmission. Each data segment can be transmitted independently using the UDP User Datagram Protocol (UDP). The receiver reassembles the data segments after receiving them.

[0059] UDP is a connectionless transport layer protocol suitable for applications with high real-time requirements, such as video streaming. UDP does not guarantee reliable data transmission, but it offers low latency and high transmission speeds, making it suitable for transmitting large amounts of data.

[0060] Graphical data often contains a large amount of redundant information. Compression can significantly reduce the amount of data and improve transmission efficiency. In practical applications, the compression algorithm can be lossless (such as ZIP) or lossy (such as JPEG). The specific choice depends on the data type and quality requirements, and there is no restriction on this.

[0061] In this embodiment, the TCP transmission control protocol is used to transmit graphic and text data. TCP is a connection-oriented transport layer protocol that ensures reliable data transmission. TCP ensures data integrity and sequence through mechanisms such as acknowledgment, retransmission, and flow control, making it suitable for transmitting data with high reliability requirements.

[0062] In the edge gateway system provided by the embodiments of the present invention, the proprietary protocol can be designed to select different transmission strategies based on different data types. This flexibility enables the protocol to select the most appropriate transmission method based on the characteristics and requirements of each data type, thereby optimizing transmission efficiency and reliability.

[0063] In some embodiments, the cloud processing module is further configured to: Parsing the data length field corresponding to the current target data packet through the standardized private protocol to determine the byte length of the current target data packet; In the case that the parsing of the current target data packet fails, the data of the byte length of the current target data packet is skipped, and the data length field corresponding to the next target data packet is parsed continuously through the standardized private protocol.

[0064] The data length field is part of the data packet header and is used to indicate the payload length of the data packet. By parsing the data length field, the cloud processing module can determine the specific length of the current data packet (in bytes). Because data may be damaged during transmission, it cannot be correctly parsed. Therefore, in this embodiment, if the cloud processing module fails to parse the current target data packet, the cloud processing module skips the data of the byte length of the current data packet based on the value of the data length field and no longer attempts to parse the entire data packet. After skipping the current data packet, the cloud processing module continues to parse and process subsequent data packets using a standardized private protocol to ensure the normal operation of the system.

[0065] The edge gateway system provided by the embodiment of the present invention reduces data processing delays caused by single data packet errors by skipping data packets that cannot be parsed and continuing to process subsequent data packets.

[0066] In some embodiments, the cloud processing module is further configured to: Based on the data attributes corresponding to each of the data queues, the data in each of the data queues is processed in parallel by a stream processing engine.

[0067] The stream processing engine is a core component in the data processing workflow, responsible for real-time processing of data in message queues. Specifically, the stream processing engine dynamically allocates computing resources based on the data attributes of the data queues (such as data type, priority, and business scenario), enabling parallel processing of multiple queues. Furthermore, the stream processing engine determines the data processing logic based on the data attributes of the data queues. For example, the video data queue requires video stream decoding and analysis; the graphic data queue requires image recognition and text extraction; and the signaling data queue requires protocol parsing and status updates.

[0068] In the edge gateway system provided by an embodiment of the present invention, the stream processing engine configured in the cloud processing module uses parallel processing technology to parallel process the data in the queue according to the data attributes of each data queue, thereby improving the efficiency of data processing.

[0069] refer to Figure 2 Based on the edge gateway system described above, an embodiment of the present invention further provides a data processing method, including: In step 210 , the protocol adaptation module performs protocol identification on the collected original data packets, and parses and converts the original data packets into target data packets in a unified data format based on the identified protocol type.

[0070] Raw data packets refer to unprocessed data packets obtained from various business systems, network devices, sensors, and other sources. This data may come from different business scenarios, such as network traffic data, transaction records, and device logs. In actual applications, data transmission is based on specific standardized data formats, transmission rules, and interaction processes.

[0071] In this embodiment, after receiving raw data packets collected from various data sources, data characteristics (such as packet header information and specific field formats) are analyzed to determine the protocol used to transmit these raw data packets. After identifying the protocol type of these raw data packets, the relevant data is parsed from the raw data packets based on the protocol type. The raw data from different sources and protocols is then converted into target data packets in a unified, standardized data format.

[0072] In step 220 , the data transmission module transmits the target data packet to the cloud processing module through a standardized private protocol based on the data priority and data attributes corresponding to the target data packet.

[0073] It should be understood that proprietary protocols refer to custom communication protocols, rather than publicly available standards such as HTTP or TCP / IP. Standardization refers to the implementation of unified specifications within proprietary protocols. By developing and applying a set of common rules, formats, and standards, data, systems, or processes can be interoperable and consistent across different environments and systems.

[0074] In other words, although this embodiment defines a proprietary protocol based on its own needs, it incorporates standardized elements into its design and implementation. This protocol retains the flexibility and ability to meet specific needs of proprietary protocols while also achieving a certain degree of standardization to improve interoperability and compatibility.

[0075] Here, the standardized private protocol includes: a protocol header and a data packet.

[0076] The protocol header contains the protocol identifier (Tag), protocol version (Version), data type (DataType), priority (Priority), sequence number (Sequence), and timestamp (Timestamp). The Tag field is an 8-bit fixed string that serves as the protocol identifier; the Version field is an 8-bit field that identifies the protocol version (e.g., v1.1, v2.1); the DataType field is a 4-bit field that identifies the data format (e.g., 0000 for video, 0001 for graphics, 0010 for control signaling); the Priority field is a 4-bit field that indicates the data transmission priority (0 for highest, 15 for lowest); the Sequence field is a 32-bit field that is primarily used for fragmented transmission of large files, such as video, to prevent data reordering; and the Timestamp field is a 64-bit field that represents a millisecond-level timestamp.

[0077] A data packet contains data format, data length, and data content. Data format refers to the encoding format of the transmitted data; data length refers to the byte length of the data content; and data content is a variable length that stores specific data content (such as a compressed data file).

[0078] Here, data priority is used to measure the real-time requirements and importance of data. High-priority data is typically processed and transmitted first to ensure efficient system operation and smooth business operations. In this embodiment, the data priority of each target data packet can be determined based on the service type of the data source corresponding to the original data packet to which it belongs. For example, alarm information from security monitoring is transmitted first to ensure that it reaches the cloud for timely processing.

[0079] Here, data attributes refer to data types and data sources. In this embodiment, by understanding the attributes of data, data transmission can be better performed.

[0080] In step 230 , the cloud processing module receives the target data packet through the standardized private protocol; and based on the data attributes corresponding to the target data packet, caches the data in the target data packet into a corresponding data queue.

[0081] It should be understood that the target data packets are all data encapsulated through a standardized private protocol, so the cloud processing module uniformly receives the target data packets through a standardized private protocol.

[0082] In addition, due to the diversity of data sources, which may contain multiple types of data (such as video, graphics, and signaling), the cloud processing module also dynamically routes the target data packets to different message queues based on the data attributes corresponding to the target data packets to ensure that the data enters the subsequent processing process (such as real-time analysis and long-term storage) according to business needs.

[0083] In the data processing method provided by the embodiment of the present invention, the protocol adaptation module performs protocol identification on the collected original data packets, and parses the original data packets into target data packets in a unified data format based on the identified protocol type, thereby realizing the adaptation of the edge gateway system to multiple protocols. The data transmission module transmits the target data packets to the cloud processing module through a standardized private protocol based on the data priority and data attributes corresponding to the target data packets, thereby improving the data collection throughput and reducing the transmission delay. Finally, the cloud processing module receives the target data packets through a standardized private protocol; and according to the data attributes corresponding to the target data packets, caches the data in the target data packets into the corresponding data queue, thereby realizing rapid processing of the received data.

[0084] In some embodiments, the data transmission module transmits the target data packet to the cloud processing module through a standardized private protocol based on the data priority and data attributes corresponding to the target data packet, including: The data transmission module marks, for a plurality of target data packets having the same target address and data priority, data type fields corresponding to the plurality of target data packets based on the data attributes; The data transmission module simultaneously transmits the plurality of target data packets to the cloud processing module through a standardized private protocol based on the data type field.

[0085] Here, the destination address refers to the final destination of the data packet, usually an IP address or device identifier.

[0086] In this embodiment, when multiple target data packets share the same destination address and data priority, they are treated as a group and processed and transmitted uniformly. Prior to transmission, the data transmission module labels the data type field of each target data packet based on the data attributes. Finally, a standardized private protocol is used to carry and transmit mixed data types such as video, graphics, and signaling, reducing network overhead. During transmission, the labeling of the data type field enables the simultaneous transmission of multiple data types based on the same protocol.

[0087] The data processing method provided by the embodiment of the present invention simultaneously transmits multiple target data packets with the same target address and data priority through a standardized private protocol according to the identifier of the data type field corresponding to the target data packet, thereby reducing the number of network interactions and improving transmission efficiency while ensuring business consistency.

[0088] It should be understood that the data processing method provided in the embodiment of the present invention and the edge gateway system described above can be referenced to each other, and will not be described in detail here.

[0089] Figure 3An example of a physical structure diagram of an electronic device is shown below. Figure 3 As shown, the electronic device may include: a processor 310, a communications interface 320, a memory 330, and a communication bus 340, wherein the processor 310, the communications interface 320, and the memory 330 communicate with each other via the communication bus 340. The processor 310 may call logic instructions in the memory 330 to execute a data processing method, which includes: The protocol adaptation module performs protocol identification on the collected original data packets, and parses the original data packets into target data packets in a unified data format based on the identified protocol type; The data transmission module transmits the target data packet to the cloud processing module through a standardized private protocol based on the data priority and data attributes corresponding to the target data packet; The cloud processing module receives the target data packet through the standardized private protocol; and based on the data attributes corresponding to the target data packet, caches the data in the target data packet to a corresponding data queue.

[0090] Furthermore, the logic instructions in the aforementioned memory 330 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the portion that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product, stored in a storage medium, includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, a mobile hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0091] On the other hand, the present invention further provides a computer program product, comprising a computer program. The computer program may be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can perform the data processing method provided by each of the above methods, the method comprising: The protocol adaptation module performs protocol identification on the collected original data packets, and parses the original data packets into target data packets in a unified data format based on the identified protocol type; The data transmission module transmits the target data packet to the cloud processing module through a standardized private protocol based on the data priority and data attributes corresponding to the target data packet; The cloud processing module receives the target data packet through the standardized private protocol; and based on the data attributes corresponding to the target data packet, caches the data in the target data packet to a corresponding data queue.

[0092] In another aspect, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to perform the data processing method provided by the above methods, the method comprising: The protocol adaptation module performs protocol identification on the collected original data packets, and parses the original data packets into target data packets in a unified data format based on the identified protocol type; The data transmission module transmits the target data packet to the cloud processing module through a standardized private protocol based on the data priority and data attributes corresponding to the target data packet; The cloud processing module receives the target data packet through the standardized private protocol; and based on the data attributes corresponding to the target data packet, caches the data in the target data packet to a corresponding data queue.

[0093] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they may be located in one location or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of the present embodiment. Persons of ordinary skill in the art will be able to understand and implement the present invention without inventive effort.

[0094] Through the above description of the embodiments, those skilled in the art will clearly understand that each embodiment can be implemented using software plus a necessary general-purpose hardware platform, or of course, hardware. Based on this understanding, the essence of the above technical solution, or the portion that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a magnetic disk, or an optical disk, and includes a number of instructions for causing a computer device (such as a personal computer, server, or network device) to execute the methods described in each embodiment or certain portions of the embodiments.

[0095] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. An edge gateway system, characterized in that: It includes a protocol adaptation module, a data transmission module, and a cloud processing module, among which: The protocol adapter module is used to perform protocol identification on the collected original data packets and parse and convert the original data packets into target data packets in a unified data format based on the identified protocol type; The data transmission module is configured to transmit the target data packet to the cloud processing module through a standardized private protocol based on the data priority and data attributes corresponding to the target data packet; The cloud processing module is used to receive the target data packet through the standardized private protocol; and based on the data attributes corresponding to the target data packet, cache the data in the target data packet to a corresponding data queue.

2. The edge gateway system according to claim 1, characterized in that: The data transmission module is further used for: For a plurality of target data packets having the same target address and data priority, marking data type fields corresponding to the plurality of target data packets based on the data attributes; Based on the data type field, multiple target data packets are simultaneously transmitted to the cloud processing module through a standardized private protocol.

3. The edge gateway system according to claim 2, characterized in that: The data transmission module is further used to: When the data type field is video data, a fragment transmission strategy is enabled and the corresponding data packet is transmitted to the cloud processing module via the UDP protocol; When the data type field is graphic data, a compression transmission strategy is enabled and the corresponding data packet is transmitted to the cloud processing module through the TCP protocol.

4. The edge gateway system according to claim 1, wherein: The cloud processing module is further used to: Parsing the data length field corresponding to the current target data packet through the standardized private protocol to determine the byte length of the current target data packet; In the case that the parsing of the current target data packet fails, the data of the byte length of the current target data packet is skipped, and the data length field corresponding to the next target data packet is parsed continuously through the standardized private protocol.

5. The edge gateway system according to claim 1, wherein: The cloud processing module is further used to: Based on the data attributes corresponding to each of the data queues, the data in each of the data queues is processed in parallel by a stream processing engine.

6. The edge gateway system according to claim 1, characterized in that: The protocol adaptation module is further used to: Extracting protocol feature data of the original data packet; Determining the protocol fingerprint information of the original data packet according to the protocol characteristic data; Determine whether there is target protocol fingerprint information matching the protocol fingerprint information of the original data packet in a preset protocol fingerprint library; The preset protocol fingerprint library includes protocol fingerprint information corresponding to multiple protocol types; If target protocol fingerprint information matching the protocol fingerprint information of the original data packet exists in the preset protocol fingerprint library, determining that the protocol type corresponding to the target protocol fingerprint information is the protocol type of the original data packet; When there is no target protocol fingerprint information matching the protocol fingerprint information of the original data packet in the preset protocol fingerprint library, the protocol feature data of the original data packet is output to the front-end display interface, and the protocol type of the original data packet fed back by the user based on the front-end display interface is received.

7. The edge gateway system according to claim 1, characterized in that: The protocol adaptation module is further used to: Calling a corresponding parsing plug-in based on the identified protocol type, and applying protocol configuration parameters corresponding to the identified protocol type to convert the original data packet into a target data packet in a unified data format; The protocol configuration parameters are configured by the user based on the front-end display interface.

8. The edge gateway system according to any one of claims 1 to 7, characterized in that: The standardized private protocol includes: a protocol header and a data packet; wherein the protocol header includes a protocol identifier, a protocol version, a data type, a priority, a sequence number and a timestamp; and the data packet includes a data format, a data length and data content.

9. A data processing method, characterized in that: Applied to the edge gateway system according to any one of claims 1 to 8, the edge gateway system includes a protocol adaptation module, a data transmission module and a cloud processing module, and the method includes: The protocol adaptation module performs protocol identification on the collected original data packets, and parses the original data packets into target data packets in a unified data format based on the identified protocol type; The data transmission module transmits the target data packet to the cloud processing module through a standardized private protocol based on the data priority and data attributes corresponding to the target data packet; The cloud processing module receives the target data packet through the standardized private protocol; and based on the data attributes corresponding to the target data packet, caches the data in the target data packet to a corresponding data queue.

10. The data processing method according to claim 9, characterized in that: The data transmission module transmits the target data packet to the cloud processing module through a standardized private protocol based on the data priority and data attributes corresponding to the target data packet, including: The data transmission module marks, for a plurality of target data packets having the same target address and data priority, data type fields corresponding to the plurality of target data packets based on the data attributes; The data transmission module simultaneously transmits the plurality of target data packets to the cloud processing module through a standardized private protocol based on the data type field.

Images