An edge computing gateway multi-protocol output method and system for video stream parsing recognition

Abstract

The application relates to the technical field of industrial internet, in particular to a multi-protocol output method of an edge computing gateway for video stream analysis and identification, which comprises the following steps: step S1, converting multi-mode data of a source end into a binary data stream, adding metadata, mapping the binary data stream into a Data Matrix data array, and then performing tiled display of the source end; step S2, transmitting display signals of the source end to an edge computing gateway through a physical video interface to perform high-frequency sampling of the video stream; step S3, processing the video stream, restoring original binary data packets, and obtaining restored binary data packets; and outputting the restored binary data packets of step S3 to a downstream application end. The application further discloses a multi-protocol output system of an edge computing gateway for video stream analysis and identification. The application can convert general data into visual signals, has high throughput, high fault tolerance and multi-protocol adaptation capability, and is safe, reliable, and low in cost.

Description

Technical Field

[0001] This invention relates to the field of industrial internet technology, and in particular to a multi-protocol output method and system for edge computing gateways used for video stream parsing and recognition. Background Technology

[0002] In current server operation and maintenance and industrial data acquisition, there is a common problem that some servers or industrial control equipment, such as various dedicated data acquisition servers, are difficult to obtain underlying data directly through conventional API interfaces due to completely physically isolated local area networks or closed hardware designs, limited interface permissions, or high maintenance costs.

[0003] Existing technologies typically require complex hardware modifications or software intrusions into existing servers or networks, such as installing network gateways and communicating via radio, laser, or serial port protocols like RS232 / RS485. This not only increases implementation costs but may also affect system stability. Furthermore, in some high-security scenarios, physical isolation is essential, and direct connection via traditional network protocols poses security risks. Although screen capture technology based on Optical Character Recognition (OCR) exists, it has significant shortcomings in practical applications. First, OCR technology is only suitable for recognizing visible text and cannot effectively transmit binary files, images, or complex non-text data. Second, there is often a frame rate asynchrony between video stream transmission and the acquisition end, which can easily lead to the loss of critical data when the source data display duration is extremely short. Finally, existing simple screen recognition solutions lack effective data integrity verification and packet loss recovery mechanisms, making it difficult to meet the accuracy requirements of industrial-grade data acquisition. Summary of the Invention

[0004] To address the aforementioned issues, this invention provides a multi-protocol output method for an edge computing gateway used for video stream parsing and recognition. This method can convert general data into visual signals, possesses high throughput, high fault tolerance, and multi-protocol adaptability, and is also secure, reliable, and cost-effective.

[0005] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0006] A multi-protocol output method for an edge computing gateway for video stream parsing and recognition includes the following steps:

[0007] S1. Extract the multi-modal data from the source end, convert it into a binary data stream and add metadata to obtain the original binary data packet, map the original binary data packet into a Data Matrix data array, and display it in a tiled manner at the source end;

[0008] S2. The display signal from the source end is transmitted to the edge computing gateway via the physical video interface for high-frequency sampling of the video stream;

[0009] S3. Process the video stream to restore the original binary data packets, thereby obtaining restored binary data packets;

[0010] S4. Output the restored binary data packet from step S3 to the downstream application.

[0011] Further, in step S1, the multi-modal data, including video, text, images, and sensor values, is split into multiple binary data streams. A redundancy check packet is calculated based on the data fault tolerance rate, and the redundancy check packet is placed into a binary data stream queue. From the binary data stream queue, each original binary data element is extracted one by one. Using the DataMatrix code structure, the original binary data is constructed into Data Matrix data, and metadata including the data format type, total number of blocks, and current block number is embedded in the metadata area of ​​the Data Matrix data header. The constructed Data Matrix data array is tiled and mapped onto the screen display area for tiled display at the source end.

[0012] Furthermore, in step S2, a video stream transmission link and an edge acquisition environment are established, and a one-way physical connection is established between the video output port of the source end and the video capture card of the edge computing gateway through a video cable; a high-frequency sampling rate is configured in the edge computing gateway, and the screen signal of the source end is captured in real time to form a video stream, which is a continuous time-series image stream.

[0013] Furthermore, in step S3, the steps of processing the video stream include:

[0014] S3.1 Input the video stream to the edge computing gateway;

[0015] S3.2 The edge computing gateway sequentially performs frame splitting, Data Matrix array recognition, binary data reconstruction, and redundancy correction decoding on the video stream to obtain the restored binary data packet;

[0016] Further, in step S3.2, after discretizing the video stream into a static frame sequence, the image processing algorithm is used to locate and identify the tiled Data Matrix code array in each frame to obtain the Data Matrix code array; the metadata in each code block in the Data Matrix code array is parsed, and the scattered binary fragments are sequentially spliced ​​into a complete binary data packet according to the total number of blocks and the sequence number; the integrity of the spliced ​​binary data packet is verified by the error correction algorithm of the Data Matrix and the error correction redundancy to obtain the restored binary data packet.

[0017] Furthermore, in the integrity verification in step S3.2, when data incompleteness is detected, the redundant verification packet is used to perform reverse calculation and repair with the existing data packet to restore the incomplete data; when no data incompleteness is detected, the restored binary data packet is output.

[0018] Furthermore, in step S4, dynamic type parsing and interface distribution are performed according to the metadata definition, and the restored binary data packet is output to the downstream application through the standard communication interface.

[0019] Furthermore, in step S4, the steps of dynamic type resolution and interface dispatch include:

[0020] S4.1 Read the data format definition field of the metadata of the restored binary data packet, and transcode the restored binary data packet into the originally defined type;

[0021] S4.2 Dynamically select the output channel based on the data type of the restored binary data packet.

[0022] Furthermore, the parsed binary data packet is signed using a digital certificate, and then encrypted again after signing.

[0023] An edge computing gateway multi-protocol output system for video stream parsing and recognition, comprising a method for implementing an edge computing gateway multi-protocol output for video stream parsing and recognition, including:

[0024] A multi-modal data visualization encoding module is used for data acquisition at the source end and maps the original binary data packets into a Data Matrix data array;

[0025] The edge computing gateway is used for data acquisition by the multi-modal data visualization encoding module and for generating restored binary data packets;

[0026] Downstream applications are used for data reception by the edge computing gateway.

[0027] The beneficial effects of this invention are:

[0028] In environments with limited physical interfaces, physical isolation, or indirect communication, compared to traditional OCR screen recognition technology, this invention adopts an encoding strategy based on a Data Matrix structure link and a binary universal stream. This breaks the limitation of only being able to transmit text data, and unifies the transmission of multi-modal data such as log files, images, and sensor values ​​into visual signals, achieving full data type compatibility. Combined with a redundancy check mechanism and a high-frequency sampling strategy, it can effectively resist signal jitter, noise interference, and packet loss caused by frame rate asynchrony during video transmission. Even in cases of partial occlusion or missing keyframes, the original data can still be restored losslessly through error correction algorithms, significantly improving the accuracy and robustness of data acquisition. It also supports multiple physical interfaces such as HDMI, VGA, and DisplayPort, exhibiting extremely high hardware compatibility and achieving a transmission rate of over 30KB / s, meeting the needs of industrial-grade real-time data acquisition. Through a metadata-driven dynamic parsing and distribution mechanism, the gateway can flexibly route the restored data to a cloud database or industrial PLC, achieving a seamless transformation from a closed display terminal to a smart IoT node. This invention enables highly reliable, non-intrusive data acquisition from dedicated servers that are closed, physically isolated, or have limited interfaces. It requires no hardware modifications to the source device or the opening of underlying software permissions. A one-way data transmission channel can be built simply through the video output interface, effectively solving the data silo problem of black-box devices in industrial sites, while maximizing the stability of the source system and the security of the intranet. Attached Figure Description

[0029] Figure 1 This is a flowchart of a preferred embodiment of the multi-protocol output method for an edge computing gateway used for video stream parsing and recognition according to the present invention.

[0030] Figure 2 This is a schematic diagram of the screen encoding layout of a multi-protocol output method for an edge computing gateway used for video stream parsing and recognition, according to a preferred embodiment of the present invention.

[0031] Figure 3 This is a block diagram of a preferred embodiment of the edge computing gateway multi-protocol output system for video stream parsing and recognition according to the present invention. Detailed Implementation

[0032] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0033] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used herein in the description of the invention is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. The term "and / or" as used herein includes any and all combinations of one or more of the associated listed items.

[0034] Please also see Figures 1 to 3 A preferred embodiment of the present invention provides a multi-protocol output method for an edge computing gateway for video stream parsing and recognition, comprising the following steps:

[0035] S1. Extract the multimodal data from the source end and convert it into a binary data stream. Calculate redundant data packets based on redundancy and add them to the binary data stream. Map the original binary data packets into a Data Matrix array and display it in a tiled format at the source end. In this embodiment, a source-end multimodal data visualization encoding module is constructed to extract the original business data from the target device to obtain multimodal data, such as text, images, or sensor values. Figure 2 The diagram shown is a schematic of the screen encoding layout based on the Data Matrix structure linking and metadata definition in this embodiment.

[0036] In step S1, the multi-modal data includes video, text, images, and sensor values. Before constructing the Data Matrix data array from the original binary data stream array using the Data Matrix code structure, the error correction redundancy of the Data Matrix is ​​set. In this embodiment, the error correction redundancy is set to approximately 26% to ensure data recovery even in cases of partial image occlusion or noise interference. Furthermore, metadata including the data format type, total number of blocks, and current block number is embedded in the metadata area of ​​each Data Matrix data header, and the Data Matrix data array is tiled and mapped onto the screen display area for tiled display at the source end.

[0037] In this embodiment, when splitting large files, the Reed-Solomon algorithm, also known as the Data Matrix error correction algorithm, is used to calculate all packets. If the error correction capability is 20%, the actual data sent will be 20% more. For example, a 10K file is split into 1K data packets, which theoretically would be split into 10 packets, but in reality, with error correction, it becomes 12 packets. This way, if any two packets are lost, the data can be recovered.

[0038] S2. The display signal from the source end is transmitted to the edge computing gateway through the physical video interface for high-frequency sampling of the video stream.

[0039] In step S2, a video stream transmission link and edge acquisition environment are established, and a one-way physical connection is established between the video output port of the source end and the video capture card of the edge computing gateway through a video cable; a high-frequency sampling rate is configured in the edge computing gateway, and the screen signal of the source end is captured in real time to form a video stream, which is a continuous time-series image stream.

[0040] This embodiment establishes a video streaming transmission link and edge acquisition environment using HDMI, VGA, DVI, or DisplayPort video cables. The edge computing gateway is configured with a video sampling rate of no less than 25-30 frames per second.

[0041] S3. Process the video stream to restore the original binary data packets, thereby obtaining the restored binary data packets.

[0042] In step S3, the steps for processing the video stream include:

[0043] S3.1 Input the video stream to the edge computing gateway.

[0044] The S3.2 edge computing gateway sequentially performs frame splitting, Data Matrix array recognition, binary data reconstruction, and redundancy correction decoding on the video stream to obtain the restored binary data packet.

[0045] In step S3.2, after discretizing the video stream into a static frame sequence, an image processing algorithm is used to locate and identify the tiled Data Matrix code array in each frame to obtain the Data Matrix code array. The metadata in each code block of the Data Matrix code array is parsed, and the scattered binary fragments are sequentially assembled into a complete binary data packet according to the total number of blocks and the sequence number. The integrity of the assembled binary data packet is verified using the Data Matrix error correction algorithm and error correction redundancy to obtain the restored binary data packet. In this embodiment, the image processing algorithm is the localization and recognition algorithm built into the Data Matrix algorithm.

[0046] In the integrity verification in step S3.2, when data incompleteness is detected, the redundant verification packet is used to perform reverse calculation and repair with the existing data packet to restore the incomplete data; when no data incompleteness is detected, the restored binary data packet is output.

[0047] S4. Output the restored binary data packet from step S3 to the downstream application.

[0048] In step S4, dynamic type parsing and interface distribution are performed based on the metadata definition, and the restored binary data packet is output to the downstream application through the standard communication interface.

[0049] In step S4, the dynamic type resolution and interface dispatch steps include:

[0050] S4.1 Reads the data format definition fields of the metadata of the restored binary data packet and converts the restored binary data packet into the original defined type; the original defined type includes JSON string, Modbus register value or original file.

[0051] S4.2 Dynamically select the output channel based on the data type of the restored binary data packet. In this embodiment, the output channel is dynamically selected based on the data type of the restored data packet. If it is a control command, it is sent via the RS232 / RS485 serial port; if it is structured data or a file, it is pushed via TCP / IP network in the form of HTTP or FTP.

[0052] The binary data packet is signed using a digital certificate after being parsed and restored, and then encrypted again after signing.

[0053] In this embodiment, the restored binary data packet is securely encapsulated after parsing, and a digital certificate mechanism is introduced to sign the decoded and restored data to ensure the authenticity of the data source; custom encryption is supported, allowing users to load custom encryption algorithms to encrypt the data packet content a second time to prevent the data from being stolen in subsequent network transmission.

[0054] like Figure 3 As shown, this embodiment also discloses an edge computing gateway multi-protocol output system for video stream parsing and recognition, used to implement an edge computing gateway multi-protocol output method for video stream parsing and recognition, including:

[0055] The multi-modal data visualization encoding module is used for data acquisition at the source end and maps the raw binary data packets into a Data Matrix array. The multi-modal data visualization encoding module interfaces with the source end, which is a closed, dedicated server. The source-side multi-modal data visualization encoding module communicates with the edge computing gateway via a physical video cable.

[0056] The edge computing gateway is used for data acquisition by the multi-modal data visualization encoding module and generates restored binary data packets. The edge computing gateway includes a video acquisition unit, a processing unit, and a security and protocol conversion module.

[0057] Downstream applications are used for data reception by the edge computing gateway. These downstream applications include a cloud platform and database, as well as an industrial PLC. The cloud platform and database communicate with the edge computing gateway via network cables, while the industrial PLC communicates with the edge computing gateway via serial cables.

[0058] This embodiment illustrates a multi-protocol output method for edge computing gateways used for video stream parsing and recognition, in conjunction with a specific industrial application scenario:

[0059] Combination Figure 3 The goal is to build an edge computing data acquisition environment for traffic inspection equipment manufacturing enterprises, with the aim of extracting real-time equipment operating parameters and log files from a local area network that cannot grant network access.

[0060] During the implementation at the data source end, we deployed a lightweight data front-end component device within the local area network. This component employs a binary-first processing strategy to overcome the limitation of traditional OCR, which can only recognize text.

[0061] The device first reads the underlying business data. Whether it's key-value pairs in the database, analog values ​​from sensors, or log files generated by the system, all are first converted into a unified binary data stream. The system encapsulates metadata before the binary data. This metadata explicitly defines the business attributes of the current data, with fields including the current data format definition, the total number of file blocks, the current block number, and a checksum. The system uses Data Matrix data array codes as the carrier and leverages their structural linking function to divide the encapsulated large data packets into several smaller data blocks. Based on the screen resolution, such as 1920×1080, the system displays these Data Matrix code blocks flatly on the screen. At the current resolution, a single frame can carry approximately 1KB of effective data. Simultaneously, the system uniformly sets the error correction redundancy to approximately 26%, meaning that even if there is noise or partial corruption in the transmitted image, the algorithm can ensure data readability.

[0062] During the physical connection and signal transmission phase, we use standard video cables to connect the device's video output port to the video input port of the edge computing gateway.

[0063] To accommodate different hardware environments, the connection cable can be HDMI, VGA, DVI, or DisplayPort (DP). If the source refresh rate and the gateway acquisition rate are synchronized at 25 to 60 frames per second, the theoretical data transfer rate can be stably maintained at 30KB / s to 70KB / s. This speed not only meets the needs of regular sensor data acquisition but also the real-time transmission requirements of log files and even small images.

[0064] During the data decoding and post-processing stage, the edge computing gateway, as an independent hardware entity, is equipped with a high-performance processor to perform decoding tasks.

[0065] The gateway captures video streams at a high frame rate, identifies the Data Matrix array in each frame, and extracts binary segments from it. Based on the sequence number in the metadata, the gateway reassembles multiple consecutive frames or multiple data blocks within the same frame into a complete binary packet. Subsequently, the gateway reads the data type field in the metadata for dynamic type mapping. If the data type is identified as a Modbus register, the gateway automatically converts the binary data into a register value and sends it to the PLC via its built-in RS485 interface; if the data type is identified as JSON or XML, the gateway converts it into a string and pushes it to the cloud via HTTP POST through the Ethernet interface; if the data type is identified as a raw binary file, it is transferred via the FTP protocol. Through this implementation, this embodiment is no longer limited to specific known formats, but achieves WYSIWYG universal data transmission through metadata-driven methods, successfully transforming a closed, non-networked display terminal into a high-throughput, highly compatible intelligent IoT data routing node.

[0066] This embodiment enables highly reliable, non-intrusive data acquisition from closed, physically isolated, or interface-restricted dedicated networks. It requires no hardware modifications to the source devices or granting access to underlying software. A one-way data transmission channel can be built solely through the video output interface, effectively solving the data silo problem of black-box devices in industrial settings. At the same time, it maximizes the stability of the source system and the security of the intranet, meeting the accuracy requirements of industrial-grade data acquisition.

Claims

1. A multi-protocol output method for an edge computing gateway used for video stream parsing and recognition, characterized in that, Includes the following steps: S1. Extract the multi-modal data from the source end, convert it into a binary data stream and add metadata to obtain the original binary data packet, map the original binary data packet into a Data Matrix data array, and display it in a tiled manner at the source end; S2. The display signal from the source end is transmitted to the edge computing gateway via the physical video interface for high-frequency sampling of the video stream; S3. Process the video stream to restore the original binary data packets, thereby obtaining restored binary data packets; S4. Output the restored binary data packet from step S3 to the downstream application.

2. The multi-protocol output method for an edge computing gateway for video stream parsing and recognition according to claim 1, characterized in that: In step S1, the multi-modal data, including video, text, images, and sensor values, is split into multiple binary data streams, and a redundancy check packet is calculated based on the data fault tolerance rate, and the redundancy check packet is placed into the binary data stream queue. Each original binary data element is taken out one by one from the binary data stream queue. The original binary data is constructed into Data Matrix data through the Data Matrix code structure. Metadata including data format type, total number of blocks and current block number is embedded in the metadata area of ​​the Data Matrix data header. The constructed DataMatrix array is tiled and mapped onto the screen display area for tiling display at the source end.

3. The multi-protocol output method for an edge computing gateway for video stream parsing and recognition according to claim 1, characterized in that: In step S2, a video stream transmission link and edge acquisition environment are set up, and a one-way physical connection is established between the video output port of the source end and the video acquisition card of the edge computing gateway through a video cable; The edge computing gateway is configured with a high-frequency sampling rate and the screen signal of the source end is captured in real time to form a video stream, which is a continuous time-series image stream.

4. The multi-protocol output method for an edge computing gateway for video stream parsing and recognition according to claim 2, characterized in that: In step S3, the steps for processing the video stream include: S3.1 Input the video stream to the edge computing gateway; S3.2 The edge computing gateway sequentially performs frame splitting, Data Matrix array recognition, binary data reconstruction, and redundancy correction decoding on the video stream to obtain the restored binary data packet.

5. The multi-protocol output method for an edge computing gateway for video stream parsing and recognition according to claim 4, characterized in that: In step S3.2, after the video stream is discretized into a static frame sequence, the Data Matrix code array in each frame is located and identified using an image processing algorithm to obtain the Data Matrix code array; the metadata in each code block of the Data Matrix code array is parsed, and the scattered binary fragments are sequentially spliced ​​into a complete binary data packet according to the total number of blocks and the sequence number; the integrity of the spliced ​​binary data packet is verified by the error correction algorithm of the Data Matrix and the error correction redundancy to obtain the restored binary data packet.

6. The multi-protocol output method for an edge computing gateway for video stream parsing and recognition according to claim 5, characterized in that: In the integrity verification in step S3.2, when data incompleteness is detected, the redundant verification packet is used to perform reverse calculation and repair with the existing data packet to restore the incomplete data; when no data incompleteness is detected, the restored binary data packet is output.

7. The multi-protocol output method for an edge computing gateway for video stream parsing and recognition according to claim 1, characterized in that: In step S4, dynamic type parsing and interface distribution are performed according to the metadata definition, and the restored binary data packet is output to the downstream application through the standard communication interface.

8. The multi-protocol output method for an edge computing gateway for video stream parsing and recognition according to claim 7, characterized in that: In step S4, the dynamic type resolution and interface dispatch steps include: S4.1 Read the data format definition field of the metadata of the restored binary data packet, and transcode the restored binary data packet into the originally defined type; S4.2 Dynamically select the output channel based on the data type of the restored binary data packet.

9. A multi-protocol output method for an edge computing gateway for video stream parsing and recognition according to claim 7, characterized in that: The digital certificate is used to sign the parsed binary data packet, and then the signed data packet is encrypted again.

10. A multi-protocol output system for an edge computing gateway for video stream parsing and recognition, used to implement the multi-protocol output method for an edge computing gateway for video stream parsing and recognition as described in claim 1, characterized in that, This includes a multi-modal data visualization encoding module, an edge computing gateway, and downstream application terminals. A multi-modal data visualization encoding module is used for data acquisition at the source end and maps the original binary data packets into a Data Matrix data array; The edge computing gateway is used for data acquisition by the multi-modal data visualization encoding module and for generating restored binary data packets; Downstream applications are used for data reception by the edge computing gateway.

Images