Low-delay video transmission method and system for remote control of engineering machinery

Abstract

The embodiment of the invention discloses a low-delay video transmission method and system for remote control of engineering machinery. The system comprises the remotely controlled engineering machinery, a client for remotely controlling a cockpit, a server and a photoelectric switch, the engineering machinery adopts a WIFI (Wireless Fidelity) 6 access network, and the rest adopt optical fiber communication; the video stream pushing end generates video data by adopting H.264 and transmits the video data by adopting UDP (User Datagram Protocol); and decoding and displaying by a video receiving end. According to the low-delay video transmission scheme provided by the embodiment of the invention, a feasible method is provided for remote control low-delay video transmission of engineering machinery, and the method is mainly a method for constructing a network architecture and optimizing video encoding and decoding and network transmission by hardware and software. In the aspect of network architecture construction, a new generation of WIFI 6 is used for transmitting wireless signals, other parts are in wired connection with a gigabit network cable through optical fibers, free movement of engineering machinery is guaranteed, and stability and high speed of network signals are also guaranteed.

Description

Technical Field

[0001] The present invention relates to the technical field of remote control engineering machinery, and particularly relates to a low-latency video transmission method and system for remote control of engineering machinery. Background Art

[0002] The degree of intelligence and automation of engineering machinery is constantly improving. Especially in the fields of construction, mining, and other construction areas, remote control technology has increasingly become an important operation method. Through remote control, operators can effectively manage complex and dangerous construction tasks in a safe and comfortable environment.

[0003] The remote control of engineering machinery will improve production efficiency, reduce labor intensity, enhance safety levels, and save costs. In the existing remote control of engineering machinery, video images provide the main visual feedback, and the performance of video transmission directly affects the safety and efficiency of operation. Especially in a dynamic environment, low-latency and high-quality video transmission are particularly important, which not only relates to the decision-making ability of the operator but also directly affects the operation results. In order to achieve precise control of engineering machinery, requirements such as low transmission latency, clear pictures, and small network bandwidth occupancy are put forward for video transmission.

[0004] Currently, in the prior art of video stream transmission for remote control of engineering machinery, it mainly relies on the following methods:

[0005] (1) 5G transmission: Using the 5G network for video transmission, the low-latency characteristic of 5G technology enables the remote control system to transmit video streams in real time. At the same time, the 5G network provides a relatively high bandwidth, which can stably transmit high-definition video streams.

[0006] (2) Radio wave transmission: A communication method that uses electromagnetic waves to propagate in the air to transmit information. The control end or device encodes information such as video signals into radio wave signals through modulation technology and then transmits them through an antenna.

[0007] (3) Wireless transmission: Wi-Fi uses a wireless access point (Access Point, AP) as the core node to provide wireless communication services for connected devices. The video signal is digitized and encoded into binary data, and then converted into a signal form suitable for wireless transmission through modulation technology.

[0008] During the video transmission of remote control of construction machinery, although 5G network transmission can provide functions such as low latency, high bandwidth, and wide coverage, construction machinery generally operates in some remote areas or underground working environments, and the coverage of 5G signals is still limited, unable to provide a stable network connection, which will affect the reliability of the remote control system. At the same time, many current devices do not fully support 5G networks, especially in some old construction machinery. To ensure the smooth operation of the system, equipment upgrades or replacements may be required, which will increase the cost and implementation difficulty of the project. In addition, although 5G technology has advantages in encryption and authentication, network security is still an important issue during the remote control of construction machinery. Security threats such as hacker attacks and signal hijacking may affect the security of the remote control system, resulting in system out-of-control or misoperation of mechanical equipment.

[0009] Radio wave transmission provides a flexible and convenient operation method. However, the propagation of radio waves is easily interfered by the surrounding environment. Near buildings, metal obstacles, or other electromagnetic wave sources (such as power lines, large mechanical equipment, etc.), the signal may attenuate or distort. In addition, weather factors (such as heavy rain, heavy snow, thunderstorms, etc.) may also cause signal attenuation, affecting the quality of the video stream. Additionally, the effective transmission distance of radio waves is usually limited. In complex construction site environments, such as underground operation areas or areas with a large number of obstacles, the coverage of wireless signals is still restricted. Long-distance transmission may cause signal weakening, thereby affecting the stability of the video stream. There may be various devices and electrical systems in the environment where construction machinery is located, and the electromagnetic radiation of these devices will interfere with radio wave signals, affecting the signal quality. For example, large equipment such as construction machinery and generators may cause electromagnetic interference to radio waves, thus affecting the stability of video stream transmission.

[0010] Based on this, the existing technologies cannot fully meet the low-latency and high-quality transmission of video streams in the remote control of construction machinery. Summary of the Invention

[0011] Aiming at the technical defects existing in the prior art, the purpose of the embodiments of the present invention is to provide a low-latency video transmission method and system for the remote control of construction machinery.

[0012] To achieve the above purpose, in the first aspect, the embodiments of the present invention provide a low-latency video transmission method for the remote control of construction machinery, which is applicable to a low-latency video transmission system. The low-latency video transmission system includes a construction machinery to be remotely controlled, a client of a remote control cockpit, a server, and an optical and electrical switch; the construction machinery accesses the network using WIFI6, and fiber optic communication is used between the client, the server, and the optical and electrical switch; the low-latency video transmission method includes:

[0013] The video streaming end uses H.264 to generate video data to be transmitted and uses the UDP protocol to transmit the video data outward;

[0014] The video receiving end receives and decodes the video data to obtain decoded data and displays the decoded data.

[0015] As a preferred implementation manner of the present application, the method further includes:

[0016] In the process of transmitting the video data using the UDP protocol, a method combining an error correction mechanism and a retransmission mechanism is introduced.

[0017] As a specific implementation manner of the present application, the video streaming end uses the UDP protocol to transmit the video data outward, specifically:

[0018] The video data is divided into multiple data blocks;

[0019] Using Reed-Solomon coding, redundant data is generated through a specific algorithm; the redundant data is used to recover lost data packets;

[0020] The multiple data blocks and the redundant data are sent to the video receiving end through the UDP protocol.

[0021] As a specific implementation manner of the present application, sending the multiple data blocks and the redundant data to the video receiving end through the UDP protocol, specifically:

[0022] Select a transmission path based on the Dijkstra algorithm;

[0023] Different data blocks are sent to the video receiving end using different transmission paths.

[0024] Wherein, each data block includes video data with a serial number or a timestamp identifier; specifically, the video receiving end obtains the decoded data as follows:

[0025] Receive multiple data blocks through different transmission paths;

[0026] Sort and reorganize the multiple data blocks through the serial number or the timestamp identifier to obtain a reorganized data packet;

[0027] Merge and restore the reorganized data packet to obtain decoded data.

[0028] Further, as a preferred implementation manner of the present application, before obtaining the reorganized data packet, the method further includes:

[0029] Perform lost packet detection on the reorganized data packet;

[0030] If there are lost packets in the reorganized data packet, the redundant data is used for recovery.

[0031] In a second aspect, an embodiment of the present invention provides a low-latency video transmission system for remote control of construction machinery, including the construction machinery to be remotely controlled, a client of a remote control cockpit, a server, and an optical-electrical switch; the construction machinery accesses the network using WIFI6, and fiber-optic communication is used between the client, the server, and the optical-electrical switch; the construction machinery includes a video streaming end, which is used to generate video data to be transmitted using H.264 and transmit the video data outward using the UDP protocol;

[0032] The client and the server form a video receiving end, which is used to receive and decode the video data to obtain decoded data and display the decoded data.

[0033] The low-latency video transmission solution of the embodiment of the present invention provides a feasible method for low-latency video transmission in the remote control of construction machinery, mainly including the construction of the network architecture and the optimization methods of hardware and software for video encoding and decoding and network transmission.

[0034] In terms of the construction of the network architecture, a new generation of WIFI6 is used to transmit wireless signals, and other parts are connected by wire through optical fibers and gigabit network cables, which not only ensures the free movement of the construction machinery but also ensures the stability and high speed of the network signal. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] In order to more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the following will briefly introduce the drawings required for use in the description of the specific embodiments or the prior art.

[0036] Figure 1 is the architecture diagram of the low-latency video transmission system for remote control of construction machinery provided by the embodiment of the present invention;

[0037] Figure 2 and Figure 3 is the schematic diagram of the communication between the wireless transmitting end and the construction machinery;

[0038] Figure 4 is the video encoding and decoding transmission flow chart;

[0039] Figure 5 is the overall control video stream transmission flow chart;

[0040] Figure 6 is the flow chart of the low-latency video transmission method for remote control of construction machinery provided by the embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0041] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0042] It should be understood that when used in this specification and the appended claims, the terms "comprising" and "including" indicate the presence of the described features, wholes, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components, and / or their combinations.

[0043] The inventive concept of the present invention is:

[0044] The existing technologies cannot fully meet the low-latency and high-quality transmission of video streams in the remote control of construction machinery. It is necessary to build a video stream low-latency transmission system that can effectively adapt to the remote control system of construction machinery to ensure the safe and efficient operation of the remote control of engineering technology.

[0045] The present invention builds a network communication architecture based on fiber optic and WIFI6 technologies to construct a low-latency video transmission system for the remote control of construction machinery. At the same time, encoding and decoding algorithms are used at the sending end and the receiving end to quickly encode and decode the video stream, optimize network transmission, and finally achieve low-latency video transmission between the construction machinery and the control cockpit.

[0046] The present invention includes technologies such as the communication architecture between construction machinery and the server, video encoding and decoding, and network transmission optimization, aiming to provide a solution that can maintain low latency during the remote control video transmission process. This technology provides stable, low-latency, and high-quality visual support for the remote control of various construction machinery and equipment.

[0047] Please refer to Figure 1 , the embodiments of the present invention provide a low-latency video transmission system for the remote control of construction machinery, including the construction machinery to be remotely controlled, the client of the remote control cockpit, the server, and the optical and electrical switch; the construction machinery accesses the network using WIFI6, and fiber optic communication is used between the client, the server, and the optical and electrical switch; the construction machinery includes a video streaming end, which is used to generate video data to be transmitted using H.264 and transmit the video data outward using the UDP protocol.

[0048] The client and the server form a video receiving end, which is used to receive and decode the video data to obtain decoded data and display the decoded data.

[0049] In remote-controlled construction machinery, for the construction machinery to operate freely, the way for the construction machinery to access the network can only be through wireless connection. The WIFI6 connection is adopted in the embodiments of the present invention. Compared with 4 / 5G technology, WIFI6 has lower costs. Compared with the previous generation of WIFI, it can provide faster data transmission, smoother video streams and lower latency at a lower cost.

[0050] WiFi6 introduces the OFDMA (Orthogonal Frequency Division Multiple Access) technology, which can divide the channel into multiple sub-channels, enabling multiple devices to share the same channel simultaneously, thereby improving network efficiency. When the number of controlled construction machinery increases, it will not affect the video transmission efficiency, as Figure 2 and Figure 3 shown. WiFi6 also introduces the bidirectional MU-MIMO (Multi-User Multiple Input Multiple Output) technology. Compared with WIFI5 which only supports downlink MU-MIMO, WIFI6 allows two-way communication with multiple devices simultaneously, reducing the latency when video data and other data are transmitted simultaneously.

[0051] For other parts, wired connections are used with optical fibers to ensure a gigabit network speed, minimizing latency and enhancing anti-interference. Among them, the number of groups of the optical and electrical switches can be flexibly adjusted according to the site, enabling the coverage area of the wireless network to change according to requirements.

[0052] Based on Figure 1 the network architecture shown, the embodiments of the present invention provide a low-latency video transmission optimization scheme, which will be introduced in detail below.

[0053] Video codec delay and network transmission delay are the main parts of the video transmission system delay. Therefore, on the premise of limited network bandwidth and ensuring clear video quality, reducing video codec delay and optimizing network transmission are the main goals for reducing video transmission delay.

[0054] I. Reducing Video Codec Delay

[0055] Reducing video codec delay can be achieved through two aspects: hardware acceleration and coding algorithm optimization. In terms of hardware, using dedicated decoder and encoder chips or GPUs can effectively accelerate the video codec process and significantly reduce the processing time. In terms of coding algorithms, adopting efficient compression algorithms and optimized coding parameter settings to reduce redundant data transmission can further reduce the delay. In addition, reasonably selecting suitable bitrate, resolution, and frame rate parameters can also optimize the latency performance while ensuring video quality. The video codec transmission flow chart is as Figure 4 shown.

[0056] Furthermore, regarding the hardware aspect, the following is an explanation:

[0057] The video encoding and decoding selects H.264. Compared with the new generation of encodings such as H.265 and AV1, although it occupies more space, it has lower requirements for the performance of hardware encoding and decoding, stronger compatibility, and can be used on more devices. In particular, some of the existing construction machinery that can be remotely controlled are transformed from traditional construction machinery. And since the final client used in the present invention is a browser, H.264 encoding does not require additional transcoding processing, which is conducive to reducing latency.

[0058] The present invention uses hardware encoding. The specific implementation is to use a camera that supports the GMSL2 high-speed serial port, then connect it to a video capture card, and use the Orin platform with the Linux system to perform video encoding and push the stream to the server. The push stream format selects RTSP. The server is installed with the video server software Mediamtx, and then the client accesses the corresponding pull stream address. The overall control of the video stream transmission process is as Figure 5 shown. The host used by the client should support H.264 hardware decoding. After receiving the stream, the client writes web page code and uses JavaScript to draw the video image on the canvas. In this way, compared with directly playing the video using the video tag, it has better compatibility with the browser, less loss of picture quality, and similar latency. In addition, when encoding the video, the resolution needs to be controlled at 1920*1080@30FPS or lower, and at the same time, while meeting the visual requirements, the bit rate should be minimized to reduce latency.

[0059] II. Optimize network transmission

[0060] Regarding the optimization of network transmission, the embodiments of the present invention are mainly implemented from the following two aspects:

[0061] 1. Video transmission based on UDP protocol

[0062] In the present invention, the User Datagram Protocol (UDP) is used for video transmission. As a connectionless protocol, the main advantage of UDP is that it does not require establishing a connection and confirming the receipt of data packets, thus achieving more efficient transmission and lower latency. Although UDP does not have a retransmission mechanism, which means that even if some data packets are lost, the video stream can continue to be transmitted. Therefore, UDP is particularly suitable for real-time communication scenarios such as video and voice transmission.

[0063] Although UDP has the advantage of low latency, relevant technologies still need to be introduced to mitigate the impact of data packet loss on video quality. The present invention adopts a method that combines an error correction mechanism and a retransmission mechanism to improve the reliability of data transmission.

[0064] Specifically, the error correction mechanism adopts Forward Error Correction (FEC). This mechanism realizes the detection and correction of errors by embedding redundant information in the original data. The sender uses a specific algorithm to encode the data to generate data packets containing redundant information, while the receiver can correct the errors in the data or recover the lost parts through this redundant information. The core concept of forward error correction is that even if some data packets are lost at the receiver, the missing data can be recovered through the redundant data packets without requesting retransmission, thus avoiding delays and bandwidth consumption. The implementation steps of the forward error correction algorithm are as follows:

[0065] (1) Data chunking: The video data is segmented into multiple data chunks (or data packets) and sent to the receiver one by one.

[0066] (2) Adding redundant information: Using Reed-Solomon coding, redundant data sufficient to recover the lost data packets is generated through a specific algorithm and sent together with the original data.

[0067] (3) Sending data: All data chunks are sent to the receiver through the UDP protocol.

[0068] (4) Lost packet detection: The receiver checks the integrity of the received data packets. If no data packets are lost, the received data is directly decoded.

[0069] (5) Lost packet recovery: For the lost data packets, the receiver recovers them based on the redundant information. The amount of redundant information depends on the coding method and the number of lost data packets.

[0070] (6) Completing video data: The client decodes all the received data (including the original data and the redundant data) to restore the complete video stream.

[0071] 2. Video transmission based on the multi-path transmission method

[0072] The multi-path transmission technology can transmit data simultaneously through multiple different network paths to improve the transmission reliability, bandwidth, and delay control ability, and enhance fault tolerance. During the transmission of the remote control video stream of construction machinery, multi-path transmission effectively reduces the impact of packet loss or congestion on a single path on the video quality, thereby improving the stability and smoothness of video transmission.

[0073] The video stream transmission on construction machinery is based on splitting the stream data into multiple parts and transmitting them in parallel through independent network paths (such as different physical links and Wi-Fi access points). The selection of network transmission paths is implemented based on the Dijkstra algorithm to find the shortest path in the network topology and forward the data. The receiving end is responsible for recombining and merging the data parts from different paths to ensure the integrity of the data and present it in the correct order.

[0074] Among them, the specific process of the video stream transmission for the remote control of construction machinery is as follows:

[0075] (1) Data segmentation at the sending end: The video stream is compressed and segmented into multiple data packets. Each data packet carries part of the video data and is marked with a sequence number or timestamp.

[0076] (2) Multi-path sending: The sending end will select multiple paths to send the data packets. For example, send some data packets through one path and other data packets through another path. The selection of the transmission path is dynamically evaluated by the Dijkstra path algorithm.

[0077] (3) Data transmission in the network: The data packets reach the receiving end through different paths. The network conditions on each path will affect the transmission speed and reliability of the data.

[0078] (4) Data recombination at the receiving end: After receiving the data packets from multiple paths, the receiving end sorts and recombines them through the sequence number or timestamp. If some data packets are lost, the receiving end may obtain redundant data from other paths or use forward error correction technology to recover the lost data.

[0079] (5) Video playback: The receiving end merges and restores the decoded data stream. If some data is lost, the receiving end uses redundant data or error correction data to repair it, thus maintaining the stability of the video stream.

[0080] Based on the same inventive concept, the embodiments of the present invention provide a low-latency video transmission method for the remote control of construction machinery, which is applicable to the system of the foregoing embodiments. As Figure 6 shown, the low-latency video transmission method includes:

[0081] S1, the video streaming end generates the video data to be transmitted using H.264 and transmits the video data outward using the UDP protocol;

[0082] S2, the video receiving end receives and decodes the video data, obtains the decoded data, and displays the decoded data.

[0083] Among them, in the process of transmitting the video data between the video streaming end and the video receiving end using the UDP protocol, a method combining an error correction mechanism and a retransmission mechanism is introduced.

[0084] Furthermore, the video streaming end uses the UDP protocol to transmit the video data outward, specifically as follows:

[0085] The video data is divided into multiple data blocks;

[0086] Using Reed-Solomon coding, redundant data is generated through a specific algorithm; the redundant data is used to recover lost data packets;

[0087] Multiple data blocks and redundant data are sent to the video receiving end through the UDP protocol.

[0088] Furthermore, sending multiple data blocks and redundant data to the video receiving end through the UDP protocol is specifically as follows:

[0089] Based on the Dijkstra algorithm, a transmission path is selected;

[0090] Different data blocks are sent to the video receiving end using different transmission paths.

[0091] Among them, each data block includes video data with a serial number or timestamp identifier; specifically, the video receiving end obtains the decoded data as follows:

[0092] Receive multiple data blocks through different transmission paths;

[0093] Sort and reorganize multiple data blocks through the serial number or timestamp identifier to obtain a reorganized data packet;

[0094] Perform lost packet detection on the reorganized data packet;

[0095] If there are lost packets in the reorganized data packet, use the redundant data for recovery;

[0096] Merge and restore the reorganized data packet to obtain the decoded data.

[0097] It can be known from the above description that the low-latency video transmission solution of the embodiment of the present invention provides a feasible method for low-latency video transmission in remote control of construction machinery, mainly including the construction of the network architecture and the optimization methods of video encoding and decoding, and network transmission by hardware and software.

[0098] In terms of network architecture construction, a new generation of WIFI6 is used to transmit wireless signals, and other parts are connected by fiber optic cables and gigabit network cables, which not only ensures the free movement of construction machinery but also ensures the stability and high speed of network signals.

[0099] In terms of video encoding / decoding and optimizing network transmission, a hardware architecture is proposed. By using a camera supporting the gmsl2 serial port, together with a capture card and the Orin platform, high-quality and high-speed video encoding / decoding is achieved. The UDP protocol transmission and multi-path transmission methods are adopted to ensure low latency and stability of video transmission.

[0100] As described above, the above are only specific embodiments of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art within the technical scope disclosed by the present invention can easily think of various equivalent modifications or substitutions, and these modifications or substitutions should be covered within the protection scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.

Claims

1. A low-latency video transmission method for remote control of construction machinery, characterized in that: The low-latency video transmission method is applicable to a low-latency video transmission system, which includes a remotely controlled engineering machine, a client for remotely controlling a cockpit, a server, and an optoelectronic switch; the engineering machine uses WIFI6 to access the network, and optical fiber communication is used between the client, the server, and the optoelectronic switch; The low-delay video transmission method comprises: The video streaming end uses H.264 to generate the video data to be transmitted, and uses the UDP protocol to transmit the video data externally; The video receiving end receives and decodes the video data to obtain decoded data, and displays the decoded data.

2. The method according to claim 1, characterized in that The method further comprises: In the process of transmitting the video data using the UDP protocol, an error correction mechanism combined with a retransmission mechanism is introduced.

3. The method according to claim 2, characterized in that The video streaming end uses the UDP protocol to transmit the video data externally, specifically: Splitting the video data into a plurality of data blocks; Using Reed-Solomon coding, redundant data is generated through a specific algorithm; the redundant data is used to recover lost data packets; Multiple data blocks and redundant data are sent to the video receiving end via the UDP protocol.

4. The method according to claim 3, characterized in that Send multiple data blocks and redundant data to the video receiving end through the UDP protocol, specifically: Select the transmission path based on Dijkstra algorithm; Different data blocks are sent to the video receiving end using different transmission paths.

5. The method according to claim 3, characterized in that Each data block includes video data with a sequence number or timestamp; the decoded data obtained by the video receiving end is specifically: receiving a plurality of data blocks via different transmission paths; Sorting and reassembling multiple data blocks by sequence numbers or timestamp identifiers to obtain reassembled data packets; The reassembled data packets are merged and restored to obtain decoded data.

6. The method according to claim 5, characterized in that Before obtaining the reassembled data packet, the method further includes: Performing packet loss detection on the reassembled data packet; If there is packet loss in the reassembled data packet, redundant data is used for recovery.

7. A low-latency video transmission system for remote control of engineering machinery, comprising a remotely controlled engineering machinery, a client for remotely controlling a cockpit, a server, and an optoelectronic switch; characterized in that: The engineering machinery uses WIFI6 to access the network, and optical fiber communication is used between the client, server and optoelectronic switch; the engineering machinery includes a video streaming terminal, which is used to use H.264 to generate video data to be transmitted, and uses UDP protocol to transmit the video data externally; The client and the server constitute a video receiving end, which is used to receive and decode the video data, obtain decoded data, and display the decoded data.

8. The system according to claim 7, characterized in that The video streaming end and the video receiving end are connected by introducing an error correction mechanism and a retransmission mechanism to transmit video data.

9. The system according to claim 7, characterized in that The video streaming end transmits video data externally, specifically: Splitting the video data into a plurality of data blocks; Using Reed-Solomon coding, redundant data is generated through a specific algorithm; the redundant data is used to recover lost data packets; Select the transmission path based on Dijkstra algorithm; Different data blocks are sent to the video receiving end using different transmission paths.

10. The system according to claim 9, characterized in that Each data block includes video data with a sequence number or timestamp; the decoded data obtained by the video receiving end is specifically: receiving a plurality of data blocks via different transmission paths; Sorting and reassembling multiple data blocks by sequence numbers or timestamp identifiers to obtain reassembled data packets; Performing packet loss detection on the reassembled data packet; If there is packet loss in the reassembled data packet, redundant data is used for recovery; The reassembled data packets are merged and restored to obtain decoded data.

Images