Video transmission and instruction interaction system and method for industrial remote control

Abstract

The invention relates to a video transmission and instruction interaction system and method for industrial remote control, and the system and method achieve the low-bandwidth transmission in a manner of reducing transmission data under the ultra-low time delay requirement when video image transmission is carried out, except for the first I frame, in the whole period of P frame transmission. An intelligent packet loss processing strategy (static neglect and dynamic / key priority retransmission) effectively utilizes the bandwidth, avoids invalid retransmission, and ensures the picture integrity of a key area to the maximum extent, thereby realizing ultra-low-delay high-reliability transmission.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of industrial automation control and remote operation technology, in particular to a video transmission and instruction interaction system and method for industrial remote control. BACKGROUND

[0002] Industrial remote control needs to transmit the computer interface (mostly special software UI, including large-area static background such as drawings, framework and small-area dynamic elements such as parameters, status lights) of the equipment in real time and return the operation instructions (mouse, keyboard).

[0003] The existing industrial remote control transmission has the following problems: High latency: relying on buffer, periodic I frame (large volume, slow transmission and easy packet loss), complex retransmission mechanism, resulting in obvious operation delay.

[0004] Bandwidth fluctuation sensitive: burst code rate (especially I frame) is easy to exceed limited public network bandwidth (50M / 100M), resulting in packet loss, screen flashing and lag.

[0005] Insufficient reliability: general packet loss recovery (such as full frame retransmission, FEC) is low in efficiency and does not consider the characteristics of industrial UI (static area tolerates packet loss).

[0006] Uncontrollable instruction delay: control instructions are not prioritized with video streams and may be delayed when the network is congested.

[0007] Therefore, there is an urgent need for a solution that takes into account the characteristics of industrial control UI and can achieve ultra-low latency, high reliability transmission and interaction under limited bandwidth. SUMMARY

[0008] In view of the problems existing in the prior art, the purpose of the present application is to provide a low-latency video transmission and instruction interaction system and method for industrial remote control, which reduces the end-to-end video transmission delay.

[0009] To achieve the above purpose, the technical solution adopted by the present application is as follows: A video transmission and instruction interaction system for industrial remote control, comprising a device end and a control end, the device end and the control end being connected by network communication; The device end is provided with A video capture module for capturing the screen image of the control computer of the equipment; An image analysis module for identifying static areas, dynamic areas and key areas, the key areas being preset by the control end; A video encoder is configured to encode video images transmitted from a device to a control terminal. The first frame of the transmitted video images is encoded using an I frame, and subsequent frames are continuously encoded using P frames. Static regions in the video images are encoded using skip encoding or using a higher QP value. Dynamic regions or regions of interest are encoded using a lower QP value. An audio capture and synchronization module is configured to capture device audio and pack audio data blocks with the same PTS as corresponding video image frames in the same network transmission unit. A network sending module is configured to send the packed video data packets and audio data packets, and receive NACK / FEC requests and instruction confirmations from the control terminal. The control terminal is provided with A network receiving module is configured to receive audio and video data packets. A packet loss detection and processing module is configured to detect lost video packets, and determine the type of the lost data packets using key region, static region, and dynamic region mapping information. For static region packet loss, the corresponding region data of the previous frame is used for filling. For dynamic region packet loss and key region dynamic, NACK retransmission is preferred. A NACK request for the lost packet is immediately sent, and a short retransmission timeout is set. If the retransmitted packet is not received within the timeout period, or the cumulative packet loss affects key decoding, a local / entire frame error correction frame is requested from the device terminal. A video decoder is configured to decode video streams and process error concealment. An audio decoding and synchronization module is configured to unpack audio data and video frames, and perform audio-visual synchronization playback using the same PTS carried by the video frames and the audio data. A display module is configured to display the decoded video images, and an audio playback module is configured to play the synchronized audio.

[0010] The video encoder of the device terminal estimates the available bandwidth and network conditions in real time during the transmission of the video images, and dynamically adjusts the overall or regional QP value.

[0011] The device terminal is further provided with a bandwidth limiter configured to split the P frame data into at least two sub-data blocks when the P frame data exceeds a bandwidth threshold, and send the sub-data blocks in consecutive transmission periods. Each sub-data block carries a split marker and a sequence number. The control terminal is further provided with a frame reassembler configured to reassemble a complete frame according to the split marker and the sequence number when the split marker is detected.

[0012] The device terminal is further provided with a sending buffer module configured to design a very small sending buffer or bypass the sending buffer using a zero-copy technology.

[0013] The control end is also provided with an instruction input and sending module for capturing user mouse and keyboard operations and sending them to the device end through a high-priority channel.

[0014] The image analysis module, the video encoder, the bandwidth limiter, and the audio capture and synchronization module of the device end are implemented on an FPGA.

[0015] A video transmission and instruction interaction method for industrial remote control comprises the following steps: Step 1, initialization: the device end sends a complete I frame, the control end successfully receives, decodes and displays, and establishes static region information; Step 2, device end continuous transmission cycle: Capture a screen frame; Image analysis: update the static region, dynamic region, and focus region mapping; Video encoding: use P frame encoding, skip or high QP value encoding for the static region, and low QP value encoding for the dynamic region and the focus region; estimate the available bandwidth and network status in real time, and dynamically adjust the overall or regional QP value; Audio capture and synchronization: capture audio, bind the same PTS as the current video frame, and package; Send a data packet; When receiving the NACK / FEC request of the control end, resend the packet or generate a sending error correction frame according to the request content; Step 3, control end continuous processing cycle: Receive a data packet, and preferentially process control instructions and execute them; Detect video packet loss, and when packet loss occurs, perform intelligent packet loss processing: classify the packet loss region according to the region mapping information; if it is static region packet loss, ignore or simply hide it; if it is dynamic region packet loss or focus region packet loss: immediately send a NACK request; if it is not recovered after timeout, send an FEC / I frame request; Video decoding: decode the video data packet; when static region packet loss occurs, use available packets and error concealment technology for decoding; Audio and video unpacking and synchronization: synchronize and play audio and video using PTS, display the picture, and play the sound.

[0016] In step 2, when the P frame data volume exceeds the bandwidth threshold, the device end splits the P frame data into at least two sub-data blocks for continuous transmission cycle sending, and each sub-data block carries a split marker and a sequence number. In step 3, when receiving a data packet, the control end detects whether the data packet carries a split marker, if it carries a split marker, buffers the data, and when data packets with the same split marker are all received, recombines them into a complete frame.

[0017] In step 3, when the control end captures the user operation instruction, the user operation instruction is sent to the device end through a high-priority channel.

[0018] After the above scheme is adopted, when the video image is transmitted, except the first I frame, other full cycle P frame transmission is transmitted, under the requirement of ultra-low delay (that is, cannot be cached or as little as possible), low bandwidth transmission is realized by reducing the transmission data (I frame data is much larger than P frame), end-to-end delay is significantly reduced, intelligent packet loss processing strategy (static ignoring, dynamic / key priority retransmission) effectively utilizes the bandwidth, avoids invalid retransmission, maximally guarantees the completeness of the key area picture, and thus the ultra-low delay and high reliability transmission are realized. BRIEF DESCRIPTION OF DRAWINGS

[0019] Figure 1 It is a principle block diagram of the system of the application; Figure 2 It is an image analysis and processing flowchart of the application; Figure 3 It is a schematic diagram of audio and video synchronous transmission; Figure 4 It is a processing flowchart of the bandwidth limiter; Figure 5 It is a processing flowchart of the frame reorganizer. DETAILED DESCRIPTION

[0020] As shown in the figure, Figure 1 The application discloses a video transmission and instruction interaction system for industrial remote control, which comprises a device end and a control end, and the device end and the control end are connected through network communication. The application realizes the system and method for realizing ultra-low delay and high reliability video transmission and instruction interaction of an industrial device control computer remote operation interface under the condition of limited public network bandwidth, and is especially suitable for the scene of remote monitoring and operation of unmanned workshops and unmanned devices.

[0021] In the application, the device end is provided with a video capture module, an image analysis module, a video encoder, an audio capture and synchronization module, a bandwidth limiter, a sending buffer module and a network sending module.

[0022] The video capture module is used for capturing the screen image of the device control computer.

[0023] As shown in the figure, Figure 2As shown, the image analysis module uses video image change statistics (non-active areas automatically identified by continuously counting the pixel change rate of N frames) combined with segmentation techniques (such as edge detection, template matching, machine learning) to identify large static areas (pipeline diagrams, frames, etc.) and dynamic areas (parameter values, status lights, control buttons) in the UI (computer screen image for control). The image analysis module is also used to identify key areas in the UI, which are pre-set by the control end. All key areas form a key area set. Of course, static areas can also be manually pre-set.

[0024] The video encoder is used to encode the video images transmitted by the device end to the control end: the first frame of the transmitted video image is encoded using an I frame, and the subsequent frames are continuously encoded using P frames; for static areas in the video image, skip encoding or using a higher QP value for encoding is used to greatly save the code rate; for dynamic areas or areas of interest, a lower QP value is used for encoding to ensure clarity. During transmission, the available bandwidth and network conditions are estimated in real time, and the overall or regional QP value is dynamically adjusted to control the output code rate. The QP value is a quantization index in the encoding process. The lower the QP, the better the detail, and the larger the data. During system operation, periodic I frames are disabled, and only the first I frame is initialized. Only when a request is received from the control end, complete / local I frame encoding is performed. The video encoder in the present application supports H.264 / H.265 / AV1.

[0025] As shown in Figure 3 The audio capture and synchronization module is used to capture device audio and package audio data blocks with the same uniform time point (PTS) and corresponding video image frames in a network transmission unit (such as an RTP packet) to ensure that audio and video data are "born and die together" on the network.

[0026] As shown in Figure 4 The bandwidth limiter is used to split P frame data into at least two sub-data blocks for transmission in consecutive transmission periods when the P frame data exceeds the bandwidth threshold, each sub-data block carrying a split marker and a sequence number to allow the control end to reassemble the complete frame based on the split marker and the sequence number.

[0027] The sending buffer module is designed with a very small sending buffer or bypasses the sending buffer using zero-copy technology to reduce latency.

[0028] The network sending module sends the packaged video data packets and audio data packets, and receives NACK / FEC requests and instruction confirmations from the control end.

[0029] The control end is provided with a network receiving module, a frame reassembler, a packet loss detection and processing module, a video decoder, an audio decoding and synchronization module, an instruction input and sending module, a display module, and an audio playback module.

[0030] The network receiving module is used for receiving audio and video data packets and control instructions, and the network receiving module performs highest priority (DiffServ EF) processing on the control instruction packets to ensure instant response.

[0031] As shown in Figure 5 The frame reorganizer is used for reorganizing a complete frame according to the split marker and the sequence number when the split marker is detected.

[0032] The packet loss detection and processing module is used for detecting lost video packets and judging whether the lost data packets mainly affect a key area, a static area or a dynamic area by using key area, static area and dynamic area mapping information, and if the lost data packets mainly affect the key area, the lost data packets are key area packet loss, if the lost data packets mainly affect the dynamic area, the lost data packets are dynamic area packet loss, and if the lost data packets mainly affect the static area, the lost data packets are static area packet loss. The key area, static area and dynamic area mapping information can be transmitted from a device end or calculated by a control end.

[0033] For static area packet loss, the static area packet loss is ignored, not processed or filled (simple hiding) by using corresponding area data of a previous frame, and retransmission or error correction is not requested.

[0034] For dynamic area packet loss and key area dynamic, the first choice is NACK retransmission: an NACK request for the lost packet is immediately sent, and a short retransmission timeout is set. If the retransmission packet is not received within the timeout time or the accumulated packet loss affects key decoding, a local / complete frame error correction frame (FEC or low-latency I / P frame) is requested from the device end, and the request range is as accurate as possible to the affected area.

[0035] The video decoder is used for decoding a video stream.

[0036] The audio decoding and synchronization module is used for unpacking and separating audio data and video frames, and performing audio and video synchronization playback by using the same PTS (display time stamp) carried by the video frames and the audio data.

[0037] The instruction input and sending module is used for capturing user mouse and keyboard operations and sending the user mouse and keyboard operations to the device end through a high-priority channel.

[0038] The display module is used for displaying a decoded video picture, and the audio playing module is used for playing synchronized audio.

[0039] The network of the application supports a public network / IP network and supports DiffServ QoS, and ensures that a control instruction channel has the highest priority. QoS (Quality of Service) refers to a mechanism for providing differentiated services for different types of services or data streams when a network transmits data. It manages and optimizes network resources (such as bandwidth, delay and packet loss rate) to ensure that key services or real-time applications can obtain the required service performance.

[0040] The image analysis module, the video encoder, the bandwidth limiter, and the audio capture and synchronization module at the device end are implemented on the FPGA. The use of the FPGA hardware can greatly improve the processing speed, significantly improve the encoding efficiency, and reduce the CPU load and the latency.

[0041] Based on the same inventive concept, the application further discloses a video transmission and instruction interaction method for industrial remote control, which comprises the following steps: Step 1, initialization: the device end sends a complete I frame, the control end successfully receives, decodes and displays, and static region information (initial or continuous update) is established.

[0042] Step 2, device end continuous transmission cycle: Capture a screen frame.

[0043] Image analysis: update the static region, the dynamic region, and the focus region mapping.

[0044] Video encoding: use P frame encoding, skip or high QP value encoding for the static region, and low QP value encoding for the dynamic region and the focus region; real-time estimation of available bandwidth and network condition, and dynamic adjustment of overall or regional QP value.

[0045] Audio capture and synchronization: capture audio, bind the same PTS as the current video frame, and package.

[0046] Send data packets (video, packaged audio).

[0047] When receiving the NACK / FEC request of the control end, the retransmission packet or the error correction frame is generated and sent according to the request content (packet number, region). The error correction frame has a special information header, and when the control end receives and decodes the error correction frame, the error correction frame is directly used to replace the original region image. The error correction frame includes a complete error correction frame and a partial error correction frame, the complete error correction frame is a complete I frame, and the partial error correction frame is a partial I frame.

[0048] Step 3, control end continuous processing cycle: Receive data packets, and preferentially process control instructions and execute.

[0049] Detect video packet loss, and when the packet loss occurs, perform intelligent packet loss processing: classify the packet loss region according to the region mapping information. If the packet loss is in the static region, the packet loss is ignored or simply hidden. If the packet loss is in the dynamic region or the focus region: immediately send a NACK request; if the packet loss is not recovered after timeout, send an FEC / I frame request (indicate the region).

[0050] Video decoding: decode the video data packet; when the static region packet loss occurs, use the available packet and the error concealment technology for decoding.

[0051] Audio and video unpacking and synchronization: audio and video are synchronized by using PTS to play audio and video, display pictures and play sound.

[0052] In the above step 2, when the P frame data exceeds the bandwidth threshold, the device end splits the P frame data into at least two sub data blocks for transmission in a continuous transmission period, and each sub data block carries a segmentation mark and a sequence number. In step 3, the control end detects whether the data packet carries a segmentation mark when receiving the data packet, and if the data packet carries a segmentation mark, the data is buffered, and when the data packets with the same segmentation mark are all received, the complete frame is recombined.

[0053] In the above step 3, when the control end captures the user operation instruction, the user operation instruction is sent to the device end through a high-priority channel.

[0054] The application splits the P frame transmission into at least two sub data blocks for periodical transmission when the frame data exceeds the bandwidth threshold, and the sub data block carries a unique segmentation mark and a sequence number.

[0055] The application has the following effects: Smooth user experience: clear (dynamic area), stable, low-latency operation picture is provided under limited bandwidth (50M / 100M), and audio and picture are synchronized.

[0056] High reliability / anti-packet loss: intelligent packet loss processing strategy (static ignore, dynamic priority retransmission) effectively utilizes bandwidth, avoids invalid retransmission, and maximally guarantees the completeness of key area picture. The device end limits speed to reduce packet loss from the source.

[0057] High bandwidth utilization: static area optimization saves a lot of bandwidth, and bandwidth-aware encoding and speed limiter ensure efficient and smooth utilization of bandwidth.

[0058] Ultra-low latency: I frame burst is eliminated, a small buffer, device end speed limit, control instruction priority, and hardware acceleration work together to significantly reduce end-to-end latency (ideal <64ms) and achieve smooth operation.

[0059] Strong scene adaptability: industrial control UI features are deeply optimized.

[0060] Good scalability: support for mainstream encoding standards, FPGA acceleration to improve performance.

[0061] Cost-effective: make full use of existing public networks to reduce dependence on dedicated lines; FPGA solution can provide cost-effective hardware acceleration.

[0062] The above is only an embodiment of the present application, and does not limit the technical scope of the present application. Any slight modification, equivalent change and modification of the above embodiment according to the technical essence of the present application are still within the technical scope of the present application.

Claims

1. A video transmission and instruction interaction system for industrial remote control, comprising a device end and a control end, and the device end and the control end are connected through network communication; characterized in that, the device end is provided with a video capture module for capturing the screen image of the device control computer; an image analysis module for identifying static areas, dynamic areas and key areas, wherein the key areas are pre-set by the control end; a video encoder for encoding the video image transmitted from the device end to the control end; the first frame of the transmitted video image is encoded using I frame, and the subsequent frames are continuously encoded using P frame; the static area in the video image is encoded using skip encoding or using a higher QP value; the dynamic area or the focus area is encoded using a lower QP value; an audio capture and synchronization module for capturing device audio and packing audio data blocks with the same PTS and corresponding video image frames in the same network transmission unit; a network sending module for sending the packed video data packet and audio data packet, and receiving NACK / FEC request and instruction confirmation from the control end; the control end is provided with a network receiving module for receiving audio and video data packets; a packet loss detection and processing module for detecting lost video packets and determining the type of lost data packets using key area, static area and dynamic area mapping information; for static area packet loss, ignore or use the corresponding area data of the previous frame to fill; for dynamic area packet loss and key area dynamic, the first choice is NACK retransmission: immediately send a NACK request for the lost packet, and set a short retransmission timeout; if the retransmitted packet is not received within the timeout time, or the cumulative packet loss affects key decoding, request a local / whole frame error correction frame from the device end; a video decoder for decoding video stream and processing error concealment; an audio decoding and synchronization module for unpacking audio data and video frames, and playing audio and video in synchronization using the same PTS carried by the video frames and the audio data; a display module for displaying the decoded video image, and an audio playing module for playing the synchronized audio.

2. The video transmission and command interaction system for industrial remote control according to claim 1, characterized in that, The video encoder of the device end estimates the available bandwidth and network status in real time during the transmission of the video image, and dynamically adjusts the overall or regional QP value.

3. The video transmission and command interaction system for industrial remote control according to claim 1, characterized in that, The device end is also provided with a bandwidth limiter for splitting the P frame data into at least two sub-data blocks when the P frame data exceeds the bandwidth threshold, and sending the sub-data blocks in consecutive transmission periods, each sub-data block carrying a split marker and a sequence number; the control end is also provided with a frame reorganizer for reorganizing a complete frame according to the split marker and the sequence number when the split marker is detected.

4. The video transmission and command interaction system for industrial remote control according to claim 1, characterized in that, The device end is also provided with a sending buffer module for designing a very small sending buffer or bypassing the sending buffer using zero-copy technology.

5. The video transmission and command interaction system for industrial remote control according to claim 1, characterized in that, The control end is also provided with an instruction input and sending module for capturing user mouse and keyboard operations and sending them to the device end through a high-priority channel.

6. The video transmission and command interaction system for industrial remote control according to claim 1, characterized in that, The image analysis module, video encoder, bandwidth limiter, and audio capture and synchronization module of the device end are implemented on FPGA.

7. A video transmission and command interaction method for industrial remote control, characterized in that, The steps include: Step 1, initialization: the device end sends a complete I frame, the control end successfully receives, decodes and displays, and establishes static region information; Step 2, the device end continuously transmits in a cycle: Capture a screen frame; Image analysis: update the static region, dynamic region and focus region mapping; Video encoding: use P frame encoding, skip or high QP value encoding for the static region, and low QP value encoding for the dynamic region and focus region; estimate the available bandwidth and network status in real time, and dynamically adjust the overall or regional QP value; Audio capture and synchronization: capture audio, bind the same PTS as the current video frame and package; Send data packets; When receiving the NACK / FEC request of the control end, decide to resend packets or generate and send error correction frames according to the request content; Step 3, the control end continuously processes in a cycle: Receive data packets, and preferentially process control instructions and execute them; Detect video packet loss, and when packet loss occurs, perform intelligent packet loss processing: classify the packet loss region according to the region mapping information; if it is static region packet loss, ignore or simply hide; if it is dynamic region packet loss or focus region packet loss: immediately send a NACK request; if it is not recovered after timeout, send an FEC / I frame request; Video decoding: decode the video data packets; when static region packet loss occurs, use available packets and error concealment technology for decoding; Audio and video unpacking and synchronization: use PTS to synchronize and play audio and video, display the picture and play the sound.

8. The video transmission and command interaction method for industrial remote control according to claim 7, characterized in that, In step 2, when the P frame data volume exceeds the bandwidth threshold, the device end splits the P frame data into at least two sub-data blocks and sends them in a continuous transmission cycle, each sub-data block carrying a split marker and a sequence number; In step 3, when receiving data packets, the control end detects whether the data packets carry a split marker, if they carry a split marker, buffers the data, and when the data packets with the same split marker are all received, recombines them into a complete frame.

9. The video transmission and command interaction method for industrial remote control according to claim 7, characterized in that, In step 3, when the control end captures a user operation instruction, it sends the user operation instruction to the device end through a high-priority channel.

Images