Video transmission system and surgical robot system

Abstract

The present invention provides a video transmission system and a surgical robot system. The video transmission system includes a video encoder and a video decoder. The video encoder includes a data encoding module, a forward error correction encoding module, and a first transmission control module. The data encoding module is configured to encode video data to obtain video encoded data. The forward error correction encoding module is configured to perform forward error correction encoding on the video encoded data to obtain forward error correction encoded data. The video decoder is configured to decode the forward error correction encoded data to obtain corresponding video data. The first transmission control module is configured to dynamically adjust the forward error correction redundancy and / or the video frame rate according to network packet loss information, video frame loss information, and network available bandwidth. The present invention can adaptively adjust the forward error correction redundancy and the video frame rate when the network packet loss rate is high and the network fluctuates greatly, so that the remote video can be displayed with low latency and smoothly.

Description

Technical Field

[0001] The present invention relates to the technical field of video transmission, and particularly to a video transmission system and a surgical robot system. Background Art

[0002] In remote surgery, the real-time transmission of surgical area videos is very important, which directly affects the accuracy of remote operations. Currently, in order to reduce network bandwidth, the left and right eye images are compressed by a hardware encoder (H265 protocol) and then transmitted over the network to the decoder at the opposite end. After decompression, they are output to the binocular display screen of the doctor's console to form a 3D effect. Any decline in network performance (increase in packet loss rate and large fluctuations) will affect the real-time transmission of images.

[0003] Current encoders and decoders mainly focus on the encoding and decoding time and compression efficiency, and only use a simple transmission control protocol. When the packet loss rate is high, frame loss will occur. The encoding end will retransmit the H265 frames that have not been received within the timeout period, resulting in a large transmission delay and possibly image stuttering at the opposite end, seriously affecting the smoothness of the doctor's operation.

[0004] In addition, if the network fluctuates greatly, the output of the decoded image will also be fast and slow, affecting the accuracy of the doctor's operation. Moreover, in the prior art, video data is generally not encrypted during transmission, which may lead to the leakage of patient privacy. Summary of the Invention

[0005] The purpose of the present invention is to provide a video transmission system and a surgical robot system, which can adaptively adjust the forward error correction redundancy and video frame rate when the network packet loss rate is high and the network fluctuates greatly, so that the remote video can still be displayed with low latency and smoothly.

[0006] To achieve the above object, the present invention provides a video transmission system, including a video encoder and a video decoder connected by communication. The video encoder is configured to be disposed at a video sending end, and the video decoder is configured to be disposed at a video receiving end; the video encoder includes a data encoding module, a forward error correction encoding module, and a first transmission control module; the data encoding module is configured to encode the acquired video data using a preset video encoding protocol to obtain video encoded data and send it to the forward error correction encoding module; the forward error correction encoding module is configured to perform forward error correction encoding on the video encoded data according to the forward error correction redundancy configured by the first transmission control module to obtain forward error correction encoded data including redundant data; the video decoder is configured to decode the forward error correction encoded data to obtain corresponding video data, and feedback network packet loss information and video frame loss information to the first transmission control module; the first transmission control module is configured to dynamically adjust the forward error correction redundancy and / or the video frame rate according to the network available bandwidth configured by it and the network packet loss information and the video frame loss information feedback by the video decoder.

[0007] Optionally, the first transmission control module is configured to analyze the network packet loss information to obtain a network packet loss rate, and increase the forward error correction redundancy when the network packet loss rate rises, and decrease the forward error correction redundancy when the network packet loss rate drops.

[0008] Optionally, when the network packet loss rate rises or drops, the first transmission control module is configured to increase or decrease the forward error correction redundancy according to the following rule:

[0009] r≥p / (1-p)

[0010] Wherein, r is the forward error correction redundancy, and p is the network packet loss rate.

[0011] Optionally, when the forward error correction redundancy calculated based on the network packet loss information is greater than or equal to the maximum allowable forward error correction redundancy corresponding to the network available bandwidth and video frame loss occurs, the first transmission control module is configured to reduce the video frame rate and adjust the forward error correction redundancy of the forward error correction encoding module to the maximum allowable forward error correction redundancy.

[0012] Optionally, when the forward error correction redundancy calculated based on the network packet loss information is less than the maximum allowable forward error correction corresponding to the network available bandwidth and its decrease amplitude is greater than or equal to a preset amplitude, and the video frame rate has been reduced before, the first transmission control module is configured to increase the video frame rate.

[0013] Optionally, the single decrease amplitude of the forward error correction redundancy is greater than or equal to a first preset minimum amplitude, and the absolute value of the single adjustment amplitude of the video frame rate is greater than or equal to a second preset minimum amplitude.

[0014] Optionally, the first transmission control module is configured to send an alarm message when the video frame rate is a preset minimum frame rate and the forward error correction redundancy calculated based on the network packet loss information is greater than or equal to the maximum allowable forward error correction redundancy corresponding to the network available bandwidth.

[0015] Optionally, the first transmission control module is further configured to allow a user to set the network available bandwidth, and / or allow the user to configure the data encoding module and the forward error correction encoding module, and / or allow the user to view the network packet loss information, video frame loss information, and forward error correction redundancy in real time, and / or allow the user to configure the screen display information.

[0016] Optionally, the video encoder further includes a network sending module, and the network sending module is configured to encrypt the forward error correction encoded data by using a preset encryption algorithm to obtain forward error correction encrypted encoded data and send the forward error correction encrypted encoded data to the video decoder by using a preset high-speed secure communication protocol.

[0017] Optionally, the video decoder includes a network receiving module, a forward error correction decoding module, a data decoding module, and a second transmission control module; the network receiving module is configured to decrypt the received forward error correction encrypted encoded data to obtain corresponding forward error correction encoded data and send the data to the forward error correction decoding module, and send a network packet loss notification to the second transmission control module when a packet is lost; the forward error correction decoding module is configured to perform forward error correction decoding on the forward error correction encoded data to obtain corresponding video encoded data and send the data to the data decoding module, and send a video frame loss notification to the second transmission control module when a frame is lost; the data decoding module is configured to decode the received video encoded data to obtain corresponding video data; the second transmission control module is configured to transmit the network packet loss information and the video frame loss information to the first transmission control module through the preset high-speed secure communication protocol.

[0018] Optionally, the second transmission control module is further configured to allow a user to configure the network receiving module, the forward error correction decoding module, and the data decoding module, and / or allow the user to view the network packet loss information, video frame loss information, and forward error correction redundancy in real time, and / or allow the user to configure the screen display information.

[0019] Optionally, the video transmission system provided by the present invention further includes a cloud server, and the video encoder and the video decoder perform data interaction through the cloud server.

[0020] To achieve the above object, the present invention further provides a surgical robot system, including a remote end, a local end, and the video transmission system described in any one of the above.

[0021] Compared with the prior art, the video transmission system and the surgical robot system provided by the present invention have the following beneficial effects: Since the first transmission control module in the video encoder of the present invention can dynamically adjust the forward error correction redundancy and / or the video frame rate in real time according to the available network bandwidth configured therein, as well as the network packet loss information and video frame loss information fed back by the video decoder, it can be seen that the present invention can dynamically understand the network packet loss situation and video frame loss situation through the dynamic feedback of network packet loss and video frame loss, and then dynamically adjust the forward error correction redundancy and / or the video frame rate according to the actual packet loss situation and video frame loss situation, so as to be able to flexibly utilize the forward error correction technology to enhance the packet loss resistance and frame loss resistance capabilities during the video data transmission process under different network conditions, enabling the system to adapt to network changes, ensuring that the video data is lost as little as possible during the transmission process, providing an important guarantee for the smooth display of the video, effectively coping with the complex and changeable network packet loss situation, and avoiding video stuttering and unplayable problems caused by a large number of frame losses.

[0022] In addition, since the video frame rate determines the smoothness of the video, the available network bandwidth determines the data transmission ability, and the forward error correction redundancy affects the packet loss resistance ability and transmission efficiency of the data, and these three factors restrict and influence each other. By dynamically balancing these three factors, the present invention can adjust the video frame rate and / or the forward error correction redundancy according to the available network bandwidth situation under the condition of limited network bandwidth, while ensuring a certain smoothness of the video, reasonably utilizing the network bandwidth resources to add appropriate forward error correction redundancy information to improve the reliability of the video data transmission. For example, when the network bandwidth is sufficient, the video frame rate can be appropriately increased and the forward error correction redundancy can be increased to improve the video quality and the packet loss and frame loss resistance capabilities; when the network bandwidth is tight, the video frame rate can be reduced and the forward error correction redundancy can be adjusted to ensure that the video can be played smoothly basically and there will be no network congestion caused by excessive redundant data. This dynamic balancing mechanism can optimize the video transmission effect in a complex network environment and effectively improve the overall performance and adaptability of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 It is a scene diagram of remote transmission of video data during remote surgery.

[0024] Figure 2 It is a block structure diagram of the video transmission system provided by an embodiment of the present invention.

[0025] Figure 3 It is a working flowchart of the data encoding module provided by an embodiment of the present invention.

[0026] Figure 4 The working flowchart of the forward error correction encoding module provided by an embodiment of the present invention.

[0027] Figure 5 The secure connection flowchart between the network sending module and the network receiving module provided by an embodiment of the present invention.

[0028] Figure 6 The working flowchart of the network sending module provided by an embodiment of the present invention.

[0029] Figure 7 The working flowchart of the network receiving module provided by an embodiment of the present invention.

[0030] Figure 8 The working flowchart of the forward error correction decoding module provided by an embodiment of the present invention.

[0031] Figure 9 The working flowchart of the second transmission control module provided by an embodiment of the present invention.

[0032] Figure 10 The working flowchart of the first transmission control module provided by an embodiment of the present invention.

[0033] Figure 11 The working flowchart of the data decoding module provided by an embodiment of the present invention.

[0034] Figure 12 The block structure diagram of the video transmission system provided by another embodiment of the present invention.

[0035] Figure 13 The secure connection flowchart between the network sending module and the network receiving module and the cloud server provided by an embodiment of the present invention.

[0036] Among them, the reference numerals are explained as follows: image host - 10; video cables - 20a, 20b, 20c; encoder - 30; local doctor control cart - 40; networks - 50a, 50b; cloud server - 60; decoder - 70; remote doctor control cart - 80; video encoder - 100; data encoding module - 110; forward error correction encoding module - 120; first transmission control module - 130; network sending module - 140; video decoder - 200; network receiving module - 210; forward error correction decoding module - 220; data decoding module - 230; second transmission control module - 240; local end - 300; remote end - 400. Detailed implementation manners

[0037] The following further elaborates on the video transmission system and surgical robot system proposed by the present invention in conjunction with the accompanying drawings and specific embodiments. According to the following description, the advantages and features of the present invention will be clearer.

[0038] The core idea of the present invention is to provide a video transmission system and a surgical robot system that can adaptively adjust the forward error correction redundancy and video frame rate when the network packet loss rate is high and the network fluctuates greatly, so that the remote video can still be displayed with low latency and smoothly.

[0039] It should be noted that the video encoder in the video transmission system provided by the present invention can be deployed at the local end of the surgical robot system, and the video decoder can be deployed at the remote end of the surgical robot system. It should also be noted that, as can be understood by those skilled in the art, the video transmission system provided by the present invention can be applied not only to the remote transmission scenario of surgical videos, but also to other remote video transmission scenarios.

[0040] Please refer to Figure 1 , which is a diagram of the remote transmission scenario of video data in remote surgery. As Figure 1 shown, the image host 10 transmits the dual-channel endoscope image to the encoder 30 through the video cable 20a. While the encoder 30 transmits the dual-channel endoscope image to the stereoscopic monitor on the local doctor control cart 40 through the video cable 20b, it also encodes the dual-channel endoscope image using a video coding protocol and sends it to the cloud server 60 through the network 50a. The decoder 70 obtains the video coding data stream from the cloud server 60 through the network 50b, decodes it through the corresponding video decoding protocol, and then transmits it to the binocular monitor on the remote doctor control cart 80 through the video cable 20c.

[0041] To achieve the above idea, the present invention provides a video transmission system. Please refer to Figure 2 , which is a block structure diagram of the video transmission system provided by an embodiment of the present invention. As Figure 2As shown in the figure, the video transmission system provided by the present invention includes a video encoder 100 and a video decoder 200 that are communicatively connected. The video encoder 100 is configured to be disposed at the video sending end, and the video decoder 200 is configured to be disposed at the video receiving end. The video encoder 100 includes a data encoding module 110, a forward error correction encoding module 120, and a first transmission control module 130. The data encoding module 110 is configured to encode the acquired video data using a preset video encoding protocol to obtain video encoded data and send it to the forward error correction encoding module 120. The forward error correction encoding module 120 is configured to perform forward error correction encoding on the video encoded data according to the forward error correction redundancy configured by the first transmission control module 130 to obtain forward error correction encoded data including redundant data. The video decoder 200 is configured to decode the forward error correction encoded data to obtain corresponding video data, and feedback network packet loss information and video frame loss information to the first transmission control module 130. The first transmission control module 130 is configured to dynamically adjust the forward error correction redundancy and / or video frame rate according to the network available bandwidth configured by it and the network packet loss information and the video frame loss information feedback by the video decoder 200.

[0042] Since the first transmission control module 130 in the video encoder 100 of the present invention can dynamically adjust the forward error correction redundancy and / or video frame rate in real time according to the network packet loss information and video frame loss information feedback by the video decoder 200 and the network available bandwidth, it can be seen that the present invention can dynamically feedback through network packet loss and video frame loss to understand the network packet loss situation and video frame loss situation in real time, and then dynamically adjust the forward error correction redundancy and / or video frame rate according to the actual packet loss situation and video frame loss situation. Therefore, it can flexibly use the forward error correction technology to enhance the anti-packet loss and anti-frame loss capabilities during the video data transmission process under different network conditions, enabling the system to adapt to network changes, ensuring that video data is minimized during transmission, providing an important guarantee for the smooth display of videos, effectively coping with complex and changeable network packet loss situations, and avoiding video stuttering and unplayable problems caused by a large number of frame losses.

[0043] In addition, since the video frame rate determines the smoothness of the video, the available network bandwidth determines the data transmission capacity, and the forward error correction redundancy affects the packet loss resistance and transmission efficiency of the data, these three factors restrict and influence each other. By dynamically balancing these three factors, the present invention can adjust the video frame rate and / or the forward error correction redundancy according to the available network bandwidth under the condition of limited network bandwidth, and while ensuring a certain smoothness of the video, reasonably utilize the network bandwidth resources to add appropriate forward error correction redundancy information to improve the reliability of video data transmission. For example, when the network bandwidth is sufficient, the video frame rate can be appropriately increased and the forward error correction redundancy can be increased to improve the video quality, packet loss resistance and frame loss resistance; when the network bandwidth is tight, the video frame rate is reduced and the forward error correction redundancy is adjusted to ensure that the video can be played smoothly basically and network congestion will not be caused by excessive redundant data. This dynamic balancing mechanism can optimize the video transmission effect in a complex network environment and effectively improve the overall performance and adaptability of the system.

[0044] Specifically, the preset video coding protocol can be, but is not limited to, the H265 protocol, the H266 protocol, etc. The specific content on how to encode video data using preset video coding protocols such as the H265 protocol and the H266 protocol can refer to the relevant content in the video coding technology field well-known to those skilled in the art, and will not be elaborated here. It should also be noted that the data coding module 110 can loop out HDMI (a digital interface standard for audio and video data) video locally. Additionally, it should be noted that the forward error correction coding module 120 adds a certain amount of redundant information to the video coding data by adopting FEC (Forward Error Correction) technology (such as Reed-Solomon coding, fountain code, etc.), enabling the video receiving end (video decoder 200) to detect and correct possible errors without sending a retransmission request to the sending end (video encoder 100), thereby avoiding the delay caused by retransmission. Furthermore, it should be noted that the data coding module 110 supports multiple coding formats (such as H265, H266), coding frequencies, and can dynamically switch the coding format and coding frequency according to the instructions of the first transmission control module 130 to adapt to different network conditions. The forward error correction coding module 120 can switch between different forward error correction algorithms such as Reed-Solomon coding and fountain code according to the instructions of the first transmission control module 130.

[0045] Please continue to refer to Figure 2 , such as Figure 2As shown, in some exemplary embodiments, the video encoder 100 further includes a network transmission module 140. The network transmission module 140 is configured to encrypt the forward error correction encoded data using a preset encryption algorithm to obtain forward error correction encrypted encoded data and send the forward error correction encrypted encoded data to the video decoder 200 using a preset high-speed secure communication protocol. Thus, by encrypting the forward error correction encoded data using a preset encryption algorithm, the security during the video data transmission can be effectively guaranteed, and data leakage can be effectively prevented. By sending the forward error correction encoded data to the video decoder 200 using a preset high-speed secure communication protocol, high-speed data transmission can be ensured.

[0046] Specifically, the preset encryption algorithm can be, but is not limited to, a symmetric encryption algorithm (such as the AES algorithm, the Advanced Encryption Standard algorithm), and the preset high-speed secure communication protocol can be, but is not limited to, the DTLS protocol (Datagram Transport Layer Security protocol). The preset high-speed secure communication protocol supports dynamically configuring encryption certificates, adjusting encryption strength, and transmission parameters in different network environments to ensure the security of data transmission.

[0047] Please continue to refer to Figure 2 , as Figure 2 shown, in some exemplary embodiments, the video decoder 200 includes a network reception module 210, a forward error correction decoding module 220, a data decoding module 230, and a second transmission control module 240. The network reception module 210 is configured to decrypt the received forward error correction encrypted encoded data to obtain the corresponding forward error correction encoded data and send it to the forward error correction decoding module 220, and send a network packet loss notification to the second transmission control module 240 when a packet is lost. The forward error correction decoding module 220 is configured to perform forward error correction decoding on the forward error correction encoded data to obtain the corresponding video encoded data and send it to the data decoding module 230, and send a video frame loss notification to the second transmission control module 240 when a frame is lost. The data decoding module 230 is configured to decode the received video encoded data to obtain the corresponding video data. The second transmission control module 240 is configured to transmit the network packet loss information and the video frame loss information to the first transmission control module 130 through the preset high-speed secure communication protocol.

[0048] Specifically, the network receiving module 210 uses a decryption algorithm corresponding to a preset encryption algorithm to decrypt the received forward error correction encrypted coded data to obtain the corresponding forward error correction coded data, and sends a network packet loss notification to the second transmission control module 240 when a packet is lost. The forward error correction decoding module 220 performs forward error correction decoding on the received forward error correction coded data, restores the data according to the redundant data, and sends the restored valid data (video coded data) to the data decoding module 230, and sends a frame loss notification to the second transmission control module 240 when a frame is lost. The data decoding module 230 decodes the received video coded data to obtain the corresponding video data, and outputs the video data to a display device (such as the binocular monitor of the remote doctor control cart 80) for display.

[0049] Please continue to refer to Figure 3 , which is the workflow diagram of the data coding module provided by an embodiment of the present invention. As Figure 3 shown, after receiving the HDMI signal, the data coding module 110 first decodes the HDMI signal to extract video data; then encodes the video data using video coding protocols such as H265 / H266 to obtain a video bitstream; then encapsulates the video bitstream; and finally outputs the encapsulated video bitstream (i.e., video coded data).

[0050] Please continue to refer to Figure 4 , which is the workflow diagram of the forward error correction coding module provided by an embodiment of the present invention. As Figure 4 shown, the forward error correction coding module 120 first reads the forward error correction redundancy configured by the first transmission control module 130; then reads the video coded data sent by the data coding module 110; then divides the video coded data into blocks; then performs forward error correction coding to generate redundant data; then combines the video coded data and the redundant data to obtain forward error correction coded data; and finally outputs the forward error correction coded data to the network sending module 140.

[0051] Please continue to refer to Figure 5 , which is the secure connection flowchart between the network sending module and the network receiving module provided by an embodiment of the present invention. As Figure 5 shown, the network sending module 140 first starts a listening service and waits for a client (video receiving end) to connect. At this time, the network receiving module 210 is in a standby state, ready to process incoming DTLS handshake requests. The network receiving module 210 actively initiates a DTLS connection request and sends initial handshake information to the network sending module 140. After receiving the DTLS handshake request from the network receiving module 210, the network sending module 140 accesses the client list, thus completing the secure connection between the network sending module 140 and the network receiving module 210.

[0052] Please continue to refer to Figure 6 , which is the flowchart of the network sending module provided by an embodiment of the present invention. As Figure 6 shown, the network sending module 140 first reads the forward error correction encoded data, then encrypts the forward error correction encoded data using a preset encryption algorithm (such as the AES symmetric encryption algorithm), and then sends the encrypted forward error correction encoded data (forward error correction encrypted encoded data) to the network receiving module 210 that is DTLS handshake connected to it (i.e., sends data to the DTLS client).

[0053] Please continue to refer to Figure 7 , which is the flowchart of the network receiving module provided by an embodiment of the present invention. As Figure 7 shown, the network receiving module 210 first performs a DTLS connection handshake with the network sending module 140, then receives the forward error correction encrypted encoded data sent by the network sending module 140, and decrypts the received forward error correction encrypted encoded data (such as AES decryption). If there is a packet loss, a packet loss notification is sent, and finally the decrypted forward error correction encoded data is output.

[0054] Please continue to refer to Figure 8 , which is the flowchart of the forward error correction decoding module provided by an embodiment of the present invention. As Figure 8 shown, the forward error correction decoding module 220 first reads the forward error correction encoded data; then separates the received forward error correction encoded data into original data (video encoded data) and redundant data; then verifies whether the currently received data volume meets the decoding condition. If the decoding condition is met, forward error correction decoding is performed, and the decoded valid data is reorganized into complete video encoded data in sequence; if the decoding condition is not met, a timeout determination is made. If a timeout occurs, a video frame loss notification is sent.

[0055] Please continue to refer to Figure 9 , which is the flowchart of the second transmission control module provided by an embodiment of the present invention. As Figure 9 shown, the second transmission control module 240 first initiates a connection request to the first transmission control module 130, then counts network packet losses and video frame losses, and finally sends network packet loss statistical data (network packet loss information) and video frame loss statistical data (video frame loss information) to the first transmission control module 130.

[0056] In some exemplary embodiments, the first transmission control module 130 is configured to parse the network packet loss information to obtain the network packet loss rate, increase the forward error correction redundancy when the network packet loss rate rises, and decrease the forward error correction redundancy when the network packet loss rate drops. Thus, by increasing the forward error correction redundancy when the network packet loss rate rises, network packet loss can be effectively reduced; by decreasing the forward error correction redundancy when the network packet loss rate drops, the network bandwidth requirement can be reduced.

[0057] In some exemplary embodiments, the first transmission control module 130 is configured to increase or decrease the forward error correction redundancy according to the following rules when the network packet loss rate rises or drops:

[0058] r≥p / (1-p)

[0059] where r is the forward error correction redundancy and p is the network packet loss rate.

[0060] Thus, by increasing or decreasing the forward error correction redundancy according to the above rules, it can be ensured that the lost data can be compensated by the forward error correction technology.

[0061] In some exemplary embodiments, the single-drop amplitude of the forward error correction redundancy is greater than or equal to a first preset minimum amplitude. Thus, by setting the minimum change amplitude of the forward error correction redundancy (i.e., the first preset minimum amplitude), it is possible to avoid frequently changing the forward error correction redundancy in small amplitudes, which may affect the stability of the system. It should be noted that the present invention does not limit the specific value of the first preset minimum amplitude, and the specific value of the first preset minimum amplitude can be set according to actual requirements.

[0062] In some exemplary embodiments, the first transmission control module 130 is configured to reduce the video frame rate when the forward error correction redundancy calculated based on the network packet loss information is greater than or equal to the maximum allowable forward error correction redundancy corresponding to the network available bandwidth and video frame drops occur.

[0063] Thus, by reducing the video frame rate when the forward error correction redundancy obtained based on the network packet loss information is greater than or equal to the maximum allowable forward error correction redundancy corresponding to the network available bandwidth and video frame drops occur, the network bandwidth requirement can be reduced by reducing the frame rate, thereby allowing further increase in the forward error correction redundancy. It should be noted that as can be understood by those skilled in the art, the video frame rate can be reduced by reducing the encoding frequency of the data encoding module 110 and the sending frequency of the network sending module 140.

[0064] Specifically, theoretically, the percentage increase b in network bandwidth overhead is equal to the forward error correction redundancy r. However, considering the forward error correction coding overhead (control information + packet padding), the actual network bandwidth overhead is slightly larger, that is, b ≥ p / (1 - p) + Δ, which means b = r + Δ. It is necessary to evaluate Δ according to the specific forward error correction algorithm adopted.

[0065] Assume that the traffic of video data is t and the available network bandwidth is w. If t*(1 + b) ≥ w (that is, t*(1 + r + Δ) ≥ w) and video frame loss occurs, then the video frame rate is reduced to reduce the network bandwidth requirement, and then the increased forward error correction redundancy is used to overcome packet loss.

[0066] It should be noted that when the traffic is t, the maximum allowable forward error correction redundancy r corresponding to the available network bandwidth w max = ((w / t) - 1 - Δ). If t*(1 + r + Δ) ≥ w, it indicates that the forward error correction redundancy r obtained based on network packet loss information is greater than or equal to the maximum allowable forward error correction redundancy r corresponding to the available network bandwidth. max .

[0067] In some exemplary embodiments, the video frame rate is greater than or equal to a preset minimum frame rate and less than or equal to a preset maximum frame rate. Thus, by setting the video frame rate to be greater than or equal to the preset minimum frame rate, it is possible to avoid poor video quality caused by too low a frame rate, which may affect surgical operations. By setting the video frame rate to be less than or equal to the preset maximum frame rate, it is possible to avoid exceeding the available network bandwidth due to too high a video frame rate, ensuring the stability of video data transmission and further improving the anti-packet-loss and anti-frame-loss capabilities during video data transmission.

[0068] It should be noted that as can be understood by those skilled in the art, the present invention does not limit the specific values of the preset minimum frame rate and the preset maximum frame rate. The specific values of the preset minimum frame rate and the preset maximum frame rate can be set according to actual requirements.

[0069] In some exemplary embodiments, the first transmission control module 130 is configured to increase the video frame rate when the forward error correction redundancy calculated based on the network packet loss information is less than the maximum allowable forward error correction redundancy corresponding to the available network bandwidth, its decrease amplitude is greater than or equal to a preset amplitude, and the video frame rate has been decreased before. Since when the network packet loss rate decreases, resulting in a decrease in the required forward error correction redundancy and the decrease amplitude exceeds the preset amplitude, it indicates a significant improvement in network stability. At this time, gradually restoring the video frame rate can reallocate the released network bandwidth to the original video data, thereby improving the video quality, avoiding waste of network bandwidth resources, achieving a dynamic balance between video quality and transmission reliability, and at the same time avoiding fluctuations caused by frequent adjustment of the video frame rate and preventing frame freezing or flickering caused by frequent increase and decrease of the video frame rate.

[0070] Specifically, if t*(1 + b) < w, and the decrease amplitude of the forward error correction redundancy r is greater than or equal to a preset amplitude (e.g., 10%), the video frame rate can be gradually restored at a certain amplitude to restore the video quality. It should be noted that, as can be understood by those skilled in the art, the video frame rate can be increased by increasing the encoding frequency of the data encoding module 110 and the sending frequency of the network sending module 140.

[0071] Please continue to refer to Figure 10 , which is the flowchart of the operation of the first transmission control module provided by an embodiment of the present invention. As Figure 10 shown, if the maximum allowable forward error correction redundancy corresponding to the available network bandwidth meets the forward error correction redundancy required by the packet loss rate, but there are still dropped frames, it indicates that the actual network bandwidth between the two may be insufficient, and the frame rate needs to be decreased to balance the situation of the actual network bandwidth reduction. The frame rate also needs to be decreased to balance the network bandwidth demand when the network bandwidth is insufficient. If the packet loss rate decreases, the forward error correction redundancy should be correspondingly decreased to reduce the network bandwidth demand; if the frame rate is not the maximum frame rate at this time, the frame rate needs to be gradually restored.

[0072] In some exemplary embodiments, the absolute value of the single - time adjustment amplitude of the video frame rate is greater than or equal to a second preset minimum amplitude. Thus, by setting the absolute value of the single - time adjustment amplitude of the video frame rate to be greater than or equal to the second preset minimum amplitude, it is possible to avoid repeated fine - tuning of the video frame rate in a short period of time, reduce frame freezing or jumping of the video image, and thus effectively ensure the video quality.

[0073] In some exemplary embodiments, the first transmission control module 130 is configured to send an alarm message when the video frame rate is the preset minimum frame rate and the forward error correction redundancy calculated based on the network packet loss information is greater than or equal to the maximum allowable forward error correction redundancy corresponding to the network available bandwidth. Since when the video frame rate is the preset minimum frame rate and the forward error correction redundancy calculated based on the network packet loss information is greater than or equal to the maximum allowable forward error correction redundancy corresponding to the network available bandwidth, it is no longer possible to increase the forward error correction redundancy by reducing the video frame rate, that is, at this time the system will not be able to solve the problem of too high network packet loss rate. Therefore, by sending an alarm message, it is convenient for the user to take corresponding measures in time. Specifically, the first transmission control module 130 can output the alarm message through the Web server.

[0074] In some exemplary embodiments, the first transmission control module 130 is further configured to allow the user to set the network available bandwidth, and / or allow the user to configure the data encoding module 110, the forward error correction encoding module 120, and the network sending module 140, and / or allow the user to view the network packet loss information, video frame loss information, and forward error correction redundancy in real time, and / or allow the user to configure the screen display information. Therefore, by setting the first transmission control module 130 to have the Web server function, it is possible to support the user (administrator) to configure the data encoding module 110, the forward error correction encoding module 120, and the network sending module 140, and at the same time support the user (administrator) to view or output dynamic information such as network packet loss (network packet loss rate), video frame loss (video frame loss rate), and forward error correction redundancy, and configure the screen display information (OSD information) such as the data display, curve, and overlay display parameters (such as overlay parameters, overlay positions, overlay areas, and layouts) of the Web page for the video data (such as the surgical field image).

[0075] In some exemplary embodiments, the second transmission control module 240 is further configured to allow a user to configure the network receiving module 210, the forward error correction decoding module 220, and the data decoding module 230, and / or to allow the user to view network packet loss information, video frame loss information, and forward error correction redundancy in real time, and / or to allow the user to configure screen display information. Thus, by setting the second transmission control module 240 to have the function of a Web server, it is possible to support the user (administrator) to configure the network receiving module 210, the forward error correction decoding module 220, and the data decoding module 230, and to view or output dynamic information such as network packet loss (network packet loss rate), video frame loss (video frame loss rate), and forward error correction redundancy. And the display form can be configured, including the data display on the Web page, curves, and the superimposed display parameters (such as superimposition parameters, superimposition positions, superimposition areas, and layouts, etc.) of video data (such as surgical field images), etc., which are screen display information (OSD information). And this OSD information can be sent to the data decoding module 230 to achieve the fusion output display of the OSD information and the video data (surgical field images).

[0076] Please continue to refer to Figure 11 , which is a flowchart of the operation of the data decoding module provided by an embodiment of the present invention. As Figure 11 shown, the data decoding module 230 first decodes the received video stream (video encoded data), then reads the OSD information, fuses the decoded video data with the OSD information, encodes the fused data into an HDMI signal, and finally outputs the obtained HDMI signal.

[0077] In summary, it can be seen that Figure 2The working principle of the video transmission system shown is as follows: The data encoding module 110 of the video encoder 100 encodes the HDMI video signal B1 into H265 / H266 data B2 (video encoded data) according to the video frame rate configuration or encoding configuration B3. The forward error correction encoding module 120 of the video encoder 100 uses forward error correction technologies and algorithms such as Reed-Solomon coding and fountain codes, and adds redundant information (redundant data) to the video encoded data according to the forward error correction encoding configuration B4 to generate forward error correction encoded data B5, so that the video receiving end (video decoder 200) can detect and correct errors without retransmission. The first transmission control module 130 of the video encoder 100 is used to receive statistical data B7 such as network packet loss and video frame loss feedback from the video decoder 200, as well as the current available network bandwidth information, and dynamically adjust the forward error correction redundancy, the encoding frequency of the data encoding module 110, and the sending frequency of the network sending module 140 (i.e., adjust the video frame rate). The first transmission control module 130 also has the function of a Web server, supporting the administrator to configure the data encoding module 110, the forward error correction encoding module 120, and the network sending module 140, view or output dynamic information such as network packet loss rate / frame loss rate and forward error correction redundancy, and can configure screen display information (OSD information) such as data display, curves, and overlay settings of video data (such as surgical field images) on the Web page, including overlay parameters, overlay positions, areas, and layouts. The network sending module 140 of the video encoder 100 uses symmetric encryption (such as the AES algorithm) and a high-speed secure communication protocol (such as the DTLS secure transmission protocol) to encrypt the encoded data according to the high-speed secure communication server configuration B6 and transmit it at high speed as network high-speed secure communication data B8 (forward error correction encrypted encoded data). The network receiving module 210 of the video decoder 200 is used to receive the network high-speed secure communication data B8, decrypt it according to the high-speed secure communication client configuration B10 and send it to the forward error correction decoding module 220, and send a network packet loss notification B9 to the second transmission control module 240 of the video decoder 200. The second transmission control module 240 is used to send the network packet loss data feedback by the network receiving module 210 and the frame loss data feedback by the forward error correction decoding module 220 to the first transmission control module 130 of the video encoder 100; The second transmission control module 240 also has the function of a Web server, supporting the administrator to configure the data decoding module 230, the forward error correction decoding module 220, and the network receiving module 210, view or output dynamic information such as network packet loss rate / frame loss rate and forward error correction redundancy, and can configure screen display information (i.e., OSD information) such as data display, curves, and overlay settings of video data (such as surgical field images) on the Web page, including overlay parameters, overlay positions, areas, and layouts, and send this information to the data decoding module 230 to achieve fusion output with the video data (such as surgical field images).The forward error correction decoding module 220 is configured to decode the forward error correction encoded data B11 sent by the network receiving module 210 based on the forward error correction decoding configuration B13, recover the data according to the redundant information, and send the valid data (video encoded data B15) to the data decoding module 230. Meanwhile, the frame loss information B12 is sent to the second transmission control module 240. The data decoding module 230 is configured to decode the video encoded data B15 based on the decoding configuration B14. If there is OSD information, according to the configuration of the data decoding module 230 / OSD information B14, the video signal B16 obtained by fusing the OSD information and the video data (such as the surgical field image) is output to the display device.

[0078] Please continue to refer to Figure 12 , which is a block diagram of the video transmission system provided by another embodiment of the present invention. As Figure 12 shown, the main difference between the video transmission system provided in this embodiment and the Figure 2 video transmission system shown is that the video transmission system provided in this embodiment further includes a cloud server 60, and the video encoder 100 and the video decoder 200 perform data interaction through the cloud server 60. Since the cloud server 60 usually deploys a multi-node redundant architecture and can automatically switch the optimal path, thus, by setting up the cloud server 60 to achieve data interaction between the video encoder 100 and the video decoder 200, the continuity and stability of data transmission can be effectively ensured, the data transmission delay can be effectively reduced, and the stringent requirements for real-time performance in remote surgery can be met. In addition, since the cloud server 60 supports elastic resource allocation, the network bandwidth and computing resources can be dynamically adjusted according to the data traffic, without frequent local hardware upgrades, effectively reducing costs.

[0079] Specifically, in this embodiment, the network sending module 140 is configured to send the forward error correction encrypted encoded data to the cloud server 60 by using a preset high-speed secure communication protocol; the second transmission control module 240 is configured to transmit the network packet loss information and the video frame loss information to the cloud server 60 through the preset high-speed secure communication protocol; the cloud server 60 is configured to send the forward error correction encrypted encoded data to the network receiving module 210, and transmit the network packet loss information and the video frame loss information to the first transmission control module 130 through the preset high-speed secure communication protocol.

[0080] Furthermore, as Figure 12 shown, the working principle of the video transmission system provided in this embodiment is the same as that of Figure 2The working principles of the video transmission systems shown are generally similar. The difference lies in that, in this embodiment, the network sending module 140 first encrypts the encoded data using symmetric encryption (such as the AES algorithm) and a high-speed secure communication protocol (such as the DTLS secure transmission protocol) according to the high-speed secure communication server configuration B6 to obtain the network high-speed secure communication data B8, and then transmits the network high-speed secure communication data B8 to the cloud server 60 at high speed. The cloud server 60 then transmits the received network high-speed secure communication data B8 to the network receiving module 210. The second transmission control module 240 first uploads the statistical data B7 such as network packet loss and video frame loss to the cloud server 60, and the cloud server 60 then sends the statistical data B7 such as network packet loss and video frame loss to the first transmission control module 130.

[0081] Please continue to refer to Figure 13 , which is a flowchart of the secure connection between the network sending module and the network receiving module and the cloud server provided by an embodiment of the present invention. As Figure 13 shown, both the network sending module 140 and the network receiving module 210 actively initiate DTLS connection requests to the cloud server 60. After completing the DTLS connection handshake with the cloud server 60, the network sending module 140 and the network receiving module 210 can communicate with the cloud server 60.

[0082] It should be noted that for more content about the video transmission system provided in this embodiment, reference can be made to the relevant description of the video transmission system shown above for adaptive understanding, and details will not be elaborated here. Figure 2 shown above for adaptive understanding, and details will not be elaborated here.

[0083] Based on the same inventive concept, the present invention also provides a surgical robot system, including a remote end 400, a local end 300, and the video transmission system described in any of the above embodiments. Among them, the video encoder 100 is disposed on the local end 300, and the video decoder 200 is disposed on the remote end 400. Since the surgical robot system provided by the present invention and the video transmission system provided by the present invention belong to the same inventive concept, the surgical robot system provided by the present invention has at least all the beneficial effects of the video transmission system provided by the present invention. For specific details, reference can be made to the relevant description above, and details will not be elaborated here one by one.

[0084] In summary, compared with the prior art, the video transmission system and the surgical robot system provided by the present invention have the following beneficial effects: (1) By dynamically adjusting the forward error correction redundancy and / or the video frame rate, the present invention can perform adaptive adjustment according to changes in the network state. When the network environment is poor, the forward error correction redundancy is increased to effectively recover lost data. When the network environment is good, the forward error correction redundancy is reduced to save network bandwidth resources and improve transmission efficiency. This dynamic adjustment mechanism can avoid stuttering or frame loss caused by maintaining high resolution when the network bandwidth is insufficient, thereby reasonably utilizing network resources and providing high-quality video when the network bandwidth is sufficient.

[0085] (2) When the network packet loss rate is too high, the video frame rate is reduced to reduce the network bandwidth requirement, so as to ensure that a higher forward error correction redundancy can be provided, thereby further improving the packet loss and frame loss resistance capabilities and ensuring video quality.

[0086] (3) The transmission control module (the first transmission control module 130, the second transmission control module 240) can implement the Web server function, thereby allowing users to configure the parameters of each module (the data encoding module 110, the forward error correction encoding module 120, the network sending module 140, the network receiving module 210, the forward error correction decoding module 220, the data decoding module 230), view dynamic information (the forward error correction redundancy, the available network bandwidth, the video frame rate), and set the screen display information (overlay parameters, overlay positions, areas, and layouts, etc.), which can facilitate users to manage and monitor the operating state of the system.

[0087] (4) Through the network packet loss information and video frame loss information fed back by the video decoder 200 (the video receiving end), and combined with the available network bandwidth, the network condition can be judged more accurately, thereby more reasonably adjusting the forward error correction redundancy and the video frame rate, and improving the adaptability and robustness of the system.

[0088] It should be noted that the above description is only a description of the preferred embodiments of the present invention, and does not limit the scope of the present invention in any way. Any changes and modifications made by those of ordinary skill in the art of the present invention according to the above disclosure shall fall within the protection scope of the present invention.

Claims

1. A video transmission system, characterized in that: A video encoder and a video decoder are included in communication connection, wherein the video encoder is configured to be arranged at a video transmitting end, and the video decoder is configured to be arranged at a video receiving end; The video encoder includes a data encoding module, a forward error correction encoding module and a first transmission control module; The data encoding module is configured to encode the acquired video data using a preset video encoding protocol to obtain video encoding data and send the video encoding data to the forward error correction encoding module; The forward error correction encoding module is configured to perform forward error correction encoding on the video encoding data according to the forward error correction redundancy configured by the first transmission control module to obtain forward error correction encoded data containing redundant data; The video decoder is configured to decode the forward error correction coded data to obtain corresponding video data, and feed back network packet loss information and video frame loss information to the first transmission control module; The first transmission control module is configured to dynamically adjust the forward error correction redundancy and / or the video frame rate according to the available network bandwidth configured therein and the network packet loss information and the video frame loss information fed back by the video decoder; The first transmission control module is configured to reduce the video frame rate when the forward error correction redundancy calculated based on the network packet loss information is greater than or equal to the maximum allowed forward error correction redundancy corresponding to the network available bandwidth and video frame loss occurs.

2. The video transmission system according to claim 1, characterized in that: The first transmission control module is configured to parse the network packet loss information to obtain a network packet loss rate, and increase the forward error correction redundancy when the network packet loss rate increases, and reduce the forward error correction redundancy when the network packet loss rate decreases.

3. The video transmission system according to claim 2, characterized in that: The first transmission control module is configured to increase or decrease the forward error correction redundancy according to the following rules when the network packet loss rate increases or decreases: r≥p / (1-p) Among them, r is the forward error correction redundancy, and p is the network packet loss rate.

4. The video transmission system according to claim 1, characterized in that: The first transmission control module is configured to adjust the forward error correction redundancy of the forward error correction encoding module to the maximum allowable forward error correction redundancy when the forward error correction redundancy calculated based on the network packet loss information is greater than or equal to the maximum allowable forward error correction redundancy corresponding to the network available bandwidth and video frame loss occurs.

5. The video transmission system according to claim 1, characterized in that: The first transmission control module is configured to increase the video frame rate when the forward error correction redundancy calculated based on the network packet loss information is less than the maximum allowed forward error correction redundancy corresponding to the network available bandwidth and its decrease amplitude is greater than or equal to a preset amplitude, and the video frame rate has previously decreased.

6. The video transmission system according to claim 1, characterized in that: The single decrease amplitude of the forward error correction redundancy is greater than or equal to a first preset minimum amplitude, and the absolute value of the single adjustment amplitude of the video frame rate is greater than or equal to a second preset minimum amplitude.

7. The video transmission system according to claim 1, characterized in that: The first transmission control module is configured to send an alarm message when the video frame rate is a preset minimum frame rate and the forward error correction redundancy calculated based on the network packet loss information is greater than or equal to the maximum allowed forward error correction redundancy corresponding to the network available bandwidth.

8. The video transmission system according to claim 1, characterized in that: The first transmission control module is also configured to allow the user to set the network available bandwidth, and / or allow the user to configure the data encoding module and the forward error correction encoding module, and / or allow the user to view network packet loss information, video frame loss information and forward error correction redundancy in real time, and / or allow the user to configure screen display information.

9. The video transmission system according to claim 1, characterized in that: The video encoder also includes a network sending module, which is configured to encrypt the forward error correction encoded data using a preset encryption algorithm to obtain forward error correction encrypted encoded data and send the forward error correction encrypted encoded data to the video decoder using a preset high-speed secure communication protocol.

10. The video transmission system according to claim 9, characterized in that: The video decoder includes a network receiving module, a forward error correction decoding module, a data decoding module and a second transmission control module; The network receiving module is configured to decrypt the received forward error correction encrypted data to obtain corresponding forward error correction encoded data and send it to the forward error correction decoding module, and send a network packet loss notification to the second transmission control module when a packet is lost; The forward error correction decoding module is configured to perform forward error correction decoding on the forward error correction encoded data to obtain corresponding video encoded data and send it to the data decoding module, and send a video frame loss notification to the second transmission control module when a frame is lost; The data decoding module is configured to decode the received video encoding data to obtain corresponding video data; The second transmission control module is configured to transmit the network packet loss information and the video frame loss information to the first transmission control module via the preset high-speed secure communication protocol.

11. The video transmission system according to claim 10, characterized in that: The second transmission control module is also configured to allow the user to configure the network receiving module, the forward error correction decoding module and the data decoding module, and / or allow the user to view network packet loss information, video frame loss information and forward error correction redundancy in real time, and / or allow the user to configure screen display information.

12. The video transmission system according to claim 1, characterized in that: It also includes a cloud server, and the video encoder and the video decoder exchange data through the cloud server.

13. A surgical robot system, characterized in that: A video transmission system comprising a remote end, a local end and any one of claims 1 to 12.

Images