Image Transmission Method and Device Based on Telemedicine, and Surgical Robot

Abstract

The present invention provides an image transmission method, device and surgical robot based on telemedicine, including: obtaining the currently available bandwidth; calculating the bandwidth required for encoding and transmitting the images captured by the endoscope (target bandwidth); determining whether the currently available bandwidth meets the target bandwidth; if it meets, encoding the images captured by the endoscope; otherwise, reducing the image encoding quality level and encoding at the adjusted image encoding quality level; decoding the encoded data to obtain a decoded image; and enhancing the decoded image to obtain a target image. By judging the bandwidth and selecting the corresponding image encoding quality level for encoding, it can ensure that the encoded data can be smoothly and real-time remotely transmitted and has a high image quality; by enhancing the decoded image, the details and textures of the finally displayed target image can be clearly shown, solving the problem of how to improve the quality of telemedicine image transmission.

Description

Technical Field

[0001] The present invention relates to the technical field of medical information processing, and particularly to an image transmission method and apparatus based on telemedicine, and a surgical robot. Background Art

[0002] With the continuous development of telemedicine technology, endoscopic surgery has been widely used in the medical field. The process of image transmission in endoscopic surgery is generally as follows: First, at the operating room trolley end, the image captured by the endoscope is encoded to obtain encoded data; then, the encoded data is transmitted to the remote operation end in real time through the network; next, the remote operation end decodes the encoded data to restore the image; finally, the received image is displayed on the operating trolley to facilitate the doctor to perform surgical operations according to the displayed image.

[0003] However, due to limited bandwidth conditions, existing image transmission methods need to perform compression processing on the image at the encoding end. However, the compressed image data will lose detailed information, making it difficult to achieve high-quality endoscopic image transmission. Moreover, due to the specificity of the shooting environment of endoscopic surgery, the video images captured by it have significant characteristics such as uneven illumination, high dynamic range, and large noise compared with ordinary images. As a result, after the image undergoes encoding, transmission, and decoding, the final displayed image effect is quite different from the original captured content, seriously affecting the doctor's judgment and diagnosis of the condition.

[0004] In addition, when the network condition is poor, existing image transmission methods will have problems such as image freezing, mosaics, and blurring, which cannot meet the requirements of real-time transmission. Moreover, existing image transmission methods do not fully highlight the key diagnostic information when processing high-quality medical images, which may cause key details to be lost during compression and transmission, affecting the accuracy of remote diagnosis. Summary of the Invention

[0005] The purpose of the present invention is to provide an image transmission method and apparatus based on telemedicine, and a surgical robot, so as to at least solve the problem of how to improve the quality of telemedicine image transmission.

[0006] To solve the above technical problems, the present invention provides an image transmission method based on telemedicine, including:

[0007] Obtain the currently available bandwidth;

[0008] Calculate the bandwidth required for encoding and transmitting the image captured by the endoscope at a preset image encoding quality level to obtain the target bandwidth;

[0009] Determine whether the currently available bandwidth meets the target bandwidth;

[0010] If the current available bandwidth meets the target bandwidth, encode the endoscopic captured image at a preset image encoding quality level to obtain encoded data;

[0011] If the current available bandwidth does not meet the target bandwidth, reduce the image encoding quality level until it adapts to the current available bandwidth, and encode the endoscopic captured image at the adjusted image encoding quality level to obtain encoded data;

[0012] After receiving the remotely transmitted encoded data, decode the encoded data to obtain a decoded image;

[0013] Use an image enhancement algorithm to enhance the decoded image to obtain a target image.

[0014] Optionally, in the above image transmission method based on telemedicine, the method for obtaining the current available bandwidth includes:

[0015] Obtain the real-time bandwidth information of the network;

[0016] Estimate the current available bandwidth based on the real-time bandwidth information.

[0017] Optionally, in the above image transmission method based on telemedicine, the method for estimating the current available bandwidth based on the real-time bandwidth information includes:

[0018] Estimate the current available bandwidth using the Kalman filtering algorithm based on all historical bandwidth information;

[0019] Or, estimate the current available bandwidth using the BBR congestion control algorithm based on the historical bandwidth information of a preset time period.

[0020] Optionally, in the above image transmission method based on telemedicine, the method for calculating the bandwidth required for encoding and transmitting the endoscopic captured image at a preset image encoding quality level to obtain the target bandwidth includes:

[0021] According to the preset image encoding quality level, query the corresponding encoding parameters, where the encoding parameters include the compression ratio, resolution, and frame rate;

[0022] Calculate the required transmission bandwidth based on the original data information and encoding parameters of the endoscopic captured image to obtain the target bandwidth.

[0023] Optionally, in the above image transmission method based on telemedicine, the original data information of the endoscopic captured image includes the pixel depth;

[0024] The calculation method of the target bandwidth is:

[0025] Original data rate (Mbps) = Resolution (pixels) × Frame rate (fps) × Pixel depth (bits / pixel) ÷ 10 6

[0026] Target bandwidth (Mbps) = Original data rate (Mbps) ÷ Compression ratio.

[0027] Optionally, in the image transmission method based on telemedicine, the method of using an image enhancement algorithm to enhance the decoded image to obtain a target image includes:

[0028] Adopting contrast-limited adaptive histogram equalization to perform adaptive illumination correction on the decoded image;

[0029] And / or, using a denoising convolutional neural network to perform denoising processing on the decoded image;

[0030] And / or, adopting an enhanced super-resolution generative adversarial network to perform super-resolution reconstruction on the decoded image.

[0031] Optionally, in the image transmission method based on telemedicine, the method of using an image enhancement algorithm to enhance the decoded image to obtain a target image includes the following steps performed in sequence:

[0032] Adopting contrast-limited adaptive histogram equalization to perform adaptive illumination correction on the decoded image;

[0033] Using a denoising convolutional neural network to perform denoising processing on the decoded image;

[0034] Adopting an enhanced super-resolution generative adversarial network to perform super-resolution reconstruction on the decoded image.

[0035] Optionally, in the image transmission method based on telemedicine, the method of adopting contrast-limited adaptive histogram equalization to perform adaptive illumination correction on the decoded image includes:

[0036] According to the local brightness characteristics of the decoded image, adaptively adjust the contrast limit parameter of the CLAHE algorithm, where the adjustment range of the contrast limit parameter is 0.01~0.1;

[0037] Adopt a sliding window to traverse the decoded image, and implement the CLAHE algorithm on the sub-region within each sliding window, where the length and width of the sliding window are 32~128 pixels.

[0038] Optionally, in the image transmission method based on telemedicine, the method of adopting an enhanced super-resolution generative adversarial network to perform super-resolution reconstruction on the decoded image includes:

[0039] During the training phase, an edge-aware loss is added to the enhanced super-resolution generative adversarial network, and the edge-aware loss function is as follows:

[0040]

[0041] Wherein, represents the decoded image, represents the real high-resolution image; Sobel represents the Sobel operator, which is used to extract edge details.

[0042] Optionally, in the image transmission method based on telemedicine, after obtaining the target image, the image transmission method based on telemedicine further includes:

[0043] Fusing the target image with the digital organ model to obtain a fused image;

[0044] Overlaying the diagnosis and treatment information in the form of a picture-in-picture on the fused image to obtain a display image.

[0045] Optionally, in the image transmission method based on telemedicine, the method of fusing the target image with the digital organ model to obtain a fused image includes:

[0046] Obtaining a three-dimensional organ model;

[0047] Based on the binocular stereo vision principle, calculating the disparity between the left and right images of the endoscope to restore the depth information of the scene and obtain the point cloud data of the scene;

[0048] Using the real-time dynamic elastic registration technology, and using the point cloud data to register the three-dimensional organ model to the surgical scene to obtain a digital organ model;

[0049] Based on the binocular vision principle, converting the digital organ model into the endoscope coordinate system to fuse the target image with the digital organ model and obtain a fused image.

[0050] To solve the above technical problems, the present invention also provides an image transmission device based on telemedicine, which is used to implement the image transmission method based on telemedicine as described in any one of the above, and the image transmission device based on telemedicine includes:

[0051] A bandwidth calculation module, which is used to obtain the current available bandwidth and calculate the bandwidth required for encoding and transmitting the endoscope captured image at a preset image encoding quality level to obtain a target bandwidth;

[0052] A bandwidth matching module, which is used to determine whether the current available bandwidth meets the target bandwidth, and when the current available bandwidth does not meet the target bandwidth, reduce the image encoding quality level until it adapts to the current available bandwidth;

[0053] An image encoding module, configured to encode an endoscope-captured image at an image encoding quality level to obtain encoded data;

[0054] A data decoding module, configured to receive the remotely transmitted encoded data and decode the encoded data to obtain a decoded image;

[0055] An image processing module, configured to enhance the decoded image using an image enhancement algorithm to obtain a target image.

[0056] Optionally, in the image transmission device based on telemedicine, the image transmission device based on telemedicine further includes: a fusion display module, configured to fuse the target image with a digital organ model to obtain a fused image; and further configured to superimpose diagnosis and treatment information on the fused image in a picture-in-picture form to obtain a display image.

[0057] To solve the above technical problems, the present invention further provides a surgical robot, including a memory, a processor, and an executable program stored on the memory and capable of being run by the processor; when the processor runs the executable program, it executes the image transmission method based on telemedicine described in any one of the above.

[0058] The image transmission method, device, and surgical robot based on telemedicine provided by the present invention include: obtaining the currently available bandwidth; calculating the bandwidth required for encoding and transmitting an endoscope-captured image at a preset image encoding quality level to obtain a target bandwidth; determining whether the currently available bandwidth meets the target bandwidth; if the currently available bandwidth meets the target bandwidth, encoding the endoscope-captured image at the preset image encoding quality level to obtain encoded data; if the currently available bandwidth does not meet the target bandwidth, reducing the image encoding quality level until it adapts to the currently available bandwidth, and encoding the endoscope-captured image at the adjusted image encoding quality level to obtain encoded data; after receiving the remotely transmitted encoded data, decoding the encoded data to obtain a decoded image; using an image enhancement algorithm to enhance the decoded image to obtain a target image. By determining whether the currently available bandwidth meets the target bandwidth and selecting the corresponding image encoding quality level for encoding, it is possible to ensure smooth real-time remote transmission of the encoded data while having a high image quality; by performing image enhancement processing on the decoded image, it is possible to improve the endoscope-captured image and restore the image details compressed during encoding, so that the finally displayed target image can clearly show the detailed texture, solving the problem of how to improve the quality of telemedicine image transmission. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] Figure 1 It is a flowchart of the image transmission method based on telemedicine provided in this embodiment;

[0060] Figure 2 Schematic diagram of the device connection structure for remote medical surgery;

[0061] Figure 3 Schematic diagram of the process of adaptive light correction provided by this embodiment;

[0062] Figure 4 Schematic diagram of the denoising process of the denoising convolutional neural network provided by this embodiment

[0063] Figure 5 Flowchart of the complete remote - medical - based image transmission method provided by this embodiment;

[0064] Figure 6 Schematic diagram of the display image in the form of picture - in - picture provided by this embodiment;

[0065] Figure 7 Schematic diagram of the structure of the remote - medical - based image transmission device provided by this embodiment. Detailed implementation manners

[0066] The following further elaborates on the remote - medical - based image transmission method, device, and surgical robot proposed by the present invention in conjunction with the accompanying drawings and specific embodiments. It should be noted that the accompanying drawings are all in a very simplified form and use non - precise scales, only for conveniently and clearly assisting in explaining the purpose of the embodiments of the present invention. In addition, the structures shown in the accompanying drawings are often part of the actual structures. Specifically, the accompanying drawings need to show different focuses, and sometimes different scales are used.

[0067] It should be noted that the "first", "second", etc. in the description, claims, and drawings of the present invention are used to distinguish similar objects, so as to describe the embodiments of the present invention, rather than to describe a specific order or sequence. It should be understood that such structures can be interchanged under appropriate circumstances. In addition, the terms "including" and "having" and any of their variations are intended to cover non - exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units does not necessarily have to be limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or are inherent to these processes, methods, products, or devices.

[0068] This embodiment provides a remote - medical - based image transmission method, as Figure 1 shown, including:

[0069] S1. Obtain the currently available bandwidth;

[0070] S2. Calculate the bandwidth required for encoding and transmitting the images captured by the endoscope at a preset image encoding quality level to obtain the target bandwidth;

[0071] S3. Determine whether the current available bandwidth meets the target bandwidth;

[0072] S4-1. If the current available bandwidth meets the target bandwidth, encode the endoscopic captured image at a preset image coding quality level to obtain encoded data;

[0073] S4-2. If the current available bandwidth does not meet the target bandwidth, reduce the image coding quality level until it adapts to the current available bandwidth, and encode the endoscopic captured image at the adjusted image coding quality level to obtain encoded data;

[0074] S5. After receiving the remotely transmitted encoded data, decode the encoded data to obtain a decoded image;

[0075] S6. Use an image enhancement algorithm to perform enhancement processing on the decoded image to obtain a target image.

[0076] The image transmission method based on telemedicine provided in this embodiment can, by determining whether the current available bandwidth meets the target bandwidth and selecting an appropriate image coding quality level for encoding, ensure smooth real-time remote transmission of the encoded data while having a high image quality; by performing image enhancement processing on the decoded image, it can improve the endoscopic captured image and restore the image details compressed during encoding, enabling the finally displayed target image to clearly show the detailed texture, thus solving the problem of how to improve the quality of telemedicine image transmission.

[0077] In practical applications, the implementation order of steps S1 and S2 can be swapped or carried out simultaneously. It should be noted that changes in the step order without violating the gist of this application should also fall within the protection scope of this application.

[0078] Furthermore, in this embodiment, the method for obtaining the current available bandwidth in step S1 includes:

[0079] S11. Obtain the real-time bandwidth information of the network.

[0080] In practical applications, network detectors or other devices can be used to obtain the real-time broadband information of the network. Preferably, the real-time obtained broadband information can be statistically analyzed and stored to facilitate obtaining historical broadband information. The bandwidth information includes but is not limited to bandwidth, data transmission rate, delay, bandwidth-delay product, throughput, packet loss rate, and jitter, etc.

[0081] S12. Estimate the current available bandwidth based on the real-time bandwidth information.

[0082] Specifically, in this embodiment, the current available bandwidth can be estimated using the Kalman filtering algorithm based on all historical bandwidth information; alternatively, the current available bandwidth can also be estimated using the BBR congestion control algorithm based on the historical bandwidth information within a preset time period (such as the previous 5 minutes before the current moment).

[0083] In the actual application process, when estimating the current available bandwidth using the Kalman filtering algorithm, all historical broadband information of the current day can be selected for estimation and calculation to avoid the problem of low efficiency caused by the calculation of a large amount of historical data. Of course, if there is less historical broadband information for the current day, all historical bandwidth information for the previous several days can also be selected for estimation and calculation to ensure the accuracy of the calculation results.

[0084] Moreover, in the actual application process, when estimating the current available bandwidth using the BBR congestion control algorithm, the length of the time period can be set according to actual requirements and hardware conditions, and it is not strictly limited to 5 minutes. It can be 10 minutes, 15 minutes, 30 minutes, etc.

[0085] Kalman filtering is an algorithm that uses a linear system state equation to optimally estimate the system state through system input and output observation data. Since it is easy to implement in computer programming and can update and process the data collected on-site in real time, Kalman filtering is currently the most widely used filtering method and is applicable to data estimation in telemedicine. Those skilled in the art can obtain the specific implementation method of estimating the current available bandwidth using the Kalman filtering algorithm based on the existing technology, and this application will not elaborate on it further.

[0086] BBR (Bottleneck Bandwidth and Round-trip propagation time) is a congestion control algorithm based on bandwidth and delay feedback. BBR congestion control is an autonomous and automatic control algorithm based on feedback. The control of the rate is determined by the algorithm rather than network events, and the core of the algorithm is "no queuing". The advantages of BBR include strong packet loss resistance, low latency, strong preemption ability, and smooth transmission. Those skilled in the art can obtain the specific implementation method of estimating the current available bandwidth using the BBR congestion algorithm based on the existing technology, and this application will not elaborate on it either.

[0087] Further, in this embodiment, in step S2, the method for calculating the bandwidth required for encoding and transmitting the endoscopic captured image at a preset image coding quality level to obtain the target bandwidth includes:

[0088] S21, query the corresponding coding parameters according to the preset image coding quality level, where the coding parameters include compression ratio, resolution, and frame rate.

[0089] Specifically, in this embodiment, the image encoding quality levels include high definition level, standard definition level, and smooth level. Among them, the encoding parameters corresponding to the high definition level include a compression ratio of 10:1, a resolution of 1920×1080, and a frame rate of 30 frames per second (fps); the encoding parameters corresponding to the standard definition level include a compression ratio of 20:1, a resolution of 640×480, and a frame rate of 25 frames per second (fps); the encoding parameters corresponding to the smooth level include a compression ratio of 30:1, a resolution of 480×360, and a frame rate of 20 frames per second (fps).

[0090] S22. Calculate the required transmission bandwidth according to the original data information and encoding parameters of the endoscopic captured image to obtain the target bandwidth.

[0091] Specifically, in this embodiment, the original data information of the endoscopic captured image includes pixel depth. And, a calculation method for the target bandwidth is given as follows:

[0092] Original data rate (Mbps) = Resolution (pixels) × Frame rate (fps) × Pixel depth (bits / pixel) ÷ 10 6

[0093] Target bandwidth (Mbps) = Original data rate (Mbps) ÷ Compression ratio.

[0094] In this embodiment, the encoding parameters such as the compression ratio, resolution, and frame rate corresponding to the image encoding quality level are range values. For example, the value range of the compression ratio is 5 to 40, the value range of the resolution is 480P to 1080P, and the value range of the frame rate is 15 to 60fps, etc.

[0095] Of course, in other embodiments, the image encoding quality level can be set to three levels: high, medium, and low. Among them, the encoding parameters corresponding to the high level are: a compression ratio of 20, a resolution of 4k, and a frame rate of 60fps; the encoding parameters corresponding to the medium level are a compression ratio of 30, a resolution of 1080P, and a frame rate of 30fps; the encoding parameters corresponding to the low level are a compression ratio of 40, a resolution of 720P, and a frame rate of 24fps.

[0096] Then, in this embodiment, when the original data rate of the endoscopic captured image is 50Mbps, the target bandwidths corresponding to the high, medium, and low levels are 2.5Mbps, 1.67Mbps, and 1.25Mbps in sequence.

[0097] It should be noted that the specific values in the above examples are only used to illustrate the implementation manner of the present application, but the protection scope of the present application is not limited thereto. In practical applications, the image encoding quality level and the calculation method of the corresponding target bandwidth can be set according to actual needs.

[0098] Further, in this embodiment, in step S3, it is determined whether the current available bandwidth meets the target bandwidth.

[0099] Specifically, in this embodiment, if the current available bandwidth is greater than or equal to the target bandwidth, it is considered that the current available bandwidth meets the target bandwidth; conversely, if the current available bandwidth is less than the target bandwidth, it is considered that the current available bandwidth does not meet the target bandwidth.

[0100] And, in this embodiment, in step S4, the method for encoding the endoscopic captured image according to the judgment result includes:

[0101] S4-1, if the current available bandwidth meets the target bandwidth, encode the endoscopic captured image at a preset image encoding quality level to obtain encoded data;

[0102] S4-2, if the current available bandwidth does not meet the target bandwidth, reduce the image encoding quality level until it adapts to the current available bandwidth, and encode the endoscopic captured image at the adjusted image encoding quality level to obtain encoded data.

[0103] That is to say, in this embodiment, the preset image encoding quality level is usually the highest level, so as to ensure that the image data is encoded and transmitted with the highest quality. In the case of insufficient current available bandwidth, the image encoding quality level can be gradually reduced, and the adjusted target bandwidth can be recalculated to determine whether the current available bandwidth meets the new target bandwidth until the current available bandwidth meets the new target bandwidth. Thus, while ensuring that the image data can be transmitted smoothly, the image quality can be ensured as much as possible, avoiding problems such as image freezing, mosaics, and blurring, providing clear and smooth imaging support for doctors, and improving the coherence and accuracy of remote diagnosis.

[0104] It should be noted that in practical applications, when reducing the image encoding quality level, at least one of the encoding parameters such as the compression ratio, resolution, and frame rate is correspondingly reduced.

[0105] And, when encoding the endoscopic captured image, compression encoding methods such as H.264 or H.265 can be used to encode the image. The specific implementation manner of image encoding is well known to those skilled in the art, and this application will not elaborate on it.

[0106] In practical applications, such as Figure 2As shown, the above steps (step S1 to step S4) are usually performed at the operating room trolley end. After image encoding is completed and the encoded data is obtained, the encoded data can be sent from the operating room trolley end to the remote operation end in a remote transmission manner, and the subsequent steps are performed by the remote operation end. Among them, the specific implementation manner of remote transmission is also well known to those skilled in the art, and the present application will not elaborate on this.

[0107] Further, in this embodiment, in step S5, after receiving the encoded data transmitted remotely, the encoded data is decoded to obtain a decoded image.

[0108] Specifically, in practical applications, it is necessary to decode in a manner corresponding to the encoding to restore the complete original image data. The specific implementation manner of image decoding is well known to those skilled in the art, and the present application will not elaborate on this.

[0109] Further, in this embodiment, in step S6, the decoded image is enhanced using an image enhancement algorithm to obtain a target image.

[0110] Among them, Contrast Limited Adaptive Histogram Equalization (CLAHE) can be used to perform adaptive illumination correction on the decoded image to improve the problem of uneven illumination.

[0111] CLAHE is a very classic histogram equalization algorithm, and its main function is to enhance the contrast of the image while being able to suppress noise.

[0112] In this embodiment, as Figure 3 shown, first, according to the local brightness characteristics of the decoded image, the contrast limit parameter of the CLAHE algorithm is adaptively adjusted. Among them, the adjustment range of the contrast limit parameter is 0.01 to 0.1, so that the contrast gain of the central bright region is reduced and the gain of the edge dark region is increased; then, a sliding window ( Figure 3 the small square in) is used to traverse the decoded image, and the CLAHE algorithm is applied to each sub-region within the sliding window to balance the overall brightness of the image and highlight the details of the dark tissues. Among them, the length and width of the sliding window are 32 to 128 pixels.

[0113] Considering that the larger the value set for the contrast limit parameter, the stronger the enhancement effect. However, in the medical scenario, there is a problem of high noise. If the contrast limit parameter is set too large, the noise will also be amplified, resulting in image distortion. If the contrast limit parameter is set too small, the enhancement effect will not be significant. Therefore, in this embodiment, the adjustment range of the contrast limit parameter is restricted to 0.01 to 0.1, so that the decoded image is slightly enhanced and is suitable for the medical scenario (uneven illumination, high dynamic range, high noise). At the same time, the length and width of the sliding window are restricted to 32 to 128 pixels, so that the sliding window can cover different resolution scenarios and improve the applicability of the CLAHE algorithm.

[0114] In a specific embodiment, the range of the contrast limit parameter can be 0.02 to 0.08, and the length and width of the sliding window can be 48 to 96 pixels. Of course, in other embodiments, the contrast limit parameter and the length and width of the sliding window can be adjusted according to actual needs, and this application does not limit this.

[0115] This embodiment is based on the CLAHE algorithm, which can adaptively adjust parameters according to the local brightness characteristics of the image, so that the contrast gain of the central highlight area is reduced and the gain of the edge dark area is increased, thereby balancing the overall brightness of the image and highlighting the details of the dark tissue.

[0116] And / or, a denoising convolutional neural network is used to denoise the decoded image.

[0117] In this embodiment, first, a denoising convolutional neural network needs to be constructed, and the denoising convolutional neural network is trained according to the noise characteristics of the endoscopic image. Then, the trained denoising convolutional neural network is used to denoise the decoded image (if the decoded image has undergone adaptive illumination correction, the denoising convolutional neural network here denoises the decoded image after adaptive illumination correction).

[0118] Specifically, in this embodiment, a training data set is constructed. The training data set needs to contain a large number of labeled noisy and noise-free image pairs, such as 10,000 surgical video images or 20,000 surgical video images. And, as Figure 4 shown, the denoising convolutional neural network can be implemented based on the U-Net structure, so that the input to the denoising convolutional neural network is the noisy image, and the output of the denoising convolutional neural network is the noise-removed image.

[0119] Of course, in other embodiments, the number of images in the training data set can be set according to the actual situation, and even the existing images can be processed by the sample construction method to expand the data set. And, the method of constructing the denoising convolutional neural network and the structure of the constructed denoising convolutional neural network are well known to those skilled in the art, and this application will not elaborate.

[0120] And / or, an Enhanced Super-Resolution Generative Adversarial Network (ESRGAN) is used to perform super-resolution reconstruction on the decoded image to make the image details clearer.

[0121] In this embodiment, during the training phase, in order to increase the texture clarity of the surgical instruments and tissue structures, an edge-aware loss is added to the enhanced super-resolution generative adversarial network. The edge-aware loss function is:

[0122]

[0123] Wherein, represents the decoded image, represents the real high-resolution image; Sobel represents the Sobel operator, which is used to extract edge details.

[0124] During training, the weight of the loss function can be set to 0.1 to 0.5. In a specific embodiment, the weight of the loss function can be set to 0.2 to 0.4. In actual applications, the weight of the loss function can be reasonably set according to actual needs, and the present application does not limit this.

[0125] In addition, the training data uses the high and low resolution pairs of endoscopic images. For example, the low resolution is 480~1080P and the high resolution is 1080P~4K. Another example is that the low resolution is 780P~1080P and the high resolution is 1440P~4K. Of course, the resolution of the high and low resolution images used for training can be set according to actual needs, and the present application does not limit this.

[0126] In the inference application phase, the input to the enhanced super-resolution generative adversarial network is the low-resolution decoded image, and the output of the enhanced super-resolution generative adversarial network is the high-resolution image after resolution reconstruction. Generally, the resolution of the image output by the enhanced super-resolution generative adversarial network is not lower than 720P. For example, the resolution can specifically be 720P or 4K.

[0127] In actual applications, super-resolution algorithms such as FSRCNN (Fast Super-Resolution Convolutional Neural Network) can also be used to achieve the resolution improvement of the decoded image.

[0128] In this embodiment, through technologies such as adaptive illumination correction, denoising processing, and super-resolution reconstruction, problems such as uneven illumination, noise, and resolution of endoscopic images can be effectively improved, key diagnostic information can be highlighted, detailed textures can be retained, the clarity and detail fidelity of the images can be improved, and high-quality diagnostic basis can be provided for doctors.

[0129] Preferably, in this embodiment, step S6 is specifically to perform enhancement processing on the decoded image using a multi-stage image enhancement algorithm, which includes adaptive illumination correction, denoising processing, and super-resolution reconstruction performed in sequence.

[0130] First, contrast-limited adaptive histogram equalization is used to perform adaptive illumination correction on the decoded image.

[0131] Through adaptive illumination correction, the effect of the decoded image can be enhanced to a certain extent and the illumination brightness can be balanced. However, there is also enhanced noise in the enhanced image, and the display effect may not meet the requirements of surgical operations. Therefore, in this embodiment, after adaptive illumination correction, a denoising convolutional neural network is further used to perform denoising processing on the image after adaptive illumination correction.

[0132] Through denoising processing, it can be ensured that the noise in the image is completely removed, ensuring the accuracy of the image content. However, some texture details may be lost during the denoising process, and the resolution of the image may be relatively low, unable to clearly display the detailed textures of the endoscopic captured image. Therefore, after denoising processing, an enhanced super-resolution generative adversarial network is further used to perform super-resolution reconstruction on the image after denoising processing.

[0133] Through super-resolution reconstruction, the resolution of the image can be further improved, so that after the decoded image passes through the multi-stage image enhancement algorithm, the output target image has uniform illumination, high clarity, high resolution, no noise, can clearly and accurately reflect the environmental content captured by the endoscope, retains detailed textures, is conducive to assisting doctors in diagnosis and treatment, and provides high-quality diagnostic basis for doctors.

[0134] In practical applications, one or more of the above image enhancement algorithms can be selected according to actual needs to perform enhancement processing on the decoded image. Moreover, based on the image enhancement algorithm provided in this embodiment, histogram equalization, high-pass filtering, wavelet transform, and other enhancement methods can be selected according to actual needs to perform enhancement processing on the decoded image, and this application does not make any restrictions. Among them, the specific implementation methods of histogram equalization, high-pass filtering, and wavelet transform are well-known to those skilled in the art, and this application will not elaborate further.

[0135] Preferably, in this embodiment, as Figure 5 shown, after obtaining the target image, the image transmission method based on telemedicine further includes:

[0136] S7, fuse the target image with the digital organ model to obtain a fused image.

[0137] Specifically, in this embodiment, first, obtain a three-dimensional organ model. In practical applications, the three-dimensional organ model corresponding to the organ in the endoscope captured image can be obtained from a database, so as to obtain the point cloud data of the three-dimensional organ model. Among them, the three-dimensional organ model is a standard organ model drawn by drawing software such as Sketch, Pixelmator Pro, Procreate, Adobe Illustrator, etc.; the three-dimensional organ model includes but is not limited to a three-dimensional heart model, a three-dimensional chest and lung model, a three-dimensional kidney model, etc. In practical applications, the organ type can be determined according to the position of the organ in the endoscope captured image, and then the corresponding three-dimensional organ model can be called in the database to provide a model basis for subsequent personalized registration.

[0138] Then, based on the binocular stereo vision principle, calculate the disparity of the left and right images of the endoscope to restore the depth information of the scene and obtain the point cloud data of the scene (endoscope environment).

[0139] Next, use the real-time dynamic elastic registration technology, and use the point cloud data to register the three-dimensional organ model to the surgical scene to obtain a digital organ model. In practical applications, considering that there are differences in the size, shape, texture and other details of each organ in different human bodies, and even the organs are squeezed and deformed, etc., therefore, in order to provide more accurate model guidance, in this embodiment, the dynamic elastic registration technology is adopted to adjust the three-dimensional organ model with the point cloud data calculated from the endoscope captured image, so that the finally generated digital organ model can be completely consistent with the organ captured by the endoscope.

[0140] Finally, based on the binocular vision principle, convert the digital organ model to the endoscope coordinate system to fuse the target image with the digital organ model to obtain a fused image. In this way, the planar image captured by the endoscope can be stereoscopically displayed, which is more convenient for doctors' diagnosis, observation and operation, thus effectively reducing the surgical risk.

[0141] S8, superimpose the diagnosis and treatment information on the fused image in the form of a picture-in-picture to obtain a display image.

[0142] Specifically, in this embodiment, the diagnosis and treatment information includes but is not limited to patient information, surgical step prompt information, simulated surgical instruments, pathological analysis results, etc.; among them, the patient information includes name, age, gender, etc., and the surgical step prompt information includes surgical methods, etc. These diagnosis and treatment information can be retrieved from the database.

[0143] In practical applications, such as Figure 6As shown, the diagnosis and treatment information can be floatingly displayed in the upper right corner of the fused image, and the content is presented in the form of a picture-in-picture. Also, the diagnosis and treatment information can be changed in real time according to the surgical process to assist the doctor in diagnosis and treatment. Of course, the display position and size of the diagnosis and treatment information in the fused image can be adjusted according to actual needs, and the present application does not limit this.

[0144] In this embodiment, by fusing the enhanced endoscopic captured image with the three-dimensional organ model and superimposing and displaying the diagnosis and treatment information, more intuitive and comprehensive surgical scene information can be provided for the doctor, improving the accuracy and safety of the surgery.

[0145] This embodiment also provides an image transmission device based on telemedicine for implementing the image transmission method based on telemedicine as described above, as Figure 7 shown, the image transmission device based on telemedicine includes:

[0146] A bandwidth calculation module for obtaining the currently available bandwidth and calculating the bandwidth required for encoding and transmitting the endoscopic captured image at a preset image encoding quality level to obtain a target bandwidth;

[0147] A bandwidth matching module for determining whether the currently available bandwidth meets the target bandwidth, and when the currently available bandwidth does not meet the target bandwidth, reducing the image encoding quality level until it adapts to the currently available bandwidth;

[0148] An image encoding module for encoding the endoscopic captured image at the image encoding quality level to obtain encoded data;

[0149] A data decoding module for receiving the encoded data transmitted remotely and decoding the encoded data to obtain a decoded image;

[0150] An image processing module for enhancing the decoded image using an image enhancement algorithm to obtain a target image.

[0151] Preferably, in this embodiment, the image transmission device based on telemedicine further includes: a fusion display module for fusing the target image with the digital organ model to obtain a fused image; and also for superimposing the diagnosis and treatment information in the form of a picture-in-picture on the fused image to obtain a display image.

[0152] In practical applications, the bandwidth calculation module, the bandwidth matching module, and the image encoding module are usually set at the operating room trolley end; the data decoding module, the image processing module, and the fusion display module are usually set at the remote operation end.

[0153] The image transmission method and device based on telemedicine provided in this embodiment break through the limitations of low bandwidth utilization or image quality degradation caused by traditional fixed encoding parameters by real-time monitoring network bandwidth and dynamically adjusting encoding parameters such as compression ratio, resolution, and frame rate, achieving a dynamic balance between bandwidth and image quality.

[0154] The image transmission method and device based on telemedicine provided in this embodiment adopt a multi-stage image enhancement algorithm, including technologies such as adaptive illumination correction, denoising processing, and super-resolution reconstruction, which can effectively improve problems such as uneven illumination, noise, and resolution of endoscopic images, highlight key diagnostic information, retain detailed textures, improve the clarity and detail fidelity of images, and provide high-quality diagnostic basis for doctors.

[0155] The image transmission method and device based on telemedicine provided in this embodiment break through the limitation of traditional endoscopes only transmitting single images by overlaying auxiliary information such as three-dimensional organ models and instrument path planning, providing more comprehensive surgical decision-making support for doctors.

[0156] In practical applications, by real-time monitoring network bandwidth and dynamically adjusting the compression ratio, resolution, and frame rate, smooth transmission of 720P images can be achieved at a 3Mbps bandwidth (the existing image transmission method requires at least 5Mbps). Also, by overlaying three-dimensional organ models and surgical instrument path planning, doctors can intuitively see the real-time position and preset path of surgical instruments in the patient's body, reducing blind operations during surgery in a visual way, effectively reducing the risk of accidentally injuring important tissues or organs, and thus improving the surgical success rate and patient safety.

[0157] Also, this embodiment further provides a surgical robot, including a memory, a processor, and an executable program stored on the memory and capable of being run by the processor; when the processor runs the executable program, it executes the above-mentioned image transmission method based on telemedicine.

[0158] It should be noted that the various embodiments in this specification are described in a progressive manner. Each embodiment focuses on the differences from other embodiments. The same or similar parts among the various embodiments can be referred to each other. In addition, the different parts among the various embodiments can also be combined and used with each other, and the present invention does not limit this.

[0159] The image transmission method, device and surgical robot based on telemedicine provided in this embodiment include: obtaining the currently available bandwidth; calculating the bandwidth required for encoding and transmitting the images captured by the endoscope at a preset image encoding quality level to obtain a target bandwidth; determining whether the currently available bandwidth meets the target bandwidth; if the currently available bandwidth meets the target bandwidth, encoding the images captured by the endoscope at the preset image encoding quality level to obtain encoded data; if the currently available bandwidth does not meet the target bandwidth, reducing the image encoding quality level until it adapts to the currently available bandwidth, and encoding the images captured by the endoscope at the adjusted image encoding quality level to obtain encoded data; after receiving the remotely transmitted encoded data, decoding the encoded data to obtain decoded images; and using an image enhancement algorithm to perform enhancement processing on the decoded images to obtain target images. By determining whether the currently available bandwidth meets the target bandwidth and selecting an appropriate image encoding quality level for encoding, it is possible to ensure smooth real-time remote transmission of the encoded data while having a high image quality; by performing image enhancement processing on the decoded images, it is possible to improve the images captured by the endoscope and restore the image details compressed during encoding, so that the finally displayed target images can clearly show the detailed texture, solving the problem of how to improve the quality of telemedicine image transmission.

[0160] The above description is only a description of the preferred embodiments of the present invention, and does not limit the scope of the present invention in any way. Any changes and modifications made by those of ordinary skill in the art of the present invention based on the above disclosure shall fall within the protection scope of the claims.

Claims

1. An image transmission method based on telemedicine, characterized in that, Including: Obtain the currently available bandwidth; Calculate the bandwidth required for encoding and transmitting the endoscopic captured image at a preset image encoding quality level to obtain the target bandwidth; Determine whether the currently available bandwidth meets the target bandwidth; If the currently available bandwidth meets the target bandwidth, encode the endoscopic captured image at the preset image encoding quality level to obtain encoded data; If the currently available bandwidth does not meet the target bandwidth, reduce the image encoding quality level until it adapts to the currently available bandwidth, and encode the endoscopic captured image at the adjusted image encoding quality level to obtain encoded data; After receiving the remotely transmitted encoded data, decode the encoded data to obtain a decoded image; Use an image enhancement algorithm to perform enhancement processing on the decoded image to obtain a target image, including: performing adaptive illumination correction on the decoded image using contrast-limited adaptive histogram equalization; using a denoising convolutional neural network to perform denoising processing on the decoded image; Perform super-resolution reconstruction on the decoded image using an enhanced super-resolution generative adversarial network.

2. The image transmission method based on telemedicine according to claim 1, wherein The method for obtaining the currently available bandwidth includes: Obtain the real-time bandwidth information of the network; Estimate the currently available bandwidth based on the real-time bandwidth information.

3. The image transmission method based on telemedicine according to claim 2, wherein The method for estimating the currently available bandwidth based on the real-time bandwidth information includes: Estimate the currently available bandwidth using the Kalman filter algorithm based on all historical bandwidth information; Or, estimate the currently available bandwidth using the BBR congestion control algorithm based on the historical bandwidth information within a preset time period.

4. The image transmission method based on telemedicine according to claim 1, wherein The method for calculating the bandwidth required for encoding and transmitting the endoscopic captured image at a preset image encoding quality level to obtain the target bandwidth includes: Query the corresponding encoding parameters according to the preset image encoding quality level, and the encoding parameters include compression ratio, resolution, and frame rate; Calculate the required transmission bandwidth based on the original data information and encoding parameters of the endoscopic captured image to obtain the target bandwidth.

5. The image transmission method based on telemedicine according to claim 1, wherein The method for performing adaptive illumination correction on the decoded image using contrast-limited adaptive histogram equalization includes: Adaptively adjust the contrast limit parameter of the CLAHE algorithm according to the local brightness characteristics of the decoded image, where the adjustment range of the contrast limit parameter is 0.01 to 0.1; Traverse the decoded image using a sliding window, and apply the CLAHE algorithm to each sub-region within the sliding window, where the length and width of the sliding window are 32 to 128 pixels.

6. The image transmission method based on telemedicine according to claim 1, wherein The method for performing super-resolution reconstruction on the decoded image using an enhanced super-resolution generative adversarial network includes: In the training stage, add an edge-aware loss to the enhanced super-resolution generative adversarial network, and the edge-aware loss function is: Among them, represents the decoded image, represents the true high-resolution image; Sobel represents the Sobel operator, which is used to extract edge details.

7. The image transmission method based on telemedicine according to claim 1, wherein After obtaining the target image, the image transmission method based on telemedicine further includes: Fuse the target image with a digital organ model to obtain a fused image; Overlay the diagnosis and treatment information in a picture-in-picture form on the fused image to obtain a display image.

8. The image transmission method based on telemedicine according to claim 7, wherein The method for fusing the target image with a digital organ model to obtain a fused image includes: Obtain a three-dimensional organ model; Based on the principle of binocular stereo vision, calculate the disparity of the left and right endoscopic images to restore the depth information of the scene and obtain the point cloud data of the scene; Utilize the real-time dynamic elastic registration technology and the point cloud data to register the three-dimensional organ model to the surgical scene to obtain the digital organ model; Based on the principle of binocular vision, convert the digital organ model into the endoscopic coordinate system to fuse the target image with the digital organ model to obtain the fused image.

9. An image transmission device based on telemedicine, which is used to implement the image transmission method based on telemedicine according to any one of claims 1 to 8, characterized in that, The image transmission device based on telemedicine includes: A bandwidth calculation module, configured to obtain the currently available bandwidth and calculate the bandwidth required for encoding and transmitting the endoscopic captured image at a preset image coding quality level to obtain the target bandwidth; A bandwidth matching module, configured to determine whether the currently available bandwidth meets the target bandwidth, and when the currently available bandwidth does not meet the target bandwidth, reduce the image coding quality level until it adapts to the currently available bandwidth; An image coding module, configured to code the endoscopic captured image at the image coding quality level to obtain the coded data; A data decoding module, configured to receive the remotely transmitted coded data and decode the coded data to obtain the decoded image; An image processing module, configured to perform enhancement processing on the decoded image by using an image enhancement algorithm to obtain the target image.

10. The image transmission device based on telemedicine according to claim 9, wherein, The image transmission device based on telemedicine further includes: a fusion display module, configured to fuse the target image with the digital organ model to obtain the fused image; and further configured to superimpose the diagnosis and treatment information on the fused image in a picture-in-picture form to obtain the display image.

11. A surgical robot, characterized in that, It includes a memory, a processor, and an executable program stored on the memory and capable of being run by the processor; when the processor runs the executable program, it executes the image transmission method based on telemedicine according to any one of claims 1 to 8.

Images