Image transmission method and device based on telemedicine and surgical robot

Abstract

The invention provides an image transmission method and device based on telemedicine and a surgical robot. The image transmission method comprises the following steps: acquiring a current available bandwidth; calculating the bandwidth (target bandwidth) required for encoding and transmitting the image shot by the endoscope; judging whether the current available bandwidth meets the target bandwidth or not; if yes, encoding the image shot by the endoscope; otherwise, reducing the image coding quality grade, and coding under the adjusted image coding quality grade; decoding the coded data to obtain a decoded image; and enhancing the decoded image to obtain a target image. By judging the bandwidth and selecting the corresponding image coding quality grade for coding, the coding data can be ensured to be smoothly and remotely transmitted in real time, and the image quality is relatively high; the decoded image is enhanced, so that the finally displayed target image can clearly display detail textures, and the problem of how to improve the transmission quality of the remote medical image is solved.

Description

Technical Field

[0001] The present invention relates to the technical field of medical information processing, and in particular to an image transmission method and device based on telemedicine, and a surgical robot. Background Art

[0002] With the continuous development of telemedicine technology, endoscopic surgery has been widely used in the medical field. The process of image transmission in endoscopic surgery is roughly as follows: first, the image captured by the endoscope is encoded at the operating room trolley to obtain encoded data; then, the encoded data is transmitted to the remote operation end in real time through the network; then, the remote operation end decodes the encoded data to restore the image; finally, the received image is displayed on the operation trolley so that the doctor can perform the operation according to the displayed image.

[0003] However, due to limited bandwidth conditions, existing image transmission methods require image compression processing at the encoding end. However, the compressed image data will lose detailed information, making it difficult to achieve high-quality endoscopic image transmission. Moreover, due to the specificity of the endoscopic surgery shooting environment, the video images captured have significant characteristics such as uneven lighting, high dynamic range, and high noise compared to ordinary images. As a result, after the image is encoded, transmitted, and decoded, the final displayed image effect is quite different from the original shooting content, which seriously affects the doctor's judgment and diagnosis of the disease.

[0004] In addition, when the network conditions are poor, the existing image transmission methods will have problems such as image freeze, mosaic, blur, etc., which cannot meet the needs of real-time transmission. Moreover, the existing image transmission methods fail to fully highlight the key diagnostic information when processing high-quality medical images, which may cause key details to be lost during compression and transmission, affecting the accuracy of remote diagnosis. Summary of the invention

[0005] The purpose of the present invention is to provide a telemedicine-based image transmission method and device, and a surgical robot, so as to at least solve the problem of how to improve the quality of telemedicine image transmission.

[0006] In order to solve the above technical problems, the present invention provides an image transmission method based on telemedicine, comprising: Get the current available bandwidth; Calculate the bandwidth required for encoding and transmitting the images captured by the endoscope at a preset image encoding quality level to obtain a target bandwidth; Determine whether the current available bandwidth meets the target bandwidth; If the currently available bandwidth meets the target bandwidth, the image captured by the endoscope is encoded at a preset image encoding quality level to obtain encoded data; If the current available bandwidth does not meet the target bandwidth, the image encoding quality level is reduced until it is adapted to the current available bandwidth, and the endoscope-captured image is encoded at the adjusted image encoding quality level to obtain encoded data; After receiving the remotely transmitted encoded data, the encoded data is decoded to obtain a decoded image; The decoded image is enhanced using an image enhancement algorithm to obtain the target image.

[0007] Optionally, in the telemedicine-based image transmission method, the method for obtaining the current available bandwidth includes: Get real-time bandwidth information of the network; Estimate the current available bandwidth based on real-time bandwidth information.

[0008] Optionally, in the telemedicine-based image transmission method, the method for estimating the current available bandwidth according to the real-time bandwidth information includes: Based on all historical bandwidth information, the Kalman filter algorithm is used to estimate the current available bandwidth; Or, based on historical bandwidth information in a preset time period, the BBR congestion control algorithm is used to estimate the current available bandwidth.

[0009] Optionally, in the telemedicine-based image transmission method, the method of calculating the bandwidth required for encoding and transmitting the endoscopic image at a preset image encoding quality level to obtain the target bandwidth includes: According to the preset image encoding quality level, query the corresponding encoding parameters, the encoding parameters including compression ratio, resolution and frame rate; According to the original data information and encoding parameters of the endoscope image, the required transmission bandwidth is calculated to obtain the target bandwidth.

[0010] Optionally, in the telemedicine-based image transmission method, the raw data information of the image captured by the endoscope includes pixel depth; The target bandwidth is calculated as follows: Raw data rate (Mbps) = resolution (pixels) × frame rate (fps) × pixel depth (bits / pixel) ÷ 10 6 Target bandwidth (Mbps) = original data rate (Mbps) ÷ compression ratio.

[0011] Optionally, in the telemedicine-based image transmission method, the method of enhancing the decoded image using an image enhancement algorithm to obtain a target image includes: Adaptive illumination correction is performed on the decoded image using contrast-limited adaptive histogram equalization; and / or, performing denoising processing on the decoded image using a denoising convolutional neural network; And / or, using an enhanced super-resolution generative adversarial network to reconstruct the decoded image with super-resolution.

[0012] Optionally, in the telemedicine-based image transmission method, the method of enhancing the decoded image using an image enhancement algorithm to obtain a target image comprises the following steps performed in sequence: Adaptive illumination correction is performed on the decoded image using contrast-limited adaptive histogram equalization; Denoising the decoded image using a denoising convolutional neural network; An enhanced super-resolution generative adversarial network is used to reconstruct the decoded image in super-resolution.

[0013] Optionally, in the telemedicine-based image transmission method, the method of using contrast-limited adaptive histogram equalization to perform adaptive illumination correction on the decoded image includes: According to the local brightness characteristics of the decoded image, the contrast limit parameter of the CLAHE algorithm is adaptively adjusted, where the adjustment range of the contrast limit parameter is 0.01~0.1; A sliding window is used to traverse the decoded image, and the CLAHE algorithm is implemented on the sub-region within each sliding window, where the length and width of the sliding window are 32~128 pixels.

[0014] Optionally, in the telemedicine-based image transmission method, the method of using an enhanced super-resolution generative adversarial network to perform super-resolution reconstruction on a decoded image includes: In the training phase, an edge-aware loss is added to the enhanced super-resolution generative adversarial network, and the edge-aware loss function is:

[0015] in, Represents the decoded image, represents a real high-resolution image; Sobel represents the Sobel operator, which is used to extract edge details.

[0016] Optionally, in the telemedicine-based image transmission method, after obtaining the target image, the telemedicine-based image transmission method further includes: fusing the target image with the digital organ model to obtain a fused image; The diagnosis and treatment information is superimposed on the fused image in a picture-in-picture format to obtain a display image.

[0017] Optionally, in the telemedicine-based image transmission method, the method of fusing the target image with the digital organ model to obtain a fused image includes: Obtain three-dimensional organ models; Based on the principle of binocular stereo vision, the parallax of the left and right images of the endoscope is calculated to restore the depth information of the scene and obtain the point cloud data of the scene; Using real-time dynamic elastic registration technology and point cloud data, the three-dimensional organ model is registered to the surgical scene to obtain a digital organ model; Based on the binocular vision principle, the digital organ model is converted into the endoscope coordinate system to fuse the target image with the digital organ model to obtain a fused image.

[0018] In order to solve the above technical problems, the present invention further provides an image transmission device based on telemedicine, which is used to implement the image transmission method based on telemedicine as described in any one of the above items, and the image transmission device based on telemedicine includes: The bandwidth calculation module is used to obtain the current available bandwidth and calculate the bandwidth required for encoding and transmitting the endoscope image at a preset image encoding quality level to obtain the target bandwidth; The bandwidth matching module is used to determine whether the current available bandwidth meets the target bandwidth, and if the current available bandwidth does not meet the target bandwidth, reduce the image encoding quality level until it is adapted to the current available bandwidth; An image encoding module, used for encoding the image captured by the endoscope at an image encoding quality level to obtain encoded data; A data decoding module, used for receiving the coded data transmitted remotely and decoding the coded data to obtain a decoded image; The image processing module is used to enhance the decoded image using an image enhancement algorithm to obtain a target image.

[0019] Optionally, in the telemedicine-based image transmission device, the telemedicine-based image transmission device also includes: a fusion display module, used to fuse the target image with the digital organ model to obtain a fused image; and also used to superimpose diagnosis and treatment information on the fused image in a picture-in-picture format to obtain a display image.

[0020] In order to solve the above technical problems, the present invention also provides a surgical robot, comprising a memory, a processor, and an executable program stored in the memory and capable of being run by the processor; when the processor runs the executable program, the image transmission method based on telemedicine as described in any one of the above items is executed.

[0021] The telemedicine-based image transmission method and device, and surgical robot provided by the present invention include: obtaining the current available bandwidth; calculating the bandwidth required for encoding and transmitting an image captured by an endoscope at a preset image encoding quality level to obtain a target bandwidth; judging whether the current available bandwidth meets the target bandwidth; if the current available bandwidth meets the target bandwidth, encoding the image captured by the endoscope at the preset image encoding quality level to obtain encoded data; if the current available bandwidth does not meet the target bandwidth, reducing the image encoding quality level until it is adapted to the current available bandwidth, and encoding the image captured by the endoscope at the adjusted image encoding quality level to obtain encoded data; after receiving the remotely transmitted encoded data, decoding the encoded data to obtain a decoded image; and enhancing the decoded image using an image enhancement algorithm to obtain a target image. By judging whether the current available bandwidth meets the target bandwidth and selecting the corresponding image encoding quality level for encoding, it is possible to ensure that the encoded data can be smoothly transmitted remotely in real time while having high image quality; by performing image enhancement processing on the decoded image, it is possible to improve the image captured by the endoscope and restore the image details compressed during encoding, so that the target image finally displayed can clearly display the detailed texture, solving the problem of how to improve the quality of telemedicine image transmission. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 A flowchart of the telemedicine-based image transmission method provided in this embodiment; Figure 2 A schematic diagram of the equipment connection structure for telemedicine surgery; Figure 3 A schematic diagram of the process of adaptive illumination correction provided in this embodiment; Figure 4 Schematic diagram of the denoising process of the denoising convolutional neural network provided in this embodiment Figure 5 A flowchart of a complete telemedicine-based image transmission method provided in this embodiment; Figure 6 A schematic diagram of a display image in a picture-in-picture format provided in this embodiment; Figure 7 This is a schematic diagram of the structure of the telemedicine-based image transmission device provided in this embodiment. DETAILED DESCRIPTION

[0023] The following is a further detailed description of the telemedicine-based image transmission method and device, and the surgical robot proposed in the present invention in conjunction with the accompanying drawings and specific embodiments. It should be noted that the accompanying drawings are all in a very simplified form and use non-precise proportions, which are only used to conveniently and clearly assist in explaining the purpose of the embodiments of the present invention. In addition, the structure shown in the accompanying drawings is often a part of the actual structure. In particular, the emphasis that each drawing needs to show is different, and sometimes different proportions are used.

[0024] It should be noted that the terms "first", "second", etc. in the specification, claims and drawings of the present invention are used to distinguish similar objects in order to describe the embodiments of the present invention, rather than to describe a specific order or sequence. It should be understood that the structures used in this way can be interchanged under appropriate circumstances. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0025] This embodiment provides an image transmission method based on telemedicine, such as Figure 1 As shown, including: S1, obtain the current available bandwidth; S2, calculating the bandwidth required for encoding and transmitting the endoscope image at a preset image encoding quality level to obtain a target bandwidth; S3, determining whether the current available bandwidth meets the target bandwidth; S4-1, if the current available bandwidth meets the target bandwidth, encoding the image captured by the endoscope at a preset image encoding quality level to obtain encoded data; S4-2, if the current available bandwidth does not meet the target bandwidth, the image encoding quality level is reduced until it is adapted to the current available bandwidth, and the endoscope captured image is encoded at the adjusted image encoding quality level to obtain encoded data; S5, after receiving the remotely transmitted encoded data, decoding the encoded data to obtain a decoded image; S6, using an image enhancement algorithm to enhance the decoded image to obtain a target image.

[0026] The telemedicine-based image transmission method provided in this embodiment can ensure that the encoded data can be smoothly and remotely transmitted in real time while having high image quality by judging whether the current available bandwidth meets the target bandwidth and selecting the corresponding image encoding quality level for encoding. By performing image enhancement processing on the decoded image, the image captured by the endoscope can be improved and the image details compressed during encoding can be restored, so that the target image finally displayed can clearly display the detailed texture, thereby solving the problem of how to improve the quality of telemedicine image transmission.

[0027] In practical applications, the order of implementation of step S1 and step S2 can be swapped, or performed simultaneously. It should be noted that the change of the order of steps without violating the main purpose of this application should also fall within the scope of protection of this application.

[0028] Furthermore, in this embodiment, in step S1, the method for obtaining the current available bandwidth includes: S11, obtaining real-time bandwidth information of the network.

[0029] In practical applications, devices or equipment such as network detectors can be used to obtain real-time broadband information of the network. Preferably, the real-time broadband information can be counted and stored to obtain historical broadband information. Bandwidth information includes but is not limited to bandwidth, data transmission rate, delay, bandwidth-delay product, throughput, packet loss rate, and jitter.

[0030] S12, estimating the current available bandwidth according to the real-time bandwidth information.

[0031] Specifically, in this embodiment, the current available bandwidth can be estimated using a Kalman filter algorithm based on all historical bandwidth information; or, the current available bandwidth can be estimated using a BBR congestion control algorithm based on historical bandwidth information for a preset time period (such as 5 minutes before the current moment).

[0032] In actual application, when using the Kalman filter algorithm to estimate the current available bandwidth, all historical bandwidth information of the day can be selected for estimation calculation to avoid the inefficiency caused by the calculation of a large amount of historical data. Of course, if the total historical bandwidth information of the day is less, all historical bandwidth information of the previous few days can also be selected for estimation calculation to ensure the accuracy of the calculation result.

[0033] Also, in actual application, when using the BBR congestion control algorithm to estimate the current available bandwidth, the length of the time period can be set according to actual needs and hardware conditions. It is not strictly limited to 5 minutes, and can be 10 minutes, 15 minutes, 30 minutes, etc.

[0034] Kalman filtering is an algorithm that uses the linear system state equation to optimally estimate the system state through the system input and output observation data. Because it is easy to implement through computer programming and can update and process the data collected on site in real time, Kalman filtering is currently the most widely used filtering method and is suitable for data estimation in telemedicine. Those skilled in the art can obtain a specific implementation method for estimating the current available bandwidth using the Kalman filtering algorithm based on the existing technology, and this application will not go into details.

[0035] BBR (Bottleneck Bandwidth and Round-trip propagation time) is a congestion control algorithm based on bandwidth and delay feedback. BBR congestion control is an autonomous automatic control algorithm based on feedback. The control of the rate is determined by the algorithm rather than by network events. The core of the algorithm is "no queuing". The advantages of BBR include strong anti-packet loss capability, low latency, strong preemption capability and smooth transmission. Based on the existing technology, those skilled in the art can obtain a specific implementation method for estimating the current available bandwidth using the BBR congestion algorithm, and this application will not go into details.

[0036] Further, in this embodiment, in step S2, the method of calculating the bandwidth required for encoding and transmitting the image captured by the endoscope at a preset image encoding quality level to obtain the target bandwidth includes: S21, querying corresponding encoding parameters according to a preset image encoding quality level, where the encoding parameters include compression ratio, resolution and frame rate.

[0037] Specifically, in this embodiment, the image encoding quality levels include high-definition level, standard-definition level and smooth level. The encoding parameters corresponding to the high-definition level include a compression ratio of 10:1, a resolution of 1920×1080, and a frame rate of 30 frames per second (fps); the encoding parameters corresponding to the standard-definition level include a compression ratio of 20:1, a resolution of 640×480, and a frame rate of 25 frames per second (fps); the encoding parameters corresponding to the smooth level include a compression ratio of 30:1, a resolution of 480×360, and a frame rate of 20 frames per second (fps).

[0038] S22, calculating the required transmission bandwidth according to the original data information and encoding parameters of the image captured by the endoscope to obtain the target bandwidth.

[0039] Specifically, in this embodiment, the raw data information of the image captured by the endoscope includes pixel depth. Also, a method for calculating the target bandwidth is given as follows: Raw data rate (Mbps) = resolution (pixels) × frame rate (fps) × pixel depth (bits / pixel) ÷ 10 6 Target bandwidth (Mbps) = original data rate (Mbps) ÷ compression ratio.

[0040] In this embodiment, the encoding parameters such as compression ratio, resolution and frame rate corresponding to the image encoding quality level are range values. For example, the compression ratio is 5~40, the resolution is 480P~1080P, the frame rate is 15~60fps, etc.

[0041] Of course, in other embodiments, the image encoding quality level can be set to three levels: high, medium, and low, where the encoding parameters corresponding to the high level are: compression ratio of 20, resolution of 4k, and frame rate of 60fps; the encoding parameters corresponding to the medium level are compression ratio of 30, resolution of 1080P, and frame rate of 30fps; the encoding parameters corresponding to the low level are compression ratio of 40, resolution of 720P, and frame rate of 24fps.

[0042] Then, in this embodiment, when the original data rate of the image captured by the endoscope is 50 Mbps, the target bandwidths corresponding to the high, medium, and low levels are 2.5 Mbps, 1.67 Mbps, and 1.25 Mbps, respectively.

[0043] It should be noted that the specific values ​​of the above examples are only used to illustrate the implementation of the present application, but the protection scope of the present application is not limited thereto. In practical applications, the image coding quality level and the corresponding target bandwidth calculation method can be set according to actual needs.

[0044] Furthermore, in this embodiment, step S3 determines whether the current available bandwidth meets the target bandwidth.

[0045] Specifically, in this embodiment, if the current available bandwidth is greater than or equal to the target bandwidth, it is considered that the current available bandwidth meets the target bandwidth; conversely, if the current available bandwidth is less than the target bandwidth, it is considered that the current available bandwidth does not meet the target bandwidth.

[0046] And, in this embodiment, in step S4, according to the judgment result, the method for encoding the image captured by the endoscope includes: S4-1, if the current available bandwidth meets the target bandwidth, encoding the image captured by the endoscope at a preset image encoding quality level to obtain encoded data; S4-2, if the current available bandwidth does not meet the target bandwidth, the image encoding quality level is reduced until it is adapted to the current available bandwidth, and the endoscope-captured image is encoded at the adjusted image encoding quality level to obtain encoded data.

[0047] That is to say, in this embodiment, the preset image encoding quality level is usually the highest level, so that the image data can be encoded and transmitted with the highest quality. In the case of insufficient available bandwidth, the image encoding quality level can be gradually reduced, and the adjusted target bandwidth can be recalculated to determine whether the current available bandwidth meets the new target bandwidth, until the current available bandwidth meets the new target bandwidth, so that the image data can be transmitted smoothly while ensuring the image quality as much as possible, avoiding problems such as image freeze, mosaic, blur, etc., providing doctors with clear and smooth image support, and improving the consistency and accuracy of remote diagnosis.

[0048] It should be noted that, in practical applications, when the image encoding quality level is reduced, at least one of the encoding parameters such as compression ratio, resolution and frame rate is correspondingly reduced.

[0049] Furthermore, when encoding the image captured by the endoscope, a compression encoding method such as H.264 or H.265 can be used to encode the image. The specific implementation method of image encoding is well known to those skilled in the art, and this application will not go into details.

[0050] In practical applications, such as Figure 2 As shown, the above steps (step S1 to step S4) are usually performed on the operating room trolley. After completing the image encoding and obtaining the encoded data, the encoded data can be sent from the operating room trolley to the remote operation end by remote transmission, and the remote operation end performs subsequent steps. Among them, the specific implementation method of remote transmission is also well known to those skilled in the art, and this application will not repeat it.

[0051] Furthermore, in this embodiment, in step S5, after receiving the remotely transmitted encoded data, the encoded data is decoded to obtain a decoded image.

[0052] Specifically, in practical applications, decoding needs to be performed in a manner corresponding to encoding to restore the complete original image data. The specific implementation of image decoding is well known to those skilled in the art, and this application will not elaborate on it.

[0053] Furthermore, in this embodiment, in step S6, the decoded image is enhanced using an image enhancement algorithm to obtain a target image.

[0054] Among them, contrast limited adaptive histogram equalization (CLAHE) can be used to perform adaptive illumination correction on the decoded image to improve the problem of uneven illumination.

[0055] CLAHE is a very classic histogram equalization algorithm, which mainly enhances the contrast of the image while suppressing noise.

[0056] In this embodiment, if Figure 3 As shown in Figure 1, first, according to the local brightness characteristics of the decoded image, the contrast limit parameter of the CLAHE algorithm is adaptively adjusted, where the adjustment range of the contrast limit parameter is 0.01~0.1, so that the contrast gain of the central highlight area is reduced and the gain of the edge dark area is increased; then, a sliding window ( Figure 3 The decoded image is traversed by the small square in the sliding window (the small square in the figure), and the CLAHE algorithm is implemented on the sub-region in each sliding window to balance the overall brightness of the image and highlight the dark tissue details. The length and width of the sliding window are 32~128 pixels.

[0057] Considering that the larger the value of the contrast limit parameter is set, the stronger the enhancement effect is, and there is a problem of high noise in medical scenes, if the contrast limit parameter is set to a large value, the noise will also be amplified, resulting in image distortion; if the contrast limit parameter is set too small, the enhancement effect will not be significant. Therefore, this embodiment limits the adjustment range of the contrast limit parameter to 0.01~0.1, so that the decoded image is slightly enhanced, which is suitable for medical scenes (uneven illumination, high dynamic range, and high noise). At the same time, the length and width of the sliding window are limited to 32~128 pixels, so that the sliding window can cover scenes with different resolutions, thereby improving the applicability of the CLAHE algorithm.

[0058] In a specific embodiment, the contrast limit parameter may range from 0.02 to 0.08, and the length and width of the sliding window may range from 48 to 96 pixels. Of course, in other embodiments, the contrast limit parameter and the length and width of the sliding window may be adjusted according to actual needs, and this application does not limit this.

[0059] This embodiment is based on the CLAHE algorithm and can adaptively adjust parameters according to the local brightness characteristics of the image, so that the contrast gain of the central highlight area is reduced and the gain of the edge dark area is increased, thereby balancing the overall brightness of the image and highlighting the dark tissue details.

[0060] And / or, using a denoising convolutional neural network to denoise the decoded image.

[0061] In this embodiment, it is first necessary to construct a denoising convolutional neural network and train the denoising convolutional neural network according to the noise characteristics of the endoscopic image; then, the decoded image is denoised using the trained denoising convolutional neural network (if the decoded image has undergone adaptive illumination correction, the denoising convolutional neural network here denoises the decoded image after adaptive illumination correction).

[0062] Specifically, in this embodiment, a training data set is constructed, and the training data set needs to include a large number of annotated noisy and noise-free image pairs, such as 10,000 surgical video images or 20,000 surgical video images. And, Figure 4 As shown, the denoising convolutional neural network can be implemented based on the U-Net structure, so that the input of the denoising convolutional neural network is a noisy image, and the output of the denoising convolutional neural network is an image with noise removed.

[0063] Of course, in other embodiments, the number of images in the training data set can be set according to actual conditions, and even the existing images can be processed using the sample construction method to expand the data set. In addition, the method of constructing a denoising convolutional neural network and the structure of the constructed denoising convolutional neural network are well known to those skilled in the art, and will not be described in detail in this application.

[0064] And / or, an Enhanced Super-Resolution Generative Adversarial Network (ESRGAN) is used to perform super-resolution reconstruction on the decoded image to make the image details clearer.

[0065] In this embodiment, in order to increase the texture clarity of surgical instruments and tissue structures, an edge-aware loss is added to the enhanced super-resolution generative adversarial network during the training phase. The edge-aware loss function is:

[0066] in, Represents the decoded image, represents a real high-resolution image; Sobel represents the Sobel operator, which is used to extract edge details.

[0067] During training, the weight of the loss function can be set to 0.1-0.5. In a specific embodiment, the weight of the loss function can be set to 0.2-0.4. In practical applications, the weight of the loss function can be reasonably set according to actual needs, and this application does not limit this.

[0068] Also, the training data uses high and low resolution pairs of images taken by the endoscope, for example, the low resolution is 480~1080P and the high resolution is 1080P~4K, and the low resolution is 780P~1080P and the high resolution is 1440P~4K. Of course, the high and low resolution image pairs used for training can be set according to actual needs, and this application does not limit this.

[0069] In the inference application stage, the input to the enhanced super-resolution generative adversarial network is a low-resolution decoded image, and the output of the enhanced super-resolution generative adversarial network is a high-resolution image after resolution reconstruction. Usually, the resolution of the image output by the enhanced super-resolution generative adversarial network is not less than 720P, for example, the resolution can be 720P or 4K.

[0070] In practical applications, super-resolution algorithms such as FSRCNN (Fast Super-Resolution Convolutional Neural Network) can also be used to improve the resolution of decoded images.

[0071] This embodiment uses technologies such as adaptive illumination correction, denoising, and super-resolution reconstruction to effectively improve problems such as uneven illumination, noise, and resolution of endoscopic images, highlight key diagnostic information, retain detailed textures, improve image clarity and detail fidelity, and provide doctors with high-quality diagnostic evidence.

[0072] Preferably, in this embodiment, step S6 is specifically to enhance the decoded image using a multi-stage image enhancement algorithm, which includes adaptive illumination correction, denoising and super-resolution reconstruction performed in sequence.

[0073] Firstly, contrast-limited adaptive histogram equalization is used to perform adaptive illumination correction on the decoded image.

[0074] Through adaptive illumination correction, the decoded image can be enhanced and the light brightness can be balanced to a certain extent. However, the enhanced image will contain noise that is also enhanced, and the display effect may not meet the needs of surgical operations. Therefore, this embodiment further utilizes a denoising convolutional neural network after performing adaptive illumination correction to denoise the image after adaptive illumination correction.

[0075] Through denoising, the noise in the image can be completely removed, ensuring the accuracy of the image content. However, some texture details may be lost in the denoising process, and the resolution of the image may be low, and the detailed texture of the endoscopic image cannot be clearly displayed. Therefore, after the denoising, this embodiment further uses an enhanced super-resolution generative adversarial network to perform super-resolution reconstruction on the denoised image.

[0076] Through super-resolution reconstruction, the resolution of the image can be further improved, so that after the decoded image passes through the multi-stage image enhancement algorithm, the output target image has uniform illumination, high clarity, high resolution, and no noise. It can clearly and accurately reflect the environmental content captured by the endoscope and retain detailed textures, which is conducive to assisting doctors in diagnosis and treatment and providing doctors with high-quality diagnostic basis. In practical applications, one or more of the above-mentioned image enhancement algorithms can be selected to enhance the decoded image according to actual needs. In addition, based on the image enhancement algorithm provided in this embodiment, according to actual needs, enhancement methods such as histogram equalization, high-pass filtering, and wavelet transform can be selected to enhance the decoded image, and this application does not limit this. Among them, the specific implementation methods of histogram equalization, high-pass filtering, and wavelet transform are well known to those skilled in the art, and this application will not repeat them.

[0077] Preferably, in this embodiment, Figure 5 As shown, after obtaining the target image, the telemedicine-based image transmission method further includes: S7, fusing the target image with the digital organ model to obtain a fused image.

[0078] Specifically, in this embodiment, first, a three-dimensional organ model is obtained. In practical applications, the three-dimensional organ model corresponding to the organ in the endoscopic image can be obtained from the database to obtain point cloud data of the three-dimensional organ model. Among them, the three-dimensional organ model is a standard organ model drawn by drawing software such as Sketch, Pixelmator Pro, Procreate, Adobe Illustrator, etc.; the three-dimensional organ model includes but is not limited to a three-dimensional heart model, a three-dimensional chest and lung model, a three-dimensional kidney model, etc. In practical applications, the type of organ can be determined according to the position of the organ in the endoscopic image, and then the corresponding three-dimensional organ model can be called in the database to provide a model basis for subsequent personalized registration.

[0079] Then, based on the principle of binocular stereo vision, the parallax of the left and right images of the endoscope is calculated to restore the depth information of the scene and obtain the point cloud data of the scene (endoscope environment).

[0080] Next, the point cloud data is used to register the three-dimensional organ model to the surgical scene using real-time dynamic elastic registration technology to obtain a digital organ model. In practical applications, considering the differences in size, shape, texture and other details of each organ in different human bodies, and even the extrusion and deformation of the organ, in order to provide more accurate model guidance, in this embodiment, the dynamic elastic registration technology is used to adjust the three-dimensional organ model using the point cloud data calculated from the endoscopic image, so that the final generated digital organ model can be completely consistent with the organ photographed by the endoscope.

[0081] Finally, based on the binocular vision principle, the digital organ model is converted into the endoscope coordinate system to fuse the target image with the digital organ model to obtain a fused image. In this way, the flat image taken by the endoscope can be displayed in three dimensions, which is more convenient for doctors to observe and operate, thereby effectively reducing surgical risks.

[0082] S8, superimposing the diagnosis and treatment information on the fused image in a picture-in-picture format to obtain a display image.

[0083] Specifically, in this embodiment, the diagnosis and treatment information includes but is not limited to patient information, surgical step prompt information, simulated surgical instruments, pathological analysis results, etc.; among which, the patient information includes name, age, gender, etc., and the surgical step prompt information includes surgical procedures, etc. These diagnosis and treatment information can be retrieved from the database.

[0084] In practical applications, such as Figure 6 As shown, the diagnosis and treatment information can be displayed floating in the upper right corner of the fused image, and the content can be displayed in the form of picture-in-picture. In addition, the diagnosis and treatment information can change the displayed content in real time according to the progress of the operation to assist the doctor's diagnosis and treatment. Of course, the display position and size of the diagnosis and treatment information in the fused image can be adjusted according to actual needs, and this application does not limit this.

[0085] This embodiment can provide doctors with more intuitive and comprehensive surgical scene information by fusing the enhanced endoscopic images with the three-dimensional organ model and superimposing them with the diagnosis and treatment information, thereby improving the accuracy and safety of the surgery.

[0086] This embodiment also provides a telemedicine-based image transmission device, which is used to implement the telemedicine-based image transmission method as described above. Figure 7 As shown, the image transmission device based on telemedicine includes: The bandwidth calculation module is used to obtain the current available bandwidth and calculate the bandwidth required for encoding and transmitting the endoscope image at a preset image encoding quality level to obtain the target bandwidth; The bandwidth matching module is used to determine whether the current available bandwidth meets the target bandwidth, and if the current available bandwidth does not meet the target bandwidth, reduce the image encoding quality level until it is adapted to the current available bandwidth; An image encoding module, used for encoding the image captured by the endoscope at an image encoding quality level to obtain encoded data; A data decoding module, used for receiving the coded data transmitted remotely and decoding the coded data to obtain a decoded image; The image processing module is used to enhance the decoded image using an image enhancement algorithm to obtain a target image.

[0087] Preferably, in this embodiment, the telemedicine-based image transmission device also includes: a fusion display module, used to fuse the target image with the digital organ model to obtain a fused image; and also used to superimpose the diagnosis and treatment information on the fused image in a picture-in-picture format to obtain a display image.

[0088] In practical applications, the bandwidth calculation module, bandwidth matching module and image encoding module are usually set at the operating room trolley end; the data decoding module, image processing module and fusion display module are usually set at the remote operation end.

[0089] The telemedicine-based image transmission method and device provided in this embodiment breaks through the limitations of low bandwidth utilization or image quality degradation caused by traditional fixed encoding parameters by real-time monitoring of network bandwidth and dynamically adjusting encoding parameters such as compression ratio, resolution and frame rate, and achieves a dynamic balance between bandwidth and image quality.

[0090] The telemedicine-based image transmission method and device provided in this embodiment adopt a multi-stage image enhancement algorithm, including adaptive illumination correction, denoising processing, and super-resolution reconstruction techniques, which can effectively improve the problems of uneven illumination, noise, and resolution of endoscopic images, highlight key diagnostic information, retain detailed textures, improve image clarity and detail fidelity, and provide doctors with high-quality diagnostic basis.

[0091] The telemedicine-based image transmission method and device provided in this embodiment, by superimposing auxiliary information such as three-dimensional organ models and instrument path planning, breaks through the limitation of traditional endoscopes that only transmit a single image, and provides doctors with more comprehensive surgical decision support.

[0092] In actual application, by real-time monitoring of network bandwidth and dynamically adjusting compression ratio, resolution and frame rate, smooth transmission of 720P images can be achieved at 3Mbps bandwidth (the existing image transmission method requires at least 5Mbps). In addition, by superimposing 3D organ models and surgical instrument path planning, doctors can intuitively see the real-time position and preset path of surgical instruments in the patient's body, reducing blind operations during surgery in a visual way, effectively reducing the risk of accidental injury to important tissues or organs, thereby improving the success rate of surgery and patient safety.

[0093] Furthermore, the present embodiment further provides a surgical robot, comprising a memory, a processor, and an executable program stored in the memory and capable of being run by the processor; when the processor runs the executable program, the image transmission method based on telemedicine as described above is executed.

[0094] It should be noted that the various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other. In addition, the different parts between the various embodiments can also be used in combination with each other, and the present invention is not limited to this.

[0095] The telemedicine-based image transmission method and device, and surgical robot provided in this embodiment include: obtaining the current available bandwidth; calculating the bandwidth required for encoding and transmitting the endoscopic image at a preset image encoding quality level to obtain the target bandwidth; judging whether the current available bandwidth meets the target bandwidth; if the current available bandwidth meets the target bandwidth, encoding the endoscopic image at the preset image encoding quality level to obtain encoded data; if the current available bandwidth does not meet the target bandwidth, reducing the image encoding quality level until it is adapted to the current available bandwidth, and encoding the endoscopic image at the adjusted image encoding quality level to obtain encoded data; after receiving the remotely transmitted encoded data, decoding the encoded data to obtain a decoded image; and enhancing the decoded image using an image enhancement algorithm to obtain a target image. By judging whether the current available bandwidth meets the target bandwidth and selecting the corresponding image encoding quality level for encoding, it is possible to ensure that the encoded data can be smoothly transmitted remotely in real time while having high image quality; by performing image enhancement processing on the decoded image, it is possible to improve the image captured by the endoscope and restore the image details compressed during encoding, so that the target image finally displayed can clearly display the detailed texture, solving the problem of how to improve the quality of telemedicine image transmission.

[0096] The above description is only a description of the preferred embodiments of the present invention, and is not intended to limit the scope of the present invention. Any changes or modifications made by a person skilled in the art in the field of the present invention based on the above disclosure shall fall within the scope of protection of the claims.

Claims

1. A telemedicine-based image transmission method, characterized in that: include: Get the current available bandwidth; Calculate the bandwidth required for encoding and transmitting the images captured by the endoscope at a preset image encoding quality level to obtain a target bandwidth; Determine whether the current available bandwidth meets the target bandwidth; If the currently available bandwidth meets the target bandwidth, the image captured by the endoscope is encoded at a preset image encoding quality level to obtain encoded data; If the current available bandwidth does not meet the target bandwidth, the image encoding quality level is reduced until it is adapted to the current available bandwidth, and the endoscope captured image is encoded at the adjusted image encoding quality level to obtain encoded data; After receiving the remotely transmitted encoded data, the encoded data is decoded to obtain a decoded image; The decoded image is enhanced using an image enhancement algorithm to obtain the target image.

2. The telemedicine-based image transmission method according to claim 1, characterized in that: The method for obtaining the current available bandwidth includes: Get real-time bandwidth information of the network; Estimate the current available bandwidth based on real-time bandwidth information.

3. The telemedicine-based image transmission method according to claim 2, characterized in that: The method for estimating the current available bandwidth according to the real-time bandwidth information comprises: Based on all historical bandwidth information, the Kalman filter algorithm is used to estimate the current available bandwidth; Or, based on historical bandwidth information in a preset time period, the BBR congestion control algorithm is used to estimate the current available bandwidth.

4. The telemedicine-based image transmission method according to claim 1, characterized in that: The method for calculating the bandwidth required for encoding and transmitting the image captured by the endoscope at a preset image encoding quality level to obtain the target bandwidth includes: According to the preset image encoding quality level, query the corresponding encoding parameters, the encoding parameters including compression ratio, resolution and frame rate; According to the original data information and encoding parameters of the endoscope image, the required transmission bandwidth is calculated to obtain the target bandwidth.

5. The telemedicine-based image transmission method according to claim 1, characterized in that: The method of enhancing the decoded image using an image enhancement algorithm to obtain a target image includes: Adaptive illumination correction is performed on the decoded image using contrast-limited adaptive histogram equalization; and / or, performing denoising processing on the decoded image using a denoising convolutional neural network; And / or, using an enhanced super-resolution generative adversarial network to reconstruct the decoded image with super-resolution.

6. The telemedicine-based image transmission method according to claim 5, characterized in that: The method for adaptively correcting illumination of a decoded image using contrast-limited adaptive histogram equalization includes: According to the local brightness characteristics of the decoded image, the contrast limit parameter of the CLAHE algorithm is adaptively adjusted, where the adjustment range of the contrast limit parameter is 0.01~0.1; A sliding window is used to traverse the decoded image, and the CLAHE algorithm is implemented on the sub-region within each sliding window, where the length and width of the sliding window are 32~128 pixels.

7. The telemedicine-based image transmission method according to claim 5, characterized in that: The method for super-resolution reconstruction of a decoded image by using an enhanced super-resolution generative adversarial network comprises: In the training phase, an edge-aware loss is added to the enhanced super-resolution generative adversarial network, and the edge-aware loss function is: in, Represents the decoded image, represents a real high-resolution image; Sobel represents the Sobel operator, which is used to extract edge details.

8. The telemedicine-based image transmission method according to claim 1, characterized in that: After obtaining the target image, the telemedicine-based image transmission method further includes: fusing the target image with the digital organ model to obtain a fused image; The diagnosis and treatment information is superimposed on the fused image in a picture-in-picture format to obtain a display image.

9. The telemedicine-based image transmission method according to claim 8, characterized in that: The method of fusing the target image with the digital organ model to obtain a fused image comprises: Obtain three-dimensional organ models; Based on the principle of binocular stereo vision, the parallax of the left and right images of the endoscope is calculated to restore the depth information of the scene and obtain the point cloud data of the scene; Using real-time dynamic elastic registration technology and point cloud data, the three-dimensional organ model is registered to the surgical scene to obtain a digital organ model; Based on the binocular vision principle, the digital organ model is converted into the endoscope coordinate system to fuse the target image with the digital organ model to obtain a fused image.

10. A telemedicine-based image transmission device, used to implement the telemedicine-based image transmission method according to any one of claims 1 to 9, characterized in that: The telemedicine-based image transmission device comprises: The bandwidth calculation module is used to obtain the current available bandwidth and calculate the bandwidth required for encoding and transmitting the endoscope image at a preset image encoding quality level to obtain the target bandwidth; The bandwidth matching module is used to determine whether the current available bandwidth meets the target bandwidth, and if the current available bandwidth does not meet the target bandwidth, reduce the image encoding quality level until it is adapted to the current available bandwidth; An image encoding module, used for encoding the image captured by the endoscope at an image encoding quality level to obtain encoded data; A data decoding module, used for receiving the coded data transmitted remotely and decoding the coded data to obtain a decoded image; The image processing module is used to enhance the decoded image using an image enhancement algorithm to obtain a target image.

11. The telemedicine-based image transmission device according to claim 10, characterized in that: The telemedicine-based image transmission device also includes: a fusion display module, which is used to fuse the target image with the digital organ model to obtain a fused image; and is also used to superimpose diagnosis and treatment information on the fused image in a picture-in-picture format to obtain a display image.

12. A surgical robot, characterized in that: It includes a memory, a processor, and an executable program stored in the memory and capable of being run by the processor; when the processor runs the executable program, the image transmission method based on telemedicine as described in any one of claims 1 to 9 is executed.

Images