Image output method and device, surgical robot and electronic equipment

Abstract

The invention provides an image output method and device, a surgical robot and electronic equipment. The method comprises the steps that an in-vivo scene image to be output is acquired; performing light reflection detection, image quality detection and surgical instrument position detection on the to-be-output in-vivo scene image; judging whether the to-be-output in-vivo scene image is abnormal or not according to a detection result; if the judgment result is yes, correspondingly repairing the to-be-output in-vivo scene image according to the abnormal category of the to-be-output in-vivo scene image to obtain a target in-vivo scene image and outputting the target in-vivo scene image; and if the judgment result is no, directly outputting the to-be-output in-vivo scene image or performing basic preprocessing on the to-be-output in-vivo scene image and then outputting the to-be-output in-vivo scene image. According to the invention, a clearer and more complete image can be output, the smoothness and coherence of the output image are ensured, the accurate tracking of the position of the surgical instrument can be realized, and a surgeon can better observe the surgical process.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, in particular to an image output method and device, a surgical robot and an electronic device. BACKGROUND

[0002] In the use process of the surgical robot, the image display system draws the video frame from the image host using FPGA frame by frame to collect the image, and outputs the image through HDMI to display in the field of view of the doctor. Through deep learning and other technologies, the image can be repaired, and the problems of image screen, dark light and frame loss can be repaired. However, there are still the following technical problems in the image output process:

[0003] 1) When the host transmits the image, the phenomenon of frame loss occurs, because the time difference between the previous frame and the next frame is too large, the phenomenon of horizontal movement of surgical instruments and image disorder occurs, which affects the surgical effect;

[0004] 2) When the host transmits the image, because the previous frame and the current frame coincide, a fold is formed, which causes the abnormal phenomenon of bending of the instrument in the field of view, which affects the surgical effect;

[0005] 3) Under the cold light source, the surgical instruments are prone to reflection, and the white gauze is also prone to reflection, and the surgical environment is prone to halation, overexposure and other phenomena, which can cause the problem that the image of the instrument or part of the patient's body cannot be observed due to unreasonable light;

[0006] 4) In the case of insufficient light, the problem that the patient's internal area is in the field of view but cannot be clearly observed occurs;

[0007] 5) The existing technology based on deep learning is to insert frames for the whole picture, which has large calculation amount and cannot guarantee accurate tracking of the instrument position.

[0008] It should be noted that the information disclosed in the background section of the present application is only intended to deepen the understanding of the general background of the present application, and should not be regarded as acknowledging or implying in any form that the information constitutes prior art known to those skilled in the art. SUMMARY

[0009] The purpose of the present application is to provide an image output method, device, surgical robot and electronic device, which can not only output clearer and more complete images, but also ensure the smoothness and continuity of the output images, and can accurately track the position of the surgical instrument, which is beneficial to the surgeon to better observe the surgical process.

[0010] To achieve the above purpose, the present application provides an image output method, which comprises:

[0011] acquire an image of an in-vivo scene to be output;

[0012] detect reflection, image quality, and surgical instrument position for the image of the in-vivo scene to be output;

[0013] determine whether the image of the in-vivo scene to be output is abnormal based on the detection results;

[0014] if the determination result is yes, perform corresponding repair on the image of the in-vivo scene to be output according to an abnormal category of the image of the in-vivo scene to be output, to acquire an image of an in-vivo scene target and output the image of the in-vivo scene target;

[0015] if the determination result is no, directly output the image of the in-vivo scene to be output or output the image of the in-vivo scene to be output after performing basic preprocessing on the image of the in-vivo scene to be output.

[0016] Optionally, the detection of reflection for the image of the in-vivo scene to be output comprises:

[0017] perform edge detection on the image of the in-vivo scene to be output to acquire a corresponding edge image;

[0018] for each edge pixel point in the edge image, acquire a gradient estimation index of the edge pixel point according to an intensity of the edge pixel point;

[0019] for each edge pixel point in the edge image, calculate a motion vector of the edge pixel point in the edge image using a SIFT-flow algorithm, and acquire a motion estimation index of the edge pixel point according to the motion vector of the edge pixel point;

[0020] for each edge pixel point in the edge image, acquire a comprehensive estimation index of the edge pixel point according to the gradient estimation index and the motion estimation index of the edge pixel point, and determine whether the edge pixel point is a reflection point according to the comprehensive estimation index of the edge pixel point.

[0021] Optionally, the image quality detection for the image of the in-vivo scene to be output comprises:

[0022] input the image of the in-vivo scene to be output into a pre-trained quality scoring model to acquire a quality score result of the image of the in-vivo scene to be output;

[0023] determine whether the quality score of the image of the in-vivo scene to be output is less than a preset score threshold;

[0024] if yes, determine that the image quality of the image of the in-vivo scene to be output is abnormal.

[0025] Optionally, the surgery instrument position detection on the to-be-outputted in-vivo scene image comprises:

[0026] The surgery instrument recognition is performed on the to-be-outputted in-vivo scene image to obtain image tracking position information of the surgery instrument.

[0027] The position deviation is calculated according to the image tracking position information of the surgery instrument and the motion control position information of the surgery instrument.

[0028] It is judged whether the position deviation is within a preset position deviation range.

[0029] If not, it is determined that the surgery instrument position in the to-be-outputted in-vivo scene image is abnormal.

[0030] Optionally, the corresponding repair is performed on the to-be-outputted in-vivo scene image according to the abnormal category of the to-be-outputted in-vivo scene image to obtain a target in-vivo scene image, which comprises:

[0031] If the abnormal category of the to-be-outputted in-vivo scene image is reflection, the reflection area in the to-be-outputted in-vivo scene image is repaired to obtain a target in-vivo scene image.

[0032] If the abnormal category of the to-be-outputted in-vivo scene image is image quality abnormality, the to-be-outputted in-vivo scene image is repaired according to the previously outputted in-vivo scene image to obtain a target in-vivo scene image.

[0033] If the abnormal category of the to-be-outputted in-vivo scene image is surgery instrument position abnormality, the to-be-outputted in-vivo scene image is repaired according to the motion control position information of the surgery instrument to obtain a target in-vivo scene image.

[0034] Optionally, the repair on the reflection area in the to-be-outputted in-vivo scene image comprises:

[0035] The brightness of the surgery instrument reflection area in the to-be-outputted in-vivo scene image is reduced until the surgery instrument has no reflection and is visible; and / or

[0036] The cavity reflection area in the to-be-outputted in-vivo scene image is filled with pixels at a pixel level by using a priority pixel filling algorithm.

[0037] Optionally, the repair on the to-be-outputted in-vivo scene image according to the previously outputted in-vivo scene image to obtain a target in-vivo scene image comprises:

[0038] The intermediate optical flow information between the to-be-outputted in-vivo scene image and the previously outputted in-vivo scene image is obtained by using an optical flow method.

[0039] Fuse the intermediate optical flow information and the extracted relevant features of the to-be-outputted in-vivo scene image to obtain a target in-vivo scene image.

[0040] Optionally, the repairing the to-be-outputted in-vivo scene image according to the motion control position information of the surgical instrument to obtain a target in-vivo scene image comprises:

[0041] inputting the to-be-outputted in-vivo scene image and the motion control position information of the surgical instrument into a pre-trained deep learning model to obtain a target in-vivo scene image.

[0042] To achieve the above object, the present application further provides an image output device, which comprises:

[0043] an image acquisition module configured to acquire a to-be-outputted in-vivo scene image;

[0044] an image detection module configured to detect reflection, image quality and surgical instrument position of the to-be-outputted in-vivo scene image, and determine whether the to-be-outputted in-vivo scene image is abnormal according to the detection result;

[0045] an image control module configured to issue an abnormality repairing control instruction corresponding to the abnormality category of the to-be-outputted in-vivo scene image, or issue a normal processing control instruction when the to-be-outputted in-vivo scene image is normal;

[0046] an image processing module configured to repair the to-be-outputted in-vivo scene image according to the abnormality repairing control instruction issued by the image control module to obtain a target in-vivo scene image, or perform basic preprocessing on the to-be-outputted in-vivo scene image which is normal according to the normal processing control instruction issued by the image control module; and

[0047] an image output module configured to output the target in-vivo scene image, the to-be-outputted in-vivo scene image which is normal or the to-be-outputted in-vivo scene image which is normal and has been preprocessed.

[0048] Optionally, the image detection module and the image control module are arranged on a first FPGA chip, and the image processing module and the image output module are arranged on a second FPGA chip.

[0049] To achieve the above object, the present application further provides an electronic device comprising a processor and a memory, wherein the memory stores a computer program, and the computer program is executed by the processor to implement the image output method described above.

[0050] To achieve the above object, the application further provides a surgical robot comprising the image output device according to any one of the above.

[0051] Compared with the prior art, the image output method, device, surgical robot and electronic equipment provided by the application have the following beneficial effects:

[0052] The image output method provided by the application comprises the following steps: acquiring an in-vivo scene image to be output; detecting reflection, image quality and a surgical instrument position of the in-vivo scene image to be output; judging whether the in-vivo scene image to be output is abnormal according to a detection result; if the judgment result is yes, performing corresponding repair on the in-vivo scene image to be output according to an abnormal category of the in-vivo scene image to be output, so as to acquire a target in-vivo scene image and output the target in-vivo scene image; and if the judgment result is no, outputting the in-vivo scene image to be output. Thus, the image output method provided by the application can output clearer and more complete images, ensure the fluency and continuity of the output images, and accurately track the position of the surgical instrument, which is beneficial to the surgeon to better observe the surgical process, thereby helping to improve the accuracy and safety of the surgical operation.

[0053] Since the image output device, surgical robot and electronic equipment provided by the application belong to the same inventive concept as the image output method provided by the application, the image output device, surgical robot and electronic equipment provided by the application at least have all the beneficial effects of the image output method provided by the application, and specific contents can be referred to the related contents about the beneficial effects of the image output method provided by the application in the above description, so the beneficial effects of the image output device, surgical robot and electronic equipment provided by the application will not be described one by one here. BRIEF DESCRIPTION OF DRAWINGS

[0054] Figure 1 The surgical robot provided by an embodiment of the application is applied to the scene shown in the figure;

[0055] Figure 2 The patient operation end provided by an embodiment of the application is applied to the scene shown in the figure;

[0056] Figure 3 The in-vivo scene image collected by the endoscope provided by an embodiment of the application is applied to the scene shown in the figure;

[0057] Figure 4 The structure of the doctor control end provided by an embodiment of the application is applied to the scene shown in the figure;

[0058] Figure 5 Structure diagram of an image trolley according to an embodiment of the present application;

[0059] Figure 6 Hardware diagram of image acquisition and processing according to an embodiment of the present application;

[0060] Figure 7 Flow diagram of an image output method according to an embodiment of the present application;

[0061] Figure 8 Flow diagram of a reflection detection according to an embodiment of the present application;

[0062] Figure 9 Flow diagram of an image quality detection according to an embodiment of the present application;

[0063] Figure 10 Network structure diagram of a quality scoring model according to an embodiment of the present application;

[0064] Figure 11 Flow diagram of a surgical instrument position detection according to an embodiment of the present application;

[0065] Figure 12 Flow diagram of a priority pixel filling algorithm according to an embodiment of the present application;

[0066] Figure 13 Flow diagram of an image inpainting based on optical flow according to an embodiment of the present application;

[0067] Figure 14 Flow diagram of a deep learning model according to an embodiment of the present application;

[0068] Figure 15 Flow diagram of an image output method according to an embodiment of the present application;

[0069] Figure 16 Surgical instrument tracking prompt according to an embodiment of the present application;

[0070] Figure 17 Slow operation prompt according to an embodiment of the present application;

[0071] Figure 18 Block structure diagram of an image output device according to an embodiment of the present application;

[0072] Figure 19 Block structure diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0073] The image output method, device, surgical robot and electronic device provided by the present application are further described in detail below in combination with the drawings and specific embodiments. The advantages and features of the present application will be more apparent according to the following description. It should be noted that the drawings are greatly simplified and all use non-precise proportions, only to facilitate, clearly assist the purpose of the present application. In order to make the purpose, features and advantages of the present application more apparent and easy to understand, please refer to the drawings. It should be noted that the structure, proportion, size and the like shown in the drawings attached to the present specification are only used to cooperate with the content disclosed in the specification, so that those skilled in the art can understand and read, and are not used to limit the implementation conditions of the present application. Any modification of structure, change of proportion relationship or adjustment of size, as long as it is the same or similar to the effect and purpose that can be achieved by the present application, should still fall within the scope of the technical content disclosed by the present application.

[0074] The core idea of the present application is to provide an image output method, device, surgical robot and electronic device, which can not only output clearer and more complete images, but also ensure the smoothness and continuity of the output images, and can realize accurate tracking of the position of the surgical instrument 220, which is beneficial to the surgeon to better observe the surgical process.

[0075] It should be noted that the image output method provided by the present application can be applied to the image output device provided by the present application, which can be configured on the image host 310 of the surgical robot or the electronic device. The electronic device can be a personal computer, a mobile terminal, etc. The mobile terminal can be a mobile phone, a tablet computer, etc. with various operating systems. It should also be noted that, as understood by those skilled in the art, the "end" referred to in the present application refers to the end close to the lesion, i.e. the end away from the operator.

[0076] For the sake of understanding, before introducing the image output method, device, surgical robot and electronic device provided by the present application, the application scenario of the surgical robot is briefly described. Please refer to Figure 1 , which is a schematic diagram of the application scenario of the surgical robot provided by an embodiment of the present application. As Figure 1 shown, the surgical robot includes a doctor control end 100, a patient operation end 200, an image trolley 300, a tool trolley 400, an auxiliary device 500, etc.

[0077] In actual operation, the doctor can combine the in-vivo scene image provided by the image platform to operate the surgical instrument 220 (see Figure 3 ) on the patient operation end 200 (see Figure 4 ) through the master hand 110 (see Figure 3 ) on the doctor control end 100 to perform surgical operation.

[0078] Further, the image platform comprises an endoscope acquisition system, an image processing system and an image display system, wherein the endoscope 210 (see Figure 5 ) is installed on the patient operation end 200 to acquire the in-vivo scene image of the patient in real time.

[0079] Please continue to refer to Figure 2 and Figure 3 , wherein, Figure 2 is a schematic diagram of the patient operation end 200 provided by an embodiment of the present application simulating human surgery; Figure 3 is a schematic diagram of the in-vivo scene image acquired by the endoscope 210. As shown in Figure 2 , during the execution of the surgical robot, a rigid surgical instrument 220 is needed for surgical operation, which is mounted on the end of the mechanical arm 230 of the patient operation end 200. As shown in Figure 3 , the in-vivo scene image acquired by the endoscope 210 under the chest and abdominal cavity during surgery generally contains a large number of capillaries, blood, fat, target surgical organs 12 and other cavity environments. At the same time, the in-vivo scene image acquired by the endoscope 210 also includes surgical instruments 220, suture needles, gauze and other contents. As can be seen, the image acquisition environment of the endoscope 210 belongs to a complex image environment. Further, as shown in Figure 2 , the patient operation end 200 generally comprises a plurality of mechanical arms 230, wherein the end of at least one mechanical arm 230 is mounted with an endoscope 210, and the end of at least one mechanical arm 230 is mounted with a surgical instrument 220, wherein each mechanical arm 230 comprises a connected adjustment arm 231 and a tool arm 232, and the endoscope 210 and the surgical instrument 220 are mounted on the end of the tool arm 232 of the corresponding mechanical arm 230 and are extended into the patient's body through the corresponding punch card 240. It should be noted that 11 in the figure represents a human model as understood by those skilled in the art.

[0080] Please continue to refer to Figure 4 , which is a structural schematic diagram of the doctor control end 100 provided by an embodiment of the present application. As shown in Figure 4 , the doctor control end 100 comprises two master hands 110, a first display 120 and a foot pedal 130. The doctor can observe the in-vivo scene image transmitted back through the first display 120, control the movement of the mechanical arm 230, the endoscope 210 and the surgical instrument 220 of the patient operation end 200 through the operation of the master hand 110 with both hands, complete various operations, and thus achieve the purpose of performing surgery for the patient. At the same time, the operator can control part of the action through the foot pedal 130, for example, complete the input of related operations such as electrocision and electrocoagulation through the foot pedal 130.

[0081] Please continue to refer to Figure 5, which is a structural diagram of an imaging trolley 300 provided in one embodiment of the present invention. Figure 5 As shown, the imaging trolley 300 includes an imaging host 310 and a second display 320. Figure 5 As shown, the imaging trolley 300 further includes a cold light source 330 for providing light to the endoscope 210. The in vivo scene images captured by the endoscope 210 are transmitted to the image host 310 for processing and then transmitted to the first display 120 and the second display 320 for display.

[0082] Please continue to refer to Figure 6 , which is a schematic diagram of image acquisition and processing hardware provided by one embodiment of the present invention. Figure 6 As shown, the endoscope 210 collects in-vivo scene images through an ultra-high-definition camera 211 and transmits them to the image host 310 through a high-speed data communication channel 212. The endoscope 210 can be operated by the handle button 213, and the parameters of the endoscope 210 can be saved in the first SPI Flash (serial peripheral interface flash memory) storage module 215 thereon. The image host 310 is responsible for image processing of the received in-vivo scene images, and the image host 310 can be operated through the host button 312 and the operation menu 313. Specifically, the image host 310 includes an FPGA processing module 311, and the FPGA processing module 311 includes an image anomaly detection module 3111, an image anomaly processing module 3112 and an image enhancement module 3113, which can eliminate image anomalies and maintain image stability and clarity. The processed image is finally output to a high-definition display (including a first display 120 and a second display 320) through a multi-output display encoding module 314. In a surgical robot, it is generally necessary to output high-definition 3D images and 2D images. Further, as Figure 6 As shown, the endoscope 210 further includes a first SFP optical module 214 and a first power module 216, and the image host 310 further includes a second SFP optical module 315, a second SPI Flash storage module 316, and a second power module 317. It should be noted that the first SFP optical module 214 and the second SFP optical module 315 are both interface devices that convert gigabit electrical signals into optical signals.

[0083] To realize the above idea, the present invention provides an image output method, please refer to Figure 7 , which is a flow chart of an image output method provided by one embodiment of the present invention, such as Figure 7 As shown, the image output method includes the following steps:

[0084] Step S100: Acquire an in-vivo scene image to be output.

[0085] Step S200, reflection detection, image quality detection and surgical instrument 220 position detection are performed on the to-be-outputted in-vivo scene image.

[0086] Step S300, according to the detection result, it is judged whether the to-be-outputted in-vivo scene image is abnormal.

[0087] If yes, step S400 is performed, according to the abnormal category of the to-be-outputted in-vivo scene image, corresponding repair is performed on the to-be-outputted in-vivo scene image to obtain a target in-vivo scene image and output.

[0088] If no, step S500 is performed, the to-be-outputted in-vivo scene image is directly outputted or is outputted after basic preprocessing.

[0089] Thus, the image output method provided by the present application can output clearer and more complete images, ensure the fluency and continuity of the output images, and accurately track the position of the surgical instrument 220, which is conducive to better observation of the surgical process by the surgeon, thereby improving the accuracy and safety of the surgical operation.

[0090] It should be noted that, as understood by those skilled in the art, the "basic preprocessing" referred to in the present application includes but is not limited to white balance, demosaicing, gamma correction, noise reduction / sharpening, etc. The specific contents of image processing such as white balance, demosaicing, gamma correction, noise reduction / sharpening, etc. can refer to related technologies known to those skilled in the art, and will not be described here.

[0091] Specifically, according to the detection result, it is judged whether the to-be-outputted in-vivo scene image is abnormal, including:

[0092] If the to-be-outputted in-vivo scene image has at least one of reflection, image quality abnormality and surgical instrument 220 position abnormality, it is determined that the to-be-outputted in-vivo scene image is abnormal.

[0093] If the to-be-outputted in-vivo scene image does not have any one of reflection, image quality abnormality and surgical instrument 220 position abnormality, it is determined that the to-be-outputted in-vivo scene image is not abnormal.

[0094] Thus, when at least one of the reflection, the image quality abnormality, and the surgical instrument 220 position abnormality is detected in the to-be-outputted in-vivo scene image, it is determined that the to-be-outputted in-vivo scene image is abnormal; when none of the reflection, the image quality abnormality, and the surgical instrument 220 position abnormality is detected in the to-be-outputted in-vivo scene image, it is determined that the to-be-outputted in-vivo scene image is normal (i.e., not abnormal).

[0095] Further, when only the reflection exists in the to-be-outputted in-vivo scene image, it is determined that the abnormality category of the to-be-outputted in-vivo scene image is the reflection; when only the image quality abnormality exists in the to-be-outputted in-vivo scene image, it is determined that the abnormality category of the to-be-outputted in-vivo scene image is the image quality abnormality; when only the surgical instrument 220 position abnormality exists in the to-be-outputted in-vivo scene image, it is determined that the abnormality category of the to-be-outputted in-vivo scene image is the surgical instrument 220 position abnormality; when the reflection and the image quality abnormality exist in the to-be-outputted in-vivo scene image simultaneously, it is determined that the abnormality category of the to-be-outputted in-vivo scene image is the image quality abnormality and the reflection; when the reflection and the surgical instrument 220 position abnormality exist in the to-be-outputted in-vivo scene image simultaneously, it is determined that the abnormality category of the to-be-outputted in-vivo scene image is the surgical instrument 220 position abnormality and the reflection; when the image quality abnormality and the surgical instrument 220 position abnormality exist in the to-be-outputted in-vivo scene image simultaneously, it is determined that the abnormality category of the to-be-outputted in-vivo scene image is that both the image quality and the surgical instrument 220 position are abnormal; when the reflection, the image quality abnormality, and the surgical instrument 220 position abnormality exist in the to-be-outputted in-vivo scene image simultaneously, it is determined that the abnormality category of the to-be-outputted in-vivo scene image is that both the image quality and the surgical instrument 220 position are abnormal and the reflection.

[0096] Please continue to refer to Figure 8 which is a reflection detection flowchart provided by an embodiment of the present application. As shown in the figure, Figure 8 in some exemplary embodiments, the reflection detection on the to-be-outputted in-vivo scene image comprises:

[0097] performing edge detection on the to-be-outputted in-vivo scene image to obtain a corresponding edge image;

[0098] for each edge pixel point in the edge image, obtaining a gradient estimation index of the edge pixel point according to the intensity of the edge pixel point;

[0099] for each edge pixel point in the edge image, calculating a motion vector of the edge pixel point in the edge image using a SIFT-flow algorithm, and obtaining a motion estimation index of the edge pixel point according to the motion vector of the edge pixel point;

[0100] For each edge pixel point in the edge image, a comprehensive estimation index of the edge pixel point is obtained according to the gradient estimation index and the motion estimation index of the edge pixel point, and it is judged whether the edge pixel point is a highlight point according to the comprehensive estimation index of the edge pixel point.

[0101] Specifically, when the endoscope 210 collects the in-vivo scene image of the patient, a cold light source 330 is generally used for light supplement operation, which generally causes mirror reflection, i.e. highlight, and seriously affects the normal operation of the surgeon. At the same time, if there is a highly exposed area (i.e. a highly highlighted area) in the in-vivo scene image collected by the endoscope 210, it will cause a sharp change of the edge. Due to the complex scene of the robot surgery, there is a very strong reflection of light, so the present application can effectively improve the accuracy of the highlight detection result by, for each pixel point in the edge image corresponding to the in-vivo scene image to be output, obtaining a comprehensive estimation index of the edge pixel point according to the gradient estimation index and the motion estimation index of the edge pixel point, and judging whether the edge pixel point is a highlight point according to the comprehensive estimation index of the edge pixel point.

[0102] Specifically, the in-vivo scene image to be output can be first converted to grayscale, and then a canny algorithm, a sobel algorithm or other edge detection algorithm can be used to perform edge detection on the in-vivo scene image converted to grayscale to obtain the corresponding edge image. Further, the edge pixel points in the edge image can be first smoothed using a superpixel method, and then the gradient estimation index and the motion estimation index of each edge pixel point in the smoothed edge image are obtained.

[0103] It should be noted that, as can be understood by those skilled in the art, the specific algorithm process of the canny algorithm, the sobel algorithm and the SIFT-flow algorithm can refer to the related content known to those skilled in the art, which will not be described here.

[0104] Please continue to refer to Figure 9 , which is an image quality detection process schematic diagram provided by an embodiment of the present application. As Figure 9 shown, in some exemplary embodiments, the image quality detection of the in-vivo scene to be output includes:

[0105] inputting the in-vivo scene image to be output into a pre-trained quality score model to obtain a quality score result of the in-vivo scene image to be output;

[0106] judging whether the quality score of the in-vivo scene image to be output is less than a preset score threshold;

[0107] If yes, it is determined that the image quality of the in-vivo scene image to be output is abnormal.

[0108] Thus, by first using the pre-trained quality score model to obtain the quality score of the to-be-output in-vivo scene image, and then comparing it with the preset score threshold, it can be accurately and quickly judged whether the image quality of the to-be-output in-vivo scene image is abnormal (i.e., whether it is blurred).

[0109] Please continue to refer to Figure 10 , which is a network structure diagram of a quality score model provided by an embodiment of the present application. As Figure 10 indicated, the quality score model uses a multi-level feature fusion network (MLFF-Net), which includes a feature extraction module and a scoring module. The feature extraction module uses the feature extraction capability of an encoder-decoder to simulate the recovery of a distorted image by the human visual system, and the scoring module fuses feature maps at different levels (such as Figure 10 feature maps X1, X2, X3, and X4 in ), which can fully fuse the low-level and high-level features of the image, thus more accurately reflecting the actual quality of the image, thereby effectively ensuring the accuracy of the final output quality score result.

[0110] Further, as Figure 10 indicated, in order to reduce the amount of calculation, before fusing the feature maps at different levels extracted by the feature extraction module, an adaptive average pooling operation is first performed on these feature maps at different levels to obtain corresponding feature vectors, and then the feature vectors corresponding to these feature maps at different levels are fused, and the fused result is input into a fully connected layer, and the corresponding quality score result can be output. Since adaptive average pooling can more effectively extract features, making image processing more accurate, and also saving a lot of time, it not only ensures the scoring accuracy of the quality score model, but also effectively improves the scoring efficiency of the quality score model.

[0111] It should be noted that, as understood by those skilled in the art, the network structure of the quality score model can use other neural network structures other than the multi-level feature fusion network, and the specific selection can be made according to the actual situation, and the present application does not limit it. It should also be noted that, as understood by those skilled in the art, the specific training process of the quality score model can refer to the related content about the training process of the neural network model known to those skilled in the art, and therefore the specific training process of the quality score model will not be described here.

[0112] Please refer to Figure 11 , which is a schematic diagram of a surgical instrument 220 position detection process provided by an embodiment of the present application. AsFigure 11 As shown, in some exemplary embodiments, the surgical instrument 220 position detection on the to-be-outputted in-vivo scene image comprises:

[0113] The surgical instrument 220 recognition on the to-be-outputted in-vivo scene image is performed to obtain the image tracking position information of the surgical instrument 220;

[0114] According to the image tracking position information of the surgical instrument 220 and the motion control position information of the surgical instrument 220, the position deviation is calculated;

[0115] It is judged whether the position deviation is within a preset position deviation range;

[0116] If not, it is determined that the position of the surgical instrument 220 in the to-be-outputted in-vivo scene image is abnormal.

[0117] Specifically, in actual use, the motion control operation speed is in the first time response to realize the operation of the slave hand (located at the patient operation end 200) by the master hand 110 (located at the physician control end 100), during which various motion transformations (i.e. master-slave mapping) are needed to be performed to finally achieve the purpose of manipulating each joint of the slave hand, and generally the algorithm based on the motion control data needs to obtain the actual motion trajectory of the surgical instrument 220 40ms-100ms ahead of the image.

[0118] It should be noted that, as can be understood by those skilled in the art, the main purpose of the "surgical instrument 220 recognition on the to-be-outputted in-vivo scene image" in the present application is to recognize the straight rod (i.e. instrument rod) of the surgical instrument 220 in the to-be-outputted in-vivo scene image. It should also be noted that, as can be understood by those skilled in the art, the motion control position information of the surgical instrument 220 can be obtained based on the position information of the end joint of the master hand 110 and the pre-acquired master-slave mapping relationship. In addition, it should be noted that, as can be understood by those skilled in the art, the image tracking position information of the surgical instrument 220 and the motion control position information of the surgical instrument 220 are position information in the same coordinate system (e.g. robot coordinate system). Specifically, the position information of the surgical instrument 220 in the robot coordinate system can be obtained according to the position information of the surgical instrument 220 in the image coordinate system of the to-be-outputted in-vivo scene image and the related parameters of the endoscope 210, and then according to the position information of the endoscope 210 in the robot coordinate system.

[0119] Further, the preset position deviation range can be set under the premise of comprehensively considering parameters such as the movement speed and acceleration of the surgical instrument 220. In this way, such setting can further ensure that it can be accurately judged whether the image-based instrument movement tracking meets the expectation, thereby enabling the accurate tracking of the position of the surgical instrument 220.

[0120] In some exemplary embodiments, the identification of the surgical instrument 220 in the to-be-output intra-body scene image to obtain the image tracking position information of the surgical instrument 220 comprises:

[0121] The surgical instrument 220 in the to-be-output intra-body scene image is identified by using the Mean shift algorithm to obtain the image tracking position information of the surgical instrument 220.

[0122] Specifically, the Mean shift algorithm realizes the tracking of a target by performing density estimation and gradient descent on the pixels around the target. Since the Mean shift algorithm is easy to understand and implement, and does not require prior knowledge or training data, it can start the tracking process only by using some initial sample points, and the Mean shift algorithm also has good adaptability and can adapt to the scale and shape changes of the target. Therefore, by identifying the surgical instrument 220 in the to-be-output intra-body scene image by using the Mean shift algorithm to obtain the image tracking position information of the surgical instrument 220, the surgical instrument 220 can be quickly and accurately identified.

[0123] It should be noted that, as understood by those skilled in the art, in other embodiments, other algorithms other than the Mean shift algorithm can also be used to track the surgical instrument 220 in the to-be-output intra-body scene image to obtain the tracking position information of the surgical instrument 220. For details, refer to the target tracking algorithm known to those skilled in the art, which will not be listed one by one here.

[0124] In some exemplary embodiments, the corresponding repair of the to-be-output intra-body scene image according to the abnormal category of the to-be-output intra-body scene image to obtain a target intra-body scene image comprises:

[0125] If the abnormal category of the to-be-output intra-body scene image is reflection, the reflection area in the to-be-output intra-body scene image is repaired to obtain a target intra-body scene image;

[0126] If the abnormal category of the to-be-output intra-body scene image is image quality abnormality, the to-be-output intra-body scene image is repaired according to the already-output previous frame of intra-body scene image to obtain a target intra-body scene image;

[0127] If the abnormality category of the to-be-output in-vivo scene image is the abnormal position of the surgical instrument 220, the to-be-output in-vivo scene image is repaired according to the motion control position information of the surgical instrument 220 to obtain a target in-vivo scene image.

[0128] Thus, when the to-be-output in-vivo scene image has a reflection, the reflection area in the to-be-output in-vivo scene image is repaired, which can effectively remove the reflection area in the in-vivo scene image, thereby effectively ensuring the definition of the output target in-vivo scene image and effectively reducing the influence of the cold light source 330 on the surgery. When the image quality of the to-be-output in-vivo scene image is abnormal, the to-be-output in-vivo scene image is repaired according to the previously output in-vivo scene image (which can be a previously output target in-vivo scene image obtained by repair or a previously output in-vivo scene image without abnormality), which can not only effectively ensure the definition of the output target in-vivo scene image but also ensure the continuity of the output image. When the position of the surgical instrument 220 in the to-be-output in-vivo scene image is abnormal, the to-be-output in-vivo scene image is repaired according to the motion control position information of the surgical instrument 220, which can effectively ensure that the position of the surgical instrument 220 in the repaired target in-vivo scene image is more in line with the expectation, thereby effectively improving the accuracy of the position tracking of the surgical instrument 220.

[0129] In some exemplary embodiments, the repairing of the reflection area in the to-be-output in-vivo scene image includes:

[0130] reducing the brightness of the reflection area of the surgical instrument 220 in the to-be-output in-vivo scene image until the surgical instrument 220 has no reflection and is visible; and / or

[0131] performing pixel-level filling on the cavity reflection area in the to-be-output in-vivo scene image by using a priority pixel filling algorithm.

[0132] Specifically, the reflection points are divided into two types, one is the reflection point of the cavity, and the other is the reflection point of the surgical instrument 220. The reflection point of the surgical instrument 220 is easier to handle, and the brightness of the detected reflection area of the surgical instrument 220 can be reduced until the surgical instrument 220 has no highlight and is visible. For the cavity reflection area, pixel-level filling is needed according to the blood vessel texture information, and a priority pixel filling algorithm can be used to repair the cavity reflection area in the to-be-output in-vivo scene image. The area with more blood vessel texture information is identified as an area with higher priority, and pixel-level filling is performed on the area with high priority.

[0133] Please continue to refer to Figure 12Fig. 2 is a specific flowchart of a priority pixel filling algorithm provided by an embodiment of the present application. As shown in Fig. 2, when the priority pixel filling algorithm is used, the pixel-level filling is first performed on the highest priority lumen reflective region, and after the filling is completed, the priority is updated and the iteration is performed step by step until all the reflective regions in the entire lumen are processed. Figure 12

[0134] It should be noted that more details about the priority pixel filling algorithm can be referred to related technologies known by those skilled in the art, which will not be described here. It should also be noted that, as understood by those skilled in the art, the "surgical instrument 220 reflective region" refers to a region formed by reflective points in the region where the surgical instrument 220 is located, and the "lumen reflective region" refers to a region formed by reflective points in the region where the lumen is located.

[0135] Please continue to refer to Figure 13 Fig. 3 is a flowchart of an image repairing process based on the optical flow method provided by an embodiment of the present application. In some exemplary embodiments, the repairing of the to-be-output intra-body scene image based on the output previous intra-body scene image to obtain a target intra-body scene image comprises:

[0136] obtaining intermediate optical flow information between the to-be-output intra-body scene image and the output previous intra-body scene image by using the optical flow method;

[0137] fusing the intermediate optical flow information and the extracted related features of the to-be-output intra-body scene image to obtain a target intra-body scene image.

[0138] Specifically, the basic idea of the optical flow method is to identify the optical flow as the moving track of a pixel. Therefore, by obtaining the intermediate optical flow information between the to-be-output intra-body scene image and the output previous intra-body scene image by using the optical flow method, and then fusing the intermediate optical flow information and the extracted related features of the to-be-output intra-body scene image, a clear and complete target intra-body scene image can be obtained, and the continuity of the obtained target intra-body scene image and the output previous intra-body scene image can be further ensured.

[0139] ​It should be noted that, as can be understood by those skilled in the art, the specific content of how to obtain the intermediate optical flow information between the to-be-output intra-body scene image and the previously output intra-body scene image by using the optical flow method can refer to the related content in the optical flow interpolation known to those skilled in the art, and therefore will not be expanded here. It should be noted that, as can be understood by those skilled in the art, a pre-trained feature extraction network can be used to extract relevant features of the to-be-output intra-body scene image, and the specific network structure of the feature extraction network structure can refer to the related content known to those skilled in the art, and therefore will not be enumerated one by one here.

[0140] In some exemplary embodiments, the repairing the to-be-output intra-body scene image according to the motion control position information of the surgical instrument 220 to obtain a target intra-body scene image comprises:

[0141] inputting the to-be-output intra-body scene image and the motion control position information of the surgical instrument 220 into a pre-trained deep learning model to obtain a target intra-body scene image.

[0142] Therefore, by repairing the to-be-output intra-body scene image according to the motion control position information of the surgical instrument 220 using a pre-trained deep learning model, a target intra-body scene image with less deviation from the motion control data can be obtained more quickly and accurately, not only the accurate tracking of the position of the surgical instrument 220 can be realized, but also the real-time and continuity of image output can be further ensured.

[0143] Specifically, the deep learning model comprises a convolutional neural network (CNN) for extracting image features and a recurrent neural network (RNN) or a convolutional-recurrent neural network (CRNN) for capturing time series information.

[0144] Please refer to Figure 14 , which is a deep learning model flow diagram provided by an embodiment of the present application. As Figure 14 shown, the specific training process of the deep learning model comprises:

[0145] Data collection: collect a batch of training data set containing surgical instrument 220 motion control data and corresponding images (including images with deviations and corresponding repaired or corrected images, wherein the repaired or corrected images are labels), and ensure that the data set contains samples of various motion modes and image deviation conditions.

[0146] Data preprocessing: pre-process the collected training data set, including image denoising, cropping and scale normalization, and surgical instrument 220 motion control data normalization, etc., to ensure that the data has consistent format and range before inputting into the deep learning model.

[0147] Deep learning model design: design a deep learning model suitable for image bias processing. Consider using convolutional neural network (CNN) to extract image features, and combine recurrent neural network (RNN) or convolutional-recurrent neural network (CRNN) to capture time series information. The input of the model includes image frames and corresponding joint data (motion control data), and the output is the repaired or corrected image frame.

[0148] Model training: use the dataset to train the deep learning model, and use the supervised learning method to compare the repaired image frame with the label, and use the appropriate loss function (such as mean square error or structural similarity index) to guide the training process of the model.

[0149] Model verification and optimization: use independent verification dataset to verify the trained deep learning model, and adjust and optimize according to the verification result.

[0150] It should be noted that different network structures, hyperparameters and training strategies can be tried to obtain a deep learning model with better performance.

[0151] Please continue to refer to Figure 15 , which is a schematic diagram of the overall flow of the image output method provided by an embodiment of the present application. As shown in Figure 15 , in order to improve the operation efficiency and ensure the real-time performance of image output, the three detections of reflection detection, image quality detection and surgical instrument 220 position detection can be processed in parallel in the actual execution process. Further, as shown in Figure 15When the abnormality category of the to-be-outputted in-vivo scene image is image quality abnormality and reflection, reflection repair can be performed first, and then optical flow interpolation can be performed on the in-vivo scene image obtained by the reflection repair based on the previously outputted in-vivo scene image, so as to obtain the target in-vivo scene image. When the abnormality category of the to-be-outputted in-vivo scene image is surgical instrument 220 position abnormality and reflection, reflection repair can be performed first, and then motion interpolation (i.e., image repair based on motion control data by using a pre-trained deep learning model) can be performed on the in-vivo scene image obtained by the reflection repair based on the motion control position information of the surgical instrument 220, so as to obtain the target in-vivo scene image. When the abnormality category of the to-be-outputted in-vivo scene image is both image quality abnormality and surgical instrument 220 position abnormality, optical flow interpolation can be performed first based on the previously outputted in-vivo scene image, and then motion interpolation can be performed on the in-vivo scene image obtained by the optical flow interpolation based on the motion control position information of the surgical instrument 220, so as to obtain the target in-vivo scene image. When the abnormality category of the to-be-outputted in-vivo scene image is both image quality abnormality and surgical instrument 220 position abnormality and reflection, reflection repair can be performed first, and then optical flow interpolation can be performed on the in-vivo scene image obtained by the reflection repair based on the previously outputted in-vivo scene image, and finally motion interpolation can be performed on the in-vivo scene image obtained by the optical flow interpolation based on the motion control position information of the surgical instrument 220, so as to obtain the target in-vivo scene image.

[0152] In some exemplary embodiments, the image output method provided by the present application further comprises:

[0153] Tracking the surgical instrument 220 in the outputted in-vivo scene image.

[0154] Therefore, by tracking the surgical instrument 220 (specifically, the straight rod of the surgical instrument 220) in the outputted in-vivo scene image (the target in-vivo scene image obtained by repair, the to-be-outputted in-vivo scene image without abnormality, or the to-be-outputted in-vivo scene image without abnormality and after basic preprocessing), it can be indicated that the motion position of the surgical instrument 220 in the in-vivo scene image meets the expectation. At the same time, in order to avoid affecting the surgical operation, the prompt content should be simplified as much as possible.

[0155] In some exemplary embodiments, when tracking the surgical instrument 220 in the outputted in-vivo scene image, the image output method further comprises:

[0156] Adding an arm number mark to the end of the surgical instrument 220 in the outputted in-vivo scene image.

[0157] Thus, by adding the arm number mark at the end of the surgical instrument 220 in the output in-vivo scene image, not only can the doctor track the position of the surgical instrument 220 in real time, but also can the doctor clearly know which mechanical arm 230 each surgical instrument 220 is installed on, thereby more facilitating the surgeon to better observe the surgical process.

[0158] Specifically, please continue to refer to Figure 16 which is a surgical instrument 220 tracking prompt schematic diagram provided by an embodiment of the present application. The solid line rod in the figure represents the surgical instrument 220 in the output in-vivo scene image, and the dashed line rod in the figure represents the surgical instrument 220 drawn based on the motion control data. As Figure 16 shown, an arm number display function can be added at the end of the surgical instrument 220, and the position of the end of the surgical instrument 220 can be tracked in real time. In order not to affect the image display, the arm number mark is transparent enough when the surgical instrument 220 moves; when the surgical instrument 220 stops moving, the arm number mark can be clearly displayed. It should be noted that, as can be understood by those skilled in the art, the surgical instrument 220 drawn based on the motion control data can be displayed by turning on the instrument contour switch, so as to achieve the purpose of real-time prompting the motion trajectory of the surgical instrument 220.

[0159] In some exemplary embodiments, the image output method provided by the present application further comprises:

[0160] When a plurality of continuous frames of in-vivo scene images to be output are abnormal, a slow operation prompt information is sent.

[0161] Please refer to Figure 17 which is a slow operation prompt schematic diagram provided by an embodiment of the present application. As Figure 17 shown, when the image is abnormal, frame rate fluctuation may occur when performing the frame interpolation operation (optical flow interpolation or motion interpolation), so it is necessary to prompt the slow operation when the frame interpolation is frequent. At the same time, the influence of the prompt information on the doctor's operation needs to be considered, so the slow operation prompt information is sent only when a plurality of continuous frames are abnormal.

[0162] Based on the same inventive concept, the present application also provides an image output device, please refer to Figure 18 which is a block structure schematic diagram of the image output device provided by an embodiment of the present application. As Figure 18As shown, the image output device provided by the present invention includes an image acquisition module 610, an image detection module 620, an image control module 630, an image processing module 640 and an image output module 650, wherein the image acquisition module 610 is configured to obtain an image of an internal body scene to be output; the image detection module 620 is configured to perform reflection detection, image quality detection and surgical instrument 220 position detection on the image of the internal body scene to be output, and determine whether the image of the internal body scene to be output has an abnormality based on the detection results; the image control module 630 is configured to issue a corresponding abnormality repair control according to the abnormality category of the image of the internal body scene to be output. The image processing module 640 is configured to perform corresponding repairs on the in-vivo scene image to be output according to the abnormality repair control instructions issued by the image control module 630 to obtain a target in-vivo scene image, or perform basic preprocessing on the in-vivo scene image to be output without abnormalities according to the normal processing control instructions issued by the image control module 630; the image output module 650 is configured to output the target in-vivo scene image, the in-vivo scene image to be output without abnormalities, or the in-vivo scene image to be output without abnormalities after basic preprocessing. Thus, the image output device provided by the present invention performs reflection detection, image quality detection, and surgical instrument 220 position detection on the in-vivo scene image to be output, and when an abnormality is detected, performs corresponding repairs on the in-vivo scene image to be output according to the abnormality category. Thus, it can not only output a clearer and more complete image, ensure the fluency and coherence of the output image, but also accurately track the position of the surgical instrument 220, which is beneficial for the surgeon to better observe the surgical process, thereby helping to improve the accuracy and safety of the surgical operation.

[0163] Please continue to refer to Figure 18 ,like Figure 18 As shown, in some exemplary embodiments, the image detection module 620 and the image control module 630 are provided on the first FPGA chip 600, and the image processing module 640 and the image output module 650 are provided on the second FPGA chip 670. Thus, by providing the image detection module 620 and the image control module 630 on the first FPGA chip 600 and providing the image processing module 640 and the image output module 650 on the second FPGA chip 670, the image processing output speed can be greatly improved to achieve low-latency image output.

[0164] Further, in order to further improve the output speed of image processing, the detection module can be divided into three groups of hardware blocks, and the image processing module 640 can be divided into seven groups of hardware blocks. In this way, by dividing the detection module into three groups of hardware blocks, one group of image input can be realized, and three groups of parallel processing (i.e., three groups of detection of reflection light, image quality, and surgical instrument 220 position are parallel processed) are output. By dividing the image processing module 640 into seven groups of hardware blocks, the pre-distribution of the processing algorithms for the seven abnormal categories of reflection light, image quality abnormality, surgical instrument 220 position abnormality, image quality abnormality and reflection light, surgical instrument 220 position abnormality and image quality, and surgical instrument 220 position abnormality and reflection light can be performed.

[0165] In some exemplary embodiments, the image detection module 620 is configured to perform reflection light detection on the to-be-output in-vivo scene image by the following steps:

[0166] performing edge detection on the to-be-output in-vivo scene image to obtain a corresponding edge image;

[0167] for each edge pixel point in the edge image, obtaining a gradient estimation index of the edge pixel point according to the intensity of the edge pixel point;

[0168] for each edge pixel point in the edge image, using a SIFT-flow algorithm to calculate a motion vector of the edge pixel point in the edge image, and obtaining a motion estimation index of the edge pixel point according to the motion vector of the edge pixel point;

[0169] for each edge pixel point in the edge image, obtaining a comprehensive estimation index of the edge pixel point according to the gradient estimation index and the motion estimation index of the edge pixel point, and determining whether the edge pixel point is a reflection point according to the comprehensive estimation index of the edge pixel point.

[0170] In some exemplary embodiments, the image detection module 620 is configured to perform image quality detection on the to-be-output in-vivo scene image by the following steps:

[0171] inputting the to-be-output in-vivo scene image into a pre-trained quality scoring model to obtain a quality score result of the to-be-output in-vivo scene image;

[0172] determining whether the quality score of the to-be-output in-vivo scene image is less than a preset score threshold;

[0173] if yes, determining that the image quality of the to-be-output in-vivo scene image is abnormal.

[0174] In some demonstrative embodiments, the image detection module 620 is configured to perform surgical instrument 220 position detection of the to-be-output intra-body scene image by the following steps:

[0175] identifying the surgical instrument 220 in the to-be-output intra-body scene image, to obtain image tracking position information of the surgical instrument 220;

[0176] calculating a position deviation, according to the image tracking position information of the surgical instrument 220 and the motion control position information of the surgical instrument 220;

[0177] determining whether the position deviation is within a predetermined position deviation range;

[0178] if not, determining that the surgical instrument 220 position in the to-be-output intra-body scene image is abnormal.

[0179] In some demonstrative embodiments, the image processing module 640 is configured to perform corresponding repair of the to-be-output intra-body scene image, to obtain a target intra-body scene image, by the following steps:

[0180] if the abnormality category of the to-be-output intra-body scene image is reflection, repairing a reflection region in the to-be-output intra-body scene image, to obtain a target intra-body scene image;

[0181] if the abnormality category of the to-be-output intra-body scene image is image quality abnormality, repairing the to-be-output intra-body scene image according to the previously output intra-body scene image, to obtain a target intra-body scene image;

[0182] if the abnormality category of the to-be-output intra-body scene image is surgical instrument 220 position abnormality, repairing the to-be-output intra-body scene image according to the motion control position information of the surgical instrument 220, to obtain a target intra-body scene image.

[0183] In some demonstrative embodiments, the image processing module 640 is configured to reduce the brightness of the surgical instrument 220 reflection region in the to-be-output intra-body scene image, until the surgical instrument 220 is free of reflection and visible; and / or to perform pixel level filling of the cavity reflection region in the to-be-output intra-body scene image, using a priority pixel filling algorithm.

[0184] In some demonstrative embodiments, the image processing module 640 is configured to obtain intermediate optical flow information between the to-be-output intra-body scene image and the previously output intra-body scene image, using an optical flow method; and to fuse the intermediate optical flow information and the extracted relevant features of the to-be-output intra-body scene image, to obtain a target intra-body scene image.

[0185] In some exemplary embodiments, the image processing module 640 is configured to input the to-be-output in-vivo scene image and the motion control position information of the surgical instrument 220 into a pre-trained deep learning model to obtain a target in-vivo scene image.

[0186] Based on the same inventive concept, the present application further provides an electronic device, please refer to Figure 19 which is a block structure schematic diagram of an electronic device provided by an embodiment of the present application. As shown in Figure 19 the electronic device provided by the present application comprises a processor 710 and a memory 730, and the memory 730 stores a computer program, and the computer program is executed by the processor 710 to realize the image output method described above. Since the electronic device provided by the present application and the image output method provided by the present application belong to the same inventive concept, the electronic device provided by the present application at least has all the beneficial effects of the image output method provided by the present application. For details, please refer to the related content about the beneficial effects of the image output method provided by the present application in the foregoing description. Therefore, the beneficial effects of the electronic device provided by the present application will not be described here.

[0187] As shown in Figure 19 the electronic device provided by the present application further comprises a communication interface 720 and a communication bus 740, wherein the processor 710, the communication interface 720 and the memory 730 complete mutual communication through the communication bus 740. The communication bus 740 can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The communication bus 740 can be divided into an address bus, a data bus, a control bus, etc. For the convenience of representation, only one thick line is shown in the figure, but it does not mean that there is only one bus or only one type of bus. The communication interface 720 is used for communication between the electronic device and other devices.

[0188] The processor 710 referred to in the present application can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs) or other programmable logic devices, discrete gates or transistor logic components, discrete hardware components, etc. The general-purpose processor can be a microprocessor or can also be any conventional processor. The processor 710 is the control center of the electronic device and is connected to various parts of the electronic device through various interfaces and lines.

[0189] The memory 730 can be used to store the computer program, and the processor 710 realizes various functions of the electronic device by running or executing the computer program stored in the memory 730 and calling the data stored in the memory 730.

[0190] The memory 730 can include non-volatile and / or volatile memory. The non-volatile memory can include read-only memory (ROM), programmable memory (PROM), electrically programmable memory (EPROM), electrically erasable programmable memory (EEPROM) or flash memory. The volatile memory can include random access memory (RAM) or external cache memory. As an illustration but not limitation, random access memory is available in various forms, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous random access memory (SDRAM), double data rate synchronous random access memory (DDR SDRAM), enhanced synchronous random access memory (ESDRAM), synchronous link (Synchlink) dynamic random access memory (SLDRAM), memory bus (Rambus) direct random access memory (RDRAM), direct memory bus dynamic random access memory (DRDRAM), and memory bus dynamic random access memory (RDRAM), etc.

[0191] Based on the same inventive concept, the present application also provides a surgical robot comprising the image output device described above, or comprising the electronic device described above. Since the surgical robot provided by the present application comprises the image output device or the electronic device provided by the present application, the surgical robot provided by the present application also has at least all the beneficial effects of the image continuity method provided by the present application, and specific reference can be made to the related content about the beneficial effects of the image output method provided by the present application described above, so the beneficial effects of the surgical robot provided by the present application will not be described here.

[0192] In summary, compared with the prior art, the image output method, device, surgical robot and electronic device provided by the present application have the following beneficial effects:

[0193] The present application can output clearer and more complete images, ensure the fluency and continuity of the output images, and accurately track the position of the surgical instrument 220, which is conducive to the surgeon to better observe the surgical process, thereby helping to improve the accuracy and safety of the surgical operation.

[0194] It should be noted that the computer program code for performing the operations of the present application can be written in one or more programming languages or combinations of languages including an object oriented programming language such as Java, Smalltalk, C++ or conventional procedural programming languages such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider).

[0195] It should be noted that the apparatus and method disclosed in the embodiments herein can also be implemented in other ways. The apparatus embodiments described above are merely exemplary, and the schematic flow chart and block diagram of the flow chart and block diagram show the possible architectural, functional and operational scenarios of the apparatus, method and computer program product according to the embodiments herein. In this regard, each block in the flow chart or block diagram can represent a module, a program segment or a portion of code which comprises one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions noted in the blocks can occur out of the order noted in the flow chart. For example, two blocks shown in succession can in fact be executed substantially concurrently or the blocks can sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and / or flow chart illustrations, and combinations of blocks in the block diagrams and / or flow chart illustrations, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.

[0196] It should also be noted that the above description is merely illustrative of the application, and not in any way limiting the scope of the application, and any modifications, alterations, or equivalents of the application according to the above disclosure are intended to be included within the scope of the application. It will be apparent to those skilled in the art that various modifications and variations can be made to the present application without departing from the spirit or scope of the application. Thus, it is intended that the present application cover modifications and variations of this application provided they come within the scope of the application and their equivalent technology.

Claims

1. An image output method, characterized in that: include: Acquiring an in-vivo scene image to be output; performing reflection detection, image quality detection, and surgical instrument position detection on the in-vivo scene image to be output; Determining whether the in-vivo scene image to be output has an abnormality based on the detection result; If the judgment result is yes, then performing corresponding repair on the to-be-output in-vivo scene image according to the abnormal category of the to-be-output in-vivo scene image to obtain and output a target in-vivo scene image; If the judgment result is no, the to-be-outputted in-vivo scene image is directly output or the to-be-outputted in-vivo scene image is subjected to basic pre-processing before output.

2. The image output method according to claim 1, wherein: The performing reflection detection on the in-vivo scene image to be output includes: Performing edge detection on the in-vivo scene image to be output to obtain a corresponding edge image; For each edge pixel in the edge image, obtaining a gradient estimation index of the edge pixel according to the intensity of the edge pixel; For each edge pixel in the edge image, a SIFT-flow algorithm is used to calculate a motion vector of the edge pixel in the edge image, and a motion estimation index of the edge pixel is obtained based on the motion vector of the edge pixel; For each edge pixel point in the edge image, a comprehensive estimation index of the edge pixel point is obtained based on the gradient estimation index and the motion estimation index of the edge pixel point, and whether the edge pixel point is a reflective point is determined based on the comprehensive estimation index of the edge pixel point.

3. The image output method according to claim 1, wherein: The performing image quality detection on the in-vivo scene to be output includes: Inputting the to-be-output in-vivo scene image into a pre-trained quality scoring model to obtain a quality scoring result of the to-be-output in-vivo scene image; Determining whether the quality score of the to-be-outputted in-vivo scene image is less than a preset score threshold; If so, it is determined that the image quality of the in-vivo scene image to be output is abnormal.

4. The image output method according to claim 1, wherein: The detecting of the position of the surgical instrument on the in-vivo scene image to be output includes: Identifying a surgical instrument on the in-vivo scene image to be output to obtain image tracking position information of the surgical instrument; calculating a position deviation based on the image tracking position information of the surgical instrument and the motion control position information of the surgical instrument; Determining whether the position deviation is within a preset position deviation range; If not, it is determined that the position of the surgical instrument in the in-vivo scene image to be output is abnormal.

5. The image output method according to claim 1, wherein: The step of repairing the in-vivo scene image to be output according to the abnormality category of the in-vivo scene image to be output, so as to obtain a target in-vivo scene image, includes: If the abnormal category of the in-vivo scene image to be output is reflection, repairing the reflective area in the in-vivo scene image to be output to obtain a target in-vivo scene image; If the abnormal category of the in-vivo scene image to be output is abnormal image quality, repairing the in-vivo scene image to be output based on a previous frame of in-vivo scene image that has been output, to obtain a target in-vivo scene image; If the abnormal category of the in-vivo scene image to be output is abnormal position of the surgical instrument, the in-vivo scene image to be output is repaired according to the motion control position information of the surgical instrument to obtain a target in-vivo scene image.

6. The image output method according to claim 5, wherein: The repairing of the reflective area in the in-vivo scene image to be output includes: reducing the brightness of the reflective area of ​​the surgical instrument in the in-vivo scene image to be output until the surgical instrument is non-reflective and visible; and / or A priority pixel filling algorithm is used to perform pixel-level filling on the inner cavity reflective area in the in-vivo scene image to be output.

7. The image output method according to claim 5, wherein: The repairing of the to-be-outputted in-vivo scene image according to the previously outputted in-vivo scene image to obtain a target in-vivo scene image includes: Acquiring intermediate optical flow information between the to-be-output in-vivo scene image and a previously output in-vivo scene image using an optical flow method; The intermediate optical flow information and the extracted relevant features of the in-vivo scene image to be output are fused to obtain a target in-vivo scene image.

8. The image output method according to claim 5, wherein: The repairing of the to-be-outputted in-vivo scene image according to the motion control position information of the surgical instrument to obtain a target in-vivo scene image includes: The in-vivo scene image to be output and the motion control position information of the surgical instrument are input into a pre-trained deep learning model to obtain a target in-vivo scene image.

9. An image output device, characterized in that: include: An image acquisition module configured to acquire an in-vivo scene image to be output; an image detection module configured to perform reflection detection, image quality detection, and surgical instrument position detection on the to-be-output internal body scene image, and determine whether the to-be-output internal body scene image has any abnormality based on the detection results; an image control module configured to issue a corresponding abnormality repair control instruction according to the abnormality category of the in-vivo scene image to be output, or to issue a normal processing control instruction when there is no abnormality in the in-vivo scene image to be output; an image processing module configured to perform corresponding repair on the to-be-output in-vivo scene image according to the abnormal repair control instruction issued by the image control module to obtain a target in-vivo scene image, or to perform basic preprocessing on the to-be-output in-vivo scene image without abnormality according to the normal processing control instruction issued by the image control module; as well as The image output module is configured to output the target in-vivo scene image, the in-vivo scene image to be output without abnormality, or the in-vivo scene image to be output without abnormality after basic preprocessing.

10. The image output device according to claim 9, wherein: The image detection module and the image control module are arranged on a first FPGA chip, and the image processing module and the image output module are arranged on a second FPGA chip.

11. An electronic device, characterized in that: The apparatus comprises a processor and a memory, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, the image output method according to any one of claims 1 to 8 is implemented.

12. A surgical robot, characterized in that: The surgical robot includes the image output device according to claim 9 or 10, or the surgical robot includes the electronic device according to claim 11.

Images