3D laparoscopic surgery data processing method, device and equipment and storage medium

Abstract

The invention relates to a 3D laparoscopic surgery data processing method and device, equipment and a storage medium. The method comprises the following steps: framing an original operation data video stream by adopting a video processing technology to obtain laparoscope image sequences corresponding to a left camera and a right camera of a binocular laparoscope system; calibrating the laparoscope image sequence based on camera parameters to obtain a laparoscope image sequence without image distortion; and performing smoke removal and surgical instrument segmentation on the calibrated laparoscope image sequence by respectively adopting a smoke removal technology based on a dark channel prior algorithm and a surgical instrument segmentation algorithm based on a DeepLabv3 + architecture to generate a laparoscope image sequence without smoke interference and instrument shielding. According to the method, the laparoscopic surgery image is preprocessed by combining the dark channel prior algorithm and the DeepLabv3 + architecture, the problems of smoke interference and instrument shielding in the laparoscopic image can be solved at the same time, and the image quality and the performance of subsequent tasks are improved.

Description

Technical Field

[0001] This application belongs to the technical field of medical image processing, and particularly relates to a 3D laparoscopic surgery data processing method, device, equipment, and storage medium. Background Art

[0002] Laparoscopic surgery is a minimally invasive surgical technique. By making several small incisions in the patient's abdomen and inserting a laparoscope with a miniature camera and related surgical instruments, the images inside the abdominal cavity are transmitted to a monitor, and the doctor performs surgical operations by observing the images on the screen. This surgical method has the advantages of small trauma, fast recovery, mild postoperative pain, and short hospital stay. In recent years, with the application of 3D imaging technology, 3D laparoscopic surgery has further improved the accuracy and efficiency of surgery. Compared with traditional laparoscopic surgery, 3D laparoscopic surgery uses a dual-camera system to capture stereoscopic images, providing depth perception and spatial positioning advantages, greatly improving the surgical accuracy and safety.

[0003] Generally, during 3D laparoscopic surgery, a large amount of smoke is generated due to surgical operations such as electrocautery and laser ablation. This smoke will seriously reduce the quality of the endoscopic images, resulting in a blurred surgical field, which not only directly affects the doctor's visual judgment, increases the surgical risk, but also has a negative impact on subsequent image processing tasks. At the same time, the surgical instruments will frequently block the surgical area during operation, which not only affects the doctor's observation of key anatomical structures, but also interferes with the processing of surgical data. A large amount of image, video, and sensor data are generated during the surgical process, and the formats and sources of these data are different, and effective data preprocessing is required to integrate and manage the surgical data. Currently, the existing surgical data preprocessing methods mainly include the following aspects:

[0004] (1) Simple image calibration and normalization; most existing technologies only perform simple processing such as scaling and normalization on images. Although they can initially adjust the image format, they lack an effective processing mechanism for interference factors such as smoke and occlusion in complex surgical scenarios.

[0005] (2) Surgical instrument detection and segmentation; some studies use deep learning technology for the detection and segmentation of surgical instruments. The existing methods mainly focus on the instrument detection itself and lack the processing of interference factors such as smoke.

[0006] (3) Smoke removal technology; some studies try to remove smoke through image enhancement algorithms, but most of these methods are independent of instrument mask generation and lack comprehensive consideration of the combination of the two.

[0007] (4) Multimodal data fusion; in recent years, multimodal learning techniques have been introduced into surgical data analysis. By fusing multimodal data such as images and texts, the understanding of complex surgical scenarios by the model is improved. However, existing methods still rely on simple data calibration in the preprocessing stage and fail to fully explore the inherent complexity of surgical data. Summary of the Invention

[0008] The present application provides a method, device, equipment and storage medium for processing 3D laparoscopic surgical data, aiming to at least solve one of the above technical problems in the prior art to a certain extent.

[0009] To solve the above problems, the present application provides the following technical solutions:

[0010] A method for processing 3D laparoscopic surgical data, comprising:

[0011] Using video processing technology to frame the original surgical data video stream to obtain a laparoscopic image sequence corresponding to the left and right cameras of the binocular laparoscopic system;

[0012] Calibrating the laparoscopic image sequence based on camera parameters to obtain a laparoscopic image sequence with image distortion eliminated;

[0013] Using a smoke removal technology based on the dark channel prior algorithm and a surgical instrument segmentation algorithm based on the DeepLabv3+ architecture to perform smoke removal and surgical instrument segmentation on the calibrated laparoscopic image sequence respectively, and generating a laparoscopic image sequence without smoke interference and instrument occlusion.

[0014] The technical solution adopted in the embodiment of the present application further includes: the step of using video processing technology to frame the original surgical data video stream to obtain a laparoscopic image sequence corresponding to the left and right cameras of the binocular laparoscopic system is specifically:

[0015] Obtaining each frame image in the original surgical data video stream, and respectively dividing each frame image into left and right parts along the horizontal direction, corresponding to the laparoscopic images of the left and right cameras of the binocular laparoscopic system.

[0016] The technical solution adopted in the embodiment of the present application further includes: the step of calibrating the laparoscopic image sequence based on camera parameters to obtain a laparoscopic image sequence with image distortion eliminated is specifically:

[0017] Using the camera to capture N checkerboard images;

[0018] Detect the corner points of the checkerboard image, and use the calibration function of OpenCV to calculate the internal parameter matrix and distortion coefficients of the camera according to the 3D world coordinates and 2D image coordinates of the checkerboard; wherein, the internal parameter matrix of the camera includes the focal length or the position of the principal point, and the distortion coefficients include barrel or pincushion distortion;

[0019] Use the calculated internal parameter matrix and distortion coefficients to perform undistortion processing on the laparoscopic image sequence after frame division to restore the true geometric shape of the laparoscopic image sequence.

[0020] The technical solution adopted in the embodiment of this application further includes: respectively adopting a smoke removal technique based on the dark channel prior algorithm and a surgical instrument segmentation algorithm based on the DeepLabv3+ architecture to perform smoke removal and surgical instrument segmentation on the calibrated laparoscopic image sequence, including:

[0021] Calculate the minimum value of each pixel point from the RGB channels of each frame of laparoscopic image to form a dark channel image; assuming that the dark channel image represents the minimum light intensity under fog-free conditions, inversely calculate the transmittance map caused by atmospheric scattering according to the relationship between the dark channel image and the calibrated laparoscopic image, where the transmittance calculation formula is:

[0022]

[0023] where, t represents the transmittance map, I represents the calibrated laparoscopic image, A represents the atmospheric light intensity, and ω is a regulation parameter used to control the defogging intensity;

[0024] Globally analyze the dark channel image, select the brightest pixels with a set proportion in the dark channel image as the atmospheric light intensity, and combine the transmittance map and the atmospheric light intensity to calculate the defogged laparoscopic image using the image restoration formula as:

[0025]

[0026] where, J represents the defogged laparoscopic image.

[0027] The technical solution adopted in the embodiment of this application further includes: respectively adopting a smoke removal technique based on the dark channel prior algorithm and a surgical instrument segmentation algorithm based on the DeepLabv3+ architecture to perform smoke removal and surgical instrument segmentation on the calibrated laparoscopic image sequence, and further includes:

[0028] Input the calibrated laparoscopic image into the DeepLabv3+ model. The DeepLabv3+ model includes an encoder and a decoder. The encoder is based on an improved ResNet architecture and is used to extract the high-level semantic feature map of the laparoscopic image through dilated convolution and multi-scale feature fusion. The decoder is used to perform pixel-wise upsampling on the feature map output by the encoder, restore the low-resolution feature map to the same size as the input image, and output a segmentation probability map. The segmentation probability map is a multi-channel image, and each channel corresponds to the probability distribution of a category. Perform pixel-wise analysis on the segmentation probability map, map the category probability of each pixel to a specific category label, and generate the final surgical instrument segmentation mask.

[0029] Another technical solution adopted in the embodiments of this application further includes: The pixel-wise analysis of the segmentation probability map to map the category probability of each pixel to a specific category label and generate the final surgical instrument segmentation mask is specifically:

[0030] The surgical instrument segmentation mask is represented in the form of a binary image, where the surgical instrument area is marked as the foreground and the background area is marked as the background.

[0031] Another technical solution of the embodiments of this application is: A 3D laparoscopic surgery data processing device includes:

[0032] An image frame division module: used to divide the original surgical data video stream into frames by using video processing technology to obtain the laparoscopic image sequences corresponding to the left and right cameras of the binocular laparoscopic system.

[0033] An image calibration module: used to calibrate the laparoscopic image sequences based on the camera parameters to obtain the laparoscopic image sequences with image distortion removed.

[0034] An image preprocessing module: used to perform smoke removal and surgical instrument segmentation on the calibrated laparoscopic image sequences by using the smoke removal technology based on the dark channel prior algorithm and the surgical instrument segmentation algorithm based on the DeepLabv3+ architecture respectively, and generate the laparoscopic image sequences without smoke interference and instrument occlusion.

[0035] Another technical solution adopted in the embodiments of this application further includes: The image preprocessing module performs smoke removal and surgical instrument segmentation on the calibrated laparoscopic image sequences by using the smoke removal technology based on the dark channel prior algorithm and the surgical instrument segmentation algorithm based on the DeepLabv3+ architecture respectively, specifically:

[0036] Calculate the minimum value of each pixel point from the RGB channels of each frame of laparoscopic image to form a dark channel image; assuming that the dark channel image represents the minimum light intensity under fog-free conditions, inversely calculate the transmittance map caused by atmospheric scattering according to the relationship between the dark channel image and the calibrated laparoscopic image, where the transmittance calculation formula is:

[0037]

[0038] where \(t\) represents the transmittance map, \(I\) represents the calibrated laparoscopic image, \(A\) represents the atmospheric light intensity, and \(\omega\) is a tuning parameter used to control the defogging intensity;

[0039] Globally analyze the dark channel image, select the brightest pixels with a set proportion in the dark channel image as the atmospheric light intensity, combine the transmittance map and the atmospheric light intensity, and apply the image restoration formula to calculate the defogged laparoscopic image as:

[0040]

[0041] where \(J\) represents the defogged laparoscopic image;

[0042] Input the calibrated laparoscopic image into the DeepLabv3+ model. The DeepLabv3+ model includes an encoder and a decoder. The encoder is based on an improved ResNet architecture and is used to extract the high-level semantic feature map of the laparoscopic image through dilated convolution and multi-scale feature fusion; the decoder is used to perform a per-pixel upsampling operation on the feature map output by the encoder, restore the low-resolution feature map to the same size as the input image, and output a segmentation probability map; where the segmentation probability map is a multi-channel image, and each channel corresponds to the probability distribution of a category; perform a per-pixel analysis on the segmentation probability map, map the category probability of each pixel to a specific category label, and generate the final surgical instrument segmentation mask.

[0043] Another technical solution adopted in the embodiments of the present application is: a device, the device includes a processor and a memory coupled to the processor, where,

[0044] The memory stores program instructions for implementing the 3D laparoscopic surgery data processing method;

[0045] The processor is used to execute the program instructions stored in the memory to control the 3D laparoscopic surgery data processing method.

[0046] Another technical solution adopted in the embodiments of the present application is: a storage medium, storing program instructions executable by a processor, and the program instructions are used to execute the 3D laparoscopic surgery data processing method.

[0047] Compared with the prior art, the beneficial effects produced by the embodiments of the present application are as follows: The 3D laparoscopic surgery data processing method, device, equipment, and storage medium of the embodiments of the present application preprocess laparoscopic surgery images by combining the dark channel prior algorithm and the DeepLabv3+ architecture. The dark channel prior algorithm is used to quickly remove smoke, and the DeepLabv3+ architecture is used to ensure high-precision segmentation of surgical instrument masks, providing reliable support for real-time decision-making during the surgery. It can simultaneously solve the problems of smoke interference and instrument occlusion in laparoscopic images, achieving a balance between real-time performance and high precision, significantly improving the image quality and the performance of subsequent tasks. The preprocessed image data can be directly applied to surgical navigation and instrument detection tasks, improving the accuracy and safety of the surgery, and without additional adaptation, having good scalability and broad application prospects. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 is a flowchart of the 3D laparoscopic surgery data processing method according to the embodiments of the present application;

[0049] Figure 2 is a schematic diagram of the smoke removal effect according to the embodiments of the present application;

[0050] Figure 3 is a schematic diagram of the surgical instrument segmentation effect according to the embodiments of the present application;

[0051] Figure 4 is a schematic structural diagram of the 3D laparoscopic surgery data processing device according to the embodiments of the present application;

[0052] Figure 5 is a schematic structural diagram of the equipment according to the embodiments of the present application;

[0053] Figure 6 is a schematic structural diagram of the storage medium according to the embodiments of the present application. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0054] Next, the technical solutions in the embodiments of the present application will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments of the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present application.

[0055] The terms "first", "second", and "third" in this application are only used for descriptive purposes and should not be construed as indicating or implying relative importance or implicitly specifying the quantity of the indicated technical features. Thus, features defined with "first", "second", and "third" may explicitly or implicitly include at least one of such features. In the description of this application, the meaning of "a plurality" is at least two, such as two, three, etc., unless otherwise specifically and clearly defined. In the embodiments of this application, all directional indications (such as up, down, left, right, front, back...) are only used to explain the relative positional relationship and movement conditions between components in a specific posture (as shown in the drawings). If the specific posture changes, the directional indications will also change accordingly. In addition, the terms "comprising" and "having" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units is not limited to the listed steps or units, but optionally further includes steps or units not listed, or optionally further includes other steps or units inherent to these processes, methods, products, or devices.

[0056] Referring to "embodiments" herein means that the specific features, structures, or characteristics described in connection with the embodiments can be included in at least one embodiment of this application. The phrase appears in various places in the specification and does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment mutually exclusive with other embodiments. Those skilled in the art will explicitly and implicitly understand that the embodiments described herein can be combined with other embodiments.

[0057] Specifically, please refer to Figure 1 , which is a flowchart of the 3D laparoscopic surgery data processing method according to the embodiments of this application. The 3D laparoscopic surgery data processing method according to the embodiments of this application includes the following steps:

[0058] S100: Obtain the original surgical data video stream from the binocular laparoscopic system;

[0059] In this step, the binocular laparoscopic system includes two cameras, left and right, which are respectively used to capture laparoscopic images of the surgical area during the operation and generate the original surgical data video stream of the 3D laparoscopic surgery.

[0060] S110: Frame the original surgical data video stream using video processing technology to separate the laparoscopic image sequences corresponding to the left and right cameras respectively;

[0061] In this step, since the original surgical data video stream is captured by two cameras, namely the left and right cameras, during the 3D laparoscopic surgical data processing, it is first necessary to frame the original surgical data video stream to separate the laparoscopic image sequences corresponding to the left and right cameras. The specific framing method is as follows: Obtain each frame of the image in the original surgical data video stream, and divide each frame of the image into two parts horizontally, corresponding to the laparoscopic images of the left and right cameras respectively.

[0062] S120: Calibrate the framed laparoscopic image sequences based on the camera parameters to obtain a laparoscopic image sequence with eliminated image distortion;

[0063] In this step, the calibration process is based on the camera parameters, and a checkerboard is used as a calibration tool to calibrate the framed laparoscopic image sequences. The specific calibration process includes:

[0064] S121: Use the camera to capture N checkerboard images, where the number of captured checkerboard images N can be set according to the actual application scenario;

[0065] S122: Detect the corner points of the checkerboard images, and use the calibration function of OpenCV to calculate the internal parameter matrix and distortion coefficients of the camera according to the 3D world coordinates of the checkerboard and the detected 2D image coordinates, and optimize the camera parameters to minimize the reprojection error; among them, the internal parameter matrix of the camera includes the focal length or the position of the principal point, etc., and the distortion coefficients include barrel or pillow distortion, etc.;

[0066] S123: Use the calculated internal parameter matrix and distortion coefficients to perform distortion removal processing on the framed laparoscopic image sequences to restore the true geometric shape of the laparoscopic image sequences.

[0067] It can be understood that through the above framing and calibration processing, the laparoscopic image sequences have high quality and consistency, providing a reliable basis for surgical navigation and instrument detection tasks.

[0068] S130: Adopt the smoke removal technology based on the dark channel prior algorithm and the surgical instrument segmentation algorithm based on the DeepLabv3+ architecture to perform smoke removal and surgical instrument segmentation on the calibrated laparoscopic image sequences respectively, generating a laparoscopic image sequence without smoke interference and instrument occlusion;

[0069] In this step, by combining the dark channel prior algorithm and the DeepLabv3+ architecture, first, the smoke removal technology based on the dark channel prior algorithm is used to analyze the dark channel characteristics of laparoscopic images, quickly removing the interference of smoke on laparoscopic images to restore a clear surgical field of view; subsequently, the DeepLabv3+ architecture is used to perform surgical instrument mask segmentation on laparoscopic images, which can simultaneously solve the problems of smoke interference and instrument occlusion in laparoscopic images, achieving a balance between real-time performance and high precision, significantly improving the image quality and the performance of subsequent tasks, and providing reliable support for real-time decision-making during the surgical process. Specifically, as Figure 2 and Figure 3 shown Figure 2 is a schematic diagram of the smoke removal effect of an embodiment of this application, Figure 3 and

[0070] is a schematic diagram of the surgical instrument segmentation effect of an embodiment of this application. Further, for image data with smoke, this application uses the smoke removal technology based on the dark channel prior algorithm to remove smoke. The specific smoke removal process includes: First, the minimum value of each pixel is calculated from the RGB channels of each frame of laparoscopic image to form a dark channel image. The dark channel prior theory holds that in a local area of a fog-free image, at least one color channel's pixel value approaches zero. Through morphological erosion operations, the details of the dark channel are further enhanced, providing a basis for subsequent transmittance estimation. Assuming that the dark channel image represents the minimum light intensity under fog-free conditions, the transmittance map caused by atmospheric scattering is inversely calculated based on the relationship between the dark channel image and the original laparoscopic image (i.e., the calibrated laparoscopic image). The transmittance calculation formula is:

[0071]

[0072] where t represents the transmittance map, I represents the original laparoscopic image, A represents the atmospheric light intensity, and ω is a tuning parameter used to control the defogging intensity.

[0073] By globally analyzing the dark channel image, a set proportion of the brightest pixels in the dark channel image is selected as the atmospheric light intensity. The atmospheric light is the part where the light is uniformly weakened during the atomization process, and its value is usually close to the RGB values of white light. Among them, the set proportion of the brightest pixels in the embodiment of this application is 0.1%, and it can be specifically set according to the actual application scenario.

[0074] Combining the transmittance map and the atmospheric light intensity, the defogged laparoscopic image is calculated using the image restoration formula as:

[0075]

[0076] where J represents the defogged laparoscopic image.

[0077] Through the above operations, the smoke interference in laparoscopic images can be effectively removed, the clear surgical field of view can be restored, and high-quality image support can be provided for subsequent surgical navigation and instrument detection tasks.

[0078] Furthermore, for laparoscopic images with instrument occlusion, the present application uses a surgical instrument segmentation algorithm based on the DeepLabv3+ architecture for surgical instrument segmentation. The specific segmentation process includes: First, the calibrated laparoscopic image is input into the DeepLabv3+ model. The DeepLabv3+ model includes an encoder and a decoder. The encoder is based on an improved ResNet architecture and is used to extract the high-level semantic feature map of the laparoscopic image through dilated convolution and multi-scale feature fusion, thereby capturing the key information of the surgical instrument while retaining sufficient context information to support accurate segmentation. Subsequently, through the decoder part, pixel-by-pixel upsampling operation is performed on the feature map output by the encoder, restoring the low-resolution feature map to the same size as the input image, and outputting a segmentation probability map. The segmentation probability map is a multi-channel image, and each channel corresponds to the probability distribution of a category. In the post-processing stage, pixel-by-pixel analysis is performed on the segmentation probability map, and the category probability of each pixel is mapped to a specific category label through thresholding or maximum value index operation, ensuring that each pixel is assigned to the most likely category, thereby generating the final surgical instrument segmentation mask.

[0079] Specifically, the finally generated surgical instrument segmentation mask is represented in the form of a binary image, where the surgical instrument area is marked as the foreground (such as value 1), and the background area is marked as the background (such as value 0). Directly applying the generated surgical instrument segmentation mask to surgical navigation or instrument detection tasks can greatly improve the accuracy and safety of the surgery.

[0080] Based on the above, the 3D laparoscopic surgery data processing method of the present application embodiment preprocesses laparoscopic surgery images by combining the dark channel prior algorithm and the DeepLabv3+ architecture, uses the dark channel prior algorithm to quickly remove smoke, and uses the DeepLabv3+ architecture to ensure high-precision surgical instrument mask segmentation, providing reliable support for real-time decision-making during the surgery process. It can simultaneously solve the problems of smoke interference and instrument occlusion in laparoscopic images, achieving a balance between real-time performance and high precision, significantly improving the image quality and the performance of subsequent tasks. The preprocessed image data can be directly applied to surgical navigation and instrument detection tasks, improving the accuracy and safety of the surgery, and there is no need for additional adaptation, having good scalability and broad application prospects.

[0081] Please refer to Figure 4 , which is a schematic structural diagram of the 3D laparoscopic surgery data processing device according to the embodiment of the present application. The 3D laparoscopic surgery data processing method device 40 of the present application embodiment includes:

[0082] Image Framing Module 41: It is used to frame the original surgical data video stream by using video processing technology to obtain a sequence of laparoscopic images corresponding to the left and right cameras of the binocular laparoscopic system;

[0083] Image Calibration Module 42: It is used to calibrate the sequence of laparoscopic images based on camera parameters to obtain a sequence of laparoscopic images with image distortion removed;

[0084] Image Preprocessing Module 43: It is used to remove smoke and segment surgical instruments from the calibrated sequence of laparoscopic images by using smoke removal technology based on the dark channel prior algorithm and surgical instrument segmentation algorithm based on the DeepLabv3+ architecture respectively, and generate a sequence of laparoscopic images without smoke interference and instrument occlusion.

[0085] It should be noted that for the information interaction, execution process, etc. between the above-mentioned devices / units, since they are based on the same concept as the method embodiments of the present application, their specific functions and the technical effects brought can be specifically referred to the method embodiment part, and will not be elaborated here.

[0086] The device provided by the embodiments of the present application can be applied in the foregoing method embodiments. For details, please refer to the description of the above method embodiments, and will not be elaborated here.

[0087] Please refer to Figure 5 , which is a schematic structural diagram of the device of the embodiments of the present application. The device 50 includes:

[0088] A memory 51 storing executable program instructions;

[0089] A processor 52 connected to the memory 51;

[0090] The processor 52 is used to call the executable program instructions stored in the memory 51 and execute the following steps: frame the original surgical data video stream by using video processing technology to obtain a sequence of laparoscopic images corresponding to the left and right cameras of the binocular laparoscopic system; calibrate the sequence of laparoscopic images based on camera parameters to obtain a sequence of laparoscopic images with image distortion removed; respectively use smoke removal technology based on the dark channel prior algorithm and surgical instrument segmentation algorithm based on the DeepLabv3+ architecture to remove smoke and segment surgical instruments from the calibrated sequence of laparoscopic images, and generate a sequence of laparoscopic images without smoke interference and instrument occlusion.

[0091] Among them, the processor 52 can also be referred to as a CPU (Central Processing Unit). The processor 52 may be an integrated circuit chip with signal processing capabilities. The processor 52 can also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc.

[0092] Please refer to Figure 6 , which is a schematic structural diagram of the storage medium of the embodiment of the present application. The storage medium of the embodiment of the present application stores program instructions 61 that can implement the following steps: frame the original surgical data video stream using video processing technology to obtain a laparoscopic image sequence corresponding to the left and right cameras of the binocular laparoscopic system; calibrate the laparoscopic image sequence based on the camera parameters to obtain a laparoscopic image sequence with image distortion removed; respectively use the smoke removal technology based on the dark channel prior algorithm and the surgical instrument segmentation algorithm based on the DeepLabv3+ architecture to perform smoke removal and surgical instrument segmentation on the calibrated laparoscopic image sequence to generate a laparoscopic image sequence without smoke interference and instrument occlusion. Among them, the program instructions 61 can be stored in the above storage medium in the form of a software product, including several instructions to enable a device (which can be a personal computer, a server, or a network device, etc.) or a processor to execute all or part of the steps of the methods in various embodiments of the present application. The aforementioned storage medium includes: various media that can store program instructions such as USB flash drives, mobile hard disks, read-only memories (ROM, Read-Only Memory), random access memories (RAM, Random Access Memory), magnetic disks, or optical discs, or terminal devices such as computers, servers, mobile phones, and tablets. Among them, the server can be an independent server or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (Content Delivery Network, CDN), and big data and artificial intelligence platforms.

[0093] In several embodiments provided by the present application, it should be understood that the disclosed systems, devices, and methods can be implemented in other ways. For example, the system embodiments described above are merely illustrative. For example, the division of units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed couplings or direct couplings or communication connections to each other can be indirect couplings or communication connections through some interfaces, devices, or units, and can be in electrical, mechanical, or other forms.

[0094] In addition, each functional unit in various embodiments of the present application can be integrated in a processing unit, or each unit can exist physically alone, or two or more units can be integrated in one unit. The above-mentioned integrated units can be implemented in the form of hardware or in the form of software functional units. The above is only the implementation mode of the present application and does not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made by using the content of the specification and drawings of the present application, or directly or indirectly applied in other related technical fields, shall be equally included in the patent protection scope of the present application.

Claims

1. A 3D laparoscopic surgery data processing method, characterized in that, Including: Using video processing technology to frame the original surgical data video stream, obtaining laparoscopic image sequences corresponding to the left and right cameras of the binocular laparoscopic system; Calibrating the laparoscopic image sequences based on camera parameters to obtain laparoscopic image sequences with image distortion eliminated; Respectively using a smoke removal technology based on the dark channel prior algorithm and a surgical instrument segmentation algorithm based on the DeepLabv3+ architecture to perform smoke removal and surgical instrument segmentation on the calibrated laparoscopic image sequences, generating laparoscopic image sequences without smoke interference and instrument occlusion.

2. The 3D laparoscopic surgery data processing method according to claim 1, wherein The step of using video processing technology to frame the original surgical data video stream and obtaining laparoscopic image sequences corresponding to the left and right cameras of the binocular laparoscopic system is specifically: Obtaining each frame of image in the original surgical data video stream, and respectively dividing each frame of image into left and right parts along the horizontal direction, corresponding to the laparoscopic images of the left and right cameras of the binocular laparoscopic system.

3. The 3D laparoscopic surgery data processing method according to claim 2, wherein The step of calibrating the laparoscopic image sequences based on camera parameters to obtain laparoscopic image sequences with image distortion eliminated is specifically: Using the camera to capture N checkerboard images; Detecting the corner points of the checkerboard images, and using the calibration function of OpenCV to calculate the internal parameter matrix and distortion coefficients of the camera according to the 3D world coordinates and 2D image coordinates of the checkerboard; wherein, the internal parameter matrix of the camera includes the focal length or the position of the principal point, and the distortion coefficients include barrel or pillow distortion; Using the calculated internal parameter matrix and distortion coefficients to perform distortion removal processing on the framed laparoscopic image sequences to restore the true geometric shape of the laparoscopic image sequences.

4. The 3D laparoscopic surgery data processing method according to any one of claims 1 to 3, characterized in that The step of respectively using a smoke removal technology based on the dark channel prior algorithm and a surgical instrument segmentation algorithm based on the DeepLabv3+ architecture to perform smoke removal and surgical instrument segmentation on the calibrated laparoscopic image sequences includes: Calculating the minimum value of each pixel point from the RGB channels of each frame of laparoscopic image to form a dark channel image; assuming that the dark channel image represents the minimum light intensity under fog-free conditions, and inversely calculating the transmittance map caused by atmospheric scattering according to the relationship between the dark channel image and the calibrated laparoscopic image, where the transmittance calculation formula is: where t represents the transmittance map, I represents the calibrated laparoscopic image, A represents the atmospheric light intensity, and ω is a regulation parameter used to control the defogging intensity; Globally analyzing the dark channel image, selecting the brightest pixels with a set proportion in the dark channel image as the atmospheric light intensity, and combining the transmittance map and the atmospheric light intensity, and applying the image restoration formula to calculate the defogged laparoscopic image as: where J represents the defogged laparoscopic image.

5. The 3D laparoscopic surgery data processing method according to claim 1, wherein The step of respectively using a smoke removal technology based on the dark channel prior algorithm and a surgical instrument segmentation algorithm based on the DeepLabv3+ architecture to perform smoke removal and surgical instrument segmentation on the calibrated laparoscopic image sequences further includes: Input the calibrated laparoscopic image into the DeepLabv3+ model. The DeepLabv3+ model includes an encoder and a decoder. The encoder is based on an improved ResNet architecture and is used to extract the high-level semantic feature map of the laparoscopic image through dilated convolution and multi-scale feature fusion. The decoder is used to perform pixel-wise upsampling on the feature map output by the encoder, restore the low-resolution feature map to the same size as the input image, and output a segmentation probability map. The segmentation probability map is a multi-channel image, and each channel corresponds to the probability distribution of a category. Perform pixel-wise analysis on the segmentation probability map, map the category probability of each pixel to a specific category label, and generate the final surgical instrument segmentation mask.

6. The 3D laparoscopic surgery data processing method according to claim 5, wherein The pixel-wise analysis of the segmentation probability map, mapping the category probability of each pixel to a specific category label, and generating the final surgical instrument segmentation mask is specifically as follows: The surgical instrument segmentation mask is represented in the form of a binary image, where the surgical instrument area is marked as the foreground and the background area is marked as the background.

7. A 3D laparoscopic surgery data processing device, characterized in that, It includes: Image frame division module: used to divide the original surgical data video stream into frames by using video processing technology to obtain the laparoscopic image sequences corresponding to the left and right cameras of the binocular laparoscopic system. Image calibration module: used to calibrate the laparoscopic image sequences based on the camera parameters to obtain the laparoscopic image sequences with image distortion removed. Image preprocessing module: used to perform smoke removal and surgical instrument segmentation on the calibrated laparoscopic image sequences by using the smoke removal technology based on the dark channel prior algorithm and the surgical instrument segmentation algorithm based on the DeepLabv3+ architecture respectively, and generate the laparoscopic image sequences without smoke interference and instrument occlusion.

8. The 3D laparoscopic surgery data processing device according to claim 7, wherein, The image preprocessing module performs smoke removal and surgical instrument segmentation on the calibrated laparoscopic image sequences by using the smoke removal technology based on the dark channel prior algorithm and the surgical instrument segmentation algorithm based on the DeepLabv3+ architecture respectively, specifically as follows: Calculate the minimum value of each pixel point from the RGB channels of each frame of laparoscopic image to form a dark channel image. Assume that the dark channel image represents the minimum light intensity under the fog-free condition, and inversely calculate the transmittance map caused by atmospheric scattering according to the relationship between the dark channel image and the calibrated laparoscopic image. The transmittance calculation formula is: where t represents the transmittance map, I represents the calibrated laparoscopic image, A represents the atmospheric light intensity, and ω is a regulation parameter used to control the defogging intensity. Globally analyze the dark channel image, select the brightest pixels with a set proportion in the dark channel image as the atmospheric light intensity, and combine the transmittance map and the atmospheric light intensity to calculate the defogged laparoscopic image by applying the image restoration formula as: where J represents the defogged laparoscopic image. Input the calibrated laparoscopic image into the DeepLabv3+ model. The DeepLabv3+ model includes an encoder and a decoder. The encoder is based on an improved ResNet architecture and is used to extract the high-level semantic feature map of the laparoscopic image through dilated convolution and multi-scale feature fusion. The decoder is used to perform a per-pixel upsampling operation on the feature map output by the encoder, restore the low-resolution feature map to the same size as the input image, and output a segmentation probability map. The segmentation probability map is a multi-channel image, and each channel corresponds to the probability distribution of a category. Perform per-pixel analysis on the segmentation probability map, map the category probability of each pixel to a specific category label, and generate the final surgical instrument segmentation mask.

9. A device, characterized in that, The device includes a processor and a memory coupled to the processor, where the memory stores program instructions for implementing the 3D laparoscopic surgery data processing method according to any one of claims 1-6; the processor is used to execute the program instructions stored in the memory to control the 3D laparoscopic surgery data processing method.

10. A storage medium, characterized in that, Stores program instructions that can be run by the processor, and the program instructions are used to execute the 3D laparoscopic surgery data processing method according to any one of claims 1 to 6.

Images