Laparoscopic imaging method, device and system

Abstract

The invention provides a laparoscope imaging method, device and system, and the method comprises the steps: selectively obtaining the image data of various types and various positions through designing an imaging collection device with various types and different positions, and generating a high-precision three-dimensional imaging picture through integrating the obtained image data. Therefore, the imaging accuracy of the abdominal cavity image is effectively improved, and more accurate control is realized. And further, the laparoscope imaging acquisition device is controlled to move to a position where the high-precision three-dimensional imaging data is not generated, and the high-precision three-dimensional imaging data is generated based on data acquired by various types of cameras, so that image acquisition in all directions of the abdominal cavity is completed, and therefore, blind areas are eliminated, and full coverage of abdominal cavity image acquisition is realized.

Description

Technical Field

[0001] The present invention relates to laparoscopic imaging technology, and more specifically, to a laparoscopic imaging method, device and system. Background Art

[0002] Traditional laparoscopes provide two-dimensional images, lack depth information, and are insufficient in displaying the spatial relationship of the internal abdominal structure. They lack high-resolution grayscale cameras and related processing technologies, resulting in low-resolution images, making it difficult to clearly present subtle tissue structures and lesion details. They lack infrared thermal imaging data fusion processing, and image edges and details are not prominent enough, affecting doctors' accurate judgment of lesion location, size, and relationship with surrounding tissues. At the same time, collecting images only through the front image acquisition device will lead to blind spots in the overall observation of the abdominal cavity, and fail to provide comprehensive and accurate abdominal information.

[0003] Therefore, a laparoscopic imaging method is needed, which selectively acquires image data of various types and positions by designing imaging acquisition devices with various types and positions, and generates high-precision three-dimensional stereoscopic images based on the acquired image data, thereby effectively improving the accuracy of abdominal imaging and achieving more precise control. Furthermore, the laparoscopic imaging acquisition device is controlled to move to a position where high-precision three-dimensional stereoscopic imaging data is not generated, and high-precision three-dimensional stereoscopic imaging data is generated based on the data acquired by various types of cameras, thereby completing image acquisition of all positions of the abdominal cavity, thereby eliminating blind spots and achieving full coverage of abdominal image acquisition. Summary of the invention

[0004] In view of this, the purpose of the present invention is to provide a laparoscopic imaging method, device and system, which solves the problem that traditional laparoscopes are insufficient in displaying the spatial relationship of the internal structure of the abdominal cavity and have blind spots in the overall observation of the abdominal cavity, effectively improves the accuracy of abdominal imaging, eliminates imaging blind spots and achieves full coverage of abdominal image acquisition.

[0005] In order to solve the above technical problems, the proposed solutions are as follows: A laparoscopic imaging method, comprising: Acquire image data collected by a laparoscopic imaging acquisition device, wherein the image data includes: color two-dimensional image data of the front, side and rear of the laparoscopic imaging acquisition device, high-resolution grayscale image data, and infrared thermal imaging data, wherein the color two-dimensional image data of the front is collected by two color cameras arranged side by side; Generate color three-dimensional image data with depth information based on the color two-dimensional image data of the front collected by two color cameras arranged side by side at the same time; generating high-resolution color 3D image data based on the color 3D image data and the high-resolution grayscale image data at the same time; Generate edge-enhanced color 3D image data based on the high-resolution color 3D image data and the infrared thermal imaging data at the same time; Based on the lateral and rear color two-dimensional image data at the same position, the edge-enhanced color three-dimensional image data is corrected to generate a three-dimensional stereoscopic image.

[0006] Preferably, the edge-enhanced color three-dimensional image data is corrected based on the color two-dimensional image data of the side and rear at the same position to generate a three-dimensional stereoscopic imaging diagram, specifically comprising: obtaining the color two-dimensional image data of the front, side and rear color cameras of the laparoscopic imaging acquisition device when they are located at the same position through a position sensor, and correcting the depth information and RGB color information of the color three-dimensional image data based on the color two-dimensional image data of the side and rear to generate a three-dimensional stereoscopic imaging diagram.

[0007] Preferably, the same position is determined based on a measurement result of a position sensor, and a combination of measurement results of a speed sensor and a clock.

[0008] A laparoscopic imaging method, comprising: Acquire image data collected by a laparoscopic imaging acquisition device, wherein the image data includes: color two-dimensional image data of the front, side and rear of the laparoscopic imaging acquisition device, high-resolution grayscale image data, and infrared thermal imaging data, wherein the color two-dimensional image data of the front is collected by two color cameras arranged side by side; Generate color three-dimensional image data with depth information based on the color two-dimensional image data of the front collected by two color cameras arranged side by side at the same time; generating high-resolution color 3D image data based on the color 3D image data and the high-resolution grayscale image data at the same time; Generate edge-enhanced color 3D image data based on the high-resolution color 3D image data and the infrared thermal imaging data at the same time; Based on the lateral and rear color two-dimensional image data at the same position, the edge-enhanced color three-dimensional image data is corrected to generate high-precision three-dimensional stereoscopic imaging data; The laparoscopic imaging acquisition device is controlled to move to a position where the high-precision three-dimensional stereoscopic imaging data is not generated, and the high-precision three-dimensional stereoscopic imaging data is generated based on the data collected by various types of cameras, thereby completing the image acquisition of all positions of the abdominal cavity.

[0009] Preferably, the edge-enhanced color three-dimensional image data is corrected based on the color two-dimensional image data of the side and rear at the same position to generate high-precision three-dimensional stereo imaging data, specifically including: obtaining the color two-dimensional image data of the front, side and rear color cameras of the laparoscopic imaging acquisition device when they are located at the same position through a position sensor, and correcting the depth information and RGB color information of the color three-dimensional image data based on the color two-dimensional image data of the side and rear to generate high-precision three-dimensional stereo imaging data.

[0010] Preferably, the same position is determined based on a measurement result of a position sensor, and a combination of measurement results of a speed sensor and a clock.

[0011] A laparoscopic imaging acquisition device, located at the front end of the laparoscope and in a cylindrical shape, includes: a forward dual-color camera, a forward grayscale camera, a forward infrared camera, a side color camera, a rear color camera, a position sensor, a speed sensor and a clock; wherein, The forward dual-color camera is composed of two color cameras arranged side by side at the front end of the cylinder, and is used to obtain color two-dimensional image data in front of the laparoscopic imaging acquisition device in real time, so that the laparoscopic imaging analysis device can calculate depth information based on the color two-dimensional image data obtained by the forward dual-color camera at the same time; The forward grayscale camera is composed of a grayscale camera with a resolution more than twice that of the color camera, and is used to obtain high-resolution grayscale image data in front of the laparoscope imaging acquisition device in real time; The forward infrared camera is used to obtain infrared thermal imaging data in front of the laparoscope imaging acquisition device in real time; The lateral color camera is composed of 3 or 4 color cameras evenly distributed on the side of the cylinder, and is used to obtain color two-dimensional image data of the side of the laparoscopic imaging acquisition device in real time; The rear-facing color camera is composed of two color cameras arranged side by side at the rear end of the cylinder, and is used to obtain color two-dimensional image data behind the laparoscopic imaging acquisition device in real time, so that the laparoscopic imaging analysis device can calculate depth information based on the color two-dimensional image data obtained by the rear-facing dual color cameras at the same time; The position sensor is used to obtain real-time position information of the laparoscopic imaging acquisition device; The speed sensor is used to obtain the movement speed information of the laparoscopic imaging acquisition device.

[0012] A laparoscopic imaging system, comprising: the laparoscopic imaging acquisition device, the laparoscopic imaging analysis device and the laparoscopic imaging control device; The laparoscopic imaging analysis device is used to obtain image data collected by a laparoscopic imaging acquisition device, wherein the image data includes: color two-dimensional image data, high-resolution grayscale image data, and infrared thermal imaging data of the front, side and back of the laparoscopic imaging acquisition device, the color two-dimensional image data of the front is collected by two color cameras arranged side by side; based on the color two-dimensional image data of the front collected by two color cameras arranged side by side at the same time, color three-dimensional image data with depth information is generated; based on the color three-dimensional image data and the high-resolution grayscale image data at the same time, high-resolution color three-dimensional image data is generated; based on the high-resolution color three-dimensional image data and the infrared thermal imaging data at the same time, edge-enhanced color three-dimensional image data is generated; based on the color two-dimensional image data of the side and back at the same position, the edge-enhanced color three-dimensional image data is corrected to generate a three-dimensional stereoscopic imaging diagram; The laparoscopic imaging control device is used to control the laparoscopic imaging acquisition device to move to a position where the high-precision three-dimensional stereo imaging data is not generated, generate the high-precision three-dimensional stereo imaging data based on the data collected by various types of cameras, and then complete the image acquisition of all positions of the abdominal cavity.

[0013] An electronic device comprises: a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of a laparoscopic imaging method when executing the computer program.

[0014] A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that the computer program includes steps for implementing a laparoscopic imaging method when the computer program is executed by a processor.

[0015] It can be seen from the above technical solutions that the laparoscopic imaging method provided in the embodiment of the present application selectively acquires image data of various types and positions by designing imaging acquisition devices with various types and different positions, and generates high-precision three-dimensional stereoscopic images based on the acquired image data, thereby effectively improving the accuracy of abdominal imaging and achieving more precise control. Furthermore, the laparoscopic imaging acquisition device is controlled to move to a position where high-precision three-dimensional stereoscopic imaging data is not generated, and high-precision three-dimensional stereoscopic imaging data is generated based on the data acquired by various types of cameras, thereby completing image acquisition of all positions of the abdominal cavity, thereby eliminating blind spots and achieving full coverage of abdominal image acquisition. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0017] Figure 1 The process of the laparoscopic imaging method of the present invention is Figure 1 .

[0018] Figure 2 The process of the laparoscopic imaging method of the present invention is Figure 2 .

[0019] Figure 3 It is a schematic structural diagram of the laparoscopic imaging device of the present invention.

[0020] Figure 4 It is a schematic structural diagram of the laparoscopic imaging system of the present invention.

[0021] Figure 5 It is a schematic diagram of the structure of the electronic device of the present invention. DETAILED DESCRIPTION

[0022] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0023] The laparoscopic imaging method provided by the present invention is applicable to the field of image processing, and is specifically applicable to 3D image processing of multiple cameras of different positions and types, obtaining three-dimensional information of the position, and improving image accuracy based on grayscale information and infrared information.

[0024] Binocular camera technology: Based on dual cameras shooting synchronously, the depth information of the target object is calculated to achieve three-dimensional imaging of the target individual. Its core principles include the following aspects: Dual-camera synchronous shooting: Use two cameras to synchronously shoot the target object to obtain two two-dimensional images.

[0025] Image fusion: Fusion of two two-dimensional images to generate a three-dimensional image.

[0026] Depth information calculation: Calculate the three-dimensional shape of the target object by analyzing the depth information in the three-dimensional image.

[0027] Image enhancement: by adding some information or transforming data to the original image through certain means, selectively highlighting the features of interest in the image or suppressing (masking) some unnecessary features in the image, so that the image matches the visual response characteristics. In the process of image enhancement, the reasons for image degradation are not analyzed, and the processed image may not be close to the original image. Image enhancement technology can be divided into two categories: algorithms based on the spatial domain and algorithms based on the frequency domain, depending on the different spaces where the enhancement process is located.

[0028] like Figure 1 As shown, the present invention provides a laparoscopic imaging method, which specifically includes: Step 101 : acquiring image data acquired by the laparoscope imaging acquisition device 1 .

[0029] The image data include: image data collected by two color cameras 101 and 102 arranged side by side in front of the laparoscopic imaging acquisition device 1 are I c1 ,I c2 , high-resolution grayscale image data is I g , infrared thermal imaging data is I ir , the side color two-dimensional image data is I s , the rear color two-dimensional image data is I r .

[0030] Step 102, based on the color two-dimensional image data I of the front collected by two color cameras 101 and 102 at the same time, c1 ,I c2 , generate color 3D image data with depth information I 3d1 .

[0031] Specifically, based on the binocular vision principle, c1 ,I c2 Processing to generate I 3d1 , the formula can be expressed as I 3d1 =f1(I c1 ,I c2 ), where f1 represents an algorithm function for generating color three-dimensional image data with depth information based on binocular vision.

[0032] Based on binocular vision to generate depth information color 3D image data, this solution uses the following algorithm function to implement: Feature matching-based algorithms: by detecting and matching feature points in two images, such as SIFT (Scale Invariant Feature Transform) and SURF (Speeded Up Robust Feature) algorithms. Taking SIFT as an example, first detect the extreme points in the scale space in the image, determine the main direction of the feature points, generate feature descriptors, and then match the feature points between the two images based on the feature descriptors. Calculate the depth information based on the triangulation principle. The formula is d = (f×B) / (x l -x r ), d is the depth, f is the focal length of the camera, B is the baseline distance, x l 、x r are the horizontal coordinates of the corresponding points in the left and right images respectively.

[0033] Algorithms based on region matching: such as the block matching algorithm (BM). It divides an image into several blocks, searches for similar blocks in another image, determines matching points by comparing metrics such as grayscale differences between blocks, and uses triangulation to obtain depth information after finding matching points.

[0034] Semi-global matching algorithm (SGM): Consider the pixel matching cost in a global scope, and obtain an approximate global optimal solution through one-dimensional optimization of multiple scan lines. The matching cost is calculated for each pixel, and a depth map is generated through cost aggregation, disparity calculation and optimization.

[0035] Step 103, based on the color three-dimensional image data I at the same time 3d1 and high-resolution grayscale image data I g , generate high-resolution color 3D image data I 3d2 .

[0036] Specifically, combined with I 3d1 and I g Generate I 3d2 , the formula is I 3d2 =f2(I 3d1 ,I g ), f2 is a function for fusing high-resolution grayscale image data to improve the resolution of color 3D image data.

[0037] Functions and methods for fusing high-resolution grayscale image data to improve the resolution of color 3D image data. This solution is implemented using the following algorithm functions: 1. Fusion method based on interpolation Bilinear interpolation: commonly used for image scaling. When fusion is performed, the detailed information of the high-resolution grayscale image can be filled into the color 3D image data through the bilinear interpolation algorithm. For each point in the color 3D image, according to its corresponding position in the grayscale image, the weighted average of the four surrounding grayscale pixel values ​​is used to obtain the detailed supplementary value of the point. The formula is: P(x,y) = (1 - u)(1 - v)P00 + u(1 - v)P 10 + (1 - u)vP 01 + uvP 11 , where (x, y) is the coordinate of the target point, (u, v) is the relative position of the point between four adjacent pixels, P 00 ,P 10 ,P 01 ,P 11 are the values ​​of four adjacent grayscale pixels.

[0038] Bicubic interpolation: Compared with bilinear interpolation, bicubic interpolation uses 16 surrounding pixels for weighted calculation, which can provide smoother results. Its principle is to interpolate the image based on a cubic polynomial function, so that the fused color 3D image has more natural transition in details.

[0039] 2. Fusion based on wavelet transform Wavelet decomposition and reconstruction: Perform wavelet decomposition on high-resolution grayscale images and color 3D images to obtain sub-bands of different frequencies. Usually, low-frequency sub-bands contain the main structural information of the image, and high-frequency sub-bands contain detail information. For low-frequency sub-bands, weighted averaging and other methods can be used for fusion; for high-frequency sub-bands, select coefficients with larger absolute values ​​(representing more details) for fusion. Finally, reconstruct the fused image through inverse wavelet transform, thereby improving the resolution of the color 3D image data and increasing the details.

[0040] 3. Fusion based on multi-scale Retinex Multi-scale Retinex algorithm: This algorithm separates the reflection component and illumination component of the image by performing Gaussian filtering on the image at different scales. During fusion, the reflection component and illumination component information of the high-resolution grayscale image are used to adjust the corresponding components of the color 3D image to enhance the details and contrast of the color 3D image and improve the resolution.

[0041] Step 104, based on the high-resolution color 3D image data I at the same time 3d2 and infrared thermal imaging dataI ir , generate edge enhanced color 3D image data I 3d3 .

[0042] Specifically, using I 3d2 and I ir Generate I 3d3 , the formula is I 3d3 =f3(I 3d2 ,I ir ), f3 is a function for fusing infrared thermal imaging data to enhance the edge.

[0043] To enhance the edge by fusing infrared thermal imaging data, this solution uses the following algorithm function: Gradient operator method: using gradient operators such as Sobel and Prewitt, respectively, to perform high-resolution color 3D image data I 3d2 and infrared thermal imaging dataI ir Calculate the gradient and get the edge strength image.

[0044] Canny edge detection combined method: First, I 3d2 and I ir The Canny edge detection algorithm is applied, which includes steps such as Gaussian filtering denoising, gradient calculation, non-maximum suppression, dual threshold detection and edge connection. After obtaining the edge images of both images, they are fused by logical AND and OR operations. For example, logical OR operation ensures that all edges detected in the color 3D image or infrared thermal imaging image are retained in the fusion result.

[0045] Morphological operation fusion: infrared thermal imaging data I ir Perform morphological edge detection, such as using basic morphological operators such as erosion and dilation, first erosion to remove small interference areas, and then dilation to restore the edges. Structural elements such as 3×3 squares. Combine the edge image after morphological processing with the color 3D image data, and enhance the color 3D image edge by weighted superposition and other methods.

[0046] Step 105, based on the color two-dimensional image data I of the side and rear of the same position s and I r , edge enhanced color 3D image data I 3d3 Correction is performed to generate a three-dimensional stereoscopic image I final .

[0047] Specifically, the position sensor 107 is used to obtain the color two-dimensional image data I of the color cameras 101, 102, 105 and 106 at the side and rear of the laparoscope imaging acquisition device 1 when they are located at the same position. s and I r , based on the color two-dimensional image data from the side and rear s and I r Color 3D image data I 3d3 The depth information and RGB color information are corrected to generate a three-dimensional stereo image I final , the formula is I final =f4(I 3d3 ,I s ,I r ), f4 is a function for correcting the depth information and RGB color information of the edge-enhanced color three-dimensional image data using the side and rear image data.

[0048] The side and rear image data are used to correct the depth information and RGB color information of the edge-enhanced color 3D image data. This solution is implemented using the following algorithm function: 1. Based on feature matching and correction Feature point matching algorithm: SIFT (Scale Invariant Feature Transform), ORB (Oriented FAST and Rotated BRIEF) and other algorithms are used to match the side and rear color 2D images (I s ,I r ) and edge-enhanced color 3D image data (I 3d3 ) is projected onto the image of the corresponding plane to detect and match feature points. For example, the ORB algorithm first uses the FAST algorithm to detect corner points, and then calculates the BRIEF descriptor to match feature points. The transformation matrix (such as the affine transformation matrix H) is calculated by matching point pairs, and the I 3d3 The depth information and RGB color information are spatially transformed and corrected.

[0049] Based on optical flow method: If it is assumed that the movement of objects in the scene is consistent, the optical flow method (such as Lucas-Kanade optical flow method) can be used. 3d3 The optical flow field is calculated between the projected images to obtain the motion vectors of the pixels. Based on these motion vectors, 3d3 The depth and RGB information of the corresponding position in the image are adjusted to correct the information deviation caused by the difference in viewing angle.

[0050] Based on deep fusion and color correction Depth information fusion: When the position and posture information of the side and rear images are known, the depth information (which can be obtained by binocular vision method) is combined with I 3d3 For example, using the weighted average method, for different depth estimates d at the same spatial position 3d3 ,d s ,d r , calculate the new depth d new =w1× d 3d3 +w 2 ×d s +w3×d r , w1, w2, and w3 are weights, and the weights can be determined based on factors such as sensor accuracy.

[0051] Color correction function: By comparing the side and rear images with I 3d3 The color information of the overlapping area in the image is used to establish a color correction model. If a polynomial regression model is used, let I 3d3 The RGB value of 3d3 ,G3d3 ,B 3d3 ), and the correction formula R is obtained by learning samples in the overlapping area. new =a1×R 3d3 +a2×G 3d3 +a3×B 3d3 +b1, G new =a4×R 3d3 +a5×G 3d3 +a6×B 3d3 +b2, B new =a7×R 3d3 +a8×G 3d3 +a9×B 3d3 +b3, where a i and b i is the model parameter.

[0052] The same position is determined based on the measurement result of the position sensor 107 , and the measurement results of the speed sensor 108 and the clock 109 .

[0053] Combining the above steps, the three-dimensional imaging diagram I final The calculation process can be expressed as: final =f4(f3(f2(f1(I c1 ,I c2 ),I g ),I ir ),I s ,I r ).

[0054] The technical solution of the present invention solves the problem that traditional laparoscopes are insufficient in displaying the spatial relationship of the internal structure of the abdominal cavity, and effectively improves the accuracy of abdominal imaging.

[0055] like Figure 2 As shown, the present invention provides a laparoscopic imaging method, which specifically includes: Step 201 : acquiring image data acquired by the laparoscope imaging acquisition device 1 .

[0056] The image data include: image data collected by two color cameras 101 and 102 arranged side by side in front of the laparoscopic imaging acquisition device 1 are I c1 ,I c2 , high-resolution grayscale image data is I g , infrared thermal imaging data is I ir , the side color two-dimensional image data is I s , the rear color two-dimensional image data is I r .

[0057] Step 202, based on the color two-dimensional image data I of the front collected by the two color cameras 101 and 102 at the same time, c1 ,I c2 , generate color 3D image data with depth information I 3d1 .

[0058] Step 203, based on the color three-dimensional image data I at the same time 3d1 and high-resolution grayscale image data I g , generate high-resolution color 3D image data I 3d2 .

[0059] Step 204, based on the high-resolution color 3D image data I at the same time 3d2 and infrared thermal imaging dataI ir , generate edge enhanced color 3D image data I 3d3 .

[0060] Step 205: based on the color two-dimensional image data I of the side and rear of the same position s and I r , edge enhanced color 3D image data I 3d3 Correction is performed to generate high-precision three-dimensional stereoscopic image I final .

[0061] The same position is determined based on the measurement result of the position sensor 107 , and the measurement results of the speed sensor 108 and the clock 109 .

[0062] Combining the above steps, high-precision three-dimensional imaging Figure I final The calculation process can be expressed as: final =f4(f3(f2(f1(I c1 ,I c2 ),I g ),I ir ),I s ,I r ).

[0063] Step 206, control the laparoscope imaging acquisition device 1 to move to a position where the high-precision three-dimensional imaging data I is not generated. final Based on the data collected by various types of cameras, new high-precision three-dimensional imaging data I is generated. final , thereby completing the image acquisition of all positions of the abdominal cavity.

[0064] Specifically, the position movement of the laparoscopic imaging acquisition device is precisely controlled through relevant mechanical transmission devices (such as electric rails, robotic arms, etc.). These devices can be positioned according to preset coordinates or marked position information that has not been collected but has the conditions to generate high-precision data, ensuring that they reach the designated point accurately.

[0065] For each type of camera (such as ordinary optical cameras, near-infrared cameras, etc.), it is necessary to ensure that their parameter settings are reasonable, and the focal length, exposure, etc. are adjusted to a state suitable for collecting clear data. Use synchronous or sequential acquisition methods to allow different cameras to obtain images, depth and other information from their respective perspectives, and use supporting image synthesis and three-dimensional reconstruction algorithms to integrate these collected data to generate high-precision three-dimensional stereo imaging data.

[0066] Finally, the imaging acquisition device is continuously moved along the planned path, and the above data acquisition and generation process is continuously repeated until all positions of the abdominal cavity are covered, thus completing the image acquisition of the entire abdominal cavity.

[0067] The technical solution of the present invention solves the problem of blind spots in overall abdominal cavity observation, eliminates imaging blind spots and achieves full coverage of abdominal cavity image acquisition.

[0068] Based on the same concept of a laparoscopic imaging acquisition method provided in the foregoing text of the present invention, the present invention also provides a laparoscopic imaging acquisition device 1, such as Figure 3 As shown, the device is located at the front end of the laparoscope and is cylindrical, and includes: forward dual color cameras 101 and 102, a forward grayscale camera 103, a forward infrared camera 104, a side color camera 105, a rear color camera 106, a position sensor 107, a speed sensor 108 and a clock 109. Among them, The forward-pointing dual-color cameras 101 and 102 are composed of two color cameras located side by side at the front end of the cylinder, and are used to obtain color two-dimensional image data in front of the laparoscopic imaging acquisition device 1 in real time, so that the laparoscopic imaging analysis device 2 can calculate the depth information based on the color two-dimensional image data obtained by the forward-pointing dual-color cameras 101 and 102 at the same time.

[0069] The forward grayscale camera 103 is composed of a grayscale camera with a resolution more than twice that of the color cameras 101 and 102 , and is used to obtain high-resolution grayscale image data in front of the laparoscopic imaging acquisition device 1 in real time.

[0070] The forward infrared camera 104 is used to obtain infrared thermal imaging data in front of the laparoscope imaging acquisition device 1 in real time.

[0071] The lateral color camera 105 is composed of 3 or 4 color cameras (105-1, 105-2, 105-3, 105-4) evenly distributed on the side of the cylinder, and is used to obtain color two-dimensional image data on the side of the laparoscopic imaging acquisition device 1 in real time.

[0072] The rear-pointing color camera 106 is composed of two color cameras (106-1, 106-2) arranged side by side at the rear end of the cylinder, and is used to obtain color two-dimensional image data behind the laparoscopic imaging acquisition device 1 in real time, so that the laparoscopic imaging analysis device 2 can calculate depth information based on the color two-dimensional image data obtained by the rear-pointing dual color cameras 106 at the same time.

[0073] The position sensor 107 is used to obtain real-time position information of the laparoscopic imaging acquisition device 1 .

[0074] The speed sensor 108 is used to obtain the movement speed information of the laparoscopic imaging acquisition device 1 .

[0075] Based on the same concept of a laparoscopic imaging acquisition method provided in the previous text of the present invention, the present invention also provides a laparoscopic imaging system, such as Figure 4 As shown, the system includes: a laparoscopic imaging acquisition device 1, a laparoscopic imaging analysis device 2 and a laparoscopic imaging control device 3.

[0076] The laparoscopic imaging analysis device 2 is used to obtain image data collected by the laparoscopic imaging acquisition device 1, wherein the image data includes: color two-dimensional image data, high-resolution grayscale image data, and infrared thermal imaging data of the front, side and back of the laparoscopic imaging acquisition device, the color two-dimensional image data of the front is collected by two side-by-side color cameras 101 and 102 respectively; based on the color two-dimensional image data of the front collected by the two side-by-side color cameras at the same time, color three-dimensional image data with depth information is generated; based on the color three-dimensional image data and high-resolution grayscale image data at the same time, high-resolution color three-dimensional image data is generated; based on the high-resolution color three-dimensional image data and infrared thermal imaging data at the same time, edge-enhanced color three-dimensional image data is generated; based on the color two-dimensional image data of the side and back at the same position, the edge-enhanced color three-dimensional image data is corrected to generate a high-precision three-dimensional stereoscopic image.

[0077] The laparoscopic imaging control device 3 is used to control the laparoscopic imaging acquisition device 1 to move to a position where high-precision three-dimensional stereo imaging data is not generated, generate high-precision three-dimensional stereo imaging data based on data collected by various types of cameras, and then complete image acquisition of all positions of the abdominal cavity.

[0078] Based on the same concept of a laparoscopic imaging method provided in the foregoing text of the present invention, the present invention also provides an electronic device, such as Figure 5 As shown, the device includes: a memory 1001, a processor 1002, and a computer program stored in the memory 1001 and executable on the processor 1002. When the processor 1002 executes the computer program, the steps of the laparoscopic imaging method are implemented.

[0079] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. The terms "include", "comprises" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the statement "comprises a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0080] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0081] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0082] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0083] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0084] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

[0085] The flowcharts and block diagrams in the accompanying drawings show the possible implementation architectures, functions and operations of the methods, devices and computer program products according to multiple embodiments of the present invention. In this regard, each box in the flowchart and block diagram can represent a module, a program segment or a portion of a code, containing one or more computer executable instructions for implementing logical functions. It should also be noted that in some alternative implementations, the functions marked in the boxes can also occur in an order different from that marked in the accompanying drawings. It should also be noted that each box or combination of boxes in the block diagram and flowchart can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

Claims

1. A laparoscopic imaging method, characterized in that: include: Acquire image data collected by a laparoscopic imaging acquisition device, wherein the image data includes: color two-dimensional image data of the front, side and rear of the laparoscopic imaging acquisition device, high-resolution grayscale image data, and infrared thermal imaging data, wherein the color two-dimensional image data of the front is collected by two color cameras arranged side by side; Generate color three-dimensional image data with depth information based on the color two-dimensional image data of the front collected by two color cameras arranged side by side at the same time; generating high-resolution color 3D image data based on the color 3D image data and the high-resolution grayscale image data at the same time; Generate edge-enhanced color 3D image data based on the high-resolution color 3D image data and the infrared thermal imaging data at the same time; Based on the lateral and rear color two-dimensional image data at the same position, the edge-enhanced color three-dimensional image data is corrected to generate a three-dimensional stereoscopic image.

2. The laparoscopic imaging method according to claim 1, characterized in that: The method of correcting the edge-enhanced color three-dimensional image data based on the color two-dimensional image data of the side and rear at the same position to generate a three-dimensional stereoscopic imaging diagram specifically includes: obtaining the color two-dimensional image data of the front, side and rear color cameras of the laparoscopic imaging acquisition device when they are located at the same position through a position sensor, and correcting the depth information and RGB color information of the color three-dimensional image data based on the color two-dimensional image data of the side and rear to generate a three-dimensional stereoscopic imaging diagram.

3. The laparoscopic imaging method according to claim 1, characterized in that: The same position is determined based on the measurement result of the position sensor and the measurement results of the speed sensor and the clock.

4. A laparoscopic imaging method, characterized in that: include: Acquire image data collected by a laparoscopic imaging acquisition device, wherein the image data includes: color two-dimensional image data of the front, side and rear of the laparoscopic imaging acquisition device, high-resolution grayscale image data, and infrared thermal imaging data, wherein the color two-dimensional image data of the front is collected by two color cameras arranged side by side; Generate color three-dimensional image data with depth information based on the color two-dimensional image data of the front collected by two color cameras arranged side by side at the same time; generating high-resolution color 3D image data based on the color 3D image data and the high-resolution grayscale image data at the same time; Generate edge-enhanced color 3D image data based on the high-resolution color 3D image data and the infrared thermal imaging data at the same time; Based on the lateral and rear color two-dimensional image data at the same position, the edge-enhanced color three-dimensional image data is corrected to generate high-precision three-dimensional stereoscopic imaging data; The laparoscopic imaging acquisition device is controlled to move to a position where the high-precision three-dimensional stereoscopic imaging data is not generated, and the high-precision three-dimensional stereoscopic imaging data is generated based on the data collected by various types of cameras, thereby completing the image acquisition of all positions of the abdominal cavity.

5. The laparoscopic imaging method according to claim 4, characterized in that: The method of correcting the edge-enhanced color three-dimensional image data based on the color two-dimensional image data of the side and rear at the same position to generate high-precision three-dimensional stereoscopic imaging data specifically includes: obtaining the color two-dimensional image data of the front, side and rear color cameras of the laparoscopic imaging acquisition device when they are located at the same position through a position sensor, and correcting the depth information and RGB color information of the color three-dimensional image data based on the color two-dimensional image data of the side and rear to generate high-precision three-dimensional stereoscopic imaging data.

6. The laparoscopic imaging method according to claim 4, characterized in that: The same position is determined based on the measurement result of the position sensor and the measurement results of the speed sensor and the clock.

7. A laparoscopic imaging acquisition device, characterized in that: Located at the front end of the laparoscope, it is cylindrical and includes: a forward dual-color camera, a forward grayscale camera, a forward infrared camera, a side color camera, a rear color camera, a position sensor, a speed sensor and a clock; among them, The forward dual-color camera is composed of two color cameras arranged side by side at the front end of the cylinder, and is used to obtain color two-dimensional image data in front of the laparoscopic imaging acquisition device in real time, so that the laparoscopic imaging analysis device can calculate depth information based on the color two-dimensional image data obtained by the forward dual-color camera at the same time; The forward grayscale camera is composed of a grayscale camera with a resolution more than twice that of the color camera, and is used to obtain high-resolution grayscale image data in front of the laparoscope imaging acquisition device in real time; The forward infrared camera is used to obtain infrared thermal imaging data in front of the laparoscope imaging acquisition device in real time; The lateral color camera is composed of 3 or 4 color cameras evenly distributed on the side of the cylinder, and is used to obtain color two-dimensional image data of the side of the laparoscopic imaging acquisition device in real time; The rear-facing color camera is composed of two color cameras arranged side by side at the rear end of the cylinder, and is used to obtain color two-dimensional image data behind the laparoscopic imaging acquisition device in real time, so that the laparoscopic imaging analysis device can calculate depth information based on the color two-dimensional image data obtained by the rear-facing dual color cameras at the same time; The position sensor is used to obtain real-time position information of the laparoscopic imaging acquisition device; The speed sensor is used to obtain the movement speed information of the laparoscopic imaging acquisition device.

8. A laparoscopic imaging system, characterized in that: include: The laparoscopic imaging acquisition device, laparoscopic imaging analysis device and laparoscopic imaging control device according to claim 7; The laparoscopic imaging analysis device is used to obtain image data collected by a laparoscopic imaging acquisition device, wherein the image data includes: color two-dimensional image data, high-resolution grayscale image data, and infrared thermal imaging data of the front, side and back of the laparoscopic imaging acquisition device, the color two-dimensional image data of the front is collected by two color cameras arranged side by side; based on the color two-dimensional image data of the front collected by two color cameras arranged side by side at the same time, color three-dimensional image data with depth information is generated; based on the color three-dimensional image data and the high-resolution grayscale image data at the same time, high-resolution color three-dimensional image data is generated; based on the high-resolution color three-dimensional image data and the infrared thermal imaging data at the same time, edge-enhanced color three-dimensional image data is generated; based on the color two-dimensional image data of the side and back at the same position, the edge-enhanced color three-dimensional image data is corrected to generate a three-dimensional stereoscopic imaging diagram; The laparoscopic imaging control device is used to control the laparoscopic imaging acquisition device to move to a position where the high-precision three-dimensional stereo imaging data is not generated, generate the high-precision three-dimensional stereo imaging data based on the data collected by various types of cameras, and then complete the image acquisition of all positions of the abdominal cavity.

9. An electronic device, comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the steps of the laparoscopic imaging method as described in any one of claims 1 to 3 or any one of claims 4 to 6 are implemented.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: The computer program comprises the steps of the laparoscopic imaging method as claimed in any one of claims 1 to 3 or any one of claims 4 to 6 when executed by a processor.

Images