Binocular 3D endoscope target size detection method, system, device and medium

Abstract

The present invention discloses a binocular 3D endoscope target size detection method, system, device and medium. The method includes adjusting the viewing angle of the binocular 3D endoscope and obtaining a binocular image sequence of the object to be measured captured by the binocular 3D endoscope during the viewing angle adjustment process; wherein the binocular image sequence is composed of binocular images of the object to be measured at multiple viewing angles; based on the binocular images at each viewing angle, the size information of the object to be measured at each viewing angle is calculated and obtained; the size information at all viewing angles is data fused to obtain the target size of the object to be measured. The present invention realizes the size measurement of the object to be measured. Through multi-viewpoint data fusion, the target size obtained is more accurate and more consistent with the actual situation of the object, and can adapt to the size measurement of objects in different environments and complex shapes.

Description

Technical Field

[0001] The present invention relates to the field of medical equipment, and in particular to a binocular 3D endoscope target size detection method, system, device and medium. Background Art

[0002] A binocular 3D endoscope is an endoscopic device that uses binocular stereo vision to achieve three-dimensional imaging. It uses two cameras (or sensors) to capture images of the same scene from different angles. Using computer algorithms, it generates a disparity map, which infers the depth of the object and generates a three-dimensional image.

[0003] The three-dimensional images generated by binocular 3D endoscopes help doctors more clearly determine the distance, shape, and size of objects. During surgery, doctors can use 3D images to more accurately locate lesions, organs, or tissues, reducing the risk of misoperation. In a variety of procedures, including internal medicine, surgery, otolaryngology, and urology, binocular 3D endoscopes provide doctors with a clearer field of view. Especially in minimally invasive surgeries, they can precisely locate lesions, reducing patient trauma and postoperative recovery time.

[0004] Traditional two-dimensional endoscopes can only provide flat images. Therefore, binocular 3D endoscopes have significant advantages over traditional two-dimensional endoscopes, especially in precise positioning, depth perception and stereoscopic observation.

[0005] However, binocular 3D endoscopes also have some defects and limitations. When calculating parallax, they need to use the baseline of the binocular vision system, which refers to the physical distance between the two cameras. A shorter baseline may result in smaller parallax, thereby reducing the accuracy of depth measurement. A longer baseline will result in a larger parallax during measurement, but it may also bring larger errors, especially when measuring distant objects. When part of the object is blocked, the parallax calculation may become difficult, and the depth information of the blocked part cannot be accurately calculated. In addition, the surface of the object may have a complex structure or too many details, which makes it difficult to calculate the parallax of the local area, affecting the accuracy of the overall size measurement. Summary of the Invention

[0006] In view of this, the present invention provides a binocular 3D endoscope target size detection method, system, device and medium to solve the problem that the existing binocular 3D endoscope is easily affected by factors such as inappropriate baseline, object occlusion and complex object structure when measuring target size, resulting in low overall measurement accuracy.

[0007] The present invention provides a binocular 3D endoscope target size detection method, the method comprising:

[0008] Adjusting the viewing angle of the binocular 3D endoscope and obtaining a binocular image sequence of the object to be measured captured by the binocular 3D endoscope during the viewing angle adjustment process; wherein the binocular image sequence is composed of binocular images of the object to be measured at multiple viewing angles;

[0009] Calculating the size information of the object to be measured at each viewing angle according to the binocular image at each viewing angle;

[0010] The size information at all viewing angles is fused to obtain the target size of the object to be measured.

[0011] Optionally, the binocular 3D endoscope includes a first camera and a second camera that are fixedly connected;

[0012] The adjusting of the viewing angle of the binocular 3D endoscope includes:

[0013] Using a linear translation method, the first camera and the second camera are translated together, so that the viewing angle of the binocular 3D endoscope changes, thereby obtaining multiple viewing angles of the binocular 3D endoscope;

[0014] And / or, a linear translation method is used to translate the object to be measured, so that the viewing angle of the binocular 3D endoscope changes, thereby obtaining multiple viewing angles of the binocular 3D endoscope.

[0015] Optionally, the size information includes depth information, length information and width information;

[0016] The step of calculating the size information of the object to be measured at each viewing angle based on the binocular image at each viewing angle includes:

[0017] Select the binocular image under any one viewing angle, and pre-process the selected binocular image;

[0018] Performing feature extraction and feature matching on the preprocessed binocular image in sequence to obtain a first eigenvalue and a second eigenvalue of the object to be measured in the preprocessed binocular image;

[0019] Performing parallax calculation based on the first eigenvalue and the second eigenvalue to obtain a parallax value of the object to be measured at a selected viewing angle;

[0020] Reconstructing a disparity map according to the disparity value of the object to be measured at the selected viewing angle, and calculating the depth information of the object to be measured at the selected viewing angle;

[0021] Performing three-dimensional reconstruction based on the depth information of the object to be measured at the selected viewing angle to obtain the three-dimensional coordinates of each object point in the object to be measured at the selected viewing angle;

[0022] Calculating the length information and the width information of the object to be measured at the selected viewing angle according to the three-dimensional coordinates of all object points in the object to be measured at the selected viewing angle;

[0023] The binocular images at each viewing angle are traversed, and the size information of the object to be measured at each viewing angle is calculated respectively using the same method.

[0024] Optionally, each of the binocular images includes a first camera image and a second camera image;

[0025] The first eigenvalue is specifically a first horizontal coordinate value of a target object point in the object to be measured in the preprocessed first camera image, and the second eigenvalue is specifically a second horizontal coordinate value of the target object point in the preprocessed second camera image.

[0026] Optionally, the binocular image under the selected viewing angle is assumed to be the i-th binocular image in the binocular image sequence, and the i-th binocular image includes the i-th first camera image and the i-th second camera image;

[0027] The specific formula for calculating the parallax value of the object to be measured at the selected viewing angle is:

[0028] ;

[0029] Among them, d i is the parallax value of the object to be measured at the i-th viewing angle, x i,L is the first horizontal coordinate value of the target object point in the object to be measured in the i-th first camera image, x i,R is the second horizontal coordinate value of the target object point in the object to be measured in the i-th second camera image.

[0030] Optionally, a specific formula for calculating the depth information of the object to be measured at a selected viewing angle is:

[0031] ;

[0032] Among them, Z i is the depth information of the object to be measured at the i-th viewing angle, f is the focal length of the two cameras in the binocular 3D endoscope, B i is the baseline length between the two cameras in the binocular 3D endoscope at the i-th viewing angle.

[0033] Optionally, each of the binocular images includes a first camera image and a second camera image; assuming that the binocular image under the selected viewing angle is the i-th binocular image in the binocular image sequence, the i-th binocular image includes the i-th first camera image and the i-th second camera image;

[0034] The three-dimensional reconstruction is performed based on the depth information of the object to be measured at the selected viewing angle to obtain the three-dimensional coordinates of each object point in the object to be measured at the selected viewing angle, including:

[0035] Selecting a j-th object point in the object to be measured, obtaining a first pixel coordinate of the j-th object point in the i-th first camera image, and calculating a first three-dimensional coordinate of the j-th object point in the coordinate system of the i-th first camera image based on the first pixel coordinate of the j-th object point and the depth information of the object to be measured at the selected viewing angle;

[0036] Converting the first three-dimensional coordinate of the j-th object point into the world coordinate system to obtain the three-dimensional coordinate of the j-th object point in the object to be measured at the selected viewing angle;

[0037] Traversing each object point in the object to be measured, and obtaining the three-dimensional coordinates of each object point in the object to be measured under the selected viewing angle according to the same method;

[0038] Alternatively, selecting a j-th object point in the object to be measured, obtaining a second pixel coordinate of the j-th object point in the i-th second camera image, and calculating a second three-dimensional coordinate of the j-th object point in the coordinate system of the i-th second camera image based on the second pixel coordinate of the j-th object point and the depth information of the object to be measured at the selected viewing angle;

[0039] Converting the second three-dimensional coordinates of the j-th object point into the world coordinate system to obtain the three-dimensional coordinates of the j-th object point in the object to be measured at the selected viewing angle;

[0040] Each object point in the object to be measured is traversed, and the three-dimensional coordinates of each object point in the object to be measured under the selected viewing angle are obtained in the same manner.

[0041] Optionally, a specific formula for calculating the first three-dimensional coordinates of the j-th object point in the coordinate system of the i-th first camera image is:

[0042] ;

[0043] in, is the first three-dimensional coordinate of the j-th object point in the coordinate system of the i-th first camera image, is the first pixel coordinate of the j-th object point in the i-th first camera image, (c x ,c y ) is the camera principal point coordinate in the binocular 3D endoscope, Z i is the depth information of the object to be measured at the i-th viewing angle, and f is the focal length of the two cameras in the binocular 3D endoscope;

[0044] Alternatively, the specific formula for calculating the second three-dimensional coordinates of the j-th object point in the coordinate system where the i-th second camera image is located is:

[0045] ;

[0046] in, is the second three-dimensional coordinate of the j-th object point in the coordinate system of the i-th second camera image, is the second pixel coordinate of the j-th object point in the i-th second camera image, (c x ,c y ) is the camera principal point coordinate in the binocular 3D endoscope, Z i is the depth information of the object to be measured at the i-th viewing angle, and f is the focal length of the two cameras in the binocular 3D endoscope.

[0047] Optionally, the calculating the size information of the object to be measured at the selected viewing angle according to the three-dimensional coordinates of all object points in the object to be measured at the selected viewing angle includes:

[0048] Traversing the three-dimensional coordinates of all object points in the object to be measured at the selected viewing angle to obtain the coordinates of the boundary points of the object to be measured at the selected viewing angle;

[0049] The length information and the width information of the object to be measured at the selected viewing angle are respectively calculated based on the coordinates of the boundary points of the object to be measured at the selected viewing angle.

[0050] Optionally, assuming that the binocular image under the selected viewing angle is the i-th binocular image in the binocular image sequence, the boundary point coordinates of the object to be measured under the i-th viewing angle include the maximum coordinate value in the x-direction, the minimum coordinate value in the x-direction, the maximum coordinate value in the y-direction, and the minimum coordinate value in the y-direction;

[0051] The specific formulas for calculating the length information and the width information of the object to be measured at the i-th viewing angle are:

[0052] ;

[0053] Among them, L x,i and L y,iare the length information and width information of the object to be measured at the i-th viewing angle, respectively, max,i 、x min,i 、y max,i and y min,i are the maximum x-direction coordinate value, minimum x-direction coordinate value, maximum y-direction coordinate value, and minimum y-direction coordinate value of the object to be measured at the i-th viewing angle, respectively.

[0054] Optionally, the target size includes a target depth, a target length and a target width of the object to be measured;

[0055] The step of fusing the size information at all viewing angles to obtain the target size of the object to be measured includes:

[0056] Performing a weighted average operation on the depth information under all viewing angles to obtain the target depth of the object to be measured;

[0057] The specific formula for performing weighted average calculation on the depth information under all viewing angles is:

[0058] ;

[0059] Wherein, Z is the target depth, Z i is the depth information of the object to be measured at the i-th viewing angle, w i is the weight under the i-th perspective, and n is the total number of perspectives;

[0060] Performing a weighted average operation on the length information at all viewing angles to obtain the target length of the object to be measured;

[0061] The specific formula for performing weighted average calculation on the length information under all viewing angles is:

[0062] ;

[0063] Among them, L x is the target length, L x,i is the length information of the object to be measured at the i-th viewing angle;

[0064] Performing a weighted average operation on the width information at all viewing angles to obtain the target width of the object to be measured;

[0065] The specific formula for performing weighted average calculation on the width information under all viewing angles is:

[0066] ;

[0067] Among them, L y is the target width, Ly,i is the width information of the object to be measured at the i-th viewing angle.

[0068] In addition, the present invention also provides a binocular 3D endoscope target size detection system, which is applied to the aforementioned binocular 3D endoscope target size detection method, and the system includes:

[0069] A viewing angle adjustment module, used to adjust the viewing angle of the binocular 3D endoscope;

[0070] An image acquisition module, configured to acquire a binocular image sequence of the object to be measured captured by the binocular 3D endoscope during the viewing angle adjustment process; wherein the binocular image sequence is composed of binocular images of the object to be measured at multiple viewing angles;

[0071] A size calculation module is used to calculate the size information of the object to be measured at each viewing angle based on the binocular image at each viewing angle;

[0072] The size fusion module is used to fuse the size information under all viewing angles to obtain the target size of the object to be measured.

[0073] In addition, the present invention also provides a binocular 3D endoscope target size detection device, including a processor, a memory, and a computer program stored in the memory and runnable on the processor, wherein the computer program implements the method steps in the aforementioned binocular 3D endoscope target size detection method when running.

[0074] In addition, the present invention also provides a computer storage medium, which includes: at least one instruction, which implements the method steps in the aforementioned binocular 3D endoscope target size detection method when the instruction is executed by a computer.

[0075] The beneficial effects of the present invention are as follows: by adjusting the viewing angle of the binocular 3D endoscope, binocular images of the object to be measured at multiple different viewing angles are collected, and image information of the binocular 3D endoscope at different viewing angles can be provided; then the binocular images at each viewing angle are processed, and the size information of the object to be measured at each viewing angle is measured. Finally, by performing data fusion on the size information at all viewing angles, the target size containing multi-angle information can be obtained, thereby realizing the size measurement of the object to be measured, and is not limited to the depth information calculated from the image at a single viewing angle. This can effectively avoid the situation where only the baseline length between the two cameras in the binocular 3D endoscope at a certain viewing angle is short or long, and can also avoid the situation where only the occlusion of the object to be measured at a certain viewing angle is considered. Through multi-view data fusion, the image information of the object to be measured with a more complex structure at different viewing angles can be fully considered. Finally, the target size obtained by data fusion has higher accuracy, is more in line with the actual situation of the object, and can adapt to the size measurement of objects in different environments and with complex shapes. BRIEF DESCRIPTION OF THE DRAWINGS

[0076] The features and advantages of the present invention will be more clearly understood by referring to the accompanying drawings, which are schematic and should not be construed as limiting the present invention in any way. In the accompanying drawings:

[0077] Figure 1 A flow chart of a binocular 3D endoscope target size detection method in embodiment 1 of the present invention is shown;

[0078] Figure 2 A flowchart of calculating size information at each viewing angle in the first embodiment of the present invention is shown;

[0079] Figure 3 The figure shows a structural diagram of a binocular 3D endoscope target size detection system in the second embodiment of the present invention. DETAILED DESCRIPTION

[0080] To make the purpose, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without making any creative efforts shall fall within the scope of protection of the present invention.

[0081] Example 1

[0082] This embodiment provides a binocular 3D endoscope target size detection method, such as Figure 1 As shown, the method includes:

[0083] S1: adjusting the viewing angle of the binocular 3D endoscope and obtaining a binocular image sequence of the object to be measured captured by the binocular 3D endoscope during the viewing angle adjustment process; wherein the binocular image sequence is composed of binocular images of the object to be measured at multiple viewing angles;

[0084] S2: Calculating the size information of the object to be measured at each viewing angle according to the binocular image at each viewing angle;

[0085] S3: Performing data fusion on the size information under all viewing angles to obtain the target size of the object to be measured.

[0086] In this embodiment, by adjusting the viewing angle of the binocular 3D endoscope, binocular images of the object to be measured at multiple different viewing angles are collected, and image information of the binocular 3D endoscope at different viewing angles can be provided; then the binocular images at each viewing angle are processed, and the size information of the object to be measured at each viewing angle is measured. Finally, by performing data fusion on the size information at all viewing angles, the target size containing multi-angle information can be obtained, thereby realizing the size measurement of the object to be measured, and is not limited to the depth information calculated from the image at a single viewing angle. This can effectively avoid the situation where only the baseline length between the two cameras in the binocular 3D endoscope at a certain viewing angle is short or long, and can also avoid the situation where only the occlusion of the object to be measured at a certain viewing angle is considered. Through multi-view data fusion, the image information of the object to be measured with a more complex structure at different viewing angles can be fully considered. Finally, the target size obtained by data fusion has higher accuracy, is more consistent with the actual situation of the object, and can adapt to the size measurement of objects in different environments and complex shapes.

[0087] Each step of the binocular 3D endoscope target size detection method of this embodiment is further described below.

[0088] In this embodiment S1, the binocular 3D endoscope includes a first camera and a second camera that are fixedly connected.

[0089] By using the first camera and the second camera, the same scene can be observed from different angles and images can be captured, thereby conveniently utilizing the parallax between the images to infer the depth information of the object and achieve size measurement.

[0090] Preferably, in this embodiment S1, adjusting the viewing angle of the binocular 3D endoscope includes:

[0091] Using a linear translation method, the first camera and the second camera are translated together, so that the viewing angle of the binocular 3D endoscope changes, thereby obtaining multiple viewing angles of the binocular 3D endoscope;

[0092] And / or, a linear translation method is used to translate the object to be measured, so that the viewing angle of the binocular 3D endoscope changes, thereby obtaining multiple viewing angles of the binocular 3D endoscope.

[0093] In this embodiment, two approaches are considered for adjusting the viewing angle of a binocular 3D endoscope: the first approach involves moving the binocular cameras (i.e., the first and second cameras) while the observed object (i.e., the object to be measured) remains stationary; the second approach involves moving the object while the binocular cameras remain stationary. When the object to be measured remains stationary while the binocular cameras move, the angle at which the binocular cameras observe the object changes as the binocular cameras move, thereby generating multiple viewing angles. When the object to be measured moves while the binocular cameras remain stationary, the angle at which the binocular cameras observe the object also changes as the object moves, similarly generating multiple viewing angles.

[0094] Of course, we can also consider that the object to be measured moves and the binocular camera also moves. In this case, it is necessary to ensure that the movement of the object to be measured and the movement of the binocular camera are not synchronized, that is, it is necessary to ensure that the two move in different directions or at different speeds. When the two move at the same time and do not move synchronously (that is, different directions or different speeds), the angle of the object to be measured observed by the binocular camera will also change, and multiple perspectives can also be generated.

[0095] In this embodiment, the linear translation method involves controlling the target (the binocular camera and / or the object to be measured) to move in a linear direction. This linear direction can be set based on the actual situation. For example, the binocular camera can be controlled to translate horizontally or vertically.

[0096] During the movement of the binocular camera and / or the object to be measured, each time it moves to a position point, a viewing angle is generated, and the binocular camera collects a binocular image under this viewing angle. Similarly, a binocular image sequence of the object to be measured collected by the binocular 3D endoscope during the viewing angle adjustment process can be obtained.

[0097] This embodiment uses a stereo disparity method when subsequently calculating depth information, and a linear translation method when providing perspective. By combining these two methods, the actual size of the object can be accurately calculated, which can effectively improve the accuracy of the final size measurement.

[0098] Preferably, if Figure 2 As shown, the size information includes depth information, length information and width information; in this embodiment S2 includes:

[0099] S21: Select the binocular image under any one viewing angle and pre-process the selected binocular image;

[0100] S22: performing feature extraction and feature matching on the preprocessed binocular image in sequence to obtain a first eigenvalue and a second eigenvalue of the object to be measured in the preprocessed binocular image;

[0101] S23: performing parallax calculation according to the first eigenvalue and the second eigenvalue to obtain a parallax value of the object to be measured at a selected viewing angle;

[0102] S24: reconstructing a disparity map according to the disparity value of the object to be measured at the selected viewing angle, and calculating the depth information of the object to be measured at the selected viewing angle;

[0103] S25: performing three-dimensional reconstruction based on the depth information of the object to be measured at the selected viewing angle to obtain the three-dimensional coordinates of each object point in the object to be measured at the selected viewing angle;

[0104] S26: Calculating the length information and the width information of the object to be measured at the selected viewing angle according to the three-dimensional coordinates of all object points in the object to be measured at the selected viewing angle;

[0105] S27: Traverse the binocular image at each viewing angle, and calculate the size information of the object to be measured at each viewing angle respectively using the same method.

[0106] When using stereo disparity to calculate depth information at each viewing angle, preprocessing the binocular images effectively improves image quality and enhances image features, reducing the impact of noise and other interference factors on subsequent feature extraction and matching. This highlights useful information in the images, facilitating more accurate extraction of the first and second eigenvalues, and thereby improving the accuracy of depth information. After preprocessing the binocular images, feature extraction can extract representative features, reducing data volume and improving subsequent processing efficiency. Feature matching ensures that the extracted features are features of the same object points in both binocular images, ensuring the reliability of subsequent calculations and the reliability of depth information calculated using stereo disparity. Disparity calculation and disparity map reconstruction are then used to accurately measure the depth information of the object under test at each viewing angle. Once the depth information at each viewing angle is calculated, three-dimensional reconstruction can be performed using this depth information, obtaining the three-dimensional coordinates of each object point in the object under test at each viewing angle. These three-dimensional coordinates are not only related to the object's depth information but also directly related to its length and width information. Therefore, the three-dimensional coordinates can be calculated to obtain the length and width information of the object under test at each viewing angle.

[0107] Specifically, in the above step S21, the pre-processing performed on the selected binocular image includes but is not limited to image processing methods such as denoising processing and grayscale processing.

[0108] The binocular image at each viewing angle includes a first camera image and a second camera image.

[0109] Specifically, in step S22, the feature extraction and matching methods can employ features such as the SIFT (Speeded Up Robust Features) scale-invariant feature transform, the SURF (Speeded Up Robust Features) accelerated robust feature method, or the ORB (Oriented Fast and Rotated BRIEF) oriented fast features and rotated binary robust independent basis features method. This embodiment employs the SURF feature extraction method, which extracts SURF features from the preprocessed binocular images (including the first and second camera images), constructs a Hessian matrix to generate interest points, and then constructs a scale space to locate feature points and generate descriptors. By comparing the descriptors of the feature points in the two images, similar feature point pairs are found and used as matching points, thus achieving complete feature matching.

[0110] Preferably, for feature matching, the first and second eigenvalues ​​of the same target object point in the two images are extracted. When the position of the same target object point (i.e., the matching point) in the two images is matched, an interpolation method can be used to refine the position of the matching point to the sub-pixel level. Bundle adjustment is then used to globally optimize the postures of the two cameras and the coordinates of the object to be measured in the two camera images, eliminating the accumulation of local errors and improving overall measurement accuracy. Common interpolation methods include bilinear interpolation and bicubic interpolation. Taking bilinear interpolation as an example, it calculates the grayscale value of the sub-pixel position using the grayscale values ​​of the four surrounding pixels, thereby more accurately determining the specific position of the matching point within the pixel and reducing errors caused by pixel discreteness. While ensuring the precise position of the matching point, the first and second eigenvalues ​​obtained in the two images are more accurate and reliable. The bundle adjustment method is used to globally optimize the postures of the two cameras and the coordinates of the object to be measured in the two camera images. Its specific implementation method adopts conventional methods and is not limited in this embodiment.

[0111] Specifically, in the above step S22, the first eigenvalue is specifically the first horizontal coordinate value of the target object point in the object to be measured in the first camera image after preprocessing, and the second eigenvalue is specifically the second horizontal coordinate value of the target object point in the second camera image after preprocessing.

[0112] In a binocular vision system, the optical axes of the two cameras (typically left and right cameras) are typically parallel and on the same horizontal plane. Therefore, for the same target object point, the vertical coordinate in the left and right images varies relatively little, while the difference in the horizontal coordinate is primarily due to the different viewing angles of the left and right cameras. This difference directly reflects the object's depth information, or parallax. Therefore, this embodiment uses the first and second horizontal coordinate values ​​in the two images as the first and second eigenvalues, respectively, for subsequent parallax and depth calculations. This conforms to the basic principle of binocular vision, which uses the horizontal difference between the left and right images to obtain depth information. The calculated results are more accurate and reliable.

[0113] The target object point can be selected from all object points of the object to be measured according to actual conditions.

[0114] Specifically, in the above step S23, it is assumed that the binocular image under the selected viewing angle is the i-th binocular image in the binocular image sequence, and the i-th binocular image includes the i-th first camera image and the i-th second camera image;

[0115] The specific formula for calculating the parallax value of the object to be measured at the selected viewing angle is:

[0116] ;

[0117] Among them, d i is the parallax value of the object to be measured at the i-th viewing angle, x i,L is the first horizontal coordinate value of the target object point in the object to be measured in the i-th first camera image, x i,R is the second horizontal coordinate value of the target object point in the object to be measured in the i-th second camera image.

[0118] Specifically, in the above step S24, the specific formula for calculating the depth information of the object to be measured under the selected viewing angle is:

[0119] ;

[0120] Among them, Z i is the depth information of the object to be measured at the i-th viewing angle, f is the focal length of the two cameras in the binocular 3D endoscope, B i is the baseline length between the two cameras in the binocular 3D endoscope at the i-th viewing angle.

[0121] The depth information under each viewing angle is calculated by the above formula, which is more in line with the working principle of binocular 3D endoscope. The method is simple and the results are accurate and reliable.

[0122] Among them, the focal length and baseline length of the two cameras are inherent parameters of the binocular 3D endoscope and can be known in advance.

[0123] Specifically, the above step S25 includes two methods. The first method is to use the first camera as a reference and calculate using information in the first camera image; the second method is to use the second camera as a reference and calculate using information in the second camera image.

[0124] In the first method, S25 includes:

[0125] S25A1: Select the jth object point in the object to be measured, obtain a first pixel coordinate of the jth object point in the i-th first camera image, and calculate a first three-dimensional coordinate of the jth object point in the coordinate system of the i-th first camera image based on the first pixel coordinate of the jth object point and the depth information of the object to be measured at the selected viewing angle;

[0126] S25A2: Convert the first three-dimensional coordinate of the j-th object point to the world coordinate system to obtain the three-dimensional coordinate of the j-th object point in the object to be measured at the selected viewing angle;

[0127] S25A3: Traverse each object point in the object to be measured, and obtain the three-dimensional coordinates of each object point in the object to be measured under the selected viewing angle according to the same method.

[0128] In the second method, S25 includes:

[0129] S25B1: Selecting a j-th object point in the object to be measured, obtaining a second pixel coordinate of the j-th object point in the i-th second camera image, and calculating a second three-dimensional coordinate of the j-th object point in the coordinate system of the i-th second camera image based on the second pixel coordinate of the j-th object point and the depth information of the object to be measured at the selected viewing angle;

[0130] S25B2: Convert the second three-dimensional coordinates of the j-th object point to the world coordinate system to obtain the three-dimensional coordinates of the j-th object point in the object to be measured at the selected viewing angle;

[0131] S25B3: Traverse each object point in the object to be measured, and obtain the three-dimensional coordinates of each object point in the object to be measured under the selected viewing angle according to the same method.

[0132] The two methods above share the same principle, and you can choose one based on your needs. Taking the first method as an example, it utilizes the principle of triangulation, and can use feature matching points and camera parameters to construct the three-dimensional coordinates of an object in space. This method is a non-contact measurement method, which can avoid measurement errors caused by contact, has high measurement accuracy, and can obtain three-dimensional coordinate data in a short time. It has high measurement efficiency and is easy to implement automated measurement.

[0133] For the first method, in step S25A1, the specific formula for calculating the first three-dimensional coordinates of the j-th object point in the coordinate system of the i-th first camera image is:

[0134] ;

[0135] in, is the first three-dimensional coordinate of the j-th object point in the coordinate system of the i-th first camera image, is the first pixel coordinate of the j-th object point in the i-th first camera image, (c x ,c y ) is the camera principal point coordinate in the binocular 3D endoscope, Z i is the depth information of the object to be measured at the i-th viewing angle, and f is the focal length of the two cameras in the binocular 3D endoscope;

[0136] For the second method, in step S25B1, the specific formula for calculating the second three-dimensional coordinates of the j-th object point in the coordinate system where the i-th second camera image is located is:

[0137] ;

[0138] in, is the second three-dimensional coordinate of the j-th object point in the coordinate system of the i-th second camera image, is the second pixel coordinate of the j-th object point in the i-th second camera image, (c x ,c y ) is the camera principal point coordinate in the binocular 3D endoscope, Z i is the depth information of the object to be measured at the i-th viewing angle, and f is the focal length of the two cameras in the binocular 3D endoscope.

[0139] Based on the parameters of the two cameras in the binocular 3D endoscope, the first three-dimensional coordinate or the second three-dimensional coordinate can be obtained using the principle of triangulation. The principle is simple and easy to implement.

[0140] Taking the first method as an example, the first three-dimensional coordinate of the j-th object point in the coordinate system of the i-th first camera image is The specific process of converting the first three-dimensional coordinates into the world coordinate system in step S25A2 is as follows:

[0141] ;

[0142] in, is the coordinate of the first three-dimensional coordinate of the j-th object point converted to the world coordinate system, representing the three-dimensional coordinate of the j-th object point at the i-th viewing angle; R L and T L are respectively the rotation matrix and the translation matrix for transforming the i-th coordinate system of the second camera image into the world coordinate system;

[0143] By expanding the above equation, we can get The three coordinate values ​​in .

[0144] The process of step S25B2 in the second method is similar to this and will not be repeated here.

[0145] The three-dimensional coordinates of each object point at the selected viewing angle are achieved according to the above method.

[0146] Specifically, the above step S26 includes:

[0147] S261: Traversing the three-dimensional coordinates of all object points in the object to be measured at the selected viewing angle to obtain the coordinates of the boundary points of the object to be measured at the selected viewing angle;

[0148] S262: Calculate the length information and the width information of the object to be measured at the selected viewing angle according to the coordinates of the boundary points of the object to be measured at the selected viewing angle.

[0149] The above method can directly and efficiently describe the boundary of the object to be measured, and then fit the boundary of the object to be measured, and efficiently calculate the length and width information.

[0150] In this embodiment, the coordinates of the boundary points are obtained by traversing the above step S261, which can be implemented by edge detection, contour extraction or bounding box methods.

[0151] Assume that the binocular image under the selected viewing angle is the i-th binocular image in the binocular image sequence, and the boundary point coordinates of the object to be measured under the i-th viewing angle include the maximum coordinate value in the x-direction, the minimum coordinate value in the x-direction, the maximum coordinate value in the y-direction, and the minimum coordinate value in the y-direction;

[0152] In step S262 of this embodiment, the specific formula for calculating the length information and the width information of the object to be measured at the i-th viewing angle is:

[0153] ;

[0154] Among them, L x,i and L y,i are the length information and width information of the object to be measured at the i-th viewing angle, respectively, max,i 、x min,i 、y max,i and y min,i are the maximum x-direction coordinate value, minimum x-direction coordinate value, maximum y-direction coordinate value, and minimum y-direction coordinate value of the object to be measured at the i-th viewing angle, respectively.

[0155] Preferably, the target size includes a target depth, a target length and a target width of the object to be measured;

[0156] This embodiment S3 includes:

[0157] S31: performing a weighted average operation on the depth information under all viewing angles to obtain the target depth of the object to be measured;

[0158] The specific formula for performing weighted average calculation on the depth information under all viewing angles is:

[0159] ;

[0160] Wherein, Z is the target depth, Z i is the depth information of the object to be measured at the i-th viewing angle, w i is the weight under the i-th perspective, and n is the total number of perspectives;

[0161] S32: performing a weighted average operation on the length information at all viewing angles to obtain the target length of the object to be measured;

[0162] The specific formula for performing weighted average calculation on the length information under all viewing angles is:

[0163] ;

[0164] Among them, L x is the target length, L x,i is the length information of the object to be measured at the i-th viewing angle;

[0165] S33: performing a weighted average operation on the width information under all viewing angles to obtain the target width of the object to be measured;

[0166] The specific formula for performing weighted average calculation on the width information under all viewing angles is:

[0167] ;

[0168] Among them, L y is the target width, L y,i is the width information of the object to be measured at the i-th viewing angle.

[0169] In the process of data fusion, the depth information, length information and width information from multiple perspectives are fused separately by weighted averaging, which can effectively reduce the impact of blind spots and occlusion problems, and the obtained size information is more accurate.

[0170] Example 2

[0171] A binocular 3D endoscope target size detection system is applied to the binocular 3D endoscope target size detection method of embodiment 1, such as Figure 3 As shown, the system includes:

[0172] A viewing angle adjustment module, used to adjust the viewing angle of the binocular 3D endoscope;

[0173] An image acquisition module, configured to acquire a binocular image sequence of the object to be measured captured by the binocular 3D endoscope during the viewing angle adjustment process; wherein the binocular image sequence is composed of binocular images of the object to be measured at multiple viewing angles;

[0174] A size calculation module is used to calculate the size information of the object to be measured at each viewing angle based on the binocular image at each viewing angle;

[0175] The size fusion module is used to fuse the size information under all viewing angles to obtain the target size of the object to be measured.

[0176] In this embodiment, the perspective of the binocular 3D endoscope is adjusted by the perspective adjustment module, and then the binocular images of the object to be measured at multiple different perspectives are acquired by the image acquisition module, which can provide image information of the binocular 3D endoscope at different perspectives; then, the size calculation module is used to process the binocular images at each perspective to measure the size information of the object to be measured at each perspective, and finally, the size information at all perspectives is fused by the size fusion module to obtain the target size containing multi-angle information, thereby realizing the size measurement of the object to be measured, and is not limited to the depth information calculated from the image at a single perspective. It can effectively avoid considering only the situation where the baseline length between the two cameras in the binocular 3D endoscope is short or long at a certain perspective, and also avoid considering only the situation where the object to be measured is occluded at a certain perspective. Through multi-perspective data fusion, the image information of the object to be measured with a more complex structure at different perspectives can be fully considered. Finally, the target size obtained by data fusion has higher accuracy, is more consistent with the actual situation of the object, and can adapt to the size measurement of objects in different environments and complex shapes.

[0177] The functions of each module in the binocular 3D endoscope target size detection system described in this embodiment are the same as the method steps of the binocular 3D endoscope target size detection method described in Example 1. Therefore, for details not covered in this embodiment, please refer to Example 1 and Figure 1 and Figure 2 The detailed description is omitted here.

[0178] Example 3

[0179] This embodiment also provides a binocular 3D endoscope target size detection device, including a processor, a memory, and a computer program stored in the memory and executable on the processor. When the computer program is executed, the method steps in the binocular 3D endoscope target size detection method of the first embodiment are implemented.

[0180] By using a computer program stored in a memory and running on a processor, the size measurement of the object to be measured is achieved, and is not limited to the depth information calculated from the image at a single perspective. This can effectively avoid the situation where only the baseline length between the two cameras in a binocular 3D endoscope is shorter or longer at a certain perspective, and can also avoid the situation where only the occlusion of the object to be measured at a certain perspective is considered. Through multi-perspective data fusion, the image information of the object to be measured with a more complex structure at different perspectives can be fully considered. Ultimately, the target size obtained through data fusion has higher accuracy, is more consistent with the actual situation of the object, and can adapt to the size measurement of objects in different environments and complex shapes.

[0181] The processor may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor. The processor is the control center of a computer device, connecting all parts of the entire computer device using various interfaces and circuits.

[0182] The memory can be used to store computer programs and / or models. The processor implements the various functions of the computer device by running or executing the computer programs and / or models stored in the memory and accessing the data stored in the memory. The memory may primarily include a program storage area and a data storage area. The program storage area may store an operating system and at least one application required for a function (e.g., a sound playback function, an image playback function, etc.); the data storage area may store data generated based on the use of the mobile phone (e.g., audio data, video data, etc.). Furthermore, the memory may include high-speed random access memory and non-volatile memory, such as a hard disk, internal memory, a plug-in hard disk, a SmartMedia Card (SMC), a Secure Digital (SD) card, a flash card, at least one disk storage device, a flash memory device, or other volatile solid-state storage device.

[0183] It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by a computer program. These computer programs can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0184] These computer programs can also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0185] These computer programs can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable device to implement the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0186] This embodiment further provides a computer storage medium, which includes: at least one instruction, which implements the method steps in the binocular 3D endoscope target size detection method of embodiment 1 when the instruction is executed by a computer.

[0187] By executing a computer storage medium containing at least one instruction, the size measurement of the object to be measured is achieved, and it is not limited to the depth information calculated from the image at a single perspective. This can effectively avoid the situation where the baseline length between the two cameras in the binocular 3D endoscope is short or long at a certain perspective, and can also avoid the situation where the object to be measured is occluded at a certain perspective. Through multi-perspective data fusion, the image information of the object to be measured with a more complex structure at different perspectives can be fully considered. Ultimately, the target size obtained through data fusion has higher accuracy, is more consistent with the actual situation of the object, and can adapt to the size measurement of objects in different environments and complex shapes.

[0188] Similarly, for details not yet provided in this embodiment, please refer to Embodiment 1, Embodiment 2 and Figures 1 to 3 The detailed description is omitted here.

[0189] Although the embodiments of the present invention have been described with reference to the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present invention. Such modifications and variations are all within the scope defined by the appended claims.

Claims

1. A binocular 3D endoscope target size detection method, characterized in that: The method comprises: Adjusting the viewing angle of the binocular 3D endoscope and obtaining a binocular image sequence of the object to be measured captured by the binocular 3D endoscope during the viewing angle adjustment process; wherein the binocular image sequence is composed of binocular images of the object to be measured at multiple viewing angles; Calculating the size information of the object under test at each viewing angle based on the binocular image at each viewing angle, wherein the size information includes depth information, and the depth information is calculated using a stereo disparity method; the size information also includes length information and width information; Performing data fusion on the size information under all viewing angles to obtain the target size of the object to be measured; The binocular 3D endoscope includes a first camera and a second camera that are fixedly connected; adjusting the viewing angle of the binocular 3D endoscope includes using a linear translation method to translate the first camera and the second camera together, so that the viewing angle of the binocular 3D endoscope changes, thereby obtaining multiple viewing angles of the binocular 3D endoscope; The target size includes the target depth, target length and target width of the object to be measured; The step of fusing the size information at all viewing angles to obtain the target size of the object to be measured includes: Performing a weighted average operation on the depth information under all viewing angles to obtain the target depth of the object to be measured; The specific formula for performing weighted average calculation on the depth information under all viewing angles is: ； Wherein, Z is the target depth, Z i is the depth information of the object to be measured at the i-th viewing angle, w i is the weight under the i-th perspective, and n is the total number of perspectives; Performing a weighted average operation on the length information at all viewing angles to obtain the target length of the object to be measured; The specific formula for weighted averaging the length information at all viewing angles is: ； Among them, L x is the target length, L x,i is the length information of the object to be measured at the i-th viewing angle; Performing a weighted average operation on the width information at all viewing angles to obtain the target width of the object to be measured; The specific formula for weighted averaging the width information under all viewing angles is: ； Among them, L y is the target width, L y,i is the width information of the object to be measured at the i-th viewing angle.

2. The binocular 3D endoscope target size detection method according to claim 1, characterized in that: The adjusting the viewing angle of the binocular 3D endoscope further includes: The object to be measured is translated by a linear translation method, so that the viewing angle of the binocular 3D endoscope changes, thereby obtaining multiple viewing angles of the binocular 3D endoscope.

3. The binocular 3D endoscope target size detection method according to claim 1, characterized in that: The step of calculating the size information of the object to be measured at each viewing angle based on the binocular image at each viewing angle includes: Select the binocular image under any one viewing angle, and pre-process the selected binocular image; Performing feature extraction and feature matching on the preprocessed binocular image in sequence to obtain a first eigenvalue and a second eigenvalue of the object to be measured in the preprocessed binocular image; Performing parallax calculation based on the first eigenvalue and the second eigenvalue to obtain a parallax value of the object to be measured at a selected viewing angle; Reconstructing a disparity map according to the disparity value of the object to be measured at the selected viewing angle, and calculating the depth information of the object to be measured at the selected viewing angle; Performing three-dimensional reconstruction based on the depth information of the object to be measured at the selected viewing angle to obtain the three-dimensional coordinates of each object point in the object to be measured at the selected viewing angle; Calculating the length information and the width information of the object to be measured at the selected viewing angle according to the three-dimensional coordinates of all object points in the object to be measured at the selected viewing angle; The binocular images at each viewing angle are traversed, and the size information of the object to be measured at each viewing angle is calculated respectively using the same method.

4. The binocular 3D endoscope target size detection method according to claim 3, characterized in that: Each of the binocular images includes a first camera image and a second camera image; The first eigenvalue is specifically a first horizontal coordinate value of a target object point in the object to be measured in the preprocessed first camera image, and the second eigenvalue is specifically a second horizontal coordinate value of the target object point in the preprocessed second camera image.

5. The binocular 3D endoscope target size detection method according to claim 4, characterized in that: Assume that the binocular image under the selected viewing angle is the i-th binocular image in the binocular image sequence, and the i-th binocular image includes the i-th first camera image and the i-th second camera image; The specific formula for calculating the parallax value of the object to be measured at the selected viewing angle is: ； Among them, d i is the parallax value of the object to be measured at the i-th viewing angle, x i,L is the first horizontal coordinate value of the target object point in the object to be measured in the i-th first camera image, x i,R is the second horizontal coordinate value of the target object point in the object to be measured in the i-th second camera image.

6. The binocular 3D endoscope target size detection method according to claim 5, characterized in that: The specific formula for calculating the depth information of the object to be measured at the selected viewing angle is: ； Among them, Z i is the depth information of the object to be measured at the i-th viewing angle, f is the focal length of the two cameras in the binocular 3D endoscope, B i is the baseline length between the two cameras in the binocular 3D endoscope at the i-th viewing angle.

7. The binocular 3D endoscope target size detection method according to claim 3, characterized in that: The step of calculating the size information of the object to be measured at the selected viewing angle based on the three-dimensional coordinates of all object points in the object to be measured at the selected viewing angle includes: Traversing the three-dimensional coordinates of all object points in the object to be measured at the selected viewing angle to obtain the coordinates of the boundary points of the object to be measured at the selected viewing angle; The length information and the width information of the object to be measured at the selected viewing angle are respectively calculated based on the coordinates of the boundary points of the object to be measured at the selected viewing angle.

8. The binocular 3D endoscope target size detection method according to claim 7, characterized in that: Assume that the binocular image under the selected viewing angle is the i-th binocular image in the binocular image sequence, and the boundary point coordinates of the object to be measured under the i-th viewing angle include the maximum coordinate value in the x-direction, the minimum coordinate value in the x-direction, the maximum coordinate value in the y-direction, and the minimum coordinate value in the y-direction; The specific formula for calculating the length information and the width information of the object to be measured at the i-th viewing angle is: ； Among them, L x,i and L y,i are the length information and width information of the object to be measured at the i-th viewing angle, respectively, max,i 、x min,i 、y max,i and y min,i are the maximum x-direction coordinate value, minimum x-direction coordinate value, maximum y-direction coordinate value, and minimum y-direction coordinate value of the object to be measured at the i-th viewing angle, respectively.

9. A binocular 3D endoscope target size detection system, characterized in that: Applied to the binocular 3D endoscope target size detection method according to any one of claims 1 to 8, the system comprises: A viewing angle adjustment module, used to adjust the viewing angle of the binocular 3D endoscope; An image acquisition module, configured to acquire a binocular image sequence of the object to be measured captured by the binocular 3D endoscope during the viewing angle adjustment process; wherein the binocular image sequence is composed of binocular images of the object to be measured at multiple viewing angles; A size calculation module is used to calculate the size information of the object to be measured at each viewing angle based on the binocular image at each viewing angle; The size fusion module is used to fuse the size information under all viewing angles to obtain the target size of the object to be measured.

10. A binocular 3D endoscope target size detection device, characterized in that: The method comprises a processor, a memory and a computer program stored in the memory and executable on the processor, wherein the computer program implements the method steps in the binocular 3D endoscope target size detection method according to any one of claims 1 to 8 when the computer program is executed.

11. A computer storage medium, characterized in that The computer storage medium comprises: at least one instruction, which implements the method steps in the binocular 3D endoscope target size detection method according to any one of claims 1 to 8 when the instruction is executed by a computer.

Images