Binocular 3D endoscope target size detection method, system and device and medium

Abstract

The invention discloses a binocular 3D endoscope target size detection method, system and device and a medium, and the method comprises the steps: adjusting the visual angle of a binocular 3D endoscope, and obtaining a binocular image sequence of a to-be-detected object collected by the binocular 3D endoscope in the visual angle adjustment process; wherein the binocular image sequence is composed of binocular images of the to-be-measured object at a plurality of visual angles; according to the binocular image under each visual angle, size information of the to-be-measured object under each visual angle is obtained through calculation; and carrying out data fusion on the size information under all the visual angles to obtain the target size of the to-be-measured object. According to the invention, the size measurement of the object to be measured is realized, the precision of the obtained target size is higher through multi-view data fusion, the target size is more in line with the real situation of the object, and the method can adapt to the size measurement of objects in different environments and complex shapes.

Description

Technical Field

[0001] The present invention relates to the field of medical equipment, and in particular to a binocular 3D endoscope target size detection method, system, device and medium. Background Art

[0002] Binocular 3D endoscope is an endoscope device that uses binocular stereoscopic vision principle to achieve three-dimensional imaging. It uses two cameras (or sensors) to capture images of the same scene from different angles, and combines computer algorithms to obtain disparity maps, thereby calculating the depth information of the object and generating a three-dimensional image.

[0003] The three-dimensional images generated by binocular 3D endoscopes can help doctors more clearly judge the distance, shape, size, etc. of objects. During surgery, doctors can use three-dimensional images to more accurately locate the location of lesions, organs or tissues, reducing the risk of misoperation. In various surgeries such as internal medicine, surgery, otolaryngology, and urology, binocular three-dimensional endoscopes provide doctors with a clearer field of vision, especially in minimally invasive surgery, which can accurately locate the lesion area and reduce the patient's trauma and postoperative recovery time.

[0004] Traditional two-dimensional endoscopes can only provide planar images. Therefore, binocular 3D endoscopes have significant advantages over traditional two-dimensional endoscopes, especially in precise positioning, depth perception and stereoscopic observation.

[0005] However, binocular 3D endoscopes also have some defects and limitations. When calculating parallax, they need to use the baseline of the binocular vision system, which refers to the physical distance between the two cameras. A shorter baseline may result in smaller parallax, thereby reducing the accuracy of depth measurement. A longer baseline will result in a larger parallax during measurement, but it may also bring larger errors, especially in the measurement of farther objects. When part of the object is obscured, the parallax calculation may become difficult, and the depth information of the obscured part cannot be accurately calculated. In addition, the surface of the object may be complex in structure or have too many details, which makes it difficult to calculate the parallax of the local area, affecting the accuracy of the overall size measurement. Summary of the invention

[0006] In view of this, the present invention provides a binocular 3D endoscope target size detection method, system, device and medium to solve the problem that the existing binocular 3D endoscope is easily affected by factors such as inappropriate baseline, object occlusion and complex object structure when measuring target size, resulting in low overall measurement accuracy.

[0007] The present invention provides a binocular 3D endoscope target size detection method, the method comprising:

[0008] Adjusting the viewing angle of the binocular 3D endoscope, and acquiring a binocular image sequence of the object to be measured captured by the binocular 3D endoscope during the viewing angle adjustment process; wherein the binocular image sequence is composed of binocular images of the object to be measured at multiple viewing angles;

[0009] According to the binocular image at each viewing angle, the size information of the object to be measured at each viewing angle is calculated respectively;

[0010] The size information at all viewing angles is fused to obtain the target size of the object to be measured.

[0011] Optionally, the binocular 3D endoscope comprises a first camera and a second camera that are fixedly connected;

[0012] The adjusting the viewing angle of the binocular 3D endoscope comprises:

[0013] Using a linear translation method, the first camera and the second camera are translated together, so that the viewing angle of the binocular 3D endoscope changes, and multiple viewing angles of the binocular 3D endoscope are obtained;

[0014] And / or, a linear translation method is used to translate the object to be measured, so that the viewing angle of the binocular 3D endoscope changes, thereby obtaining multiple viewing angles of the binocular 3D endoscope.

[0015] Optionally, the size information includes depth information, length information and width information;

[0016] The step of calculating the size information of the object to be measured at each viewing angle according to the binocular image at each viewing angle comprises:

[0017] Select the binocular image under any viewing angle, and pre-process the selected binocular image;

[0018] Performing feature extraction and feature matching on the preprocessed binocular image in sequence to obtain a first eigenvalue and a second eigenvalue of the object to be measured in the preprocessed binocular image;

[0019] Performing parallax calculation according to the first eigenvalue and the second eigenvalue to obtain a parallax value of the object to be measured at a selected viewing angle;

[0020] Reconstructing a disparity map according to the disparity value of the object to be measured at the selected viewing angle, and calculating the depth information of the object to be measured at the selected viewing angle;

[0021] Performing three-dimensional reconstruction according to the depth information of the object to be measured at the selected viewing angle to obtain the three-dimensional coordinates of each object point in the object to be measured at the selected viewing angle;

[0022] Calculating the length information and the width information of the object to be measured at the selected viewing angle according to the three-dimensional coordinates of all object points in the object to be measured at the selected viewing angle;

[0023] The binocular images at each viewing angle are traversed, and the size information of the object to be measured at each viewing angle is calculated respectively according to the same method.

[0024] Optionally, each of the binocular images includes a first camera image and a second camera image;

[0025] The first eigenvalue is specifically a first horizontal coordinate value of a target object point in the object to be measured in the first camera image after preprocessing, and the second eigenvalue is specifically a second horizontal coordinate value of the target object point in the second camera image after preprocessing.

[0026] Optionally, the binocular image under the selected viewing angle is assumed to be the i-th binocular image in the binocular image sequence, and the i-th binocular image includes the i-th first camera image and the i-th second camera image;

[0027] The specific formula for calculating the parallax value of the object to be measured under the selected viewing angle is:

[0028] ;

[0029] Among them, d i is the disparity value of the object to be measured at the i-th viewing angle, x i,L is the first horizontal coordinate value of the target object point in the object to be measured in the i-th first camera image, x i,R is the second horizontal coordinate value of the target object point in the object to be measured in the i-th second camera image.

[0030] Optionally, a specific formula for calculating the depth information of the object to be measured at a selected viewing angle is:

[0031] ;

[0032] Among them, Z i is the depth information of the object to be measured at the i-th viewing angle, f is the focal length of the two cameras in the binocular 3D endoscope, B i is the baseline length between the two cameras in the binocular 3D endoscope at the i-th viewing angle.

[0033] Optionally, each of the binocular images includes a first camera image and a second camera image; assuming that the binocular image under the selected viewing angle is the i-th binocular image in the binocular image sequence, the i-th binocular image includes the i-th first camera image and the i-th second camera image;

[0034] The three-dimensional reconstruction is performed according to the depth information of the object to be measured at the selected viewing angle to obtain the three-dimensional coordinates of each object point in the object to be measured at the selected viewing angle, including:

[0035] Selecting a j-th object point in the object to be measured, obtaining a first pixel coordinate of the j-th object point in the i-th first camera image, and calculating a first three-dimensional coordinate of the j-th object point in the coordinate system of the i-th first camera image according to the first pixel coordinate of the j-th object point and the depth information of the object to be measured at the selected viewing angle;

[0036] Convert the first three-dimensional coordinate of the j-th object point to the world coordinate system to obtain the three-dimensional coordinate of the j-th object point in the object to be measured under the selected viewing angle;

[0037] Traversing each object point in the object to be measured, and obtaining the three-dimensional coordinates of each object point in the object to be measured under the selected viewing angle in the same way;

[0038] Alternatively, a j-th object point in the object to be measured is selected, a second pixel coordinate of the j-th object point in the i-th second camera image is obtained, and a second three-dimensional coordinate of the j-th object point in the coordinate system of the i-th second camera image is calculated according to the second pixel coordinate of the j-th object point and the depth information of the object to be measured at the selected viewing angle;

[0039] Convert the second three-dimensional coordinates of the j-th object point to the world coordinate system to obtain the three-dimensional coordinates of the j-th object point in the object to be measured under the selected viewing angle;

[0040] Each object point in the object to be measured is traversed, and the three-dimensional coordinates of each object point in the object to be measured under the selected viewing angle are obtained in the same way.

[0041] Optionally, a specific formula for calculating the first three-dimensional coordinates of the j-th object point in the coordinate system where the i-th first camera image is located is:

[0042] ;

[0043] in, is the first three-dimensional coordinate of the j-th object point in the coordinate system of the i-th first camera image, is the first pixel coordinate of the jth object point in the i-th first camera image, (c x ,c y ) is the camera principal point coordinate in the binocular 3D endoscope, Z i is the depth information of the object to be measured at the i-th viewing angle, and f is the focal length of two cameras in the binocular 3D endoscope;

[0044] Alternatively, the specific formula for calculating the second three-dimensional coordinates of the j-th object point in the coordinate system where the i-th second camera image is located is:

[0045] ;

[0046] in, is the second three-dimensional coordinate of the j-th object point in the coordinate system of the i-th second camera image, is the second pixel coordinate of the jth object point in the i-th second camera image, (c x ,c y ) is the camera principal point coordinate in the binocular 3D endoscope, Z i is the depth information of the object to be measured at the i-th viewing angle, and f is the focal length of the two cameras in the binocular 3D endoscope.

[0047] Optionally, the calculating the size information of the object to be measured at the selected viewing angle according to the three-dimensional coordinates of all object points in the object to be measured at the selected viewing angle includes:

[0048] Traversing the three-dimensional coordinates of all object points in the object to be measured at the selected viewing angle to obtain the coordinates of the boundary points of the object to be measured at the selected viewing angle;

[0049] The length information and the width information of the object to be measured at the selected viewing angle are respectively calculated according to the boundary point coordinates of the object to be measured at the selected viewing angle.

[0050] Optionally, assuming that the binocular image under the selected viewing angle is the i-th binocular image in the binocular image sequence, the boundary point coordinates of the object to be measured under the i-th viewing angle include a maximum coordinate value in the x-direction, a minimum coordinate value in the x-direction, a maximum coordinate value in the y-direction, and a minimum coordinate value in the y-direction;

[0051] The specific formulas for calculating the length information and the width information of the object to be measured at the i-th viewing angle are:

[0052] ;

[0053] Among them, L x,i and L y,iare the length information and the width information of the object to be measured at the i-th viewing angle, respectively, max,i 、x min,i ,y max,i and min,i are respectively the maximum x-direction coordinate value, the minimum x-direction coordinate value, the maximum y-direction coordinate value and the minimum y-direction coordinate value of the object to be measured at the ith viewing angle.

[0054] Optionally, the target size includes a target depth, a target length and a target width of the object to be measured;

[0055] The step of fusing the size information at all viewing angles to obtain the target size of the object to be measured includes:

[0056] Performing a weighted average operation on the depth information under all viewing angles to obtain the target depth of the object to be measured;

[0057] The specific formula for performing weighted average calculation on the depth information under all viewing angles is:

[0058] ;

[0059] Wherein, Z is the target depth, Z i is the depth information of the object to be measured at the i-th viewing angle, w i is the weight under the i-th perspective, and n is the total number of perspectives;

[0060] Performing a weighted average operation on the length information at all viewing angles to obtain the target length of the object to be measured;

[0061] The specific formula for performing weighted average calculation on the length information at all viewing angles is:

[0062] ;

[0063] Among them, L x is the target length, L x,i is the length information of the object to be measured at the i-th viewing angle;

[0064] Performing a weighted average operation on the width information at all viewing angles to obtain the target width of the object to be measured;

[0065] The specific formula for performing weighted average calculation on the width information under all viewing angles is:

[0066] ;

[0067] Among them, L y is the target width, Ly,i is the width information of the object to be measured at the i-th viewing angle.

[0068] In addition, the present invention also provides a binocular 3D endoscope target size detection system, which is applied to the aforementioned binocular 3D endoscope target size detection method, and the system comprises:

[0069] A viewing angle adjustment module, used to adjust the viewing angle of the binocular 3D endoscope;

[0070] An image acquisition module, used to acquire a binocular image sequence of the object to be measured captured by the binocular 3D endoscope during the viewing angle adjustment process; wherein the binocular image sequence is composed of binocular images of the object to be measured at multiple viewing angles;

[0071] A size calculation module, used to calculate the size information of the object to be measured at each viewing angle according to the binocular image at each viewing angle;

[0072] The size fusion module is used to perform data fusion on the size information under all viewing angles to obtain the target size of the object to be measured.

[0073] In addition, the present invention also provides a binocular 3D endoscope target size detection device, including a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the computer program implements the method steps in the aforementioned binocular 3D endoscope target size detection method when executed.

[0074] In addition, the present invention also provides a computer storage medium, which includes: at least one instruction, which implements the method steps in the aforementioned binocular 3D endoscope target size detection method when the instruction is executed by a computer.

[0075] The beneficial effects of the present invention are as follows: by adjusting the viewing angle of the binocular 3D endoscope, binocular images of the object to be measured at multiple different viewing angles are collected, and image information of the binocular 3D endoscope at different viewing angles can be provided; then the binocular images at each viewing angle are processed, and the size information of the object to be measured at each viewing angle is measured, and finally, by performing data fusion on the size information at all viewing angles, a target size containing multi-angle information can be obtained, and the size measurement of the object to be measured is realized, and the depth information calculated from the image at a single viewing angle is not limited. This can effectively avoid the situation where only the baseline length between the two cameras in the binocular 3D endoscope at a certain viewing angle is shorter or longer, and also avoid the situation where only the occlusion of the object to be measured at a certain viewing angle is considered. Through multi-view data fusion, the image information of the object to be measured with a more complex structure at different viewing angles can be fully considered, and the target size finally obtained through data fusion has higher accuracy, is more in line with the actual situation of the object, and can adapt to the size measurement of objects in different environments and with complex shapes. BRIEF DESCRIPTION OF THE DRAWINGS

[0076] The features and advantages of the present invention will be more clearly understood by referring to the accompanying drawings, which are schematic and should not be construed as limiting the present invention in any way. In the accompanying drawings:

[0077] Figure 1 A flow chart of a binocular 3D endoscope target size detection method in Embodiment 1 of the present invention is shown;

[0078] Figure 2 A flowchart of calculating size information at each viewing angle in the first embodiment of the present invention is shown;

[0079] Figure 3 The structure diagram of a binocular 3D endoscope target size detection system in the second embodiment of the present invention is shown. DETAILED DESCRIPTION

[0080] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0081] Embodiment 1

[0082] This embodiment provides a binocular 3D endoscope target size detection method, such as Figure 1 As shown, the method includes:

[0083] S1: adjusting the viewing angle of the binocular 3D endoscope, and obtaining a binocular image sequence of the object to be measured captured by the binocular 3D endoscope during the viewing angle adjustment process; wherein the binocular image sequence is composed of binocular images of the object to be measured at multiple viewing angles;

[0084] S2: Calculate the size information of the object to be measured at each viewing angle according to the binocular image at each viewing angle;

[0085] S3: performing data fusion on the size information under all viewing angles to obtain the target size of the object to be measured.

[0086] In this embodiment, by adjusting the viewing angle of the binocular 3D endoscope, binocular images of the object to be measured at multiple different viewing angles are collected, and image information of the binocular 3D endoscope at different viewing angles can be provided; then the binocular images at each viewing angle are processed, and the size information of the object to be measured at each viewing angle is measured. Finally, by performing data fusion on the size information at all viewing angles, the target size containing multi-angle information can be obtained, and the size measurement of the object to be measured is realized, and it is not limited to the depth information calculated from the image at a single viewing angle. It can effectively avoid the situation where only the baseline length between the two cameras in the binocular 3D endoscope at a certain viewing angle is shorter or longer, and it can also avoid the situation where only the occlusion of the object to be measured at a certain viewing angle is considered. Through multi-view data fusion, the image information of the object to be measured with a more complex structure at different viewing angles can be fully considered. Finally, the target size obtained by data fusion has higher accuracy, is more in line with the actual situation of the object, and can adapt to the size measurement of objects in different environments and complex shapes.

[0087] Each step of the binocular 3D endoscope target size detection method of this embodiment is further described below.

[0088] In this embodiment S1, the binocular 3D endoscope includes a first camera and a second camera that are fixedly connected.

[0089] Through the first camera and the second camera, the same scene can be observed from different angles to capture images, thereby conveniently utilizing the parallax between the images to infer the depth information of the object and achieve size measurement.

[0090] Preferably, in this embodiment S1, adjusting the viewing angle of the binocular 3D endoscope includes:

[0091] Using a linear translation method, the first camera and the second camera are translated together, so that the viewing angle of the binocular 3D endoscope changes, and multiple viewing angles of the binocular 3D endoscope are obtained;

[0092] And / or, a linear translation method is used to translate the object to be measured, so that the viewing angle of the binocular 3D endoscope changes, thereby obtaining multiple viewing angles of the binocular 3D endoscope.

[0093] In this embodiment, two aspects are considered to adjust the viewing angle of the binocular 3D endoscope. The first aspect is that the observed object (i.e., the object to be measured) does not move while the binocular camera (i.e., the first camera and the second camera) moves. The second aspect is that the object to be measured moves while the binocular camera does not move. When the object to be measured does not move while the binocular camera moves, the angle at which the binocular camera observes the object to be measured will change as the binocular camera moves, thereby generating multiple viewing angles; when the object to be measured moves while the binocular camera does not move, the angle at which the binocular camera observes the object to be measured will also change as the object to be measured moves, thereby generating multiple viewing angles.

[0094] Of course, we can also consider that the object to be measured moves and the binocular camera also moves. At this time, it is necessary to ensure that the movement of the object to be measured and the movement of the binocular camera are not synchronized, that is, it is necessary to ensure that the two move in different directions or at different speeds. When the two move at the same time and the movements are not synchronized (that is, the moving directions are different or the moving speeds are different), the angle of the object to be measured observed by the binocular camera will also change, and multiple perspectives can also be generated.

[0095] In this embodiment, the linear translation method refers to controlling the target to be moved (the binocular camera and / or the object to be measured) to move in a certain linear direction, and the linear direction can be set according to actual conditions. For example, the binocular camera is controlled to translate in the horizontal direction, or the binocular camera is controlled to translate in the vertical direction.

[0096] During the movement of the binocular camera and / or the object to be measured, each time the binocular camera moves to a position point, a viewing angle is generated, and the binocular camera collects a binocular image under the viewing angle once. Similarly, a binocular image sequence of the object to be measured collected by the binocular 3D endoscope during the viewing angle adjustment process can be obtained.

[0097] This embodiment uses a stereo disparity method when subsequently calculating depth information, and uses a linear translation method when providing viewing angles. By combining these two methods, the actual size of the object can be accurately calculated, which can effectively improve the accuracy of the final size measurement.

[0098] Preferably, if Figure 2 As shown, the size information includes depth information, length information and width information; this embodiment S2 includes:

[0099] S21: Select the binocular image under any viewing angle, and pre-process the selected binocular image;

[0100] S22: performing feature extraction and feature matching on the preprocessed binocular image in sequence to obtain a first feature value and a second feature value of the object to be measured in the preprocessed binocular image;

[0101] S23: performing parallax calculation according to the first eigenvalue and the second eigenvalue to obtain a parallax value of the object to be measured at a selected viewing angle;

[0102] S24: reconstructing a disparity map according to the disparity value of the object to be measured at the selected viewing angle, and calculating the depth information of the object to be measured at the selected viewing angle;

[0103] S25: performing three-dimensional reconstruction according to the depth information of the object to be measured at the selected viewing angle to obtain the three-dimensional coordinates of each object point in the object to be measured at the selected viewing angle;

[0104] S26: Calculating the length information and the width information of the object to be measured at the selected viewing angle according to the three-dimensional coordinates of all object points in the object to be measured at the selected viewing angle;

[0105] S27: traverse the binocular images at each viewing angle, and calculate the size information of the object to be measured at each viewing angle respectively according to the same method.

[0106] When using the stereo disparity method to calculate the depth information at each viewing angle, firstly, by preprocessing the binocular image, the image quality can be effectively improved and the image features can be enhanced, the interference of noise and other interference factors on the subsequent feature extraction and feature matching can be reduced, and the useful information in the image can be highlighted, which is helpful to extract the first eigenvalue and the second eigenvalue more accurately in the subsequent process, thereby improving the accuracy of the depth information. After the binocular image is preprocessed, through feature extraction, representative features can be extracted, the amount of data can be reduced, and the subsequent processing efficiency can be improved; then, through feature matching, it can be ensured that the extracted features are the features of the same object points in the two binocular images, thereby ensuring that the subsequent calculation is reliable, and ensuring the reliability of the depth information calculated by the stereo disparity method. After that, the depth information of the object to be measured at each viewing angle can be accurately measured by disparity calculation and disparity map reconstruction; after calculating the depth information at each viewing angle, the depth information can be used to realize three-dimensional reconstruction, and the three-dimensional coordinates of each object point in the object to be measured at each viewing angle can be obtained. In addition to being related to the depth information of the object, the three-dimensional coordinates are also directly related to the length information and width information of the object, so the three-dimensional coordinates can be calculated to obtain the length information and width information of the object to be measured at each viewing angle.

[0107] Specifically, in the above step S21, the preprocessing performed on the selected binocular image includes but is not limited to image processing methods such as denoising processing and grayscale processing.

[0108] The binocular image at each viewing angle includes a first camera image and a second camera image.

[0109] Specifically, in the above step S22, the specific method of feature extraction and feature matching can adopt SIFT (Speeded Up Robust Features) scale-invariant feature transformation method, SURF (Speeded Up Robust Features) accelerated robust feature method or ORB (Oriented FAST and Rotated BRIEF) fast feature and rotation binary robust independent basic feature method and other feature extraction methods. This embodiment specifically adopts the SURF feature extraction method, that is, in the pre-processed binocular image (including the first camera image and the second camera image), the SURF features in the first camera image and the second camera image are respectively extracted, the Hessian matrix is ​​constructed to generate interest points, and then the scale space is constructed to locate the feature points and generate descriptors. By comparing the descriptors of the feature points in the two images, similar feature point pairs are found and used as matching points, that is, complete feature matching.

[0110] Preferably, in order to perform feature matching, the first eigenvalue and the second eigenvalue of the same target object point in the two images are extracted respectively. When the position of the same target object point (i.e., the matching point) in the two images is matched, the position of the matching point can be refined at the sub-pixel level by using an interpolation method, and then the posture of the two cameras and the coordinates of the object to be measured in the two camera images are globally optimized by using a bundle adjustment method, so as to eliminate the accumulation of local errors and improve the overall measurement accuracy. Common interpolation methods include bilinear interpolation and bicubic interpolation. Taking bilinear interpolation as an example, it calculates the grayscale value of the sub-pixel position by the grayscale value of the four surrounding pixels, so as to more accurately determine the specific position of the matching point in the pixel, reduce the error caused by the discreteness of the pixel, and ensure that the position of the matching point is accurate. The first eigenvalue and the second eigenvalue in the two images are more accurate and reliable. Among them, the bundle adjustment method is used to globally optimize the posture of the two cameras and the coordinates of the object to be measured in the two camera images. The specific implementation method adopts a conventional method, which is not limited in this embodiment.

[0111] Specifically, in the above step S22, the first eigenvalue is specifically the first horizontal coordinate value of the target object point in the object to be measured in the first camera image after preprocessing, and the second eigenvalue is specifically the second horizontal coordinate value of the target object point in the second camera image after preprocessing.

[0112] In a binocular vision system, the optical axes of two cameras (usually left and right cameras) are usually parallel and on the same horizontal plane. Therefore, for the same target object point, the change of its ordinate in the left and right images is relatively small, while the difference in the abscissa is mainly caused by the different viewing angles of the left and right cameras. This difference directly reflects the depth information of the object, that is, parallax. Therefore, in this embodiment, the first abscissa value and the second abscissa value in the two images are used as the first eigenvalue and the second eigenvalue, respectively, for subsequent parallax calculation and depth information calculation, which conforms to the basic principle of obtaining depth information through the difference in the horizontal direction of the left and right images in binocular vision, and the calculated result is more accurate and reliable.

[0113] The target object point mentioned above can be selected from all object points of the object to be measured according to actual conditions.

[0114] Specifically, in the above step S23, the binocular image under the selected viewing angle is assumed to be the i-th binocular image in the binocular image sequence, and the i-th binocular image includes the i-th first camera image and the i-th second camera image;

[0115] The specific formula for calculating the parallax value of the object to be measured under the selected viewing angle is:

[0116] ;

[0117] Among them, d i is the disparity value of the object to be measured at the i-th viewing angle, x i,L is the first horizontal coordinate value of the target object point in the object to be measured in the i-th first camera image, x i,R is the second horizontal coordinate value of the target object point in the object to be measured in the i-th second camera image.

[0118] Specifically, in the above step S24, the specific formula for calculating the depth information of the object to be measured at the selected viewing angle is:

[0119] ;

[0120] Among them, Z i is the depth information of the object to be measured at the i-th viewing angle, f is the focal length of the two cameras in the binocular 3D endoscope, B i is the baseline length between the two cameras in the binocular 3D endoscope at the i-th viewing angle.

[0121] The depth information under each viewing angle is calculated by the above formula, which is more in line with the working principle of binocular 3D endoscope. The method is simple and the result is accurate and reliable.

[0122] Among them, the focal length and baseline length of the two cameras are inherent parameters of the binocular 3D endoscope and can be known in advance.

[0123] Specifically, the above step S25 includes two methods. The first method is to use the first camera as a reference and use the information in the first camera image to calculate; the second method is to use the second camera as a reference and use the information in the second camera image to calculate.

[0124] In the first method, S25 includes:

[0125] S25A1: Select the jth object point in the object to be measured, obtain the first pixel coordinates of the jth object point in the ith first camera image, and calculate the first three-dimensional coordinates of the jth object point in the coordinate system of the ith first camera image according to the first pixel coordinates of the jth object point and the depth information of the object to be measured at the selected viewing angle;

[0126] S25A2: converting the first three-dimensional coordinate of the j-th object point into the world coordinate system to obtain the three-dimensional coordinate of the j-th object point in the object to be measured under the selected viewing angle;

[0127] S25A3: Traverse each object point in the object to be measured, and obtain the three-dimensional coordinates of each object point in the object to be measured under the selected viewing angle in the same way.

[0128] In the second method, S25 includes:

[0129] S25B1: Select the jth object point in the object to be measured, obtain the second pixel coordinates of the jth object point in the i-th second camera image, and calculate the second three-dimensional coordinates of the jth object point in the coordinate system of the i-th second camera image according to the second pixel coordinates of the jth object point and the depth information of the object to be measured at the selected viewing angle;

[0130] S25B2: converting the second three-dimensional coordinates of the j-th object point into the world coordinate system to obtain the three-dimensional coordinates of the j-th object point in the object to be measured under the selected viewing angle;

[0131] S25B3: Traverse each object point in the object to be measured, and obtain the three-dimensional coordinates of each object point in the object to be measured under the selected viewing angle in the same way.

[0132] The principles of the above two methods are the same, and one of them can be selected according to actual needs. Taking the first method as an example, the above method uses the principle of triangulation, and can use feature matching points and camera parameters to construct the three-dimensional coordinates of the object in space. This method is a non-contact measurement method, which can avoid measurement errors caused by contact, has high measurement accuracy, and can obtain three-dimensional coordinate data in a short time. It has high measurement efficiency and is easy to realize automated measurement.

[0133] For the first method, in the above step S25A1, the specific formula for calculating the first three-dimensional coordinates of the j-th object point in the coordinate system where the i-th first camera image is located is:

[0134] ;

[0135] in, is the first three-dimensional coordinate of the j-th object point in the coordinate system of the i-th first camera image, is the first pixel coordinate of the jth object point in the i-th first camera image, (c x ,c y ) is the camera principal point coordinate in the binocular 3D endoscope, Z i is the depth information of the object to be measured at the i-th viewing angle, and f is the focal length of two cameras in the binocular 3D endoscope;

[0136] For the second method, in the above step S25B1, the specific formula for calculating the second three-dimensional coordinates of the j-th object point in the coordinate system where the i-th second camera image is located is:

[0137] ;

[0138] in, is the second three-dimensional coordinate of the j-th object point in the coordinate system of the i-th second camera image, is the second pixel coordinate of the jth object point in the i-th second camera image, (c x ,c y ) is the camera principal point coordinate in the binocular 3D endoscope, Z i is the depth information of the object to be measured at the i-th viewing angle, and f is the focal length of the two cameras in the binocular 3D endoscope.

[0139] Based on the parameters of the two cameras in the binocular 3D endoscope, the first three-dimensional coordinate or the second three-dimensional coordinate can be obtained by using the triangulation principle. The principle is simple and easy to implement.

[0140] Taking the first method as an example, the first three-dimensional coordinate of the jth object point in the coordinate system of the i-th first camera image is The specific process of converting the first three-dimensional coordinates into the world coordinate system in step S25A2 is as follows:

[0141] ;

[0142] in, is the coordinate of the first three-dimensional coordinate of the j-th object point converted to the world coordinate system, representing the three-dimensional coordinate of the j-th object point at the i-th viewing angle; R L and T L are respectively the rotation matrix and the translation matrix for transforming the i-th coordinate system where the second camera image is located into the world coordinate system;

[0143] By expanding the above equation, we can get The three coordinate values ​​in .

[0144] The process of step S25B2 in the second method is similar to this and will not be repeated here.

[0145] The three-dimensional coordinates of each object point at the selected viewing angle are realized according to the above method.

[0146] Specifically, the above step S26 includes:

[0147] S261: traversing the three-dimensional coordinates of all object points in the object to be measured at the selected viewing angle to obtain the coordinates of the boundary points of the object to be measured at the selected viewing angle;

[0148] S262: According to the coordinates of the boundary points of the object to be measured at the selected viewing angle, respectively calculate the length information and the width information of the object to be measured at the selected viewing angle.

[0149] The above method can directly and efficiently describe the boundary of the object to be measured, and then fit the boundary of the object to be measured, and efficiently calculate the length information and width information.

[0150] In this embodiment, the above step S261 traverses to obtain the coordinates of the boundary points, which can be implemented by edge detection, contour extraction or bounding box methods.

[0151] Assuming that the binocular image under the selected viewing angle is the i-th binocular image in the binocular image sequence, the boundary point coordinates of the object to be measured under the i-th viewing angle include the maximum coordinate value in the x-direction, the minimum coordinate value in the x-direction, the maximum coordinate value in the y-direction, and the minimum coordinate value in the y-direction;

[0152] In the above step S262 of this embodiment, the specific formula for calculating the length information and the width information of the object to be measured at the i-th viewing angle is:

[0153] ;

[0154] Among them, L x,i and L y,i are the length information and the width information of the object to be measured at the i-th viewing angle, respectively, max,i 、x min,i ,y max,i and min,i are respectively the maximum x-direction coordinate value, the minimum x-direction coordinate value, the maximum y-direction coordinate value and the minimum y-direction coordinate value of the object to be measured at the ith viewing angle.

[0155] Preferably, the target size includes a target depth, a target length and a target width of the object to be measured;

[0156] This embodiment S3 includes:

[0157] S31: performing a weighted average operation on the depth information under all viewing angles to obtain the target depth of the object to be measured;

[0158] The specific formula for performing weighted average calculation on the depth information under all viewing angles is:

[0159] ;

[0160] Wherein, Z is the target depth, Z i is the depth information of the object to be measured at the i-th viewing angle, w i is the weight under the i-th perspective, and n is the total number of perspectives;

[0161] S32: performing a weighted average operation on the length information at all viewing angles to obtain the target length of the object to be measured;

[0162] The specific formula for performing weighted average calculation on the length information at all viewing angles is:

[0163] ;

[0164] Among them, L x is the target length, L x,i is the length information of the object to be measured at the i-th viewing angle;

[0165] S33: performing a weighted average operation on the width information at all viewing angles to obtain the target width of the object to be measured;

[0166] The specific formula for performing weighted average calculation on the width information under all viewing angles is:

[0167] ;

[0168] Among them, L y is the target width, L y,i is the width information of the object to be measured at the i-th viewing angle.

[0169] In the process of data fusion, the depth information, length information and width information under multiple perspectives are fused separately by weighted averaging, which can effectively reduce the impact of blind spots and occlusion problems, and the size information obtained is more accurate.

[0170] Embodiment 2

[0171] A binocular 3D endoscope target size detection system is applied to the binocular 3D endoscope target size detection method of embodiment 1, such as Figure 3 As shown, the system comprises:

[0172] A viewing angle adjustment module, used to adjust the viewing angle of the binocular 3D endoscope;

[0173] An image acquisition module, used to acquire a binocular image sequence of the object to be measured captured by the binocular 3D endoscope during the viewing angle adjustment process; wherein the binocular image sequence is composed of binocular images of the object to be measured at multiple viewing angles;

[0174] A size calculation module, used to calculate the size information of the object to be measured at each viewing angle according to the binocular image at each viewing angle;

[0175] The size fusion module is used to perform data fusion on the size information under all viewing angles to obtain the target size of the object to be measured.

[0176] In this embodiment, the viewing angle of the binocular 3D endoscope is adjusted by the viewing angle adjustment module, and then the binocular images of the object to be measured at multiple different viewing angles are acquired by the image acquisition module, so that the image information of the binocular 3D endoscope at different viewing angles can be provided; then the binocular images at each viewing angle are processed by the size calculation module, and the size information of the object to be measured at each viewing angle is measured, and finally the size information at all viewing angles is fused by the size fusion module, so that the target size containing multi-angle information can be obtained, and the size measurement of the object to be measured can be realized, and it is not limited to the depth information calculated by the image at a single viewing angle. It can effectively avoid the situation where only the baseline length between the two cameras in the binocular 3D endoscope at a certain viewing angle is short or long, and it can also avoid the situation where only the object to be measured is blocked at a certain viewing angle. Through multi-view data fusion, the image information of the object to be measured with a more complex structure at different viewing angles can be fully considered, and the target size finally obtained by data fusion has higher accuracy, is more in line with the actual situation of the object, and can adapt to the size measurement of objects in different environments and complex shapes.

[0177] The functions of each module in the binocular 3D endoscope target size detection system described in this embodiment are the same as the method steps of the binocular 3D endoscope target size detection method described in Example 1. Therefore, for details not included in this embodiment, please refer to Example 1 and Figure 1 and Figure 2 The detailed description will not be repeated here.

[0178] Embodiment 3

[0179] This embodiment also provides a binocular 3D endoscope target size detection device, including a processor, a memory, and a computer program stored in the memory and executable on the processor, and when the computer program is executed, the method steps in the binocular 3D endoscope target size detection method of the first embodiment are implemented.

[0180] By storing a computer program in a memory and running it on a processor, the size measurement of the object to be measured is achieved, and it is not limited to the depth information calculated from the image at a single viewing angle. This can effectively avoid the situation where only the baseline length between the two cameras in a binocular 3D endoscope is shorter or longer at a certain viewing angle, and can also avoid the situation where only the occlusion of the object to be measured at a certain viewing angle is considered. Through multi-view data fusion, the image information of the object to be measured with a more complex structure at different viewing angles can be fully considered. Finally, the target size obtained through data fusion has higher accuracy, is more in line with the actual situation of the object, and can adapt to the size measurement of objects in different environments and complex shapes.

[0181] The processor may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc. The processor is the control center of a computer device, and uses various interfaces and lines to connect various parts of the entire computer device.

[0182] The memory can be used to store computer programs and / or models. The processor realizes various functions of the computer device by running or executing the computer programs and / or models stored in the memory and calling the data stored in the memory. The memory can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system and at least one application required for a function (such as a sound playback function, an image playback function, etc.); the data storage area can store data created according to the use of the mobile phone (such as audio data, video data, etc.). In addition, the memory can include a high-speed random access memory and can also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart memory card (SmartMedia Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0183] It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by a computer program. These computer programs can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0184] These computer programs may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which implements the process Figure 1A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0185] These computer programs can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0186] This embodiment further provides a computer storage medium, which includes: at least one instruction, which implements the method steps in the binocular 3D endoscope target size detection method of embodiment 1 when the instruction is executed by a computer.

[0187] By executing a computer storage medium containing at least one instruction, the size measurement of the object to be measured is achieved, and it is not limited to the depth information calculated from the image at a single viewing angle. This can effectively avoid the situation where only the baseline length between the two cameras in a binocular 3D endoscope is shorter or longer at a certain viewing angle, and can also avoid the situation where only the occlusion of the object to be measured at a certain viewing angle is considered. Through multi-view data fusion, the image information of the object to be measured with a more complex structure at different viewing angles can be fully considered. Ultimately, the target size obtained through data fusion has higher accuracy, is more in line with the actual situation of the object, and can adapt to the size measurement of objects in different environments and complex shapes.

[0188] Similarly, for details not yet provided in this embodiment, please refer to Embodiment 1, Embodiment 2 and Figures 1 to 3 The detailed description will not be repeated here.

[0189] Although the embodiments of the present invention have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present invention, and such modifications and variations are all within the scope defined by the appended claims.

Claims

1. A binocular 3D endoscope target size detection method, characterized in that: The method comprises: Adjusting the viewing angle of the binocular 3D endoscope, and acquiring a binocular image sequence of the object to be measured captured by the binocular 3D endoscope during the viewing angle adjustment process; wherein the binocular image sequence is composed of binocular images of the object to be measured at multiple viewing angles; According to the binocular image at each viewing angle, the size information of the object to be measured at each viewing angle is calculated respectively; The size information at all viewing angles is fused to obtain the target size of the object to be measured.

2. The binocular 3D endoscope target size detection method according to claim 1, characterized in that: The binocular 3D endoscope includes a first camera and a second camera that are fixedly connected; The adjusting the viewing angle of the binocular 3D endoscope comprises: Using a linear translation method, the first camera and the second camera are translated together, so that the viewing angle of the binocular 3D endoscope changes, and multiple viewing angles of the binocular 3D endoscope are obtained; And / or, a linear translation method is used to translate the object to be measured, so that the viewing angle of the binocular 3D endoscope changes, thereby obtaining multiple viewing angles of the binocular 3D endoscope.

3. The binocular 3D endoscope target size detection method according to claim 1, characterized in that: The size information includes depth information, length information and width information; The step of calculating the size information of the object to be measured at each viewing angle according to the binocular image at each viewing angle comprises: Select the binocular image under any viewing angle, and pre-process the selected binocular image; Performing feature extraction and feature matching on the preprocessed binocular image in sequence to obtain a first eigenvalue and a second eigenvalue of the object to be measured in the preprocessed binocular image; Performing parallax calculation according to the first eigenvalue and the second eigenvalue to obtain a parallax value of the object to be measured at a selected viewing angle; Reconstructing a disparity map according to the disparity value of the object to be measured at the selected viewing angle, and calculating the depth information of the object to be measured at the selected viewing angle; Performing three-dimensional reconstruction according to the depth information of the object to be measured at the selected viewing angle to obtain the three-dimensional coordinates of each object point in the object to be measured at the selected viewing angle; Calculating the length information and the width information of the object to be measured at the selected viewing angle according to the three-dimensional coordinates of all object points in the object to be measured at the selected viewing angle; The binocular images at each viewing angle are traversed, and the size information of the object to be measured at each viewing angle is calculated respectively according to the same method.

4. The binocular 3D endoscope target size detection method according to claim 3, characterized in that: Each of the binocular images includes a first camera image and a second camera image; The first eigenvalue is specifically a first horizontal coordinate value of a target object point in the object to be measured in the first camera image after preprocessing, and the second eigenvalue is specifically a second horizontal coordinate value of the target object point in the second camera image after preprocessing.

5. The binocular 3D endoscope target size detection method according to claim 4, characterized in that: Assume that the binocular image under the selected viewing angle is the i-th binocular image in the binocular image sequence, and the i-th binocular image includes the i-th first camera image and the i-th second camera image; The specific formula for calculating the parallax value of the object to be measured under the selected viewing angle is: ； Among them, d i is the disparity value of the object to be measured at the i-th viewing angle, x i,L is the first horizontal coordinate value of the target object point in the object to be measured in the i-th first camera image, x i,R is the second horizontal coordinate value of the target object point in the object to be measured in the i-th second camera image.

6. The binocular 3D endoscope target size detection method according to claim 5, characterized in that: The specific formula for calculating the depth information of the object to be measured at the selected viewing angle is: ； Among them, Z i is the depth information of the object to be measured at the i-th viewing angle, f is the focal length of the two cameras in the binocular 3D endoscope, B i is the baseline length between the two cameras in the binocular 3D endoscope at the i-th viewing angle.

7. The binocular 3D endoscope target size detection method according to claim 3, characterized in that: The step of calculating the size information of the object to be measured at the selected viewing angle according to the three-dimensional coordinates of all object points in the object to be measured at the selected viewing angle includes: Traversing the three-dimensional coordinates of all object points in the object to be measured at the selected viewing angle to obtain the coordinates of the boundary points of the object to be measured at the selected viewing angle; The length information and the width information of the object to be measured at the selected viewing angle are respectively calculated according to the boundary point coordinates of the object to be measured at the selected viewing angle.

8. The binocular 3D endoscope target size detection method according to claim 7, characterized in that: Assuming that the binocular image under the selected viewing angle is the i-th binocular image in the binocular image sequence, the boundary point coordinates of the object to be measured under the i-th viewing angle include the maximum coordinate value in the x-direction, the minimum coordinate value in the x-direction, the maximum coordinate value in the y-direction, and the minimum coordinate value in the y-direction; The specific formula for calculating the length information and the width information of the object to be measured at the i-th viewing angle is: ； Among them, L x,i and L y,i are the length information and the width information of the object to be measured at the i-th viewing angle, respectively, max,i 、x min,i ,y max,i and min,i are respectively the maximum x-direction coordinate value, the minimum x-direction coordinate value, the maximum y-direction coordinate value and the minimum y-direction coordinate value of the object to be measured at the ith viewing angle.

9. The binocular 3D endoscope target size detection method according to claim 3, characterized in that: The target size includes the target depth, target length and target width of the object to be measured; The step of fusing the size information at all viewing angles to obtain the target size of the object to be measured includes: Performing a weighted average operation on the depth information under all viewing angles to obtain the target depth of the object to be measured; The specific formula for performing weighted average calculation on the depth information under all viewing angles is: ； Wherein, Z is the target depth, Z i is the depth information of the object to be measured at the i-th viewing angle, w i is the weight under the i-th perspective, and n is the total number of perspectives; Performing a weighted average operation on the length information at all viewing angles to obtain the target length of the object to be measured; The specific formula for performing weighted average calculation on the length information at all viewing angles is: ； Among them, L x is the target length, L x,i is the length information of the object to be measured at the i-th viewing angle; Performing a weighted average operation on the width information at all viewing angles to obtain the target width of the object to be measured; The specific formula for performing weighted average calculation on the width information under all viewing angles is: ； Among them, L y is the target width, L y,i is the width information of the object to be measured at the i-th viewing angle.

10. A binocular 3D endoscope target size detection system, characterized in that: Applied to the binocular 3D endoscope target size detection method according to any one of claims 1 to 9, the system comprises: A viewing angle adjustment module, used to adjust the viewing angle of the binocular 3D endoscope; An image acquisition module, used to acquire a binocular image sequence of the object to be measured captured by the binocular 3D endoscope during the viewing angle adjustment process; wherein the binocular image sequence is composed of binocular images of the object to be measured at multiple viewing angles; A size calculation module, used to calculate the size information of the object to be measured at each viewing angle according to the binocular image at each viewing angle; The size fusion module is used to perform data fusion on the size information under all viewing angles to obtain the target size of the object to be measured.

11. A binocular 3D endoscope target size detection device, characterized in that: The method comprises a processor, a memory and a computer program stored in the memory and executable on the processor, wherein the computer program implements the method steps in the binocular 3D endoscope target size detection method as claimed in any one of claims 1 to 9 when running.

12. A computer storage medium, characterized in that: The computer storage medium comprises: at least one instruction, which, when executed by a computer, implements the method steps in the binocular 3D endoscope target size detection method according to any one of claims 1 to 9.

Images