A curve measurement method, device and computer equipment based on a binocular camera

Abstract

The application relates to a curve measurement method, device and computer equipment based on a binocular camera. The method comprises the following steps: collecting multiple pairs of left and right views of to-be-measured objects at different time points; detecting an identification point in each pair of left and right views of to-be-measured objects to obtain first identification point coordinates and second identification point coordinates in each view; determining reference left and right views in the left and right views of to-be-measured objects, adjusting the coordinates of the identification points in all the left and right views of to-be-measured objects based on first identification point reference coordinates in the reference left and right views, and obtaining first target identification point coordinates and second target identification point coordinates in each view; performing three-dimensional coordinate conversion according to the target identification point coordinates in the left and right views of to-be-measured objects to obtain corresponding first identification point three-dimensional coordinates and second identification point three-dimensional coordinates located on a to-be-measured curve; and sequentially connecting the three-dimensional coordinates of the identification points to obtain curve information of the to-be-measured curve. The method can improve the accuracy and efficiency of curve measurement.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to a curve measurement method, apparatus and computer equipment based on a binocular camera. Background Technology

[0002] Machine vision is a rapidly developing branch of artificial intelligence. It aims to use machines to simulate human eyes for image acquisition, then process and recognize the information to ultimately perform operations through actuators. With the continuous development of computer technology, machine vision technology, with its advantages of high precision, real-time performance, automation, and intelligence, is gradually penetrating many industries such as manufacturing, agriculture, transportation, and healthcare.

[0003] Machine vision technology has a wide range of applications in medical image analysis. Traditional medical image analysis requires doctors to spend a lot of time and effort to observe and interpret the images, while machine vision technology can quickly and accurately identify and mark abnormal areas in medical images through algorithms and models, helping doctors to make diagnoses and treatments.

[0004] In clinical endoscopy, accurate assessment of lesion size is crucial for diagnosis. Spatial linear measurement of the lesion can determine its diameter, providing important guidance for surgical selection. However, lesion shapes are highly variable, and measuring surface curves can provide doctors with more three-dimensional information and calculate the perimeter of the lesion area. Current techniques for measuring lesion area curves mostly rely on uniform sampling between the two endpoints of the curve, calculating the curve length by solving for the distances of multiple line segments. However, this method is significantly affected by the sampling frequency, failing to balance accuracy and computational complexity, and also requires high image texture and exhibits poor robustness.

[0005] Currently, no effective solution has been proposed for the problems of existing technologies in curve measurement, which cannot balance accuracy and computational load, as well as poor robustness. Summary of the Invention

[0006] Therefore, it is necessary to provide a curve measurement method, apparatus, and computer equipment based on a binocular camera to address the aforementioned technical problems.

[0007] Firstly, this application provides a curve measurement method based on a binocular camera. The method includes:

[0008] Multiple left and right views of the target object are obtained at different times. Each of the left and right views of the target object includes a first marker point and a second marker point on the preset curve to be measured on the target object. The first marker point in each left and right view of the target object is located at the preset endpoint of the curve to be measured. The second marker point in different left and right views of the target object is located at different positions on the curve to be measured. The coordinates of the second marker point at all times cover the preset target point position in the curve to be measured.

[0009] The first and second marker points in each left and right view to be measured are detected respectively to obtain the coordinates of the marker points in each view;

[0010] Determine the reference left and right views in the left and right views to be measured, and adjust the coordinates of the first and second marker points in all the left and right views to be measured based on the reference coordinates of the first marker point in the reference left and right views, so as to obtain the coordinates of the first and second target marker points in each view respectively.

[0011] Perform a three-dimensional coordinate transformation on the coordinates of the first and second target markers in the left and right views to be measured, and obtain the corresponding three-dimensional coordinates of the first and second markers located on the curve to be measured.

[0012] By sequentially connecting the three-dimensional coordinates of the first marker point and each of the second marker points according to the shooting time sequence, the curve information of the curve to be measured can be obtained.

[0013] In one embodiment, the left and right views to be measured include a left view to be measured and a right view to be measured. The detection of a first marker point and a second marker point in each of the left and right views to be measured includes:

[0014] For each left and right view to be measured, an inspection is performed to determine the first and second auxiliary devices included in each view. The first auxiliary device is fixed at a preset endpoint, and the second auxiliary device at different times corresponds to different positions on the curve to be measured.

[0015] The corresponding first and second marker points are determined by the front ends of the first and second auxiliary instruments in each left and right view to be measured.

[0016] In one embodiment, the first and second marker points in each of the left and right views to be measured are detected respectively to obtain the coordinates of the marker points in each view, including:

[0017] Each left view and each right view to be measured are input into a fully trained detection neural network to detect the position information of the two auxiliary devices in each view, as well as the front end point of each auxiliary device.

[0018] The coordinates of the corresponding two-dimensional marker point in each view are obtained through the front end of the assistive device.

[0019] In one embodiment, the first and second marker points in each of the left and right views to be measured are detected respectively to obtain the coordinates of the marker points in each view, including:

[0020] For each left and right view to be measured, an error is reported if the number of marker points in the view is not equal to 2.

[0021] If the number of marker points detected in the view is equal to 2, then the coordinates of the marker points in each view are obtained.

[0022] In one embodiment, the coordinates of the first and second marker points in all the left and right views to be measured are adjusted based on the reference coordinates of the first marker point in the reference left and right views, respectively, to obtain the coordinates of the first and second target marker points in each view, including:

[0023] Compare the coordinates of the first marker point in each left and right view to be measured with the reference coordinates of the first marker point, and calculate the coordinate deviation value corresponding to the coordinates of each first marker point.

[0024] The coordinates of the first marker point are adjusted based on each coordinate deviation value until the coordinates of the first marker point in each left and right view to be measured are equal to the reference coordinates of the first marker point, thus obtaining the coordinates of the first target marker point.

[0025] The coordinates of the corresponding second marker point are adjusted based on each coordinate deviation value to obtain the coordinates of the corresponding second target marker point. The target marker point coordinates include the coordinates of the first target marker point and the coordinates of the second target marker point.

[0026] In one embodiment, the left and right views to be measured include a left view and a right view; the coordinates of the first target marker point and the second target marker point in the left and right views to be measured are transformed into three-dimensional coordinates to obtain the corresponding three-dimensional coordinates of the first marker point and the second marker point located on the curve to be measured, including:

[0027] Calculate the disparity information of the coordinates of the first target marker and the second target marker between the left view to be measured and the right view to be measured, and determine the depth information of each marker based on the disparity information;

[0028] Based on the depth information, the coordinates of the first target marker point and the second target marker point are transformed into three-dimensional coordinates to obtain the three-dimensional coordinates of the first marker point and the second marker point respectively.

[0029] In one embodiment, the three-dimensional coordinates of the first marker point and the three-dimensional coordinates of each of the second marker points are sequentially connected according to the shooting time sequence to obtain the curve information of the curve to be measured, including:

[0030] By connecting the three-dimensional coordinates of adjacent second marker points in the order of shooting time, the path information of the curve to be measured is obtained.

[0031] The sum of the line segment lengths between the three-dimensional coordinates of adjacent marker points and second marker points is calculated to obtain the length information of the curve to be measured. The curve information includes path information and length information.

[0032] In one embodiment, the method further includes:

[0033] The second auxiliary device moves along the curve to be measured according to a preset movement rule at different times. The movement rule includes a preset movement speed and a preset movement space rule. The movement space rule includes one of the following: moving from top to bottom, moving from bottom to top, moving from left to right, and moving from right to left.

[0034] Secondly, this application also provides a curve measuring device. The device includes:

[0035] The acquisition module is used to acquire multiple left and right views of the target object at different times. Each view of the target object includes a first marker point and a second marker point on a preset curve to be measured. The first marker point in each left and right view is located at a preset endpoint of the curve to be measured. The second marker points in different left and right views are located at different positions on the curve to be measured. The coordinates of the second marker points at all times cover the preset target points on the curve to be measured.

[0036] The calculation module is used to detect the first and second marker points in each of the left and right views to be measured, and obtain the coordinates of the marker points in each view; determine the reference left and right views in the left and right views to be measured, and adjust the coordinates of the first and second marker points in all the left and right views to be measured based on the reference coordinates of the first marker points in the reference left and right views, and obtain the coordinates of the first and second target marker points in each view; perform three-dimensional coordinate transformation on the coordinates of the first and second target marker points in the left and right views to be measured, and obtain the corresponding three-dimensional coordinates of the first and second marker points located on the curve to be measured.

[0037] The generation module is used to sequentially connect the three-dimensional coordinates of the first marker point and the three-dimensional coordinates of each second marker point according to the shooting time sequence to obtain the curve information of the curve to be measured.

[0038] Thirdly, this application also provides a computer device. The computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to perform the following steps:

[0039] Multiple left and right views of the target object are obtained at different times. Each of the left and right views of the target object includes a first marker point and a second marker point on the preset curve to be measured on the target object. The first marker point in each left and right view of the target object is located at the preset endpoint of the curve to be measured. The second marker point in different left and right views of the target object is located at different positions on the curve to be measured. The coordinates of the second marker point at all times cover the preset target point position in the curve to be measured.

[0040] The first and second marker points in each left and right view to be measured are detected respectively to obtain the coordinates of the marker points in each view;

[0041] Determine the reference left and right views in the left and right views to be measured, and adjust the coordinates of the first and second marker points in all the left and right views to be measured based on the reference coordinates of the first marker point in the reference left and right views, so as to obtain the coordinates of the first and second target marker points in each view respectively.

[0042] Perform a three-dimensional coordinate transformation on the coordinates of the first and second target markers in the left and right views to be measured, and obtain the corresponding three-dimensional coordinates of the first and second markers located on the curve to be measured.

[0043] By sequentially connecting the three-dimensional coordinates of the first marker point and each of the second marker points according to the shooting time sequence, the curve information of the curve to be measured can be obtained.

[0044] The aforementioned method, apparatus, and computer equipment for curve measurement based on a binocular camera first acquire multiple left and right views of the target object at different times. Each pair of views includes a first and a second marker point located on a pre-defined curve to be measured on the target object. The first marker point in each left and right view is located at a pre-defined endpoint of the curve to be measured. The second marker points in different left and right views are located at different positions on the curve to be measured, and the coordinates of the second marker points at all times cover the pre-defined target points on the curve to be measured. The first and second marker points in each left and right view are detected to obtain the coordinates of the marker points in each view. Next, a reference left and right view is determined in the left and right views to be measured, and the coordinates of the first and second marker points in all left and right views to be measured are adjusted based on the reference coordinates of the first marker points in the reference left and right views to obtain the coordinates of the first and second target marker points in each view. Finally, a three-dimensional coordinate transformation is performed on the coordinates of the first and second target marker points in the left and right views to obtain the corresponding three-dimensional coordinates of the first and second marker points located on the curve to be measured. This application allows for the flexible identification of path changes in the curve under test based on the location of the second marker point at multiple time points, without the need for additional measurement equipment, effectively reducing measurement costs and required computing resources. Furthermore, this application compensates for the offset of marker point coordinates in other left and right views under test based on the reference coordinates of the first marker point, in order to address phenomena such as image jitter in multi-frame image processing, thereby effectively improving the accuracy and robustness of the measurement. Attached Figure Description

[0045] Figure 1 This is a flowchart illustrating a curve measurement method in one embodiment;

[0046] Figure 2 This is a flowchart illustrating a curve measurement method in a preferred embodiment;

[0047] Figure 3 This is a structural block diagram of a curve measuring device in one embodiment;

[0048] Figure 4 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0049] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0050] In one embodiment, such as Figure 1As shown, a curve measurement method based on a binocular camera is provided, including the following steps:

[0051] Step S110: Obtain multiple left and right views of the target object at different times. Each of the left and right views of the target object includes a first marker point and a second marker point on the preset curve to be measured on the target object. The first marker point in each left and right view of the target object is located at a preset endpoint of the curve to be measured. The second marker point in different left and right views of the target object is located at different positions on the curve to be measured. The coordinates of the second marker point at all times cover the preset target point position on the curve to be measured.

[0052] Specifically, the left and right views to be measured in this application include a left view and a right view. The first marker point in the left view and the first marker point in the right view, acquired at the same time, are actually at the same three-dimensional coordinate point in three-dimensional space. Similarly, the second marker point in the left view and the second marker point in the right view, acquired at the same time, are also at the same three-dimensional coordinate point in three-dimensional space. However, due to lateral parallax in the left and right views captured by a binocular camera, there are slight differences in the coordinates of the first marker point in the left and right views acquired at the same time, and similarly, there are slight differences in the coordinates of the second marker point in the left and right views acquired at the same time. In practical applications, the curve to be measured is located on the target object and can be the edge curve of the target object that needs to be measured, or a curve within the range of the target object that needs to be measured. Furthermore, the marker point is a point on the curve to be measured. This marker point can be an actual marker point pre-set by relevant technicians, or it can be a virtual point used to assist in the measurement.

[0053] This embodiment requires acquiring multiple frames of the left and right views to be measured at different times in chronological order. Each left and right view includes a first marker point and a second marker point. The position of the first marker point remains constant in each left and right view, located at a preset endpoint of the curve to be measured. However, the coordinates of the second marker point in the left and right views acquired at different times are different. In practical applications, the second marker point can be controlled to move along the curve to be measured until its coordinates cover multiple preset target points on the curve at all times. During this process, the first and second marker points are photographed using a binocular camera, thereby acquiring the aforementioned multiple frames of the left and right views to be measured. The method of controlling the movement of the second marker point along the path of the curve to be measured includes, but is not limited to, having a doctor or technician mark the movement path of the second marker point sequentially on the curve to be measured, or having a doctor or technician slide surgical instruments along the path of the curve to be measured at a relatively slow speed. Since the marker point cannot completely cover the curve to be measured, multiple target points can be set in practical applications. These target points can be virtual auxiliary points or actual points preset by relevant technicians. Multiple target points that need to be focused on are preset on the curve to be tested, such as the corners of the curve, points near the corners, and parts of the curve with large fluctuations. Alternatively, multiple target points can be set on the curve to be tested according to preset point intervals. In summary, the obtained second marker points should include at least the above-mentioned target points.

[0054] Step S120: Detect the first and second marker points in each of the left and right views to be measured, and obtain the marker point coordinates in each view; determine the reference left and right views in the left and right views to be measured, and adjust the coordinates of the first and second marker points in all the left and right views to be measured based on the reference coordinates of the first marker points in the reference left and right views, and obtain the first target marker point coordinates and the second target marker point coordinates in each view.

[0055] Specifically, the first and second marker points in each left and right view to be measured are detected. In some preferred embodiments, the detection of marker points includes, but is not limited to, identifying marker points using a trained neural network to obtain the marker point coordinates corresponding to each marker point in each view. Further, since multiple left and right views to be measured are captured sequentially at multiple time points in this application, a pair of left and right views is selected as the reference left and right views. The selection strategy for the reference left and right views includes, but is not limited to, using the clearest frame in the left and right views to be measured as the reference left and right views, or using the left and right views that are at the end of the time frame as the aforementioned reference left and right views. After the reference left and right views are selected, the coordinates of the first marker point in the reference left and right views are used as the reference coordinates of the first marker point. The coordinates of the first marker point and the second marker point in the remaining frames of the left and right views to be measured are adjusted according to the reference coordinates of the first marker point to obtain the adjusted coordinates of the first and second target marker points in each left and right view to be measured. As those skilled in the art will understand based on this application, since the coordinates of the first marker point remain fixed and are all located at the preset endpoint of the curve to be measured, the coordinates of the first marker point in each frame of the left and right views to be measured captured at different times should ideally be the same. However, in practical applications, due to factors such as the patient's breathing, the overall image may shift. Therefore, it is necessary to adjust the coordinates of the first and second marker points in the remaining left and right views to be measured based on the reference left and right views. After adjustment, the coordinates of the first marker point in the first target marker point coordinates at different times should be the same. This first marker point is the marker point fixed at the preset endpoint of the curve to be measured. The adjustment method based on the reference coordinates of the first marker point includes, but is not limited to, calculating the coordinate deviation value between the coordinates of the first marker point in each left and right view to be measured and the reference coordinates of the first marker point, and adjusting the corresponding coordinates of the first marker point based on each coordinate deviation value until the coordinates of the first marker point in each left and right view to be measured are equal to the reference coordinates of the first marker point, thus obtaining the coordinates of the first target marker point; and adjusting the corresponding coordinates of the second marker point based on each coordinate deviation value to obtain the coordinates of the second target marker point.

[0056] Step S130: Perform three-dimensional coordinate transformation on the coordinates of the first target marker point and the second target marker point in the left and right views to be measured, and obtain the corresponding three-dimensional coordinates of the first marker point and the second marker point located on the curve to be measured; connect the three-dimensional coordinates of the first marker point and each of the second marker points in sequence according to the shooting time order to obtain the curve information of the curve to be measured.

[0057] Specifically, the coordinates of the first and second target marker points in each of the left and right views to be measured are transformed into three-dimensional coordinates. The two-dimensional coordinates of the first and second target marker points are converted into three-dimensional coordinates of the first and second marker points. This means that the three-dimensional coordinates of the marker points, compared to the two-dimensional coordinates, include an additional depth coordinate relative to the stereo camera. Finally, according to the shooting time sequence of each frame of the left and right views to be measured, the corresponding three-dimensional coordinates of the first and second marker points are sequentially connected. Since the three-dimensional coordinates of the first marker points should be the same at each shooting time after adjustment, the sequential connection of the three-dimensional coordinates of the first and second marker points is performed in the following order: first marker point three-dimensional coordinates, second marker point three-dimensional coordinates obtained from the first shot, second marker point three-dimensional coordinates obtained from the second shot, second marker point three-dimensional coordinates obtained from the third shot, and so on. This yields the curve information of the curve to be measured, which includes, but is not limited to, the length of the curve and the path information of the curve. In summary, those skilled in the art will understand that each measurement includes a left view and a right view. Simultaneous imaging with a binocular camera yields a set of measurement images, meaning the content captured in each measurement image is identical, differing only in the shooting angle. Each measurement includes two marker points: a first marker point and a second marker point. Due to the difference in shooting angles between the left and right views captured by the binocular camera in practical applications, the coordinates of the first marker point in the left and right views, although both indicate the same point, are different. Similarly, the coordinates of the second marker point in the left and right views, also indicating the same point, are also different. Based on the adjusted coordinates of the first target marker in the left view to be measured and the corresponding adjusted coordinates of the first target marker in the right view to be measured, a three-dimensional coordinate transformation can be performed to obtain the corresponding three-dimensional coordinates of the first marker. Similarly, based on the adjusted coordinates of the second target marker in the left view to be measured and the corresponding adjusted coordinates of the second target marker in the right view to be measured, a three-dimensional coordinate transformation can be performed to obtain the corresponding three-dimensional coordinates of the second marker.

[0058] Through steps S110 to S130, after acquiring multiple frames of the left and right views to be measured, this application adjusts the coordinates of the first and second marker points in the remaining left and right views to be measured based on the reference coordinates of the first marker point in the selected reference left and right views. This effectively addresses phenomena such as image jitter that occur during multi-frame image processing, significantly improving measurement accuracy and robustness. Furthermore, this application can flexibly obtain the path of the curve to be measured using the first and second marker points, requiring no additional equipment configuration and improving measurement efficiency.

[0059] In one embodiment, the left and right views to be measured include a left view to be measured and a right view to be measured. The step of detecting a first marker point and a second marker point in each of the left and right views to be measured includes:

[0060] For each left and right view to be measured, an inspection is performed to determine the first and second auxiliary devices included in each view. The first auxiliary device is fixed at a preset endpoint, and the second auxiliary device at different times corresponds to different positions on the curve to be measured.

[0061] The corresponding first and second marker points are determined by the front ends of the first and second auxiliary instruments in each left and right view to be measured.

[0062] Specifically, this embodiment provides a step for detecting marker points, including: firstly, detecting each left and right view to be measured to identify a first auxiliary instrument and a second auxiliary instrument included in each view. The first and second auxiliary instruments are instruments that doctors need to use during surgery or medical examinations, such as scissors and forceps. The front ends of the two auxiliary instruments are placed on the curve to be measured. The first and second marker points are obtained through the front ends of the first and second auxiliary instruments, respectively. The first auxiliary instrument is fixed at a preset endpoint, which is generally the leftmost or rightmost endpoint of the curve to be measured. The second auxiliary device is slid along the path of the curve to be measured according to a certain pattern until the coordinates of the second marker point at all times cover the preset target points on the curve to be measured. Furthermore, the moving speed and movement pattern of the second auxiliary device can be set by relevant technicians. In some preferred embodiments, if the shape of the curve to be measured is relatively complex, the moving speed of the second auxiliary device can be set to be lower, thereby more clearly capturing the path of the curve. Similarly, if the shape of the curve to be measured is relatively simple and clear, the moving speed of the second auxiliary device can be set to be higher, thereby improving the calculation speed and efficiency while ensuring accurate acquisition of the curve path. During the sliding process of the second auxiliary device, a binocular camera captures images at a preset shooting frequency, obtaining a series of left and right views of the curve to be measured. The second auxiliary device in the left and right views of the curve to be measured at different times corresponds to different positions on the curve to be measured. The leading point of the second auxiliary device, i.e., the second marker point, traverses the curve to be measured. This embodiment allows for the detection of marker points using existing surgical equipment, reducing usage costs and having virtually no other impact on the surgical cavity or lesion area. Furthermore, the marker points are marked by the doctor using existing auxiliary instruments, offering greater flexibility, enabling accurate measurement of non-edge measurement points, and significantly improving measurement accuracy in areas with weak texture.

[0063] In one embodiment, the first and second marker points in each of the left and right views to be measured are detected respectively to obtain the coordinates of the marker points in each view, including:

[0064] Each left view and each right view to be measured are input into a fully trained detection neural network to detect the position information of the two auxiliary devices in each view, as well as the front end point of each auxiliary device.

[0065] The coordinates of the corresponding two-dimensional marker point in each view are obtained through the front end of the assistive device.

[0066] Specifically, a fully trained detection neural network is used to detect each left and right view to be measured, obtaining the position information of the two assistive devices in each view. In practical applications, this detection neural network can use networks such as RCNN (Regions with CNN features) or YOLO (You Only Look Once). The aforementioned position information refers to the range of the area where the assistive device is located. The detection neural network can also detect the front point corresponding to each assistive device. This front point is generally located at the front of the assistive device's area, and the coordinates of this front point are the coordinates of the corresponding marker point.

[0067] Furthermore, in practical applications, the matching of marker points in the left and right views to be measured can be completed based on the position information of each auxiliary device. It is understood that in this application, there are two auxiliary devices, and both are located on the curve to be measured, with a clear left-right or up-down relationship. Therefore, matching the location areas of the two auxiliary devices in the left view position information with those in the right view position information is relatively simple. Since there are only two auxiliary devices in this embodiment, the scheme of directly obtaining marker points from the binocular view does not require complex image matching operations. Moreover, due to the auxiliary detection of the front point of the auxiliary device, the problem of inaccurate measurement results and high time consumption caused by mismatches in weak texture areas is effectively solved, and the accuracy of marker point recognition in weak texture areas is also improved.

[0068] In some embodiments, the first and second marker points in each of the left and right views to be measured are detected respectively to obtain the coordinates of the marker points in each view, including:

[0069] For each left and right view to be measured, an error is reported if the number of marker points in the view is not equal to 2.

[0070] If the number of marker points detected in the view is equal to 2, then the coordinates of the marker points in each view are obtained.

[0071] Specifically, in this embodiment, the marker points in the left and right view images to be measured need to be further detected. If the number of marker points in either image is not equal to 2, the measurement is paused and an error is reported. It can be understood that two points can determine a unique line segment, and by splicing adjacent line segments, a unique curve to be measured can be obtained. However, if the number of marker points is not equal to 2, the line segment to be measured cannot be formed; or multiple line segments may be obtained, making it impossible to determine a unique line segment to be measured. Therefore, if the number of marker points in any image is not equal to 2, an error is reported.

[0072] Similarly, if the number of marker points detected in the left or right view to be measured is equal to 2, the detection continues to obtain the coordinates of the marker points in each view. This embodiment allows for the detection of marker points in the left and right view images to be measured before coordinate transformation, thereby further ensuring the accuracy of the measured line segment length and improving the detection accuracy rate.

[0073] In some embodiments, the coordinates of the first and second marker points in all the left and right views to be measured are adjusted based on the reference coordinates of the first marker point in the reference left and right views, respectively, to obtain the coordinates of the first target marker point and the second target marker point in each view, including:

[0074] Compare the coordinates of the first marker point in each left and right view to be measured with the reference coordinates of the first marker point, and calculate the coordinate deviation value corresponding to the coordinates of each first marker point.

[0075] The coordinates of the first marker point are adjusted based on each coordinate deviation value until the coordinates of the first marker point in each left and right view to be measured are equal to the reference coordinates of the first marker point, thus obtaining the coordinates of the first target marker point.

[0076] The coordinates of the corresponding second marker point are adjusted based on each coordinate deviation value to obtain the coordinates of the corresponding second target marker point. The target marker point coordinates include the coordinates of the first target marker point and the coordinates of the second target marker point.

[0077] Specifically, in some preferred embodiments, the aforementioned reference left and right views can be set as the last frame of the left and right views to be measured acquired in chronological order. The coordinates of the first marker point in the aforementioned reference left and right views are used as the first marker point reference coordinates. The coordinates of the first marker points in the remaining left and right views to be measured are compared with the first marker point reference coordinates. Ideally, the first marker point reference coordinates should be the same as the first marker point coordinates in the remaining left and right views to be measured. However, in practical applications, due to factors such as patient breathing, the entire image may shift, so in most cases, the first marker point reference coordinates are not the same as the coordinates of the other first marker points. In this case, the first marker point reference coordinates should be used as the standard, and the coordinate deviation value of the remaining first marker point coordinates compared to the first marker point reference coordinates should be calculated. This coordinate deviation value is preferably the difference between the first marker point reference coordinates and the first marker point coordinates. The first marker point coordinates and the corresponding second marker point coordinates are then adjusted based on this coordinate deviation value to obtain the corresponding first target marker point coordinates and second target marker point coordinates. This embodiment effectively solves the problem of inconsistent marker point coordinates in different views caused by image jitter during curve measurement, improving the accuracy and robustness of the measurement.

[0078] In some embodiments, the left and right views to be measured include a left view to be measured and a right view to be measured; the coordinates of the first target marker point and the second target marker point in the left and right views to be measured are transformed into three-dimensional coordinates to obtain the corresponding three-dimensional coordinates of the first marker point and the second marker point located on the curve to be measured, including:

[0079] Calculate the disparity information of the coordinates of the first target marker and the second target marker between the left view to be measured and the right view to be measured, and determine the depth information of each marker based on the disparity information;

[0080] Based on the depth information, the coordinates of the first target marker point and the second target marker point are transformed into three-dimensional coordinates to obtain the three-dimensional coordinates of the first marker point and the second marker point respectively.

[0081] Specifically, in practical applications, binocular cameras are generally calibrated. Therefore, only lateral disparity exists in the left and right view images to be measured. That is, the disparity information mentioned above is the lateral disparity information of the coordinates of the marker points in the left and right views. Based on this disparity information, the depth information of each marker point coordinate can be determined. The formula for calculating the depth information is:

[0082]

[0083] Where f is the camera focal length, b is the baseline distance, d is the parallax information, and z is the depth value of the coordinates. It can be understood that after camera calibration, the camera's focal length and baseline distance are known parameters. Therefore, based on the calculated depth information of each marker point's coordinates, the coordinates of each marker point are converted into their corresponding 3D coordinates.

[0084] In some embodiments, the three-dimensional coordinates of the first marker point and the three-dimensional coordinates of each of the second marker points are sequentially connected according to the shooting time sequence to obtain the curve information of the curve to be measured, including:

[0085] By connecting the three-dimensional coordinates of adjacent second marker points in the order of shooting time, the path information of the curve to be measured is obtained.

[0086] The sum of the line segment lengths between the three-dimensional coordinates of adjacent second marker points is calculated to obtain the length information of the curve to be measured. The curve information includes path information and length information.

[0087] Specifically, after knowing the three-dimensional coordinates of the second marker point at each moment, the three-dimensional coordinates of the first marker point, the second marker point obtained from the first shot, the second marker point obtained from the second shot, the third marker point obtained from the third shot, and so on, are sequentially connected by straight lines to obtain the path information of the curve to be measured. This path information represents information such as the shape of the curve to be measured. The sum of the lengths of the line segments between the three-dimensional coordinates of each adjacent marker point is calculated to obtain the length information of the curve to be measured. The distance between adjacent marker points can be obtained by calculating the Euclidean distance between them.

[0088] In one embodiment, the method further includes:

[0089] The second auxiliary device moves along the curve to be measured according to a preset movement rule at different times. The movement rule includes a preset movement speed and a preset movement space rule. The movement space rule includes one of the following: moving from top to bottom, moving from bottom to top, moving from left to right, or moving from right to left.

[0090] Specifically, in practical applications, the second auxiliary device can move along the curve to be measured according to a preset movement pattern. This movement pattern includes a movement speed. In some embodiments, a fixed movement speed can be preset by relevant technicians, and the second auxiliary device can be controlled to move along the curve to be measured according to this movement speed. In some embodiments, different movement speeds can be set for different parts of the curve to be measured. For example, a slower movement speed can be set for parts of the curve to be measured with large changes and large fluctuations, and a faster movement speed can be set for parts of the curve to be measured with small changes. Furthermore, the above-mentioned movement pattern also includes movement space rules, which are determined by the path of the curve to be measured, i.e., its three-dimensional shape. For example, if the path of the curve to be measured is roughly vertically distributed, the above-mentioned spatial order can be set from top to bottom or from bottom to top, so that the second marker point can more comprehensively traverse the curve to be measured. Similarly, if the path of the curve to be measured is roughly horizontally distributed, the above-mentioned spatial order can be set from left to right or from right to left.

[0091] This application also provides a preferred embodiment of a curve measurement method based on a binocular camera. Figure 2 This is a flowchart illustrating a curve measurement method in a preferred embodiment.

[0092] In step S210, the first auxiliary device is fixed at the preset endpoint of the curve to be measured, and the second auxiliary device moves along the curve to be measured at a preset speed. During this process, the left and right views to be measured are acquired.

[0093] Step S220: Detect whether the number of auxiliary instruments in each view is equal to 2. If yes, proceed to step S230. If no, end the curve measurement process and perform error handling. Among them, the detection of the left and right views to be measured is mostly achieved by training a fully trained neural network. The training of the neural network includes: (1) Creating a dataset, collecting surgical scene materials containing commonly used surgical instruments (i.e., auxiliary instruments), marking the position, type and front point coordinates of the instruments, and including surgical instrument materials from multiple angles, types and scenes as much as possible; (2) Training the initial neural network based on the dataset. In this embodiment, a YOLO network is used, and a key point detection function is added to the YOLO target detection algorithm. Using the trained network, the number of instruments, the position of the instruments and the coordinates of the front point of the instruments in the left and right images can be detected respectively.

[0094] Step S230: The position information of the auxiliary devices in the left and right views to be measured is obtained by detecting the fully trained neural network. The coordinates of the first and second marker points are determined by the coordinates of the front ends of the first and second auxiliary devices, respectively. The coordinate sequence of the two auxiliary devices is output in chronological order. The target tracking algorithm is combined to maintain the one-to-one correspondence of the devices. In some preferred embodiments, the target tracking algorithm includes identifying the two auxiliary devices in each left and right view to be measured and predicting the movement direction and speed of each auxiliary device. In some preferred embodiments, after obtaining the left and right views to be measured in the current frame, the above detection processing is first performed on the left and right views to be measured in the current frame to obtain the position information of the two auxiliary devices and the coordinates of the marker points in the left and right views to be measured in the current frame. Then, the movement direction and speed of the corresponding auxiliary device in the next frame are predicted according to the detection results, so as to obtain the predicted position of the auxiliary device in the next frame. After the prediction is completed, the left and right views to be measured in the next frame transmitted back at the next moment are received. If the position of the auxiliary device in the next frame to be measured differs greatly from the predicted position, it is determined that there is an anomaly. There may be problems such as the loss of a frame of left and right views to be measured in the sequence or detection errors, and error processing is required. In other embodiments, if the position of the auxiliary device in the next frame of the left and right views to be measured is significantly different from the position predicted based on the current frame, prediction can be continued based on the obtained position of the auxiliary device in the next frame of the left and right views to be measured, or prediction can be continued based on the speed and direction predicted in the current frame. The prediction result is then compared with the obtained position of the auxiliary device in the next frame of the left and right views to be measured, and so on. If there are significant differences between the predicted position and the actual position in multiple consecutive frames, an alarm is then triggered to reduce the impact of false alarms and false detections.

[0095] Step S240: Based on the reference coordinates of the first marker point in the reference left and right views to be measured, adjust the coordinates of the first and second marker points in the remaining left and right views to be measured, obtaining the adjusted coordinates of the first and second target marker points in each view. The adjusted coordinates of the first target marker point in each left and right view to be measured should be the same.

[0096] Step S250: Based on the disparity information between the coordinates of the first target marker point and the coordinates of the second target marker point in the left and right views to be measured, perform three-dimensional coordinate transformation to obtain the corresponding three-dimensional coordinates of the first marker point and the second marker point.

[0097] Step S260: Connect the three-dimensional coordinates of each first marker point and the three-dimensional coordinates of the second marker point in sequence according to the shooting time to obtain the curve information of the curve to be measured. Specifically, by calculating the Euclidean distance between the three-dimensional coordinates of the marker points at adjacent times, the summation yields the curve length from the start point to the end point of the curve to be measured. Outputting the three-dimensional coordinates of the marker points at each time point along with the measurement results, and connecting them sequentially, displays the actual measurement route and the corresponding measurement results.

[0098] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0099] Based on the same inventive concept, this application also provides a curve measuring device for implementing the curve measuring method described above. The solution provided by this device is similar to the solution described in the above method; therefore, the specific limitations in one or more curve measuring device embodiments provided below can be found in the limitations of the curve measuring method described above, and will not be repeated here.

[0100] In one embodiment, such as Figure 3 As shown, a curve measurement device is provided, including: an acquisition module 31, a calculation module 32, and a generation module 33, wherein:

[0101] The acquisition module 31 is used to acquire multiple left and right views of the target object at different times. Each view of the left and right views of the target object includes a first marker point and a second marker point located on a preset curve to be measured on the target object. The first marker point in each left and right view of the target object is located at a preset endpoint of the curve to be measured. The second marker points in different left and right views of the target object are located at different positions on the curve to be measured. The coordinates of the second marker points at all times cover the preset target points in the curve to be measured.

[0102] The calculation module 32 is used to detect the first and second marker points in each of the left and right views to be measured, and obtain the coordinates of the marker points in each view; determine the reference left and right views in the left and right views to be measured, and adjust the coordinates of the first and second marker points in all the left and right views to be measured based on the reference coordinates of the first marker points in the reference left and right views, and obtain the coordinates of the first and second target marker points in each view; perform three-dimensional coordinate transformation on the coordinates of the first and second target marker points in the left and right views to be measured, and obtain the corresponding three-dimensional coordinates of the first and second marker points located on the curve to be measured.

[0103] The generation module 33 is used to sequentially connect the three-dimensional coordinates of the first marker point and the three-dimensional coordinates of each second marker point according to the shooting time sequence to obtain the curve information of the curve to be measured.

[0104] Each module in the aforementioned curve measuring device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the operations corresponding to each module.

[0105] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 4 As shown, the computer device includes a processor, memory, and a network interface connected via a system bus. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The database stores data related to curve measurement algorithms. The network interface communicates with external terminals via a network connection. When the computer program is executed by the processor, it implements a curve measurement method.

[0106] Those skilled in the art will understand that Figure 4The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0107] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0108] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0109] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0110] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A curve measurement method based on a binocular camera, characterized in that, The method includes: Multiple left and right views of the target object are obtained at different times. Each view in a pair of left and right views includes a first marker point and a second marker point located on a preset curve to be measured on the target object. The first marker point in each pair of left and right views is located at a preset endpoint of the curve to be measured. The second marker points in different pairs of left and right views are located at different positions on the curve to be measured. The coordinates of the second marker point at all times cover the preset target point position in the curve to be measured. The first and second marker points in each pair of left and right views to be measured are detected respectively to obtain the coordinates of the marker points in each view; Determine the reference left and right views in the left and right views to be measured, and adjust the coordinates of the first and second marker points in all the left and right views to be measured based on the reference coordinates of the first marker point in the reference left and right views, so as to obtain the coordinates of the first and second target marker points in each view respectively. Perform a three-dimensional coordinate transformation on the coordinates of the first target marker point and the second target marker point in the left and right views to be measured to obtain the corresponding three-dimensional coordinates of the first marker point and the second marker point located on the curve to be measured. By sequentially connecting the three-dimensional coordinates of the first marker point and each of the second marker points according to the shooting time sequence, the curve information of the curve to be measured is obtained.

2. The method according to claim 1, characterized in that, The left and right views to be measured include a left view and a right view to be measured. The step of detecting the first marker point and the second marker point in each pair of the left and right views to be measured includes: Each pair of the left view and the right view to be measured is detected to determine the first auxiliary device and the second auxiliary device included in each view. The first auxiliary device is fixed at the preset endpoint, and the second auxiliary device at different times corresponds to different positions on the curve to be measured. The corresponding first and second marker points are determined by the front ends of the first and second auxiliary instruments in each of the left and right views to be measured.

3. The method according to claim 1, characterized in that, The step of detecting the first and second marker points in each pair of left and right views to be measured, and obtaining the coordinates of the marker points in each view, includes: Each left view and each right view to be measured are input into a fully trained detection neural network to detect the position information of the two auxiliary devices in each view, as well as the front end point of each auxiliary device. The coordinates of the corresponding two-dimensional marker point in each view are obtained through the front end point of the auxiliary device.

4. The method according to claim 1, characterized in that, The step of detecting the first and second marker points in each pair of left and right views to be measured, and obtaining the coordinates of the marker points in each view, includes: For each of the left and right views to be measured, an error is reported if the number of the marker points in the view is not equal to 2. If the number of marker points detected in the view is equal to 2, then the coordinates of the marker points in each view are obtained.

5. The method according to claim 1, characterized in that, The step of adjusting the coordinates of the first and second marker points in all the left and right views to be measured based on the reference coordinates of the first marker point in the reference left and right views, to obtain the coordinates of the first target marker point and the second target marker point in each view, includes: The coordinates of the first marker point in each of the left and right views to be measured are compared with the reference coordinates of the first marker point, and the coordinate deviation value corresponding to each of the first marker point coordinates is calculated. The coordinates of the first marker point are adjusted based on each of the coordinate deviation values ​​until the coordinates of the first marker point in each of the left and right views to be measured are equal to the reference coordinates of the first marker point, thus obtaining the coordinates of the first target marker point. The coordinates of the corresponding second marker point are adjusted based on each of the aforementioned coordinate deviation values ​​to obtain the coordinates of the corresponding second target marker point.

6. The method according to claim 5, characterized in that, The left and right views to be measured include a left view and a right view; the step of performing a three-dimensional coordinate transformation on the coordinates of the first target marker point and the second target marker point in the left and right views to obtain the corresponding three-dimensional coordinates of the first marker point and the second marker point located on the curve to be measured includes: Calculate the disparity information between the coordinates of the first target marker point and the coordinates of the second target marker point for each pair of the left view to be measured and the right view to be measured, and determine the depth information of each marker point based on the disparity information; Based on the depth information, the coordinates of the first target marker point and the second target marker point are transformed into three-dimensional coordinates to obtain the three-dimensional coordinates of the first marker point and the second marker point.

7. The method according to claim 1, characterized in that, The process of sequentially connecting the three-dimensional coordinates of the first marker point and each of the second marker points according to the shooting time sequence to obtain the curve information of the curve to be measured includes: According to the shooting time sequence, the three-dimensional coordinates of the adjacent second marker points are connected by straight lines to obtain the path information of the curve to be measured; The sum of the line segment lengths between the three-dimensional coordinates of adjacent second marker points is calculated to obtain the length information of the curve to be measured, wherein the curve information includes the path information and the length information.

8. The method according to any one of claims 1 to 7, characterized in that, The method further includes: The second auxiliary device moves along the curve to be measured according to a preset movement rule at different times. The movement rule includes a preset movement speed and a preset movement space rule. The movement space rule includes one of the following: moving from top to bottom, moving from bottom to top, moving from left to right, and moving from right to left.

9. A curve measuring device, characterized in that, The device includes: The acquisition module is used to acquire multiple left and right views to be measured at different times. Each view in a pair of left and right views to be measured includes a first marker point and a second marker point located on a preset curve to be measured on the target object. The first marker point in each pair of left and right views to be measured is located at a preset endpoint of the curve to be measured. The second marker points in different pairs of left and right views to be measured are located at different positions on the curve to be measured. The coordinates of the second marker point at all times cover the preset target point position in the curve to be measured. The calculation module is used to detect the first and second marker points in each pair of left and right views to be measured, respectively, to obtain the coordinates of the marker points in each view; determine the reference left and right views in the left and right views to be measured, and adjust the coordinates of the first and second marker points in all the left and right views to be measured based on the reference coordinates of the first marker points in the reference left and right views, respectively, to obtain the coordinates of the first and second target marker points in each view; and perform three-dimensional coordinate transformation on the coordinates of the first and second target marker points in the left and right views to be measured, to obtain the corresponding three-dimensional coordinates of the first and second marker points located on the curve to be measured. The generation module is used to sequentially connect the three-dimensional coordinates of the first marker point and the three-dimensional coordinates of each of the second marker points according to the shooting time sequence to obtain the curve information of the curve to be measured.

10. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 8.

Images