Curve measurement method and device based on binocular camera and computer equipment

Abstract

The invention relates to a curve measurement method and device based on a binocular camera and computer equipment. Comprising the following steps: collecting multiple to-be-measured left and right views at different moments; detecting identification points in each to-be-measured left and right views to obtain a first identification point coordinate and a second identification point coordinate in each view; determining reference left and right views in the left and right views to be measured, and adjusting coordinates of identification points in all the left and right views to be measured based on first identification point reference coordinates in the reference left and right views to obtain first target identification point coordinates and second target identification point coordinates in each view; according to the coordinates of the target identification points in the left and right views to be measured, performing three-dimensional coordinate conversion to obtain corresponding first identification point three-dimensional coordinates and second identification point three-dimensional coordinates on the curve to be measured; the three-dimensional coordinates of all the identification points are connected in sequence, and curve information of the curve to be measured is obtained. By adopting the method, the accuracy and efficiency of curve measurement can be improved.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a curve measurement method, device and computer equipment based on a binocular camera. Background Art

[0002] Machine vision is a rapidly developing branch of artificial intelligence, which aims to use machines to simulate human eyes to collect images, recognize images and process information, and finally complete operations through execution devices. With the continuous development of computer technology, machine vision technology has gradually penetrated into many industries such as industry, agriculture, transportation, and medical care with its advantages of high precision, strong real-time performance, automation and intelligence.

[0003] Machine vision technology has a wide range of applications in medical image analysis. Traditional medical image analysis requires doctors to spend a lot of time and energy to observe and interpret, while machine vision technology can use algorithms and models to quickly and accurately identify and mark abnormal areas in medical images, helping doctors diagnose and treat.

[0004] In clinical endoscopy, accurate assessment of lesion size is of great significance for diagnosis. Spatial straight line measurement of lesions can obtain the diameter of lesions, which provides important guidance for surgical selection. However, the shape of lesions varies. The measurement of surface curves can provide doctors with more three-dimensional information and can also calculate the perimeter of the lesion area. In the prior art, the measurement of lesion area curves is mostly based on uniform sampling between the two end points of the curve, and the length of the curve is obtained by solving the distance of multiple line segments. However, this method is greatly affected by the sampling frequency, cannot take into account both accuracy and calculation amount, and has high requirements on image texture and poor robustness.

[0005] At present, no effective solution has been proposed for the problems of the existing technology in measuring curves, such as the inability to balance accuracy and computational complexity, as well as poor robustness. Summary of the invention

[0006] Based on this, it is necessary to provide a curve measurement method, device and computer equipment based on a binocular camera to address the above technical problems.

[0007] In a first aspect, the present application provides a curve measurement method based on a binocular camera. The method comprises:

[0008] Acquire a plurality of left and right views to be measured at different times, wherein each view in a pair of left and right views to be measured includes a first identification point and a second identification point located on a preset curve to be measured on the target object; the first identification point in each left and right view to be measured is located at a preset end point of the curve to be measured, and the second identification points in different left and right views to be measured are located at different positions on the curve to be measured, and the coordinates of the second identification point at all times cover the preset target point position in the curve to be measured;

[0009] Detect the first identification point and the second identification point in each left and right view to be measured respectively, and obtain the coordinates of the identification point in each view;

[0010] Determine the reference left and right views in the left and right views to be measured, and adjust the coordinates of the first identification point and the second identification point in all the left and right views to be measured based on the reference coordinates of the first identification point in the reference left and right views, and obtain the coordinates of the first target identification point and the second target identification point in each view respectively;

[0011] Perform three-dimensional coordinate conversion on the coordinates of the first target identification point and the second target identification point in the left and right views to be measured, and obtain the corresponding three-dimensional coordinates of the first identification point and the second identification point on the curve to be measured;

[0012] The three-dimensional coordinates of the first marking point and the three-dimensional coordinates of each second marking point are sequentially connected in the order of shooting time to obtain curve information of the curve to be measured.

[0013] In one embodiment, the left and right views to be measured include a left view to be measured and a right view to be measured, and detecting a first identification point and a second identification point in each of the left and right views to be measured respectively includes:

[0014] Detect each left view and right view to be measured, and determine the first auxiliary instrument and the second auxiliary instrument included in each view, wherein the first auxiliary instrument is fixed at a preset end point, and the second auxiliary instrument at different times corresponds to different positions on the curve to be measured;

[0015] The corresponding first identification point and second identification point are determined by the front end points of the first auxiliary tool and the second auxiliary tool in each left and right view to be measured.

[0016] In one embodiment, detecting the first marking point and the second marking point in each left and right view to be measured respectively to obtain the coordinates of the marking point in each view includes:

[0017] Input each left view to be measured and each right view to be measured into a well-trained detection neural network, and respectively detect and obtain the position information of two auxiliary instruments in each view, as well as the front end point corresponding to each auxiliary instrument;

[0018] The corresponding two-dimensional coordinates of the marking point in each view are obtained through the front end point of the auxiliary instrument.

[0019] In one embodiment, detecting the first marking point and the second marking point in each left and right view to be measured respectively to obtain the coordinates of the marking point in each view includes:

[0020] Detect each left and right view to be measured. If the number of identification points in the detected view is not equal to 2, an error is reported;

[0021] If the number of identification points detected in the view is equal to 2, the coordinates of the identification points in each view are obtained.

[0022] In one embodiment, the coordinates of the first identification point and the second identification point in all the left and right views to be measured are adjusted based on the reference coordinates of the first identification point in the reference left and right views to obtain the coordinates of the first target identification point and the second target identification point in each view, respectively, including:

[0023] Compare the coordinates of the first marking point in each left and right view to be measured with the first marking point reference coordinates respectively, and calculate the coordinate deviation value corresponding to each first marking point coordinate;

[0024] Adjust the corresponding first marking point coordinates based on each coordinate deviation value until the first marking point coordinates in each left and right view to be measured are equal to the first marking point reference coordinates, thereby obtaining the first target marking point coordinates;

[0025] The corresponding second identification point coordinates are adjusted based on each coordinate deviation value to obtain the corresponding second target identification point coordinates, wherein the target identification point coordinates include the first target identification point coordinates and the second target identification point coordinates.

[0026] In one of the embodiments, the left and right views to be measured include a left view to be measured and a right view to be measured; performing three-dimensional coordinate conversion on the coordinates of the first target identification point and the second target identification point in the left and right views to be measured to obtain the corresponding three-dimensional coordinates of the first identification point and the second identification point located on the curve to be measured, including:

[0027] Calculate the disparity information of the first target identification point coordinates and the second target identification point coordinates between each left view to be measured and the right view to be measured, and determine the depth information of each identification point according to the disparity information;

[0028] Based on the depth information, the coordinates of the first target identification point and the coordinates of the second target identification point are respectively converted into three-dimensional coordinates to obtain the three-dimensional coordinates of the first identification point and the three-dimensional coordinates of the second identification point correspondingly.

[0029] In one embodiment, the three-dimensional coordinates of the first marking point and the three-dimensional coordinates of each second marking point are sequentially connected in the order of shooting time to obtain the curve information of the curve to be measured, including:

[0030] Connect the three-dimensional coordinates of the adjacent second marking points by straight lines according to the shooting time sequence to obtain the path information of the curve to be measured;

[0031] The sum of the lengths of the line segments between the three-dimensional coordinates of the adjacent marking point and the second marking point is calculated to obtain the length information of the curve to be measured, wherein the curve information includes path information and length information.

[0032] In one embodiment, the method further comprises:

[0033] The second auxiliary device at different times moves along the curve to be measured according to a preset movement rule, wherein the movement rule includes a preset movement speed and a preset movement space rule; the movement space rule includes one of the following: moving from top to bottom, moving from bottom to top, moving from left to right, and moving from right to left.

[0034] In a second aspect, the present application also provides a curve measuring device. The device comprises:

[0035] An acquisition module is used to acquire a plurality of left and right views to be measured at different times, wherein each view in a pair of left and right views to be measured includes a first identification point and a second identification point located on a preset curve to be measured on a target object; the first identification point in each left and right view to be measured is located at a preset end point of the curve to be measured, and the second identification points in different left and right views to be measured are located at different positions on the curve to be measured, and the coordinates of the second identification point at all times cover the preset target point position in the curve to be measured;

[0036] A calculation module is used to detect the first identification point and the second identification point in each left and right view to be measured, respectively, to obtain the coordinates of the identification points in each view; determine the reference left and right views in the left and right views to be measured, and adjust the coordinates of the first identification point and the second identification point in all the left and right views to be measured based on the reference coordinates of the first identification point in the reference left and right views, to obtain the first target identification point coordinates and the second target identification point coordinates in each view; perform three-dimensional coordinate conversion on the first target identification point coordinates and the second target identification point coordinates in the left and right views to be measured, to obtain the corresponding three-dimensional coordinates of the first identification point and the second identification point located on the curve to be measured;

[0037] The generating module is used to sequentially connect the three-dimensional coordinates of the first identification point and the three-dimensional coordinates of each second identification point according to the shooting time sequence to obtain the curve information of the curve to be measured.

[0038] In a third aspect, the present application further provides a computer device. The computer device includes a memory and a processor, the memory stores a computer program, and the processor implements the following steps when executing the computer program:

[0039] Acquire a plurality of left and right views to be measured at different times, wherein each view in a pair of left and right views to be measured includes a first identification point and a second identification point located on a preset curve to be measured on the target object; the first identification point in each left and right view to be measured is located at a preset end point of the curve to be measured, and the second identification points in different left and right views to be measured are located at different positions on the curve to be measured, and the coordinates of the second identification point at all times cover the preset target point position in the curve to be measured;

[0040] Detect the first identification point and the second identification point in each left and right view to be measured respectively, and obtain the coordinates of the identification point in each view;

[0041] Determine the reference left and right views in the left and right views to be measured, and adjust the coordinates of the first identification point and the second identification point in all the left and right views to be measured based on the reference coordinates of the first identification point in the reference left and right views, and obtain the coordinates of the first target identification point and the second target identification point in each view respectively;

[0042] Perform three-dimensional coordinate conversion on the coordinates of the first target identification point and the second target identification point in the left and right views to be measured, and obtain the corresponding three-dimensional coordinates of the first identification point and the second identification point on the curve to be measured;

[0043] The three-dimensional coordinates of the first marking point and the three-dimensional coordinates of each second marking point are sequentially connected in the order of shooting time to obtain curve information of the curve to be measured.

[0044] The above-mentioned curve measurement method, device and computer equipment based on binocular camera first obtain multiple left and right views to be measured at different times, wherein each pair of views includes a first identification point and a second identification point located on the preset curve to be measured on the target object, the first identification point in each left and right view to be measured is located at the preset end point of the curve to be measured, the second identification points in different left and right views to be measured are located at different positions on the curve to be measured, and the coordinates of the second identification point at all times cover the preset target point position in the curve to be measured. The first identification point and the second identification point in each left and right view to be measured are detected respectively to obtain the coordinates of the identification points in each view. Then, the reference left and right views in the left and right views to be measured are determined, and the coordinates of the first identification point and the second identification point in all the left and right views to be measured are adjusted based on the reference coordinates of the first identification point in the reference left and right views, and the coordinates of the first target identification point and the second target identification point in each view are obtained respectively. Finally, the coordinates of the first target identification point and the second target identification point in the left and right views to be measured are converted into three-dimensional coordinates to obtain the corresponding three-dimensional coordinates of the first identification point and the second identification point on the curve to be measured. Through the present application, based on the position of the second identification point at multiple moments, the path changes of the curve to be measured can be flexibly identified without the need for additional measuring equipment, effectively reducing the measurement cost and the required computing resources; and further, in the present application, the coordinates of the identification points in the other left and right views to be measured are offset compensated according to the reference coordinates of the first identification point, so as to cope with the phenomenon of picture jitter in multi-frame image processing, thereby effectively improving the measurement accuracy and robustness. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 A schematic diagram of a flow chart of a curve measurement method in one embodiment;

[0046] Figure 2 A schematic diagram of a flow chart of a curve measurement method in a preferred embodiment;

[0047] Figure 3 is a structural block diagram of a curve measuring device in one embodiment;

[0048] Figure 4 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0049] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0050] In one embodiment, Figure 1As shown, a curve measurement method based on a binocular camera is provided, comprising the following steps:

[0051] Step S110, obtaining a plurality of left and right views to be measured at different times, wherein each view in a pair of left and right views to be measured includes a first identification point and a second identification point located on a preset curve to be measured on the target object; the first identification point in each pair of left and right views to be measured is located at a preset end point of the curve to be measured, and the second identification points in different pairs of left and right views to be measured are located at different positions on the curve to be measured, and the coordinates of the second identification point at all times cover the preset target point position in the curve to be measured.

[0052] Specifically, the left and right views to be measured in the present application include a left view to be measured and a right view to be measured. The first identification point in the left view to be measured and the first identification point in the right view to be measured collected at the same time are actually at the same three-dimensional coordinate point in the three-dimensional space. Similarly, the second identification point in the left view to be measured and the second identification point in the right view to be measured collected at the same time are actually at the same three-dimensional coordinate point in the three-dimensional space. However, due to the lateral parallax between the left and right views captured by the binocular camera, the coordinates of the first identification point in the left view to be measured and the right view to be measured collected at the same time are slightly different. Similarly, the coordinates of the second identification point in the left view to be measured and the right view to be measured collected at the same time are also slightly different. In practical applications, the curve to be measured is located on the target object, which can be an edge curve of the target object that needs to be measured, or a curve that needs to be measured within the range of the target object. Further, the identification point is a point on the curve to be measured, which can be an actual identification point pre-set by relevant technical personnel, or a virtual point for auxiliary measurement.

[0053] In this embodiment, multiple frames of left and right views to be measured at different times are collected according to the time sequence, and each left and right view to be measured includes a first identification point and a second identification point, wherein the position of the first identification point in each left and right view to be measured remains unchanged and is located at the preset end point of the curve to be measured, and the coordinates of the second identification point in the left and right views to be measured collected at different times are different, that is, in actual application, the second identification point can be controlled to move along the curve to be measured until the coordinates of the second identification point at all times cover multiple target points preset in the curve to be measured, and in this process, the first identification point and the second identification point are photographed by a binocular camera, so as to collect the above-mentioned multiple frames of left and right views to be measured. Among them, the method of controlling the second identification point to move along the path of the curve to be measured includes but is not limited to, the doctor or technician marks the moving path of the second identification point on the curve to be measured in sequence, or the doctor or technician holds the surgical instrument and slides along the path of the curve to be measured at a slower speed. Since the identification point cannot completely cover the curve to be measured, multiple target points can be set in actual application, and the target point can be a virtual auxiliary point or an actual point preset by relevant technicians. Multiple target points that need to be focused on are preset on the curve to be measured, such as the corners of the curve, points close to the corners, parts with large fluctuations in the curve to be measured, etc., or multiple target points can be set on the curve to be measured according to preset point intervals, etc. In summary, the second identification points obtained should at least include the above-mentioned target points.

[0054] Step S120, respectively detect the first identification point and the second identification point in each left and right view to be measured to obtain the coordinates of the identification points in each view; determine the benchmark left and right views in the left and right views to be measured, and adjust the coordinates of the first identification point and the second identification point in all left and right views to be measured based on the benchmark coordinates of the first identification point in the benchmark left and right views, to respectively obtain the first target identification point coordinates and the second target identification point coordinates in each view.

[0055] Specifically, the first identification point and the second identification point in each left and right view to be measured are detected. In some preferred embodiments, the detection of identification points includes but is not limited to identifying identification points through a trained neural network to obtain identification point coordinates corresponding to each identification point in each view. Further, since multiple left and right views to be measured at multiple moments are obtained by shooting in time sequence in the present application, a pair of left and right views are selected from the multiple left and right views to be measured as reference left and right views, wherein the selection strategy of the reference left and right views includes but is not limited to using the frame with the clearest picture in the left and right views to be measured as the reference left and right views, or using the left and right views that are in the termination frame in time as the above-mentioned reference left and right views, etc. After the reference left and right views are selected, the coordinates of the first identification point in the reference left and right views are used as the first identification point reference coordinates, and the first identification point coordinates and the second identification point coordinates in the left and right views to be measured in the remaining frames are adjusted according to the first identification point reference coordinates to obtain the adjusted first target identification point coordinates and the second target identification point coordinates in each left and right view to be measured. Among them, those skilled in the art can understand based on the present application that, because the coordinates of the first identification point in the present application remain stationary and are all located at the preset end points of the curve to be measured, the coordinates of the first identification point in each frame of the left and right views to be measured taken at different times should be the same in an ideal state, but in actual applications, due to the breathing of the patient to be measured, the entire picture will move, so it is necessary to adjust the first identification point coordinates and the second identification point coordinates in the remaining left and right views to be measured based on the reference left and right views, and the first identification point coordinates in the first target identification point coordinates at different times after adjustment should be the same, and the first identification point is the identification point fixed at the preset end point of the curve to be measured. The adjustment method based on the reference coordinates of the first identification point includes but is not limited to calculating the coordinate deviation value between the first identification point coordinates and the first identification point reference coordinates in each left and right view to be measured, and adjusting the corresponding first identification point coordinates based on each coordinate deviation value until the first identification point coordinates in each left and right view to be measured are equal to the first identification point reference coordinates, and obtaining the first target identification point coordinates; adjusting the corresponding second identification point coordinates based on each coordinate deviation value to obtain the corresponding second target identification point coordinates.

[0056] Step S130, performing three-dimensional coordinate conversion on the coordinates of the first target identification point and the second target identification point in the left and right views to be measured to obtain the corresponding three-dimensional coordinates of the first identification point and the second identification point on the curve to be measured; connecting the three-dimensional coordinates of the first identification point and each second identification point in sequence according to the shooting time sequence to obtain the curve information of the curve to be measured.

[0057] Specifically, the coordinates of the first target identification point and the second target identification point in each left and right view to be measured are converted into three-dimensional coordinates, and the coordinates of the first target identification point and the second target identification point in two-dimensional form are converted into the three-dimensional coordinates of the first identification point and the second identification point in three-dimensional form. It can be understood that the three-dimensional coordinates of the identification point in three-dimensional form have an additional depth coordinate relative to the binocular camera compared to the two-dimensional target identification point coordinates. Finally, according to the order of the shooting time of the left and right views to be measured in each frame, the corresponding three-dimensional coordinates of the first identification point and the three-dimensional coordinates of each second identification point are connected in sequence. Since the three-dimensional coordinates of the first identification point captured at each moment should be the same after adjustment, the three-dimensional coordinates of the first identification point and the three-dimensional coordinates of each second identification point are connected in sequence as follows: the first identification point three-dimensional coordinates, the second identification point three-dimensional coordinates obtained by the first shooting, the second identification point three-dimensional coordinates obtained by the second shooting, the second identification point three-dimensional coordinates obtained by the third shooting...are connected in sequence to obtain the curve information of the curve to be measured, wherein the curve information includes but is not limited to the length of the curve to be measured, the path information of the curve, etc. In summary, those skilled in the art can understand that each pair of left and right views to be measured includes a left view to be measured and a right view to be measured. By shooting with a binocular camera at the same time, a pair of left and right views to be measured can be obtained, that is, the contents captured in a pair of left and right views to be measured are the same, and only the shooting angles are different. Each left view to be measured and the right view to be measured include two identification points, namely, a first identification point and a second identification point. Since there is a difference in shooting angles between the left view to be measured and the right view to be measured captured by the binocular camera in actual applications, at this time, the coordinates of the first identification point in the left view to be measured and the first identification point in the right view to be measured indicate the same point, but the coordinates are different. Similarly, the coordinates of the second identification point in the left view to be measured and the second identification point in the right view to be measured indicate the same point, but the coordinates are different. According to the adjusted coordinates of the first target identification point in the left view to be measured and the corresponding adjusted coordinates of the first target identification point in the right view to be measured, a three-dimensional coordinate conversion can be performed to obtain the corresponding three-dimensional coordinates of the first identification point. Similarly, according to the adjusted coordinates of the second target identification point in the left view to be measured and the corresponding adjusted coordinates of the second target identification point in the right view to be measured, a three-dimensional coordinate conversion can be performed to obtain the corresponding three-dimensional coordinates of the second identification point.

[0058] Through steps S110 to S130, after collecting multiple frames of left and right views to be measured, the present application adjusts the coordinates of the first identification point and the second identification point in the remaining left and right views to be measured according to the reference coordinates of the first identification point in the selected reference left and right views, thereby effectively coping with the phenomenon of image jitter and the like occurring in the multi-frame image processing, and greatly improving the measurement accuracy and robustness. Furthermore, the present application can flexibly obtain the path of the curve to be measured through the first identification point and the second identification point, without requiring additional configuration of the equipment, thereby improving the measurement efficiency.

[0059] In one embodiment, the left and right views to be measured include a left view to be measured and a right view to be measured, and the detecting of the first identification point and the second identification point in each of the left and right views to be measured respectively includes:

[0060] Detect each left view and right view to be measured, and determine the first auxiliary instrument and the second auxiliary instrument included in each view, wherein the first auxiliary instrument is fixed at a preset end point, and the second auxiliary instrument at different times corresponds to different positions on the curve to be measured;

[0061] The corresponding first identification point and second identification point are determined by the front end points of the first auxiliary tool and the second auxiliary tool in each left and right view to be measured.

[0062] Specifically, this embodiment provides a detection step of identification points, including first detecting each pair of left view and right view to be measured, and determining the first auxiliary instrument and the second auxiliary instrument included in each view, wherein the first auxiliary instrument and the second auxiliary instrument are instruments that doctors need to use when performing surgery or medical examinations, such as scissors, tweezers, etc. The front end points of the two auxiliary instruments are placed on the curve to be measured, and the first identification point and the second identification point can be obtained respectively through the front end points of the first auxiliary instrument and the second auxiliary instrument, and the first auxiliary instrument is fixed at a preset endpoint to keep it still, and the preset endpoint is generally the leftmost endpoint of the curve to be measured or the rightmost endpoint of the curve to be measured. The second auxiliary device is slid along the path of the curve to be measured in a certain regularity until the coordinates of the second identification point at all times should cover the preset target point in the curve to be measured. Furthermore, the moving speed and moving regularity of the second auxiliary device can be set by relevant technical personnel. In some preferred embodiments, if the shape of the curve to be measured is relatively complex, the moving speed of the second auxiliary device can be set to be lower, so that the path of the curve to be measured can be collected more clearly. Similarly, if the shape of the curve to be measured is relatively simple and clear, the moving speed of the second auxiliary device can be set to be higher, so that the calculation speed and efficiency can be improved under the premise of ensuring accurate collection of the path of the curve to be measured. During the sliding process of the second auxiliary device, the binocular camera is used to capture according to the preset shooting frequency to obtain a series of left and right views to be measured, wherein the second auxiliary device in the left and right views to be measured at different times corresponds to different positions on the curve to be measured, and the front end point of the second auxiliary device, that is, the second identification point traverses the curve to be measured. Through this embodiment, the detection of the marking points can be completed with the help of existing surgical equipment, which reduces the cost of use and has almost no other impact on the surgical cavity or lesion area; and the above-mentioned marking points are marked by doctors using existing auxiliary instruments, which is more flexible, can accurately measure non-edge points to be measured, and greatly improves the measurement accuracy in weak texture areas.

[0063] In one embodiment, detecting the first marking point and the second marking point in each left and right view to be measured respectively to obtain the coordinates of the marking point in each view includes:

[0064] Input each left view to be measured and each right view to be measured into a well-trained detection neural network, and respectively detect and obtain the position information of two auxiliary instruments in each view, as well as the front end point corresponding to each auxiliary instrument;

[0065] The corresponding two-dimensional coordinates of the marking point in each view are obtained through the front end point of the auxiliary instrument.

[0066] Specifically, each left view to be measured and each right view to be measured are detected by a well-trained detection neural network to obtain the position information of the two auxiliary devices in each view. In practical applications, the detection neural network can use a network such as RCNN (Regions with CNN features) and YOLO (You Only Look Once). The above position information is the range of the area where the auxiliary device is located. The detection neural network can also detect the front end point corresponding to each auxiliary device. The front end point is generally located at the front end of the auxiliary device area, and the coordinates of the front end point are the coordinates of the corresponding identification point.

[0067] Furthermore, in practical applications, the matching of the identification points in the left and right views to be measured can be completed based on the position information of each auxiliary device. It can be understood that the number of auxiliary devices in the present application is two, and the two auxiliary devices are located on the curve to be measured, with a clear left-right relationship or up-down relationship. Therefore, the matching of the position area where the two auxiliary devices are located in the left view position information of this embodiment with the position area where the two auxiliary devices are located in the right view position information is relatively simple. In this embodiment, since there are only two auxiliary devices, the solution of directly obtaining identification points from the binocular view in this embodiment does not require complex image matching operations, and due to the auxiliary detection of the front end point of the auxiliary device, the problem of inaccurate and time-consuming measurement results caused by easy mismatching of image matching in weak texture areas is effectively solved, and the accuracy of identifying identification points in weak texture areas is also improved.

[0068] In some embodiments, detecting the first identification point and the second identification point in each left and right view to be measured respectively to obtain the coordinates of the identification point in each view includes:

[0069] Detect each left and right view to be measured. If the number of identification points in the detected view is not equal to 2, an error is reported;

[0070] If the number of identification points detected in the view is equal to 2, the coordinates of the identification points in each view are obtained.

[0071] Specifically, in this embodiment, the identification points in the left view image to be measured and the right view image to be measured need to be further detected. If the number of identification points in any of the left view image to be measured and the right view image to be measured is not equal to 2, the measurement is suspended and an error is reported. It can be understood that a unique line segment can be determined by two points, and the adjacent line segments are spliced ​​to obtain the unique curve to be measured. If the number of identification points is not equal to 2, the line segment to be measured cannot be formed; or multiple line segments are obtained, so it is impossible to determine that a unique line segment to be measured is obtained. Therefore, if the number of identification points in any image is not equal to 2, an error is reported.

[0072] Similarly, if the number of identification points in the left view to be measured or the right view to be measured is equal to 2, the detection is continued to obtain the coordinates of the identification points in each view. Through this embodiment, the identification points in the left view image to be measured and the right view image to be measured can be detected before the coordinate conversion is performed, thereby further ensuring the accuracy of the measurement of the length of the line segment to be measured and improving the detection accuracy.

[0073] In some embodiments, the coordinates of the first identification point and the second identification point in all the left and right views to be measured are adjusted based on the reference coordinates of the first identification point in the reference left and right views to obtain the coordinates of the first target identification point and the second target identification point in each view, respectively, including:

[0074] Compare the coordinates of the first marking point in each left and right view to be measured with the first marking point reference coordinates respectively, and calculate the coordinate deviation value corresponding to each first marking point coordinate;

[0075] Adjust the corresponding first marking point coordinates based on each coordinate deviation value until the first marking point coordinates in each left and right view to be measured are equal to the first marking point reference coordinates, thereby obtaining the first target marking point coordinates;

[0076] The corresponding second identification point coordinates are adjusted based on each coordinate deviation value to obtain the corresponding second target identification point coordinates, wherein the target identification point coordinates include the first target identification point coordinates and the second target identification point coordinates.

[0077] Specifically, in some preferred embodiments, the above-mentioned reference left and right views can be set to the last frame of the left and right views to be measured collected in chronological order. The coordinates of the first identification point in the above-mentioned reference left and right views are used as the first identification point reference coordinates, and the coordinates of the first identification points in the remaining left and right views to be measured are respectively compared with the first identification point reference coordinates. In an ideal state, the first identification point reference coordinates should be the same as the first identification point coordinates in the remaining left and right views to be measured. However, in practical applications, due to reasons such as patient breathing, the entire screen will move, so in most cases the first identification point reference coordinates are not the same as the remaining first identification point coordinates. At this time, the first identification point reference coordinates should be used as the standard, and the coordinate deviation values ​​of the remaining first identification point coordinates compared to the first identification point reference coordinates are calculated. The coordinate deviation value is preferably the difference between the first identification point reference coordinates and the first identification point coordinates, and the first identification point coordinates and the second identification point coordinates corresponding to the first identification point coordinates are adjusted accordingly based on the coordinate deviation value to obtain the corresponding first target identification point coordinates and the second target identification point coordinates. Through this embodiment, the problem of inconsistent identification point coordinates in different views caused by image jitter during curve measurement is effectively solved, and the accuracy and robustness of measurement are improved.

[0078] In some embodiments, the left and right views to be measured include a left view to be measured and a right view to be measured; performing three-dimensional coordinate transformation on the coordinates of the first target identification point and the second target identification point in the left and right views to be measured to obtain the corresponding three-dimensional coordinates of the first identification point and the second identification point located on the curve to be measured includes:

[0079] Calculate the disparity information of the first target identification point coordinates and the second target identification point coordinates between each left view to be measured and the right view to be measured, and determine the depth information of each identification point according to the disparity information;

[0080] Based on the depth information, the coordinates of the first target identification point and the coordinates of the second target identification point are respectively converted into three-dimensional coordinates to obtain the three-dimensional coordinates of the first identification point and the three-dimensional coordinates of the second identification point correspondingly.

[0081] Specifically, in practical applications, binocular cameras are generally calibrated, so there is only lateral parallax in the left view image to be measured and the right view image to be measured, that is, the above disparity information is the lateral disparity information of the coordinates of the marking points in the left view and the right view. Based on the above disparity information, the depth information of the coordinates of each marking point can be determined, and the calculation formula of the depth information is:

[0082]

[0083] Among them, f is the focal length of the camera, b is the baseline distance, d is the parallax information, and z is the depth value of the coordinate. It can be understood that after the camera is calibrated, the focal length and baseline distance of the camera are known parameters. Therefore, according to the depth information of the coordinates of each marker point calculated, the coordinates of each marker point are converted into the corresponding three-dimensional coordinates of the marker point.

[0084] In some embodiments, the three-dimensional coordinates of the first marking point and the three-dimensional coordinates of each second marking point are sequentially connected in the order of shooting time to obtain the curve information of the curve to be measured, including:

[0085] Connect the three-dimensional coordinates of the adjacent second marking points by straight lines according to the shooting time sequence to obtain the path information of the curve to be measured;

[0086] The sum of the line segment lengths between the three-dimensional coordinates of the adjacent second marking points is calculated to obtain the length information of the curve to be measured, wherein the curve information includes path information and length information.

[0087] Specifically, after the three-dimensional coordinates of the second identification point at each moment are known, the three-dimensional coordinates of the first identification point, the three-dimensional coordinates of the second identification point obtained by the first shooting, the three-dimensional coordinates of the second identification point obtained by the second shooting, the three-dimensional coordinates of the second identification point obtained by the third shooting, etc. are sequentially connected by straight lines in the order of shooting time to obtain the path information of the curve to be measured, and the path information represents the shape and other information of the curve to be measured. The sum of the lengths of the line segments between the three-dimensional coordinates of the identification points of the above-mentioned adjacent identification points is calculated to obtain the length information of the curve to be measured, wherein the distance between the three-dimensional coordinates of adjacent identification points can be obtained by calculating the Euclidean distance between the three-dimensional coordinates of adjacent identification points.

[0088] In one embodiment, the method further comprises:

[0089] The second auxiliary device at different times moves along the curve to be measured according to a preset movement rule, wherein the movement rule includes a preset movement speed and a preset movement space rule; the movement space rule includes one of the following: moving from top to bottom, moving from bottom to top, moving from left to right, and moving from right to left.

[0090] Specifically, in practical applications, the second auxiliary device can move along the curve to be measured according to a preset movement rule, and the movement rule includes a movement speed. In some embodiments, a fixed movement speed can be preset by relevant technical personnel to control the second auxiliary device to move along the curve to be measured according to the movement speed; in some embodiments, different movement speeds can be set for different parts of the curve to be measured, such as setting a slower movement speed for the part with larger changes and larger fluctuations in the curve to be measured, and setting a faster movement speed for the part with smaller changes in the curve to be measured. Further, the above movement rule also includes a movement space rule, which is determined by the path of the curve to be measured, that is, the three-dimensional shape. For example, if the path of the curve to be measured is roughly vertically distributed, the above spatial order can be set from top to bottom or from bottom to top, so that the second identification point can traverse the curve to be measured more comprehensively. Similarly, if the path of the curve to be measured is roughly horizontally distributed, the above spatial order can be set from left to right or from right to left.

[0091] The present application also provides a preferred embodiment of a curve measurement method based on a binocular camera. Figure 2 Schematic diagram of a flow chart of a curve measurement method in a preferred embodiment.

[0092] Step S210, fixing the first auxiliary device at a preset end point of the curve to be measured, and moving the second auxiliary device along the curve to be measured at a preset speed, during which the left and right views of the curve to be measured are acquired.

[0093] Step S220, detect whether the number of auxiliary instruments in each view is equal to 2, if so, jump to step S230, if not, end the curve measurement process and perform error processing. Among them, the detection of the left and right views to be measured is mostly achieved by training a complete neural network, and the training of the neural network includes: (1) making a data set, collecting surgical scene materials containing commonly used surgical instruments (i.e., auxiliary instruments), marking the position, type and front end point coordinates of the instruments, and trying to include surgical instrument materials of multiple angles, multiple types and multiple scenes; (2) training the initial neural network based on the data set. This embodiment uses the yolo network and adds a key point detection function to the yolo target detection algorithm. Using the trained network, the number of instruments, the position of the instruments and the front end point coordinates of the instruments in the left and right images can be detected respectively.

[0094] Step S230, the position information of the auxiliary device in the left and right views to be measured is obtained by detecting the trained neural network, and the coordinates of the first identification point and the second identification point are respectively determined by the coordinates of the front end points of the first auxiliary device and the second auxiliary device, and the coordinate sequence of the two auxiliary devices is output in time order, and the one-to-one correspondence of the devices is maintained in combination with the target tracking algorithm. In some preferred embodiments, the target tracking algorithm includes identifying the two auxiliary devices in each left and right view to be measured, and predicting the movement direction and speed of each auxiliary device respectively. In some preferred embodiments, after the left and right views to be measured in the current frame are obtained, the left and right views to be measured in the current frame are firstly subjected to the above-mentioned detection processing to obtain the position information and the coordinates of the identification points of the two auxiliary devices in the left and right views to be measured in the current frame, and then the moving direction and speed of the corresponding auxiliary device in the next frame are predicted according to the detection results, so as to obtain the predicted position of the auxiliary device in the next frame. After the prediction is completed, the next frame of the left and right views to be measured transmitted back at the next moment is received. If the position of the auxiliary device in the obtained left and right views of the next frame to be measured is significantly different from the predicted position, it is determined that there is an abnormality at this time, and there may be a problem such as a loss of a left and right view to be measured in a certain frame in the sequence or a detection error, and an error reporting process is required. In other embodiments, if it is detected that the position of the auxiliary equipment in the left and right views to be measured in the next frame is significantly different from the position predicted based on the current frame, the prediction can be continued based on the position of the auxiliary equipment in the left and right views to be measured in the next frame, or the prediction can be continued based on the speed and direction predicted in the current frame, and the prediction result can be compared with the position of the auxiliary equipment in the left and right views to be measured in the next frame, and so on. If there is a large difference between the predicted position and the actual position in multiple consecutive frames, an alarm process is performed to reduce the impact of false alarms and false detections.

[0095] Step S240, according to the reference coordinates of the first identification point in the reference left and right views in the left and right views to be measured, the coordinates of the first identification point and the second identification point in the remaining left and right views to be measured are adjusted to obtain the adjusted coordinates of the first target identification point and the second target identification point in each view. The adjusted coordinates of the first target identification point in each left and right view to be measured should be the same.

[0096] Step S250, performing three-dimensional coordinate conversion according to the parallax information between the coordinates of the first target identification point and the coordinates of the second target identification point in the left and right views to be measured to obtain the corresponding three-dimensional coordinates of the first identification point and the second identification point.

[0097] Step S260, connect the three-dimensional coordinates of each first identification point and the three-dimensional coordinates of the second identification point in sequence according to the shooting time to obtain the curve information of the curve to be measured. Among them, by calculating the Euclidean distance between the three-dimensional coordinates of the identification points at adjacent moments, the curve length between the starting point and the end point of the curve to be measured can be obtained after summing, and the three-dimensional coordinates of the identification points at each moment and the measurement results are output and connected in sequence to display the actual measurement route and the corresponding measurement results.

[0098] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0099] Based on the same inventive concept, the embodiment of the present application also provides a curve measuring device for implementing the curve measuring method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more curve measuring device embodiments provided below can refer to the limitations of the curve measuring method above, and will not be repeated here.

[0100] In one embodiment, Figure 3 As shown, a curve measuring device is provided, including: an acquisition module 31, a calculation module 32 and a generation module 33, wherein:

[0101] An acquisition module 31 is used to acquire a plurality of left and right views to be measured at different times, wherein each view in a pair of left and right views to be measured includes a first identification point and a second identification point located on a preset curve to be measured on a target object; the first identification point in each left and right view to be measured is located at a preset end point of the curve to be measured, and the second identification points in different left and right views to be measured are located at different positions on the curve to be measured, and the coordinates of the second identification point at all times cover the preset target point position in the curve to be measured;

[0102] The calculation module 32 is used to detect the first identification point and the second identification point in each left and right view to be measured, respectively, to obtain the coordinates of the identification points in each view; determine the reference left and right views in the left and right views to be measured, and adjust the coordinates of the first identification point and the second identification point in all the left and right views to be measured based on the reference coordinates of the first identification point in the reference left and right views, to obtain the first target identification point coordinates and the second target identification point coordinates in each view; perform three-dimensional coordinate conversion on the first target identification point coordinates and the second target identification point coordinates in the left and right views to be measured, to obtain the corresponding three-dimensional coordinates of the first identification point and the second identification point located on the curve to be measured;

[0103] The generating module 33 is used to sequentially connect the three-dimensional coordinates of the first marking point and the three-dimensional coordinates of each second marking point according to the shooting time sequence to obtain the curve information of the curve to be measured.

[0104] Each module in the above curve measurement device can be implemented in whole or in part by software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module.

[0105] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 4 As shown. The computer device includes a processor, a memory and a network interface connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data related to the curve measurement algorithm. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a curve measurement method is implemented.

[0106] Those skilled in the art will understand that Figure 4The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0107] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0108] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0109] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0110] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A curve measurement method based on a binocular camera, characterized in that: The method comprises: Acquire multiple left and right views to be measured at different times, wherein each view in a pair of the left and right views to be measured includes a first identification point and a second identification point located on a preset curve to be measured on a target object; the first identification point in each pair of the left and right views to be measured is located at a preset end point of the curve to be measured, and the second identification points in different pairs of the left and right views to be measured are located at different positions on the curve to be measured, and the coordinates of the second identification point at all times cover the preset target point position in the curve to be measured; Respectively detecting the first marking point and the second marking point in each pair of left and right views to be measured to obtain the coordinates of the marking points in each view; Determine a reference left and right view in the left and right views to be measured, and adjust the coordinates of the first identification point and the second identification point in all the left and right views to be measured based on the reference coordinates of the first identification point in the reference left and right view, to obtain the first target identification point coordinates and the second target identification point coordinates in each view respectively; Performing three-dimensional coordinate conversion on the coordinates of the first target identification point and the coordinates of the second target identification point in the left and right views to be measured, to obtain corresponding three-dimensional coordinates of the first identification point and the second identification point on the curve to be measured; The three-dimensional coordinates of the first marking point and the three-dimensional coordinates of each of the second marking points are sequentially connected in the order of shooting time to obtain the curve information of the curve to be measured.

2. The method according to claim 1, characterized in that: The left and right views to be measured include a left view to be measured and a right view to be measured, and the detecting the first identification point and the second identification point in each pair of the left and right views to be measured respectively includes: Detecting each pair of the left view to be measured and the right view to be measured, and determining the first auxiliary instrument and the second auxiliary instrument included in each view, wherein the first auxiliary instrument is fixed at the preset end point, and the second auxiliary instrument at different times corresponds to different positions on the curve to be measured; The corresponding first identification point and the second identification point are determined by the front end points of the first auxiliary tool and the second auxiliary tool in each of the left and right views to be measured.

3. The method according to claim 1, characterized in that The detecting the first identification point and the second identification point in each pair of left and right views to be measured respectively to obtain the coordinates of the identification points in each view includes: Input each of the left view to be measured and each of the right view to be measured into a well-trained detection neural network, and respectively detect and obtain the position information of the two auxiliary instruments in each view, and the front end point corresponding to each of the auxiliary instruments; The corresponding two-dimensional coordinates of the identification point in each view are obtained through the front end point of the auxiliary device.

4. The method according to claim 1, characterized in that The detecting the first identification point and the second identification point in each pair of left and right views to be measured respectively to obtain the coordinates of the identification points in each view includes: Detecting each of the left and right views to be measured, and if it is detected that the number of the identification points in the view is not equal to 2, an error processing is performed; If it is detected that the number of the identification points in the view is equal to 2, the coordinates of the identification points in each view are obtained.

5. The method according to claim 1, characterized in that The step of adjusting the coordinates of the first identification point and the second identification point in all the left and right views to be measured based on the first identification point reference coordinates in the reference left and right views to obtain the first target identification point coordinates and the second target identification point coordinates in each view respectively includes: Compare the coordinates of the first marking point in each of the left and right views to be measured with the first marking point reference coordinates respectively, and calculate the coordinate deviation value corresponding to each of the first marking point coordinates; Adjusting the corresponding first marking point coordinates based on each of the coordinate deviation values ​​until the first marking point coordinates in each of the left and right views to be measured are equal to the first marking point reference coordinates, thereby obtaining the first target marking point coordinates; Based on each of the coordinate deviation values, the corresponding coordinates of the second identification point are adjusted to obtain the corresponding coordinates of the second target identification point.

6. The method according to claim 5, characterized in that The left and right views to be measured include a left view to be measured and a right view to be measured; the three-dimensional coordinate conversion of the first target identification point coordinates and the second target identification point coordinates in the left and right views to be measured to obtain the corresponding first identification point three-dimensional coordinates and second identification point three-dimensional coordinates located on the curve to be measured includes: Calculating disparity information of the first target identification point coordinates and the second target identification point coordinates between each pair of the left view to be measured and the right view to be measured, and determining depth information of each identification point according to the disparity information; Based on the depth information, the first target identification point coordinates and the second target identification point coordinates are respectively converted into three-dimensional coordinates to obtain the first identification point three-dimensional coordinates and the second identification point three-dimensional coordinates accordingly.

7. The method according to claim 1, characterized in that The step of sequentially connecting the three-dimensional coordinates of the first identification point and the three-dimensional coordinates of each of the second identification points in the order of shooting time to obtain the curve information of the curve to be measured includes: Connecting the three-dimensional coordinates of the adjacent second marking points by straight lines according to the order of shooting time to obtain the path information of the curve to be measured; The sum of the line segment lengths between the three-dimensional coordinates of the adjacent second identification points is calculated to obtain the length information of the curve to be measured, wherein the curve information includes the path information and the length information.

8. The method according to any one of claims 1 to 7, characterized in that: The method further comprises: The second auxiliary device at different times moves along the curve to be measured according to a preset movement rule, wherein the movement rule includes a preset movement speed and a preset movement space rule; the movement space rule includes one of the following: moving from top to bottom, moving from bottom to top, moving from left to right, and moving from right to left.

9. A curve measuring device, characterized in that: The device comprises: An acquisition module is used to acquire a plurality of left and right views to be measured at different times, wherein each view in a pair of the left and right views to be measured includes a first identification point and a second identification point located on a preset curve to be measured on a target object; the first identification point in each pair of the left and right views to be measured is located at a preset end point of the curve to be measured, and the second identification points in different pairs of the left and right views to be measured are located at different positions on the curve to be measured, and the coordinates of the second identification point at all times cover the preset target point position in the curve to be measured; A calculation module, used to detect the first identification point and the second identification point in each pair of left and right views to be measured, respectively, to obtain the coordinates of the identification points in each view; determine the reference left and right views in the left and right views to be measured, and adjust the coordinates of the first identification point and the second identification point in all the left and right views to be measured based on the reference coordinates of the first identification point in the reference left and right views, to obtain the first target identification point coordinates and the second target identification point coordinates in each view, respectively; perform three-dimensional coordinate conversion on the first target identification point coordinates and the second target identification point coordinates in the left and right views to be measured, to obtain the corresponding three-dimensional coordinates of the first identification point and the second identification point located on the curve to be measured; A generation module is used to sequentially connect the three-dimensional coordinates of the first identification point and the three-dimensional coordinates of each of the second identification points in the order of shooting time to obtain the curve information of the curve to be measured.

10. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 8 are implemented.

Images