Capillary length determination method and device, electronic equipment and storage medium

Abstract

The invention relates to the technical field of steel tube size online detection, in particular to a tubular billet length determination method and device, electronic equipment and a storage medium, and the method comprises the steps: firstly obtaining a first tubular billet image and a second tubular billet image; then according to a shooting part of a corresponding camera, identifying an end edge line of the first capillary image and an end edge line of the second capillary image to obtain a first end edge line and a second end edge line; carrying out image coordinate transformation on the coordinate of the first end part edge line in the first capillary image and the coordinate of the second end part edge line in the second capillary image; taking the obtained coordinates of the first end edge line in the camera coordinate system and the obtained coordinates of the second end edge line in the camera coordinate system as first camera coordinates and second camera coordinates respectively; and finally, determining the length of the capillary tube according to the relative coordinates of the two cameras, the coordinates of the first camera and the coordinates of the second camera. According to the invention, the accuracy and stability of capillary tube length detection are ensured.

Description

Technical Field

[0001] The present invention relates to the technical field of on-line detection of steel pipe dimensions, and particularly to a method, device, electronic device and storage medium for determining the length of a rough pipe. Background Art

[0002] A rough pipe of a steel pipe refers to a hollow long-strip semi-finished product with certain dimensions and surface quality processed from an ingot or a billet through processes such as piercing during the production process of the steel pipe. It is the basis for further processing into steel pipes of various specifications and uses. The main use of the rough pipe of the steel pipe is as an intermediate raw material for further processing of the steel pipe. Through subsequent hot rolling, cold rolling, cold drawing and other processes, the rough pipe can be processed into finished steel pipes of various different specifications and uses, which are widely used in many fields such as construction, machinery, petrochemical industry, shipbuilding, and aerospace.

[0003] On-line detection during rough pipe production refers to the real-time detection of multiple parameters including the length of the rough pipe during the production process of the rough pipe, so as to timely adjust the production equipment to meet the quality requirements of the rough pipe.

[0004] Currently, the mainstream technologies for on-line detection of the outer diameter of rough pipes include: machine vision detection method, photoelectric on-line diameter measuring instrument detection method, and ultrasonic detection method. Among them, the machine vision detection method is highly regarded for its high environmental adaptability.

[0005] However, the main problem currently existing in the machine vision detection method is that the positioning of the measured end points of the rough pipe is inaccurate, resulting in a large error in the detection result.

[0006] Based on this, it is necessary to develop and design a method for determining the length of a rough pipe. Summary of the Invention

[0007] Embodiments of the present invention provide a method, device, electronic device and storage medium for determining the length of a rough pipe, which are used to solve the problem that the on-line length detection result of the rough pipe based on machine vision in the prior art is inaccurate.

[0008] In a first aspect, embodiments of the present invention provide a method for determining the length of a rough pipe, including: Obtaining a first rough pipe image and a second rough pipe image, wherein the first rough pipe image and the second rough pipe image are respectively obtained from two fixed cameras; Identifying the end edge lines of the first rough pipe image and the end edge lines of the second rough pipe image according to the shooting parts of the corresponding cameras, and obtaining a first end edge line and a second end edge line; Perform image coordinate transformation on the coordinates of the first end edge line in the first capillary image and the coordinates of the second end edge line in the second capillary image, and use the obtained coordinates of the first end edge line in the camera coordinate system and the coordinates of the second end edge line in the camera coordinate system as the first camera coordinate and the second camera coordinate respectively; Determine the capillary length according to the relative coordinates of the two cameras, the first camera coordinate, and the second camera coordinate.

[0009] In a possible implementation manner, the identifying the end edge line of the first capillary image and the end edge line of the second capillary image according to the shooting part of the corresponding camera to obtain the first end edge line and the second end edge line includes: For any one of the first capillary image and the second capillary image, perform the following steps respectively: Perform Gaussian filtering on the capillary image to obtain a smoothed capillary image; Perform gradient operation on the smoothed capillary image using an operator to obtain a gradient capillary image; Perform non-maximum suppression on the gradient capillary image to obtain a first edge capillary image; Perform double-threshold detection on the first edge capillary image to obtain a second edge capillary image; Send the second edge capillary image into a deep learning model to identify the image block where the end edge line is located.

[0010] In a possible implementation manner, the deep learning model is obtained by training according to multiple sample images, and the construction process of the sample label includes: For each sample image, perform the following steps respectively: Obtain a target edge point, where the target edge point is located on the end edge line of the sample image; Taking the target edge point as the center, obtain a plurality of first edge points from within a preset neighborhood radius; According to the first formula and the plurality of first edge points, use the least squares method to construct an edge line equation expressing the end edge line, where the first formula is:

[0011] In the formula, , , , , and are the first coefficient, the second coefficient, the third coefficient, the fourth coefficient, the fifth coefficient, and the sixth coefficient respectively, is the x-axis coordinate of the first edge point in the sample image, is the y-axis coordinate of the first edge point in the sample image; Substitute the multiple first edge points into the edge line equation respectively to obtain multiple deviation values; Take the maximum value among the multiple deviation values as the reference value; Taking the target edge point as the center, according to the edge line equation, obtain multiple second edge points, wherein the value obtained by substituting the second edge point into the edge line equation is not greater than the reference value, and the multiple second edge points are continuous; Take the smallest rectangle enclosing the multiple second edge points as the identification rectangle; Take the center coordinates of the identification rectangle, the width of the identification rectangle, and the length of the identification rectangle as the sample label.

[0012] In a possible implementation manner, performing image coordinate transformation on the coordinates of the first end edge line in the first capillary image and the coordinates of the second end edge line in the second capillary image, and taking the obtained coordinates of the first end edge line in the camera coordinate system and the coordinates of the second end edge line in the camera coordinate system as the first camera coordinate and the second camera coordinate respectively, includes: For each end edge line, perform the following steps respectively: Use the least squares method to fit the end edge line to obtain the first general equation of the ellipse describing the end edge line; Perform formula transformation on the first general equation of the ellipse, and determine the first focus and the second focus according to the result of the formula transformation; Take the midpoint of the first focus and the second focus as the capillary center point; Perform camera coordinate transformation on the coordinates of the capillary center point to obtain the camera coordinate.

[0013] In a possible implementation manner, performing formula transformation on the first general equation of the ellipse, and determining the first focus and the second focus according to the result of the formula transformation, includes: If there is a cross term in the first general equation of the ellipse, perform a rotation transformation on the first general equation of the ellipse according to the second formula to obtain the second general equation of the ellipse, where the second formula is:

[0014] In the formula, is the horizontal axis coordinate in the coordinate system after rotation transformation, is the vertical axis coordinate in the coordinate system after rotation transformation, is the horizontal axis coordinate in the original coordinate system, is the vertical axis coordinate in the original coordinate system, is the rotation transformation angle, , and are the first coefficient, the second coefficient, and the third coefficient of the first general equation of the ellipse, respectively. is the arctangent function; Otherwise, use the first general equation of the ellipse as the second general equation of the ellipse; Convert the second general equation of the ellipse by completing the square into the first standard equation of the ellipse; Determine the third focus and the fourth focus according to the first standard equation of the ellipse; If there is a cross term in the first general equation of the ellipse, perform a coordinate inverse rotation transformation on the third focus and the fourth focus according to the second formula to obtain the first focus and the second focus; Otherwise, use the third focus and the fourth focus as the first focus and the second focus.

[0015] In a possible implementation manner, the conversion of the coordinates of the capillary center point into the camera coordinate system to obtain the camera coordinate includes: Obtain the camera internal parameter matrix and the depth information; Convert the coordinates of the capillary center point according to the third formula, the camera internal parameter matrix, and the depth information to obtain the camera coordinate, where the third formula is:

[0016] In the formula, , and are the coordinates of the capillary center point on the x-axis, y-axis, and z-axis in the camera coordinate system, respectively. is the depth information, is the internal parameter matrix, is the focal length of the x-axis, is the focal length of the y-axis, and are the coordinates of the intersection point of the optical axis and the image plane on the x-axis and y-axis in the image plane, respectively. and are the coordinates of the capillary center point on the x-axis and y-axis in the image coordinate system, respectively.

[0017] In a possible implementation manner, the determination of the capillary length according to the relative coordinates of two cameras, the first camera coordinate, and the second camera coordinate includes: Determine the capillary length according to the fourth formula, the relative coordinates of two cameras, the first camera coordinate, and the second camera coordinate, where the fourth formula is:

[0018] In the formula, , and are the x-axis, y-axis, and z-axis components of the first camera coordinate, respectively. , and are the x-axis, y-axis, and z-axis components of the second camera coordinate, respectively. , and are the x-axis, y-axis, and z-axis components of the relative coordinate of the camera, respectively.

[0019] In a second aspect, an embodiment of the present invention provides a capillary length determination device for implementing the capillary length determination method described in the first aspect or any possible implementation manner of the first aspect above. The capillary length determination device includes: A capillary image acquisition module for acquiring a first capillary image and a second capillary image, where the first capillary image and the second capillary image are obtained from two fixed cameras respectively; A capillary edge recognition module for recognizing the end edge lines of the first capillary image and the second capillary image according to the shooting parts of the corresponding cameras, and obtaining a first end edge line and a second end edge line; A coordinate transformation module for performing image coordinate transformation on the coordinates of the first end edge line in the first capillary image and the coordinates of the second end edge line in the second capillary image, and using the obtained coordinates of the first end edge line in the camera coordinate system and the coordinates of the second end edge line in the camera coordinate system as the first camera coordinate and the second camera coordinate respectively; And, A capillary length determination module for determining the capillary length according to the relative coordinates of the two cameras, the first camera coordinate, and the second camera coordinate.

[0020] In a third aspect, an embodiment of the present invention provides an electronic device including a memory and a processor. A computer program that can run on the processor is stored in the memory. When the processor executes the computer program, the steps of the method described in the first aspect or any possible implementation manner of the first aspect above are implemented.

[0021] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, the steps of the method described in the first aspect or any possible implementation manner of the first aspect above are implemented.

[0022] The beneficial effects of the embodiments of the present invention compared with the prior art are: An embodiment of the present invention discloses a method for determining the length of a capillary tube. First, a first capillary tube image and a second capillary tube image are obtained, where the first capillary tube image and the second capillary tube image are respectively obtained from two fixed cameras. Then, according to the shooting parts of the corresponding cameras, the end edge lines of the first capillary tube image and the end edge lines of the second capillary tube image are identified to obtain a first end edge line and a second end edge line. Next, image coordinate transformation is performed on the coordinates of the first end edge line in the first capillary tube image and the coordinates of the second end edge line in the second capillary tube image, and the coordinates of the first end edge line in the camera coordinate system and the coordinates of the second end edge line in the camera coordinate system obtained are respectively used as the first camera coordinate and the second camera coordinate. Finally, the length of the capillary tube is determined according to the relative coordinates of the two cameras, the first camera coordinate, and the second camera coordinate. The embodiment of the present invention is based on the identification of the end edge line. By converting the edge line from the image coordinate to the camera coordinate and then combining the relative coordinates of the two cameras to determine the length of the capillary tube. Since the method of the present invention determines the length of the capillary tube based on the end edge line, the end positioning is accurate, ensuring the accuracy and stability of the capillary tube length detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following will briefly introduce the drawings required for use in the embodiments or the description of the prior art. Obviously, the following drawings are only some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0024] Figure 1 is a flowchart of the method for determining the length of a capillary tube provided by the embodiment of the present invention; Figure 2 is an application scenario diagram of the method for determining the length of a capillary tube provided by the embodiment of the present invention; Figure 3 is a functional block diagram of the device for determining the length of a capillary tube provided by the embodiment of the present invention; Figure 4 is a functional block diagram of the electronic device provided by the embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0025] In the following description, specific details such as specific system structures and technologies are presented for the purpose of illustration rather than limitation, so as to thoroughly understand the embodiments of the present invention. However, those skilled in the art should clearly understand that the present invention can also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, and methods are omitted to avoid unnecessary details from interfering with the description of the present invention.

[0026] To make the objectives, technical solutions and advantages of the present invention clearer, the following will explain through specific embodiments with reference to the accompanying drawings.

[0027] The following will elaborate on the embodiments of the present invention. This example is implemented on the premise of the technical solution of the present invention, and detailed implementation manners and specific operation procedures are given. However, the protection scope of the present invention is not limited to the following embodiments.

[0028] Figure 1 It is a flowchart of the capillary length determination method provided by the embodiment of the present invention.

[0029] As Figure 1 shown, it shows the implementation flowchart of the capillary length determination method provided by the embodiment of the present invention, which is elaborated as follows: In step 101, a first capillary image and a second capillary image are acquired, where the first capillary image and the second capillary image are respectively obtained from two fixed cameras.

[0030] Exemplarily, as Figure 2 shown, this figure shows an application scenario diagram of the capillary length determination method of the present invention. In the figure, two cameras 203 are fixed on one side of the capillary 201. Generally, the connection line of the two cameras 203 is parallel to the axis of the capillary 201. A hot metal detector 202 is provided on one side of the capillary 201. Under the indication of the hot metal detector 202, the two cameras 203 capture photos synchronously. The two capillary images obtained by the capture are analyzed visually, and combined with the positions of the two cameras 203, the length of the capillary 201 is determined.

[0031] The method of the present invention identifies the edge lines at the ends through the capillary images, then determines the center of the capillary end based on the edge lines, and finally determines the capillary length based on the centers of the two ends. Compared with the existing visual recognition methods, since this method measures the length based on the center points, the error is small and the obtained results are relatively stable.

[0032] The embodiment of the present invention elaborates in detail on the implementation process of the above principle from the following aspects.

[0033] In step 102, according to the shooting parts of the corresponding cameras, the edge lines at the ends of the first capillary image and the second capillary image are identified to obtain a first edge line at the end and a second edge line at the end.

[0034] In some embodiments, the identifying the edge lines at the ends of the first capillary image and the second capillary image according to the shooting parts of the corresponding cameras to obtain a first edge line at the end and a second edge line at the end includes: For any one of the first capillary image and the second capillary image, the following steps are respectively performed: Perform Gaussian filtering on the capillary image to obtain a smoothed capillary image; Perform gradient operation on the smoothed capillary image by using an operator to obtain a gradient capillary image; Perform non-maximum suppression on the gradient capillary image to obtain a first edge capillary image; Perform double-threshold detection on the first edge capillary image to obtain a second edge capillary image; Send the second edge capillary image into a deep learning model to identify the image block where the end edge line is located.

[0035] In some embodiments, the deep learning model is obtained by training according to a plurality of sample images. Among them, the construction process of the sample label includes: For each sample image, the following steps are respectively performed: Obtain a target edge point, where the target edge point is located on the end edge line of the sample image; Taking the target edge point as the center, obtain a plurality of first edge points from within a preset neighborhood radius; According to the first formula and the plurality of first edge points, use the least squares method to construct an edge line equation expressing the end edge line, where the first formula is:

[0036] In the formula, , , , , and are the first coefficient, the second coefficient, the third coefficient, the fourth coefficient, the fifth coefficient and the sixth coefficient respectively, is the x-axis coordinate of the first edge point in the sample image, is the y-axis coordinate of the first edge point in the sample image; Substitute the plurality of first edge points into the edge line equation respectively to obtain a plurality of deviation values; Take the maximum value among the plurality of deviation values as the reference value; Taking the target edge point as the center, according to the edge line equation, obtain a plurality of second edge points, where the value obtained by substituting the second edge point into the edge line equation is not greater than the reference value, and the plurality of second edge points are continuous; Take the smallest rectangle enclosing the plurality of second edge points as the identification rectangle; Take the center coordinates of the identification rectangle, the width of the identification rectangle, and the length of the identification rectangle as the sample label.

[0037] Exemplarily, in the present invention, the capillary image is subjected to edge detection to obtain an edge image, and then the edge image is fed into a deep learning model for end edge line recognition.

[0038] In terms of edge detection, the embodiment of the present invention first performs Gaussian filtering (Gaussian blur) on the capillary image, the purpose of which is to remove noise in the image. Then, an operator is used to perform gradient operation on the image. For example, the Sobel operator is adopted to detect edges by calculating the gradients of the image in the horizontal and vertical directions.

[0039] Specifically, two Sobel operators (horizontal and vertical) are respectively convolved with the image. One is used to detect horizontal edges, and the other is used to detect vertical edges. Then, the gradient magnitude and gradient direction are determined according to the results of the two convolutions.

[0040] In edge detection, possible edge points are initially determined by calculating the gradient magnitude and direction of the image. However, these initially detected edge points may have a certain width, and on the true edge, the gradient magnitude should be locally maximum. The purpose of non-maximum suppression is to retain only those points with locally maximum gradient magnitude among these possible edge points and suppress other non-maximum points, thereby refining the edge into a single-pixel-wide line and improving the edge positioning accuracy.

[0041] Therefore, after obtaining the gradient magnitude and gradient direction, non-maximum suppression is also required. First, the discrete gradient direction is suppressed: the gradient direction is discretized into several specific angles, usually divided into 4 or 8 directions, such as 0°, 45°, 90°, 135°, etc. This can facilitate subsequent comparison and judgment.

[0042] Then, the pixel points are traversed: starting from the upper left corner of the image, each pixel point is traversed row by row and column by column.

[0043] Next, the local gradient magnitudes are compared: for the current pixel point, according to its gradient direction, its gradient magnitude is compared with that of the adjacent pixel points in that direction. For example, if the gradient direction of the current pixel point is 0°, its gradient magnitude is compared with that of the left and right adjacent pixel points in the horizontal direction; if the gradient direction is 45°, its gradient magnitude is compared with that of the diagonally adjacent pixel points in that direction.

[0044] Then, the non-maximum points are suppressed: if the gradient magnitude of the current pixel point is not locally maximum (i.e., less than the gradient magnitude of the adjacent pixel points in its gradient direction), the gray value of this pixel point is set to 0, that is, this point is suppressed; if the gradient magnitude of the current pixel point is locally maximum, the gray value of this point remains unchanged.

[0045] Finally, a refined edge image is obtained: After processing all pixel points, only the points with local maximum gradient magnitudes are retained in the resulting image, and these points form the refined edge image.

[0046] After the foregoing steps, the edges of the above image take initial shape. However, there are still breakpoints and noise points in the image. Therefore, double-threshold detection is still required. The edge points in the image can be divided into strong edge points and weak edge points according to the magnitude of their gradient magnitudes. Double-threshold detection is to set a high threshold and a low threshold, determine the points with gradient magnitudes greater than the high threshold as strong edge points, directly exclude the points less than the low threshold, and mark the points between the low threshold and the high threshold as weak edge points. Then, by further processing the weak edge points, it is finally determined whether they are regarded as real edge points according to their connectivity with strong edge points.

[0047] A process of double-threshold detection includes: First, set the thresholds: According to the characteristics of the image and application requirements, select appropriate high threshold Th and low threshold Tl. Usually, the high threshold is 2 - 3 times the low threshold. Then, classify the edge points: Traverse each pixel point in the image and classify it according to the relationship between its gradient magnitude and the thresholds. Pixel points with gradient magnitudes greater than Th are marked as strong edge points and represented by white (or other specific colors); pixel points with gradient magnitudes less than Tl are marked as non-edge points and represented by black; pixel points with gradient magnitudes between Tl and Th are marked as weak edge points and represented by gray. Finally, connect the weak edge points: Traverse the pixels marked as weak edge points and check whether there are strong edge points in their 8-neighborhoods (i.e., the 8 adjacent pixel points around). If there is a strong edge point in the 8-neighborhood of a weak edge point, then retain the weak edge point as an edge point; if there is no strong edge point in the 8-neighborhood of a weak edge point, then delete the weak edge point, that is, do not regard it as an edge point. In this way, the weak edge points connected to strong edge points are retained to form complete edge lines, while isolated weak edge points are removed.

[0048] After the above steps, the edges of the image are extracted to form an edge capillary image.

[0049] In the recognition of the capillary end edge, the present invention applies deep learning methods to extract the end edge. The deep learning recognition methods include Faster R – CNN, YOLO. These deep learning methods for extracting the capillary end edge all require sample images to train them. These training sample images are set with labels identifying the capillary end edge. For example, one form of the label is the center point of the end edge, the length, and the width of the image block where it is located.

[0050] In order to identify a more accurate capillary end edge, in the label construction of the sample image in the embodiment of the present invention, by designating an end edge point, and then taking this end edge point as the center, searching for a plurality of edge pixels with a preset pixel radius. For example, 18 end edge pixels are searched with a radius of 10 pixels. The first formula is fitted by the searched end edge pixels using the least squares method, so as to determine the values of the six coefficients in the first formula. The first formula is:

[0051] In the formula, 、 、 、 、 and are the first coefficient, the second coefficient, the third coefficient, the fourth coefficient, the fifth coefficient and the sixth coefficient respectively, is the x-axis coordinate of the first edge point in the sample image, is the y-axis coordinate of the first edge point in the sample image.

[0052] After the construction is completed, substitute these searched edge pixels into the left side of the first line of the above formula respectively. The right side value is the deviation value. Take the maximum value in the deviation values as the reference value. Then, taking the designated end edge point as the center, find the edge pixels, and substitute the found edge pixels into the left side of the first line of the above formula. If the obtained value is less than the reference value, keep this pixel and continue to search. Otherwise, terminate the search process. It should be noted that during the search process, it is necessary to ensure that the found pixels are connected to the already found end edge pixels, so as to form a continuous edge line.

[0053] After the search is completed, take the smallest rectangle enclosing these found pixels as the identification rectangle, take the center of the identification rectangle as the label center coordinate, and take the length and width of the identification rectangle as the length and width of the label.

[0054] In this way, the deep learning model constructed through the sample image can find the edge line that conforms to the law of the first formula, laying a good image foundation for length detection.

[0055] In step 103, perform image coordinate transformation on the coordinates of the first end edge line in the first capillary image and the coordinates of the second end edge line in the second capillary image, and take the obtained coordinates of the first end edge line in the camera coordinate system and the coordinates of the second end edge line in the camera coordinate system as the first camera coordinate and the second camera coordinate respectively.

[0056] In some embodiments, performing image coordinate transformation on the coordinates of the first end edge line in the first capillary image and the coordinates of the second end edge line in the second capillary image, and using the obtained coordinates of the first end edge line in the camera coordinate system and the coordinates of the second end edge line in the camera coordinate system as the first camera coordinate and the second camera coordinate respectively, includes: For each end edge line, perform the following steps respectively: Use the least squares method to fit the end edge line to obtain the first general equation of the ellipse describing the end edge line; Perform formulation on the first general equation of the ellipse, and determine the first focus and the second focus according to the result of the formulation; Take the midpoint of the first focus and the second focus as the center point of the capillary; Perform camera coordinate system conversion on the coordinates of the center point of the capillary to obtain the camera coordinate.

[0057] In some embodiments, performing formulation on the first general equation of the ellipse, and determining the first focus and the second focus according to the result of the formulation, includes: If there is a cross term in the first general equation of the ellipse, perform a rotation transformation on the first general equation of the ellipse according to the second formula to obtain the second general equation of the ellipse, where the second formula is:

[0058] In the formula, is the horizontal axis coordinate in the coordinate system after the rotation transformation, is the vertical axis coordinate in the coordinate system after the rotation transformation, is the horizontal axis coordinate in the original coordinate system, is the vertical axis coordinate in the original coordinate system, is the rotation transformation angle, , and are the first coefficient, the second coefficient and the third coefficient of the first general equation of the ellipse respectively, is the arctangent function; Otherwise, take the first general equation of the ellipse as the second general equation of the ellipse; Formulate the second general equation of the ellipse into the first standard equation of the ellipse; Determine the third focus and the fourth focus according to the first standard equation of the ellipse; If there is a cross term in the first general equation of the ellipse, perform coordinate inverse rotation transformation on the third focus and the fourth focus according to the second formula to obtain the first focus and the second focus; Otherwise, use the third focus and the fourth focus as the first focus and the second focus.

[0059] In some embodiments, the conversion of the coordinates of the capillary center point into the camera coordinate system to obtain the camera coordinates includes: Obtain the camera internal parameter matrix and the depth information; Convert the coordinates of the capillary center point according to the third formula, the camera internal parameter matrix, and the depth information to obtain the camera coordinates, where the third formula is:

[0060] In the formula, , and are the coordinates of the capillary center point on the x-axis, y-axis, and z-axis in the camera coordinate system respectively, is the depth information, is the internal parameter matrix, is the focal length of the x-axis, is the focal length of the y-axis, and are the coordinates of the intersection point of the optical axis and the image plane on the x-axis and y-axis of the image plane respectively, and are the coordinates of the capillary center point on the x-axis and y-axis in the image coordinate system respectively.

[0061] Exemplarily, in detecting the length of the capillary, the present invention first finds the center point of the end according to the end edge line of the image, then converts the center points of the two ends into the coordinates in the corresponding camera coordinate system, and finally determines the length of the capillary according to the coordinates of the center points of the two ends in the camera coordinate system and the relative coordinates of the two cameras.

[0062] The end edge line of the capillary is an elliptical line in the image. The coordinates of the end edge line pixel points obtained through the foregoing steps can be used to fit the general ellipse equation by the least squares method. In order to distinguish from the ellipse equation in the first equation, the general ellipse equation used in this step is:

[0063] In the formula, , , , , and are the seventh coefficient, the eighth coefficient, the ninth coefficient, the tenth coefficient, the eleventh coefficient, and the twelfth coefficient respectively, is the horizontal axis coordinate of the first edge point in the sample image, is the vertical axis coordinate of the first edge point in the sample image.

[0064] After the fitting is completed, the elliptic equations corresponding to the two ends are obtained. According to these two elliptic equations, the image coordinates of the central points of the two end edges of the capillary can be found. Specifically, if there is a cross term ( ) a coordinate system rotation transformation needs to be performed. The transformation method of the present invention is carried out according to the following equation:

[0065] In the formula, is the horizontal axis coordinate in the coordinate system after the rotation transformation, is the vertical axis coordinate in the coordinate system after the rotation transformation, is the horizontal axis coordinate in the original coordinate system, is the vertical axis coordinate in the original coordinate system, is the rotation transformation angle, , and are respectively the first coefficient, the second coefficient and the third coefficient of the first general equation of the ellipse, is the arctangent function.

[0066] The transformed equation is used as the equation to be analyzed. If there is no cross term, the original equation is used as the equation to be analyzed.

[0067] Then, the equation to be analyzed is formulated. Taking the equation to be analyzed obtained after the transformation as an example, after formulation, we get:

[0068] According to this equation, the major axis and minor axis of the ellipse are obtained ( otherwise, in the following formula , are interchanged):

[0069]

[0070] In the above formula, , are respectively the major axis and minor axis (or minor axis and major axis). When , the two foci are , ; when , the two foci are , .

[0071] The two foci obtained in the above process are then inversely transformed through the transformation equation to obtain the coordinates of the two foci in the image coordinate system.

[0072] The median value of the two foci is the coordinate of the central point of the capillary end.

[0073] After the above steps, the coordinates of the center points of the two ends in the image coordinate system are obtained. Through the camera internal parameter matrix and depth information, these two coordinates are used to perform the transformation from the image coordinate system to the camera coordinate system. The embodiment of the present invention adopts the third formula:

[0074] In the formula, 、 and are the coordinates of the center point of the capillary tube on the x-axis, y-axis and z-axis in the camera coordinate system respectively, is the depth information, is the internal parameter matrix, is the focal length of the x-axis, is the focal length of the y-axis, and are the coordinates of the intersection points of the optical axis and the image plane on the x-axis and y-axis of the image plane respectively, and are the coordinates of the center point of the capillary tube on the x-axis and y-axis in the image coordinate system respectively.

[0075] The above internal parameter matrix is obtained through camera calibration. For example, Zhang Zhengyou calibration method: This method uses a planar template such as a checkerboard. By taking template images in different poses and according to the coordinate relationship between the pixel coordinates of the checkerboard corner points in the image and their coordinates in the world coordinate system, a system of equations is established to solve the camera internal parameter matrix. Specifically, the coordinates of the checkerboard corner points in the image are found through the corner detection algorithm. At the same time, the coordinates of the checkerboard corner points in the world coordinate system are known (usually assuming that the plane where the checkerboard is located is the xy plane of the world coordinate system, (z = 0)). Using the relationship of these corresponding points, a system of linear equations is constructed, and then the camera internal parameter matrix is solved.

[0076] Operation process: First, prepare a checkerboard template, place it in different positions and angles, and use the camera to take multiple checkerboard images. Then, perform corner detection on each image to extract the pixel coordinates of the checkerboard corner points. Finally, use these corner point coordinates and the known checkerboard world coordinates to calculate through the Zhang Zhengyou calibration algorithm to obtain the internal parameter matrix of the camera.

[0077] The depth information of the camera refers to the distance information between the object in the scene corresponding to the pixel point in the image and the camera. Since the camera in the embodiment of the present invention is fixed at a fixed position, the position where the capillary tube appears is also relatively determined. In addition, in some scenes, a hot metal detector is provided to detect the coordinates of the predetermined position of the capillary tube. Therefore, the depth information can be obtained relatively simply.

[0078] In step 104, the capillary length is determined based on the relative coordinates of the two cameras, the first camera coordinates, and the second camera coordinates.

[0079] In some embodiments, the determining the capillary length based on the relative coordinates of the two cameras, the first camera coordinates, and the second camera coordinates includes: Determining the capillary length according to the fourth formula, the relative coordinates of the two cameras, the first camera coordinates, and the second camera coordinates, where the fourth formula is:

[0080] In the formula, , and are the x-axis, y-axis, and z-axis components of the first camera coordinates, respectively. , and are the x-axis, y-axis, and z-axis components of the second camera coordinates, respectively. , and are the x-axis, y-axis, and z-axis components of the relative coordinates of the cameras, respectively.

[0081] Exemplarily, after obtaining the coordinates of the center point of the capillary end in the camera coordinates, combining the relative coordinates of the two cameras, and applying the fourth formula, the capillary length can be calculated:

[0082] In the formula, , and are the x-axis, y-axis, and z-axis components of the first camera coordinates, respectively. , and are the x-axis, y-axis, and z-axis components of the second camera coordinates, respectively. , and are the x-axis, y-axis, and z-axis components of the relative coordinates of the cameras, respectively.

[0083] Embodiment of the method for determining the capillary length of the present invention. First, a first capillary image and a second capillary image are obtained, where the first capillary image and the second capillary image are respectively obtained from two fixed cameras; then, according to the shooting parts of the corresponding cameras, the end edge lines of the first capillary image and the end edge lines of the second capillary image are identified to obtain a first end edge line and a second end edge line; then, image coordinate transformation is performed on the coordinates of the first end edge line in the first capillary image and the coordinates of the second end edge line in the second capillary image, and the coordinates of the first end edge line in the camera coordinate system and the coordinates of the second end edge line in the camera coordinate system obtained are respectively used as the first camera coordinate and the second camera coordinate; finally, the capillary length is determined according to the relative coordinates of the two cameras, the first camera coordinate, and the second camera coordinate. The embodiment of the present invention is based on the identification of the end edge line. By converting the edge line from the image coordinate to the camera coordinate and then combining the relative coordinates of the two cameras to determine the length of the capillary. Since the method of the present invention determines the capillary length based on the end edge line, the end positioning is accurate, ensuring the accuracy and stability of the capillary length detection.

[0084] It should be understood that the magnitudes of the sequence numbers of the steps in the above embodiments do not mean the order of execution. The order of execution of each process should be determined according to its function and internal logic, and should not constitute any limitation to the implementation process of the embodiments of the present invention.

[0085] The following is the device embodiment of the present invention. For the details not described in detail, reference can be made to the corresponding method embodiment above.

[0086] Figure 3 is the functional block diagram of the capillary length determination device provided by the embodiment of the present invention. Referring to Figure 3 , the capillary length determination device includes: a capillary image acquisition module 301, a capillary edge recognition module 302, a coordinate transformation module 303, and a capillary length determination module 304, where: The capillary image acquisition module 301 is used to acquire a first capillary image and a second capillary image, where the first capillary image and the second capillary image are respectively obtained from two fixed cameras; The capillary edge recognition module 302 is used to identify the end edge lines of the first capillary image and the end edge lines of the second capillary image according to the shooting parts of the corresponding cameras to obtain a first end edge line and a second end edge line; A coordinate transformation module 303 is configured to perform image coordinate transformation on the coordinates of the first end edge line in the first capillary image and the coordinates of the second end edge line in the second capillary image, and use the obtained coordinates of the first end edge line in the camera coordinate system and the coordinates of the second end edge line in the camera coordinate system as the first camera coordinate and the second camera coordinate respectively; A capillary length determination module 304 is configured to determine the capillary length according to the relative coordinates of the two cameras, the first camera coordinate, and the second camera coordinate.

[0087] Figure 4 It is a functional block diagram of an electronic device provided by an embodiment of the present invention. As Figure 4 shown, the electronic device 4 of this embodiment includes: a processor 400 and a memory 401, and a computer program 402 that can run on the processor 400 is stored in the memory 401. When the processor 400 executes the computer program 402, the steps in the above-mentioned various capillary length determination methods and embodiments are implemented, such as Figure 1 the steps 101 to 104 shown.

[0088] Exemplarily, the computer program 402 can be divided into one or more modules / units, and the one or more modules / units are stored in the memory 401 and executed by the processor 400 to complete the present invention.

[0089] The electronic device 4 can be a computing device such as a desktop computer, a notebook, a palm computer, and a cloud server. The electronic device 4 may include, but is not limited to, a processor 400 and a memory 401. Those skilled in the art can understand that Figure 4 this is only an example of the electronic device 4 and does not constitute a limitation on the electronic device 4. It may include more or fewer components than shown in the figure, or combine some components, or different components. For example, the electronic device 4 may further include input / output devices, network access devices, buses, etc.

[0090] The so-called processor 400 may be a Central Processing Unit (CPU), or may also be other general-purpose processors, Digital Signal Processors (DSPs), Application Specific Integrated Circuits (ASICs), Field-Programmable Gate Arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0091] The memory 401 may be an internal storage unit of the electronic device 4, such as the hard disk or memory of the electronic device 4. The memory 401 may also be an external storage device of the electronic device 4, such as a plug-in hard disk equipped on the electronic device 4, a Smart Media Card (SMC), a Secure Digital (SD) card, a Flash Card, etc. Further, the memory 401 may also include both the internal storage unit of the electronic device 4 and the external storage device. The memory 401 is used to store the computer program 402 and other programs and data required by the electronic device 4. The memory 401 may also be used to temporarily store data that has been output or is to be output.

[0092] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the above division of each functional unit and module is used as an example. In actual applications, the above functions can be allocated to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. Each functional unit and module in the embodiments can be integrated into one processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above integrated unit can be implemented in the form of hardware or in the form of a software functional unit. In addition, the specific names of each functional unit and module are only for the convenience of mutual distinction and do not limit the protection scope of this application. The specific working processes of the units and modules in the above system can refer to the corresponding processes in the foregoing method embodiments and will not be elaborated herein.

[0093] In the above embodiments, the descriptions of the various embodiments have their own emphases. For parts not detailed or recorded in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0094] Those of ordinary skill in the art can realize that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be implemented by electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are executed in a hardware or software manner depends on the specific application and design constraints of the technical solution. Professional technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the present invention.

[0095] In the embodiments provided by the present invention, it should be understood that the disclosed device / electronic device and method can be implemented in other ways. For example, the device / electronic device embodiments described above are only illustrative. For example, the division of the modules or units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed couplings or direct couplings or communication connections to each other can be through some interfaces. The indirect couplings or communication connections of the devices or units can be in electrical, mechanical or other forms.

[0096] The units described as separate components may or may not be physically separated. The components displayed as units may or may not be physical units, that is, they can be located in one place, or can be distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0097] In addition, the functional units in each embodiment of the present invention can be integrated in a processing unit, or each unit can exist physically alone, or two or more units can be integrated in one unit. The above integrated units can be implemented in the form of hardware or in the form of software functional units.

[0098] When the integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, to implement all or part of the processes in the above-described implementation manners of the present invention, it can also be completed by instructing relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by a processor, the steps of the above-described various method and apparatus implementation manners can be realized. Among them, the computer program includes computer program code, and the computer program code can be in the form of source code, object code, executable file, or some intermediate form, etc. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disc, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal, and software distribution medium, etc.

[0099] The above-described implementation manners are only used to illustrate the technical solutions of the present invention, rather than to limit them; although the present invention has been described in detail with reference to the foregoing implementation manners, those of ordinary skill in the art should understand that: they can still modify the technical solutions recorded in the foregoing implementation manners, or perform equivalent replacements on some of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the various implementation manners of the present invention, and should all be included in the protection scope of the present invention.

Claims

1. A method for determining capillary length, characterized in that: include: Acquire a first capillary tube image and a second capillary tube image, wherein the first capillary tube image and the second capillary tube image are respectively acquired from two fixed cameras; According to the shooting position of the corresponding camera, the end edge line of the first capillary tube image and the end edge line of the second capillary tube image are identified to obtain the first end edge line and the second end edge line; Performing image coordinate transformation on the coordinates of the first end edge line in the first capillary tube image and the coordinates of the second end edge line in the second capillary tube image, and using the obtained coordinates of the first end edge line in the camera coordinate system and the obtained coordinates of the second end edge line in the camera coordinate system as the first camera coordinates and the second camera coordinates, respectively; The capillary length is determined according to the relative coordinates of the two cameras, the first camera coordinates and the second camera coordinates.

2. The method for determining the capillary length according to claim 1, characterized in that: The step of identifying the end edge line of the first capillary tube image and the end edge line of the second capillary tube image according to the shooting position of the corresponding camera to obtain the first end edge line and the second end edge line includes: For any one of the first capillary tube image and the second capillary tube image, the following steps are performed respectively: Perform Gaussian filtering on the capillary image to obtain a smooth capillary image; Using an operator to perform a gradient operation on the smooth capillary image to obtain a gradient capillary image; Performing non-maximum suppression on the gradient capillary image to obtain a first edge capillary image; Performing double threshold detection on the first edge capillary tube image to obtain a second edge capillary tube image; The second edge capillary image is sent to the deep learning model to identify the image block where the end edge line is located.

3. The method for determining the capillary length according to claim 2, characterized in that: The deep learning model is obtained by training a plurality of sample images, wherein the sample label construction process includes: For each sample image, perform the following steps: Acquire a target edge point, wherein the target edge point is located on an edge line at an end of the sample image; Taking the target edge point as the center, acquiring a plurality of first edge points within a preset neighborhood radius; According to the first formula and the plurality of first edge points, an edge line equation expressing the end edge line is constructed using the least square method, wherein the first formula is: In the formula, , , , , as well as are the first coefficient, the second coefficient, the third coefficient, the fourth coefficient, the fifth coefficient and the sixth coefficient respectively, is the x-axis coordinate of the first edge point in the sample image, is the y-axis coordinate of the first edge point in the sample image; Substituting the plurality of first edge points into the edge line equation respectively to obtain a plurality of deviation values; taking the maximum value among the plurality of deviation values ​​as a reference value; Taking the target edge point as the center, and according to the edge line equation, obtaining a plurality of second edge points, wherein a value obtained by substituting the second edge point into the edge line equation is not greater than the reference value, and the plurality of second edge points are continuous; Using a minimum rectangle surrounding the plurality of second edge points as a marking rectangle; The center coordinates of the marking rectangle, the width of the marking rectangle, and the length of the marking rectangle are used as sample labels.

4. The method for determining the capillary length according to claim 1, characterized in that: The step of performing image coordinate transformation on the coordinates of the first end edge line in the first capillary tube image and the coordinates of the second end edge line in the second capillary tube image, and using the obtained coordinates of the first end edge line in the camera coordinate system and the obtained coordinates of the second end edge line in the camera coordinate system as the first camera coordinates and the second camera coordinates, respectively, comprises: For each end edge line, perform the following steps: The end edge line is fitted by the least square method to obtain the first ellipse general equation describing the end edge line; Formulate the first ellipse general equation, and determine the first focus and the second focus according to the result of the formulation; The midpoint between the first focus and the second focus is taken as the center point of the capillary tube; The coordinates of the center point of the capillary tube are transformed into a camera coordinate system to obtain camera coordinates.

5. The method for determining the capillary length according to claim 4, characterized in that: The step of formulating the first ellipse general equation and determining the first focus and the second focus according to the result of the formulation includes: If the first ellipse general equation has cross terms, the first ellipse general equation is subjected to a rotation transformation according to a second formula to obtain a second ellipse general equation, wherein the second formula is: In the formula, is the horizontal axis coordinate in the coordinate system after rotation transformation, is the vertical axis coordinate in the coordinate system after rotation transformation, is the horizontal axis coordinate in the original coordinate system, is the vertical axis coordinate in the original coordinate system, is the rotation transformation angle, , as well as are the first, second and third coefficients of the general equation of the first ellipse, is the inverse tangent function; Otherwise, taking the first ellipse general equation as the second ellipse general equation; Formulating the second ellipse general equation into the first ellipse standard equation; Determine a third focus and a fourth focus according to the first standard ellipse equation; If the first ellipse general equation has a cross term, performing a coordinate inverse rotation transformation on the third focus and the fourth focus according to the second formula to obtain the first focus and the second focus; Otherwise, the third focus and the fourth focus are used as the first focus and the second focus.

6. The method for determining the capillary length according to claim 4, characterized in that: The step of converting the coordinates of the center point of the capillary tube into a camera coordinate system to obtain the camera coordinates includes: Get the camera intrinsic parameter matrix and depth information; The coordinates of the center point of the capillary tube are transformed according to a third formula, the camera intrinsic parameter matrix and the depth information to obtain the camera coordinates, wherein the third formula is: In the formula, , as well as are the x-axis, y-axis and z-axis coordinates of the center point of the capillary tube in the camera coordinate system, is the depth information, is the internal parameter matrix, is the x-axis focal length, is the y-axis focal length, as well as are the x-axis and y-axis coordinates of the intersection of the optical axis and the image plane on the image plane, as well as They are respectively the x-axis and y-axis coordinates of the center point of the capillary tube in the image coordinate system.

7. The method for determining the capillary length according to any one of claims 1 to 6, characterized in that: The method of determining the capillary length according to the relative coordinates of the two cameras, the first camera coordinates and the second camera coordinates comprises: The capillary length is determined according to a fourth formula, the relative coordinates of the two cameras, the first camera coordinates, and the second camera coordinates, wherein the fourth formula is: In the formula, , as well as are the x-axis, y-axis and z-axis components of the first camera coordinates, respectively. , as well as are the x-axis, y-axis, and z-axis components of the second camera coordinates, , as well as They are the x-axis, y-axis, and z-axis components of the camera's relative coordinates.

8. A capillary length determination device, characterized in that: Used to implement the capillary length determination method according to any one of claims 1 to 7, the capillary length determination device comprises: A capillary tube image acquisition module, used to acquire a first capillary tube image and a second capillary tube image, wherein the first capillary tube image and the second capillary tube image are respectively acquired from two fixed cameras; A capillary tube edge recognition module is used to recognize the end edge line of the first capillary tube image and the end edge line of the second capillary tube image according to the shooting position of the corresponding camera, and obtain the first end edge line and the second end edge line; a coordinate transformation module, used for performing image coordinate transformation on the coordinates of the first end edge line in the first capillary tube image and the coordinates of the second end edge line in the second capillary tube image, and using the obtained coordinates of the first end edge line in the camera coordinate system and the obtained coordinates of the second end edge line in the camera coordinate system as the first camera coordinates and the second camera coordinates, respectively; as well as, The capillary length determination module is used to determine the capillary length according to the relative coordinates of the two cameras, the first camera coordinates and the second camera coordinates.

9. An electronic device comprising a memory and a processor, wherein the memory stores a computer program that can be run on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method as claimed in any one of claims 1 to 7 are implemented.

Images