Long material positioning method based on machine vision

Abstract

The invention provides a machine vision-based long material position positioning method, which comprises the following steps of: S100, shooting a long material conveyed along with a conveying belt through a binocular vision system to obtain an image, extracting detail features, structural features and semantic features of the image, fusing the detail features, the structural features and the semantic features, and obtaining an edge contour of the long material from the fused features through an edge detection algorithm; s200, positioning the axis of the long material on the edge of the long material through micro Hough transform; and step S300, back-projecting the axis of the long material into the three-dimensional space of the binocular view angle shot image, and outputting the parameter information of the long material.

Description

Technical Field

[0001] This invention relates to a visual positioning technology, and more particularly to a method for locating long materials based on machine vision. Background Technology

[0002] Long materials (such as round bars and flat bars) require precise acquisition of their parameter information, such as center point position and orientation angle, during processing or transportation to facilitate subsequent gripping or sorting by robotic arms. In scenarios involving the automated sorting, handling, and cutting of long materials such as steel and wood, quickly and accurately acquiring their spatial position information becomes a key issue for achieving automated control. Image recognition technology provides an effective path for long material positioning. Patent CN110517317A discloses a billet positioning method and application based on machine vision camera image processing. This method identifies the edge contour of the billet using image recognition technology and then locates its transition path based on the billet's image features. The application uses image enhancement technology to denoise the image and uses a finite state machine mechanism to locate the billet. However, this technology has the following problems: (1) For the edges of long material images under conditions such as high temperature, movement, and occlusion, simple denoising technology cannot achieve accurate contour recognition; (2) The finite state machine mechanism mainly considers the change of state or the occurrence of events during the movement of the long material to make judgments, and cannot play a direct role in the positioning of the long material. Summary of the Invention

[0003] The purpose of this invention is to provide a long material positioning method based on machine vision, comprising: Step S100: The long material being conveyed by the conveyor belt is captured by a binocular vision system to obtain an image, and the detailed features, structural features and semantic features of the image are extracted and fused. The fused features are then used to obtain the edge contour of the long material through an edge detection algorithm. Step S200: Locate the axis of the long material by differentiable Hough transform on the edge contour of the long material. Step S300: Project the axis of the long material back onto the three-dimensional space of the binocular view image and output the parameter information of the long material.

[0004] Further, in step S100, a depth map is calculated based on the left and right images acquired by the binocular vision system. The left view and the depth map are stitched together to form RGBD four-channel data. The RGBD four-channel data is input into a deep convolutional neural network, which extracts detail features, structural features, and semantic features. R, G, and B correspond to the red, green, and blue components of the pixels in the left view, and D corresponds to the depth information of the pixels in the depth map. The detail features are high-resolution features, the structural features are medium-resolution features, and the semantic features are low-resolution features.

[0005] Furthermore, in step S100, cross-attention is used to fuse detail features, structural features, and semantic features, specifically including semantic feature-guided fusion of structural features and structural feature-guided fusion of detail features; The fusion of semantic features guiding structural features includes: upsampling semantic features to the resolution of structural features; using semantic features as query vector Q, and structural features as key vector K and value vector V; calculating the similarity matrix between Q and K; weighting V according to the similarity; outputting semantically enhanced structural features at the same resolution as Q; upsampling the semantically enhanced structural features to the original resolution of the structural features, and concatenating them with the original structural features to obtain the intermediate-level fusion features. The fusion of structural features guiding detail features includes: upsampling the intermediate-level fusion features to the resolution of detail features; using the intermediate-level fusion features as query vector Q, and detail features as key vector K and value vector V; calculating the similarity between Q and K; weighting V according to the similarity; outputting structure-guided detail features at the same resolution as Q; and upsampling the output features to the original image resolution as the final fusion result.

[0006] Furthermore, step S200 specifically includes the following steps: Step S201: Obtain the position coordinates of each point on the edge of the long material. x i , y i ) and marginal confidence A i Obtain the depth value of each edge point provided by the binocular vision system. d i To obtain the gradient direction of edge points in the edge detection algorithm i ; Step S202: Initialize the Hough space, and... i and r The range of discrete formation N × M A two-dimensional accumulator array, where i Indicates the normal to the line and x Angle between axes, r This represents the distance from the line to the origin; Step S203, calculate each edge point ( x i , y i ) for parameters ( i , r Voting weight ; Step S204: Obtain the global parameter space probability density. ; Step S205: Using differentiable peak detection, the discrete extremum search is transformed into a probability density optimization problem in continuous space, and the long material is separated.

[0007] Furthermore, the voting weight in step S203 Calculate according to the following formula , in, s This indicates the bandwidth for controlling the contribution attenuation. l Indicates the directional constraint strength.

[0008] Furthermore, in step S204, the global parameter space probability density... Obtained from the following formula .

[0009] Furthermore, the specific process of step S205 is as follows: Step S2051, probability density sharpening to enhance the main peak value. , , in, α To adaptively learn the sharpness of long material edges; Step S2052: Calculate the peak coordinates using Soft-Argmax. Position the straight line. , Step S2053, iterative convergence judgment, when Or, upon reaching the maximum number of iterations, output the set of detected peak parameters. ,in, As the energy threshold, the linear parameter set is obtained based on the peak parameter set. Obtain the depth set of edge points associated with each edge. ; Step S2054: Separate long materials by clustering and grouping; Step S2055: Calculate the axis of the long material.

[0010] Further, in step S2054, the depths of edge points that satisfy the constraint conditions are determined to be of the same long material, and the constraint conditions are as follows: ,in The constraints can be represented by the thickness of the long material, etc.; the directional consistency of the edge points that satisfy the constraints is verified. Then, the directional consistency verification is passed, where, Indicates the directional deviation threshold. This indicates the calculation of standard deviation. Indicates the first jThe edge grouping set of a long material.

[0011] Furthermore, after locating a straight line, in Medium Gaussian suppression of the peak neighborhood avoids repeated detection of the same peak, i.e. , Where R represents the set rule. , k =[2,3], This refers to the width of the edge of the long material.

[0012] Further, in step S2055, the axis is obtained by calculating the center line of the circle, where the center line is the projection of the axis onto the two-dimensional image. Specifically, for a pair of straight lines... Its center line is ,in .

[0013] Compared with the prior art, the present invention has the following advantages: (1) It obtains features of different scales of long material images through deep convolutional neural networks, which enhances the robustness of contour detection; (2) It adopts a cross-attention mechanism to fuse multi-scale features, and can dynamically and adaptively weight them in different scenarios to achieve global collaborative optimization; (3) It locates the axis of long material through differentiable Hough transform, the whole process is differentiable, allowing end-to-end optimization, and the feature extractor can be optimized for the characteristics of long material.

[0014] The present invention will now be further described with reference to the accompanying drawings. Attached Figure Description

[0015] Figure 1 This is a schematic diagram of the method flow of the present invention.

[0016] Figure 2 This is a schematic diagram of a multi-scale feature fusion and edge recognition method.

[0017] Figure 3 This is a schematic diagram of a method for obtaining the axis of a long material by using a differentiable Hough transform.

[0018] Figure 4 This is a schematic diagram of a method for separating long materials. Detailed Implementation

[0019] Combination Figure 1 A machine vision-based method for long material positioning includes the following steps: Step S100: The long material being transported by the conveyor belt is captured by binocular vision to obtain an image, multi-scale features of the image are extracted and fused, and the fused features are used to obtain the edge contour of the long material through an edge detection algorithm. Step S200: Locate the axis of the long material by performing a differentiable Hough transform on the edge contour image of the long material. Step S300: Project the axis of the long material back onto the three-dimensional space of the binocular view image, and output information such as the coordinates of the center point of the long material and the orientation angle.

[0020] Step S100 specifically includes the following steps: Step S101: Capture long material scene images using a binocular vision system and perform preprocessing; Step S102: Extract and fuse multi-scale features of the scene image through a feature extraction network; Step S103: Perform edge detection on the fused feature map to obtain the edge contour map of the long material.

[0021] In step S101, two precisely calibrated cameras simultaneously capture left and right view images of the long material under strictly synchronized triggering. The acquired left and right images undergo denoising, distortion correction, and brightness / contrast adjustment. The three-dimensional spatial position information of the long material is obtained using the principle of triangulation. The disparity of corresponding pixels in the left and right images is calculated. By matching feature points in the images and combining the camera's intrinsic and extrinsic parameters, the depth information of the target point is calculated using the disparity. A depth map with the same size as the original image is generated, where each pixel value represents the distance from that point to the camera. The depth map provides important three-dimensional spatial information.

[0022] In industrial settings, measuring long materials may encounter the following problems: image distortion due to high temperatures, image blurring due to high-speed movement, occlusion of the recognition area, and non-standard spacing of the long materials. These problems cannot be overcome by simple filtering, leading to inaccurate identification of the long material contour during subsequent edge detection. Therefore, these problems can be avoided by extracting features at different scales of the image. In step S102, combined with... Figure 2 The left view (RGB) and depth map (Depth) obtained from the binocular vision system are stitched together into 4-channel data (R+G+B+D), which are then input into a deep convolutional neural network (DCNN). The DCNN is used to extract features at different levels, including detail features, structural features, and semantic features.

[0023] Detail features are extracted from the initial layers of the neural network (such as convolutional layers 1-2), which have high resolution (close to the original image) and mainly capture the detailed edge information of the image. These features form the basis of object cognition and include edge attribute features, local geometric features, texture-related features, and illumination and temperature-related features. The details and acquisition methods are shown in the table below.

[0024] Medium-resolution structural features capture the main structural information in an image. These features go beyond simple edge details but do not yet reach a complete semantic understanding level. Structural features constitute the "skeleton" of object cognition. Structural features include geometric structural features, component features, spatial relationship features, and continuity features. The features and acquisition methods are shown in the table below.

[0025] Semantic features are extracted from the terminal layers of the neural network. They have low resolution (typically 1 / 32 of the original image) and primarily capture the abstract semantic information of the image. These features go beyond visual appearance and help us understand the essential meaning of the image content. Semantic features include category information and state information. The various semantic features and their acquisition methods are shown in the table below.

[0026] In step S102, cross-attention is used to fuse detail features, structural features, and semantic features, specifically including semantic feature-guided fusion of structural features and structural feature-guided fusion of detail features.

[0027] (1) Semantic features guiding the fusion of structural features include: Upsample semantic features ( Figure 2 The algorithm uses an Upsampling module to reduce the semantic features to the structural feature resolution. Semantic features are used as the query vector (Q), and structural features are used as the key vector (K) and value vector (V). A similarity matrix between Q and K is calculated to measure the correlation between semantics and structure, such as which regions in the structural features belong to the main body of the long structure. V is weighted according to similarity to strengthen semantically relevant regions and weaken irrelevant regions, such as strengthening the main structure of the long structure and weakening background noise. The algorithm outputs semantically enhanced structural features at the same resolution as Q. The output features are upsampled to the original resolution of the structural features and concatenated with the original structural features to obtain the intermediate fusion features. (2) The fusion of structural features and detailed features includes: The mid-level fusion features are upsampled to the resolution of the detail features; the mid-level fusion features are used as the query vector (Q), and the detail features are used as the key vector (K) and value vector (V); the similarity between Q and K is calculated to identify which details are important to the current structure, such as the precise pixel-level position of the long material outline; V is weighted according to the similarity to enhance structure-related details and suppress irrelevant textures, such as enhancing the edges of the real outline and suppressing rust spots and reflective noise; the structure-guided detail features with the same resolution as Q are output; the output features are upsampled to the original image resolution as the final fusion result.

[0028] The fusion produces one or more fused feature maps with high resolution (close to the input image), which can accurately locate edges and simultaneously contain: fine edge details of detail features, main structural information of structural features, and semantic understanding of semantic features, thus enhancing the robustness of contour detection.

[0029] Step S103 uses the Canny edge detection algorithm for edge extraction, specifically including the following steps: Step S1031: Use a Gaussian kernel convolution to smooth the image and reduce noise interference with subsequent gradient calculations; the Gaussian kernel formula is as follows. , Among them, the commonly used kernel size is 3×3 or 5×5, and σ is usually taken as 1~2. The larger σ is, the blurrier the image is and the lower the edge positioning accuracy. Step S1032, using Sobel The operator calculates the gradient magnitude and direction for each pixel in the image; the gradient magnitude represents the degree of intensity change of the pixel, while the gradient direction represents the direction of change. Gradient operator: using Sobel Operators calculate levels separately and vertical The first derivative in the direction; , Gradient magnitude: , Or approximately: , Gradient direction: , Quantization is performed in four principal directions: 0° (horizontal), 45°, 90° (vertical), and 135°, for nonmaximum suppression. Step S1033: Scan the gradient image and suppress the response of non-edge regions; for each pixel, retain only the local maxima along the gradient direction; for each pixel, compare the two adjacent pixels (obtained by interpolation) along its gradient direction; if the magnitude of the current pixel is less than that of any adjacent pixel, suppress it (set it to 0), otherwise retain it; the gradient direction needs to be mapped to the nearest principal direction (e.g., 30° is mapped to the 0° direction, 60° is mapped to the 45° direction) to determine the adjacent pixels to be compared; Step S1034: Further processing is performed on the gradient image after non-maximum suppression by setting high and low thresholds; pixels above the high threshold are considered strong edges, pixels below the low threshold are considered weak edges, and pixels in between are considered possible edges; the rules are as follows: ① Strong edges (M > threshold2): Directly retained, definitely edges; ② Weak edges (threshold1 < M < threshold2): Retained if connected to strong edges, otherwise suppressed; ③ Non - edges (M < threshold1): Directly suppressed; ④ Threshold ratio: Usually take threshold2 : threshold1 = 2:1 or 3:1 (e.g., threshold1 = 50, threshold2 = 100); Step S1035, Edge connection (connectivity analysis based on 8 - neighborhood): By connecting strong - edge pixel points with adjacent possible - edge pixel points, finally form a complete edge segment.

[0030] Step S200, Locate the axis of the long - material through differentiable Hough transform for the edge contour image of the long - material. This process specifically includes the following steps: Step S201, Obtain the position coordinates of each point on the edge contour of the long - material ( x i , y i ), and the edge confidence A i , Obtain the depth value of each edge point provided by the binocular vision system d i , The edge - point gradient direction calculated by the Canny algorithm i , That is, the information of each edge point includes ( x i , y i , A i , d i , i ); Step S202, Initialize the Hough space, discretize the i range [0°, 180°] into N intervals, discretize the r range [-D, D] into M intervals, where D is the length of the image diagonal, and form a N × M two - dimensional accumulator array, where i represents the angle between the normal of the line and the x - axis, r represents the distance from the line to the origin; Step S203, Calculate each edge point ( x i ,y i ) for parameters ( i , r Voting weight ; Step S204: Obtain the global parameter space probability density. ; Step S205: Using differentiable peak detection, the discrete extremum search is transformed into a probability density optimization problem in continuous space, and the long material is separated.

[0031] In step S201, since background edge points may interfere with voting, depth map information is introduced to weight the edge point confidence. A i : , in, For reliability assessment, For estimating the depth of long materials, c This is the depth attenuation coefficient.

[0032] The formula for calculating weights is usually as follows: , in, s For learnable parameters, represents the bandwidth used to control the attenuation of the contribution. In this embodiment, considering that the gradient direction at the edge of the long material should always be perpendicular to the straight line, a gradient direction consistency penalty is added during the weight calculation process. Therefore, the weight calculation formula in step S203 is rewritten as formula (1). (1) in, l is a learnable parameter, represented as the directional constraint strength.

[0033] In step S204, the global parameter space probability density Obtained from equation (2) (2) The specific process of step S205 is as follows: Step S2051, probability density sharpening to enhance the main peak value. Suppress minor fluctuations (3) in, α To adaptively learn the sharpness of long material edges; Step S2052: Calculate the peak coordinates using Soft-Argmax. Position the straight line. (4) Step S2053, after locating a straight line, in Medium Gaussian suppression of the peak neighborhood avoids repeated detection of the same peak, i.e. , in, R The rules set, , k =[2,3], The width of the edge of the long material; Step S2054, iterative convergence judgment, when Or, upon reaching the maximum number of iterations, output the set of detected peak parameters. ,in, As the energy threshold, the linear parameter set is obtained based on the peak parameter set. Obtain the depth set of edge points associated with each edge. ; Step S2055: Separate long materials by clustering and grouping; Step S2056: Calculate the axis of the long material.

[0034] In step S2055, for the depths of edge points that satisfy the constraint conditions, it is determined that they belong to the same long material, and the constraint conditions are as follows: ,in The constraints can be represented by the thickness of the long material, etc.; the directional consistency of the edge points that satisfy the constraints is verified. Then, the directional consistency verification is passed, where, Indicates the directional deviation threshold. This indicates the calculation of standard deviation. Indicates the first j The edge grouping set of a long material.

[0035] In step S2056, the axis is obtained by calculating the center line of the circle, where the center line is the projection of the axis onto the two-dimensional image. Specifically, for a pair of straight lines that have completed the constraints of step S2055... Its center line is ,in .

[0036] Before using the differentiable Hough transform to locate the axis of the long material, the position output of the long material needs to be trained end-to-end. The training design uses the following loss function: , in, L reg For regression loss function, L peak For peak focusing loss function, L dir Let be the direction consistency loss function. , For predicted values, , This is the actual value.

Claims

1. A method for positioning a long material position based on machine vision, characterized by, The method comprises the following steps: In step S100, an image of the long material conveyed by the conveying belt is captured by a binocular vision system, and detailed features, structural features and semantic features of the image are extracted and fused, and the fused features are subjected to edge detection algorithm to obtain a long material edge profile; In step S200, the long material axis is positioned by means of differentiable Hough transform on the long material edge profile; In step S300, the long material axis is back-projected into the three-dimensional space of the binocular visual angle image, and long material parameter information is output.

2. The method of claim 1, wherein, In step S100, a depth map is calculated according to the left and right images obtained by the binocular vision system, the left view and the depth map are spliced into RGBD four-channel data, the RGBD four-channel data is input into a depth convolutional neural network, and the depth convolutional neural network is used to extract detailed features, structural features and semantic features; wherein R, G and B correspond to the red, green and blue components of the pixels in the left view, and D corresponds to the depth information of the depth image pixels; The detailed features are high-resolution features, the structural features are medium-resolution features, and the semantic features are low-resolution features.

3. The method of claim 1, wherein, In step S100, cross-attention is selected to fuse the detailed features, the structural features and the semantic features, specifically including semantic feature guided structural feature fusion and structure feature guided detailed feature fusion; The semantic feature guided structural feature fusion includes: up-sampling the semantic features to the structural feature resolution; the semantic features are used as the query vector Q, and the structural features are used as the key vector K and the value vector V; a similarity matrix of Q and K is calculated; V is weighted according to the similarity; a semantic enhanced structural feature with the same resolution as Q is output; the semantic enhanced structural feature is up-sampled to the original resolution of the structural feature, and is spliced with the original structural feature to obtain a middle layer fusion feature; The structure feature guided detailed feature fusion includes: up-sampling the middle layer fusion feature to the detailed feature resolution; the middle layer fusion feature is used as the query vector Q, and the detailed feature is used as the key vector K and the value vector V; the similarity of Q and K is calculated; V is weighted according to the similarity; a structure guided detailed feature with the same resolution as Q is output; the output feature is up-sampled to the original image resolution as the final fusion result.

4. The method of claim 1, wherein, Step S200 specifically includes the following steps: Step S201, obtaining the position coordinates of each point of the long material edge x i , y i ) and edge confidence A i , obtaining the depth value of each edge point provided by the binocular vision system d i , obtaining the gradient direction of the edge point in the edge detection algorithm i ; Step S202: Initialize the Hough space, and... θ and ρ The range of discrete formation N × M A two-dimensional accumulator array, where θ Indicates the normal to the line and x Angle between axes ρ This represents the distance from the line to the origin; Step S203, calculate each edge point ( x i , y i ) the voting weight of parameter ( θ , ρ ) to parameter ( ; Step S204, obtaining the global parameter space probability density ; In step S205, the discrete extreme value search is converted into a probability density optimization problem in continuous space by using a derivable peak value detection, and the long material is separated.

5. The method of claim 4, wherein, The voting weight in step S203 The voting weight in step S203 is calculated according to the following formula ， wherein, σ denotes a control contribution decay bandwidth, λ denotes a direction constraint strength.

6. The method of claim 4, wherein, In step S204, the global parameter space probability density is obtained from the equation 。 7. The method of claim 4, wherein, The specific process of step S205 is as follows: Step S2051, the probability density is sharpened to enhance the main peak , ， wherein, α to adaptively learn the edge definition of the long material; Step S2052, calculating the peak coordinate by using Soft-Argmax , positioning the straight line, ， Step S2053, iteration convergence judgment, when or reaches the maximum iteration number, output the detected peak value parameter set , wherein, is an energy threshold, and the straight line parameter set is obtained according to the peak value parameter set , obtain the edge point depth set associated with each edge ; In step S2054, the long material is separated by clustering grouping; In step S2055, the long material axis is calculated.

8. The method of claim 7, wherein, In step S2054, the edge point depth satisfying the constraint condition is determined as the same long material, and the constraint condition is wherein is the constraint condition, which can be represented by the long material thickness, etc.; the edge point satisfying the constraint condition is subjected to the direction consistency verification, and if it satisfies , the direction consistency verification is passed, wherein represents the direction deviation threshold value, represents the standard deviation calculation, represents the edge group set of the j th long material.

9. The method of claim 7, wherein, After locating a straight line, in the Gaussian suppresses the peak neighborhood to avoid repeated detection of the same peak, i.e. ， wherein , k = [2, 3], is the long material edge width.

10. The method of claim 7, wherein, In step S2055, the axis is obtained by calculating the circle center line, which is the projection of the axis in the two-dimensional image, specifically: for a pair of straight lines whose circle center line is where .

Images