Workpiece identification method, device and equipment and storage medium

Abstract

The invention is suitable for the technical field of image processing, and provides a method, device and equipment for recognizing a workpiece and a storage medium, and the method for recognizing the workpiece comprises the steps: obtaining a plurality of depth images of the workpiece; the depth images are corrected, so that the workpieces in the depth images have a unified size proportion; all the corrected depth images are spliced; each edge of the workpiece is extracted from the spliced depth image; carrying out geometric shape fitting on each extracted edge; and the edges are screened out from the fitted edges to be combined into the contour of the workpiece. According to the workpiece identification method provided by the embodiment of the invention, the workpiece with a relatively large size can be accurately identified.

Description

Technical Field

[0001] This application belongs to the technical field of image processing, and particularly relates to a method, device, equipment and storage medium for identifying workpieces. Background Art

[0002] With the development of industry, further improving the automation level of production, enhancing product quality and reducing human dependence are the key points for the transformation and upgrading of the steel industry. In the process of steel plate production, product handling relies on automated equipment to complete. In the past, this process relied on mechanical positioning to ensure accuracy. Later, with the development of machine vision, industrial cameras were introduced on the production line to perform positioning detection work using image recognition, making the production process more controllable and flexible, and improving production efficiency. Currently, the industrial camera vision system can only identify workpieces with relatively small sizes. Summary of the Invention

[0003] Embodiments of this application provide a method, device, equipment and storage medium for identifying workpieces, which can accurately identify workpieces with relatively large sizes.

[0004] In a first aspect, embodiments of this application provide a method for identifying workpieces, the method comprising:

[0005] Obtain multiple depth images of the workpiece;

[0006] Rectify each of the depth images so that the workpieces in each of the depth images have a unified size ratio;

[0007] Stitch the rectified depth images;

[0008] Extract each edge of the workpiece from the stitched depth image;

[0009] Perform geometric shape fitting on the extracted edges;

[0010] Select edge combinations from the fitted edges to form the contour of the workpiece.

[0011] In a possible implementation manner of the first aspect, the method further comprises:

[0012] Select the contour located at the top layer from the contours as the contour of the workpiece.

[0013] In a possible implementation manner of the first aspect, the rectifying each of the depth images so that the workpieces in each of the depth images have a unified size ratio includes:

[0014] Determine the center point coordinates of each of the depth images;

[0015] Determine the coordinate range of each of the rectified depth images according to the center point coordinates;

[0016] Determine the sampling interval for discretization processing;

[0017] According to the coordinate range and the sampling interval, discretize the coordinate data of each of the depth images.

[0018] In a possible implementation manner of the first aspect, the stitching and correcting each of the depth images includes:

[0019] Obtain the specified point coordinates of the specified point of the workpiece in each of the depth images;

[0020] Determine the surface normal vector of the workpiece in each of the depth images according to the respective specified point coordinates;

[0021] Determine the rotation matrix and the translation matrix according to the surface normal vector;

[0022] According to the rotation matrix and the translation matrix, convert the coordinate data of each of the depth images to the same image to complete the stitching.

[0023] In a possible implementation manner of the first aspect, the extracting the edge of the workpiece from the stitched depth image includes:

[0024] Obtain the height difference on both sides of each position of the stitched depth image;

[0025] Determine all the edges where the height changes according to the height difference.

[0026] In a possible implementation manner of the first aspect, the geometric shape fitting of the extracted edge includes:

[0027] Extract the edge line parameters of each of the edges;

[0028] Merge multiple edges into one edge according to the edge line growth and merging parameters and the edge line parameters.

[0029] In a possible implementation manner of the first aspect, the screening out the edge combinations from the fitted edges to form the contour of the workpiece includes:

[0030] Determine the angular and positional relationships of the fitted edges according to the geometric constraint conditions of the contour of the workpiece;

[0031] Screen out the edges that meet the angular and positional relationship constraints from the fitted edges to form the contour of the workpiece.

[0032] In a second aspect, an embodiment of the present application provides a device for identifying a workpiece, and the device includes:

[0033] An image acquisition module for acquiring multiple depth images of a workpiece;

[0034] A correction module for correcting each of the depth images so that the workpieces in each of the depth images have a unified size ratio;

[0035] A stitching module for stitching each of the corrected depth images;

[0036] An extraction module for extracting each edge of the workpiece from the stitched depth image;

[0037] A fitting module for performing geometric shape fitting on each of the extracted edges;

[0038] A screening module for screening out edge combinations from the fitted edges to form the contour of the workpiece.

[0039] In a third aspect, an embodiment of the present application provides a processing device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the method described in any one of the above first aspects is implemented.

[0040] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium storing a computer program, and when the computer program is executed by a processor, the method described in any one of the above first aspects is implemented.

[0041] In a fifth aspect, an embodiment of the present application provides a computer program product, and when the computer program product runs on a terminal device, the terminal device is caused to execute the method described in any one of the above first aspects.

[0042] The beneficial effects of the embodiments of the present application are as follows:

[0043] By acquiring multiple depth images of the workpiece, correcting the acquired depth images so that the workpieces in each depth image have a unified size ratio, stitching the corrected depth images, extracting each edge of the workpiece from the stitched depth image, performing geometric shape fitting on each of the extracted edges, and then screening out edge combinations from the fitted edges to form the contour of the workpiece, workpieces with larger sizes can be accurately identified. Description of the Drawings

[0044] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings in the following description are only some embodiments of the present application. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0045] Figure 1 It is a schematic flowchart of a method for identifying workpieces provided in an embodiment of the present application;

[0046] Figure 2 It is a schematic diagram of the camera field of view in an embodiment of the present application;

[0047] Figure 3 It is a grayscale image after stitching of binocular cameras in an embodiment of the present application;

[0048] Figure 4 It is a schematic flowchart of a method for identifying workpieces provided in another embodiment of the present application;

[0049] Figure 5 It is an output edge map in an embodiment of the present application;

[0050] Figure 6 It is a schematic flowchart of step A2 of the method for identifying workpieces provided in an embodiment of the present application;

[0051] Figure 7(a) is an original image provided in an embodiment of the present application;

[0052] Figure 7(b) is a corrected image provided in an embodiment of the present application;

[0053] Figure 7(c) is an image without void filling provided in an embodiment of the present application;

[0054] Figure 7(d) is an image with void filling provided in an embodiment of the present application;

[0055] Figure 8 It is a schematic flowchart of step A3 of the method for identifying workpieces provided in an embodiment of the present application;

[0056] Figure 9 It is a schematic diagram of the fields of view of two cameras provided in an embodiment of the present application;

[0057] Figure 10 It is a schematic flowchart of step A4 of the method for identifying workpieces provided in an embodiment of the present application;

[0058] Figure 11 It is a schematic flowchart of step A5 of the method for identifying workpieces provided in an embodiment of the present application;

[0059] Figure 12 It is a schematic flowchart of step A6 of the method for identifying workpieces provided in an embodiment of the present application;

[0060] Figure 13 It is a schematic structural diagram of a device for identifying workpieces provided in an embodiment of the present application;

[0061] Figure 14It is a schematic structural diagram of a device for identifying workpieces provided by another embodiment of the present application;

[0062] Figure 15 It is a schematic structural diagram of a correction module of a device for identifying workpieces provided by an embodiment of the present application;

[0063] Figure 16 It is a schematic structural diagram of a splicing module of a device for identifying workpieces provided by an embodiment of the present application;

[0064] Figure 17 It is a schematic structural diagram of an extraction module of a device for identifying workpieces provided by an embodiment of the present application;

[0065] Figure 18 It is a schematic structural diagram of a fitting module of a device for identifying workpieces provided by an embodiment of the present application;

[0066] Figure 19 It is a schematic structural diagram of a screening module of a device for identifying workpieces provided by an embodiment of the present application;

[0067] Figure 20 It is a schematic structural diagram of a processing device provided by an embodiment of the present application. Detailed implementation manners

[0068] In order to make the technical problems, technical solutions and beneficial effects to be solved by the present application clearer and more understandable, the following further details the present application with reference to the accompanying Figures 1 to 20 drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0069] In the following description, specific details such as specific system structures and technologies are presented for the purpose of illustration rather than limitation, so as to thoroughly understand the embodiments of the present application. However, those skilled in the art should clearly understand that the present application can also be implemented in other embodiments without these specific details. In other cases, the detailed descriptions of well-known systems, devices, circuits and methods are omitted to avoid unnecessary details from interfering with the description of the present application.

[0070] It should be understood that when used in the specification and appended claims of the present application, the term "comprising" indicates the presence of the described features, wholes, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or their combinations.

[0071] It should also be understood that the term "and / or" used in the specification and appended claims of the present application refers to any combination and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0072] As used in the description of the present application and the appended claims, the term "if" may be construed as "when" or "once" or "in response to determining" or "in response to detecting" depending on the context. Similarly, the phrase "if determined" or "if [the described condition or event] is detected" may be construed as meaning "once determined" or "in response to determining" or "once [the described condition or event] is detected" or "in response to detecting [the described condition or event]" depending on the context.

[0073] In addition, in the description of the present application and the appended claims, the terms "first", "second", "third", etc. are only used for differentiating descriptions and cannot be construed as indicating or implying relative importance.

[0074] Reference to "one embodiment" or "some embodiments" or the like described in the present application means that a specific feature, structure, or characteristic described in connection with that embodiment is included in one or more embodiments of the present application. Thus, statements such as "in one embodiment", "in some embodiments", "in other some embodiments", "in still other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized. The terms "comprising", "including", "having", and their variants mean "including but not limited to", unless otherwise specifically emphasized.

[0075] An embodiment of the present application provides a method for identifying a workpiece, which can extract the edge of the workpiece more accurately. The aforementioned workpiece can be a sheet material, a profile, or other workpieces whose profiles are located in multiple images. Taking a sheet material as an example, the embodiments of the present application will be described.

[0076] Figure 1 It is a schematic flowchart of a method for identifying a workpiece provided by an embodiment of the present application. Refer to Figure 1 , the method for identifying a workpiece provided by the embodiment of the present application includes steps A1 to A6.

[0077] Step A1: Obtain multiple depth images of the workpiece.

[0078] Workpieces such as sheet materials or profiles are relatively large in size (for example, large in length or width), and multiple images need to be taken to obtain a complete image of the workpiece. For example, use multiple depth cameras (such as binocular cameras) to take pictures of the workpiece at different positions, so as to obtain multiple depth images of the workpiece. Each depth image records a part of the contour of the workpiece, and all the depth images together record the complete contour of the workpiece.

[0079] After obtaining multiple depth images of the workpiece, preprocess each depth image. The aforementioned preprocessing includes but is not limited to denoising, smoothing, and interpolation.

[0080] Step A2: Rectify each depth image so that the workpieces in each depth image have a unified size ratio.

[0081] The characteristics of the data in the depth image are that the xy coordinates of the point cloud are arranged in order and each xy coordinate corresponds to only one height data, which can be regarded as two-dimensional data for processing. However, the acquisition method of the depth map will affect the relationship between its xy coordinates and the true coordinates. In the embodiments of the present application, due to the large size of the workpiece, in order to achieve a large field of view, a binocular camera is used to photograph the workpiece, and the imaging follows the law of perspective projection where objects closer to the camera appear larger and those farther away appear smaller.

[0082] Figure 2 is a schematic diagram of the camera's field of view in an embodiment of the present application. Refer to Figure 2 , in the camera's field of view, the parts closer to the camera and the parts farther away appear to have the same size in the camera, but the parts farther away are actually larger than the parts closer. For the convenience of subsequent processing, the data needs to be rearranged and converted into a unified proportional size, that is, each depth image is rectified to a state where the scales in the XY directions are uniform.

[0083] Step A3: Stitch the rectified depth images together.

[0084] Due to the limitations of the camera's field of view and accuracy, a single binocular camera can only photograph a part of the workpiece, and the acquired depth images of the workpiece cannot meet the requirements. Two or more binocular cameras need to be used to take pictures together to achieve a larger field of view while ensuring accuracy.

[0085] Figure 3 is a grayscale image after stitching of the binocular cameras in an embodiment of the present application. Refer to Figure 3 , for subsequent processing, multiple depth images need to be stitched into one. Specifically, multiple depth images can be calibrated and stitched into the same coordinate system to form a new complete depth image.

[0086] Step A4: Extract the edges of the workpiece from the stitched depth image.

[0087] The stitched depth image records the complete contour of the workpiece, and the contour of the workpiece has one or more edges. For a sheet, the sheet has multiple edges, and the stitched depth image records the image data of each edge of the sheet.

[0088] To identify the workpiece, all the edges of the workpiece need to be extracted from the stitched depth image. Specifically, edge extraction can be performed on the stitched depth image to obtain several groups of edges in specific directions.

[0089] Step A5: Perform geometric shape fitting on each of the extracted edges.

[0090] The edges of the workpiece may be straight line segments or arcs. Performing geometric shape fitting on each of the extracted edges is to identify the lines that may be the edges of the workpiece.

[0091] In the case of a sheet, its edges are straight line segments. After the edge map undergoes grayscale mapping processing, it can be treated as ordinary image processing, and classical traditional vision algorithms can be used for straight line extraction, such as using the EDLine algorithm for straight line extraction; of course, straight line extraction can also be performed through Hough transform, CannyLine algorithm, LSD (Line Segment Detector) algorithm, etc.

[0092] The fitted edge lines are used as candidate edges.

[0093] Step A6: Screen out edge combinations from the fitted edges to form the contour of the workpiece.

[0094] The geometric rules of the workpiece contour include the angle between edges, the distance between edges, and the positional relationship between edges. Taking a sheet as an example, its contour is a rectangle, and the geometric rules include that the angle between adjacent edges is perpendicular, the distance between opposite edges satisfies the standard size range of the sheet, and the heights of opposite edges are flush. According to the above geometric rules, edge combinations can be screened out from the edges to form the contour of the workpiece.

[0095] According to the above content, by obtaining multiple depth images of the workpiece, correcting the obtained depth images so that the workpieces in the depth images have a unified size ratio, splicing the corrected depth images, extracting the edges of the workpiece from the spliced depth image, performing geometric shape fitting on the extracted edges, and then screening out edge combinations from the fitted edges to form the contour of the workpiece, large-sized workpieces can be accurately identified.

[0096] Figure 4 It is a schematic flowchart of a method for identifying a workpiece provided by another embodiment of the present application. Refer to Figure 4 , the above method for identifying a workpiece may further include Step A7.

[0097] Step A7: Screen out the contour located at the top layer from the contours as the contour of the workpiece.

[0098] Specifically, the obtained contours are used as candidate contours, and according to the positional relationship between the contours, the contour at the top layer is detected for subsequent processing.

[0099] The contours of the obtained workpieces may overlap or be occluded, and it is necessary to detect the contour of the top layer (i.e., the topmost) for subsequent operations such as detection, positioning, and grasping. Due to the situation of incomplete photographing and incomplete recognition, it is difficult to directly determine the contour of the top layer by the total edge length of the workpiece contour. For the top workpiece (such as a steel plate), there are theoretically no other edges within its contour area, while there are other edges inside the occluded workpiece. It is relatively complex to determine the inclusion relationship between all workpiece contour areas and all lines through geometric calculations. The embodiments of the present application adopt a drawing method for detection, which is specifically as follows.

[0100] Generate a first image of a first color (such as black), and then draw the complete contour area of the workpiece (such as a steel plate) according to the edge position of the workpiece and fill the inside with a second color (such as white).

[0101] Draw all the edges of the workpiece with a color other than the second color (such as black) onto the first image.

[0102] Detect the color within each workpiece contour area (such as a rectangular area) in the first image. If there is a workpiece contour area with only the second color (such as white) inside, it is determined that this workpiece contour area is the contour area of the top layer, indicating that this contour area is not occluded, and the contour of this workpiece contour area is the contour of the top layer. As the contour of the workpiece, in this way, operations such as height measurement and surface shape detection can be performed on the area corresponding to the contour.

[0103] Figure 5 It is the output edge map of an embodiment of the present application, which is the detected edge map and the result rectangle identification. Refer to Figure 5 , there are two steel plates in the figure, and there is a stacking and occlusion situation. According to the method for identifying workpieces provided by the embodiments of the present application, the topmost steel plate can be identified based on the rectangular features and occlusion relationship and detected.

[0104] The above step A2 (correcting each depth image so that the workpieces in each depth image have a unified size ratio) may specifically include steps A21 to A24.

[0105] Step A21: Determine the center point coordinates of each depth image.

[0106] Suppose the coordinates of the data point at the j-th row and the i-th column in the point cloud of the depth image with m rows and n columns are (x i , y j , z ij ). Treat it as two-dimensional data and only consider the xy coordinates of this point. Denote the center point coordinates of the original depth image as (x C , y C ).

[0107] Step A22: Determine the coordinate range of each corrected depth image according to the center point coordinates.

[0108] To determine the size of the corrected depth image and the corresponding actual coordinates, it is necessary to count the data value range of the original depth image. Denote the maximum and minimum values of the xy coordinates as x max , y max , x min , y min . To ensure symmetry with respect to the center point coordinates (x C , y C ), the length in the x direction is selected as the longer semi-axis from the center point to the edge position, that is, X = max(x max - x C , x C - x min ). Similarly, the data Y in the y direction can be obtained. Finally, the coordinate range of the corrected depth image is obtained, that is:

[0109] x ∈ [x C - X, x C + X)

[0110] y ∈ [y C - Y, y C + Y).

[0111] Step A23: Determine the sampling interval for discretization processing.

[0112] After determining the coordinate range of the depth image, it is also necessary to determine the sampling interval for discretization processing, that is, the xy coordinate differences Δx, Δy between adjacent points. The intervals in the two directions after correction are kept equal and defined as Δxy. Since Δx, Δy vary to some extent in the original depth image, it is necessary to count all the data and determine the minimum values Δx min , Δy min to avoid data loss after correction, that is, the final Δxy = min(Δx min , Δy min ).

[0113] Step A24: Discretize the coordinate data of each of the depth images according to the coordinate range and the sampling interval.

[0114] Discretize the coordinate system data with Δxy as the sampling interval. Then the number of data in the x direction is The number of data in the y direction is The coordinates (x i , y j ) of the points in the original depth image correspond to the row and column coordinates i t , j t as:

[0115]

[0116] Then the height value of the corresponding point in the corrected depth image is z itjt = z ij 。

[0117] After processing all the data in the original depth image through the above steps and filling it into the corrected depth image, since the integer row and column values are obtained by rounding off decimals, and the sampling interval is usually smaller than the actual data interval, there will be skipped rows and columns resulting in incomplete and discontinuous data. The adjacent height values can be directly taken for filling. In actual operation, median filtering is usually directly used to obtain good results.

[0118] According to the above content, each point in the data of the depth image obtained by the binocular camera contains xyz coordinates; due to the principle of pinhole imaging, the xy coordinates of the image center point are fixed. Taking this as a reference point can ensure the alignment relationship between the corrected data and the original data, facilitating subsequent calibration and coordinate calculation; therefore, the data conversion takes the center point as a reference, calculates the xy coordinate range and resolution in the original depth image, generates the standard xy coordinates of the corrected image, and then calculates the coordinate position of each point in the original data in the corrected data one by one; due to the discrete characteristics of the data, the corrected position after floating-point calculation of the original data will deviate from the standard coordinates on the corrected image, and approximation processing is required, and the missing data caused by the processing needs to be filled; Fig. 7(a) is the original image, and Fig. 7(b) is the corrected image. It can be seen that the horizontal long strip parts in the middle and bottom are longer in actual coordinates because they are farther from the camera. Fig. 7(c) is the image without filling gaps after correction, and Fig. 7(d) is the image with filled gaps. It can be seen that due to the approximation processing of the coordinate positions of each point, many gaps appear; due to the special calculation of the data size interval, the width of these gaps is generally 1, and continuous data can be obtained after simple smoothing processing, facilitating subsequent algorithm processing.

[0119] Figure 8 is a schematic flowchart of step A3 of the method for identifying workpieces provided by an embodiment of the present application. Refer to Figure 8 above, the above step A3 (stitching the corrected depth images) may include steps A31 to A34.

[0120] Camera calibration is required for stitching the depth images. Traditional dual-camera calibration requires a large number of pictures and a professional calibration board, and the calibration process is complex and the accuracy cannot be guaranteed. The depth maps obtained by the binocular camera already contain spatial information, and this feature can be directly utilized to perform calibration using two pictures containing the same area, as follows.

[0121] Step A31: Obtain the specified point coordinates of the workpiece at the specified points in each depth image.

[0122] Assume that all the contours of the plate can be captured by two binocular cameras (such as binocular camera one and binocular camera two), then the number of depth images is two. Take the number of depth images being two as an example. Obtain the coordinates P1′, P2′, P3′ and P1″, P2″, P3″ of the corresponding points of the specified points on the plate in the two depth images.

[0123] Step A32: Determine the surface normal vectors of the workpiece in each depth image according to the coordinates of each specified point.

[0124] The surface normal vectors of the workpiece in each depth image (that is, the surface normal vectors of the workpiece in the fields of view of binocular camera one and binocular camera two) are and

[0125]

[0126] Step A33: Determine the rotation matrix and the translation matrix according to the surface normal vectors.

[0127] After determining the surface normal vectors, solve for the rotation axis n and the rotation angle θ in the Rodriguez rotation formula:

[0128]

[0129] Calculate the rotation matrix R and the translation matrix t:

[0130] R = cosθI + (1 - cosθ)nn T + sinθn^

[0131] t = P′ - RP″.

[0132] Step A34: According to the rotation matrix and the translation matrix, convert the coordinate data of each depth image to the same image to complete the stitching.

[0133] Take two depth images as an example. The relationship between a point P″ in one depth image and the corresponding point P′ in the other depth image is:

[0134] P′ = RP″ + t.

[0135] According to the above relationship, the coordinate data of each depth image can be converted to the same image, thereby realizing the stitching of each depth image. For example, convert the data P″ of the depth image captured by binocular camera two through the rotation matrix and the translation matrix, and put it into the same image as the data of the depth image captured by binocular camera one to complete the stitching.

[0136] Figure 9 is a schematic diagram of the fields of view of two cameras provided by an embodiment of the present application. Refer to Figure 9, in one example, frame 100 represents the field of view area of camera one, its center point O1, and the normal vector Frame 200 represents the field of view area of camera two, its center point O2, and the normal vector The triangular shape is a target object within the common area of the two cameras. The three points on the upper surface of this triangular shape are P1, P2, P3, and the surface normal vector is Given that the coordinate points of the three points of the workpiece in the field of view of camera one are P1′, P2′, P3′, and the coordinate points in the field of view of camera two are P1″, P2″, P3″, the surface normal vector of the workpiece in the field of view of camera one can be obtained and the surface normal vector in the field of view of camera two And, based on this, solve the coordinate transformation matrix; specifically, camera one can be used as the reference. All the data of camera two are calculated through the transformation matrix to obtain their coordinates in the coordinate system of camera one, and then the two sets of data are redrawn on one graph to complete the stitching of the depth image.

[0137] Figure 10 is a schematic flowchart of step A4 of the method for identifying a workpiece provided by an embodiment of the present application. Refer to Figure 10 , the above step A4 (extracting the edge of the workpiece from the stitched depth image) may include step A41 and step A42.

[0138] Step A41: Obtain the height difference between both sides of each position of the stitched depth image.

[0139] The height of the plate (i.e., the workpiece) in the depth image changes smoothly, and there is a sudden change in height at the edge position. Therefore, a method similar to edge extraction in traditional image processing can be adopted. By using a convolution kernel to determine the height difference between both sides of a small area, convolution operation is performed on the entire stitched depth image to obtain all the edge positions where there is a sudden change in height.

[0140] Specifically, set the size of the convolution kernel according to the edge state. For example, to extract the left edge, set the convolution kernel with a size of 5×1 as [1, 0, 0, 0, -1]. For the data D at any row j and column i ij The relationship with the original data is:

[0141] D ij = z (i-2)j ×1 + z (i-1)j ×0 + z ij ×0 + z (i+1)j ×0 + z (i+2)j ×(-1).

[0142] In practical applications, the selection of the convolution kernel should consider the edge characteristics, and the size needs to adapt to the edge transition situation.

[0143] For the convenience of classifying edges for subsequent algorithm processing, convolution kernels in different directions are set to correspond to different edges of the workpiece. Taking a rectangle as an example, four convolution kernels in different directions are set to correspond to the four edges of the rectangle, that is, the features of the upper edge and the lower edge are opposite, and the features of the left edge and the right edge are opposite.

[0144] Step A42: Determine all the edges where the height changes according to the height difference.

[0145] After the convolution result is judged by a threshold, the data with too small height difference is filtered to generate an edge map to reduce interference data. Specifically, set the height difference threshold to d, and judge D in the previous step: ij Make a judgment:

[0146]

[0147] Repeat Step A41 and Step A42 for the data at each position, and the left edge map can be obtained. The data in the non-zero area of the map are potential edge positions, and the direction of the edges is that the depth value on the left is larger than that on the right.

[0148] For a plate with left edges, right edges, upper edges, and lower edges, use [-1, 0, 0, 0, 1], [1, 0, 0, 0, -1], T [-1, 0, 0, 0, 1] T convolution kernels to repeat the above steps, and the right edge map, upper edge map, and lower edge map can be obtained.

[0149] Figure 11 is the flow schematic diagram of Step A5 of the method for identifying a workpiece provided by an embodiment of the present application. Refer to Figure 11 , the above Step A5 (geometric shape fitting of the extracted edges) may include Step A51 and Step A52.

[0150] Step A51: Extract the edge line parameters of each edge.

[0151] After the edge map is processed by gray mapping, it can be used as ordinary image processing. Classical traditional vision algorithms can be used to extract edges. In the embodiment of the present application, the EDLine algorithm is used for the first edge extraction.

[0152] Exemplarily, use the EDLine algorithm to extract the edge lines (such as straight lines) of the edge images (such as four edge images). The extracted edge line parameters (such as straight line parameters) include the starting point P s , the ending point P e , the length L, and the angle A.

[0153] Step A52: Merge multiple edges into one edge according to the edge line growth and merging parameters and the edge line parameters.

[0154] There can be multiple edge growth merging parameters. Taking a straight line as an example, the parameters for straight line growth merging include an allowable angle θ, an allowable distance d, an allowable deviation Δ, and a minimum length l.

[0155] Specifically, all straight lines with a length greater than l are screened and added to the list of lines to be grown. For example, all straight lines with a length greater than l in four edge images are respectively screened and added to four lists of lines to be grown.

[0156] Select the first line in the list, and calculate the angle (i.e., the absolute value of the angle difference), deviation (i.e., the distance between the straight lines), and distance (i.e., the minimum value of the starting point distance) of the remaining straight lines in the list relative to this line. If all are within the allowable parameters, add them to the merging list as the same line and delete them from the list of lines to be grown. Repeat the above until the lines in the list of lines to be grown cannot be merged, and multiple groups of edge merging lists can be obtained. Each group of edges is regarded as a candidate edge. For the edges of a rectangle, count the starting points with the farthest distance among them as the starting points of the candidate edges, and finally four groups of candidate edges can be obtained.

[0157] The general EDLine algorithm can only extract standard straight lines. Due to problems such as slight deformation of the edges of the steel plate (i.e., the workpiece) in actual situations, and poor imaging quality resulting in overly blurred edges, the extracted straight lines are usually incomplete, which will affect subsequent processing. The embodiments of the present application perform straight line growth merging to fit the complete edges of the workpiece as much as possible; in order to improve compatibility and avoid introducing interference, a series of verifications and determinations on the angular, distance, and positional relationships between straight lines are carried out, which can improve the correctness.

[0158] Figure 12 It is a schematic flowchart of step A6 of the method for identifying a workpiece provided by an embodiment of the present application. Refer to Figure 6 , the above step A6 (screening out edge combinations from the fitted edges to form the contour of the workpiece) may include step A61 and step A62.

[0159] Step A61: Determine the angular and positional relationships of the fitted edges according to the geometric constraint conditions of the contour of the workpiece.

[0160] According to the geometric constraint conditions of the rectangle, calculate the angles, lengths, and positional relationships of all fitted edges, and screen out the edges that can form a rectangle. Specifically, the geometric constraint conditions of the rectangle include that the adjacent sides are perpendicular to each other, the distance between the opposite sides satisfies the standard size range of the steel plate, and the heights of the opposite sides are flush.

[0161] Of course, the standard size and allowable tolerance of the workpiece can also be obtained as constraint and detection conditions.

[0162] Step A62: Select the edges that meet the angle and position relationship constraints from the fitted edges to form the contour of the workpiece.

[0163] After the above-mentioned processing, multiple groups of candidate edges can be obtained, corresponding to different sides of the workpiece respectively. It is necessary to select the edges that meet the angle and position relationship constraints from the candidate edges.

[0164] Since there may be multiple contours in the actual situation, considering possible deformations and errors, there will also be situations of overlap, occlusion, and incompleteness. It is necessary to add or modify the constraint conditions of the standard contour. Taking a rectangle as an example, the constraint conditions can be changed to: the angle between adjacent sides is close to perpendicular, the distance between opposite sides meets the standard size range of the steel plate, the adjacent sides of the opposite sides can partially exceed the opposite sides, and when there are multiple eligible sides, the innermost side is selected. After screening and pairing, several groups of contours will be formed, each group having three or four sides, that is, the edge positions of the recognized workpiece are as follows.

[0165] Taking a rectangle as an example, the candidate edges include the top edge group, the bottom edge group, the left edge group, and the right edge group.

[0166] Select the first subsequent edge of the top edge group, screen all the edges of the left edge group to meet the angle and position constraints. If there are multiple eligible ones, select the rightmost one in the left edge group; screen all the edges of the bottom edge group to meet the angle and position constraints. If there are multiple eligible ones, select the topmost one in the bottom edge group; screen all the edges of the right edge group to meet the angle and position constraints. If there are multiple eligible ones, select the leftmost one in the right edge group. If there are edges in the bottom edge group, the left edge group, and the right edge group that meet the requirements, then verify whether the other top edges in the top edge group meet the requirements and select the bottommost one. If a rectangle is successfully formed, the edges corresponding to the rectangle will be deleted from the subsequent screening. Repeat the above steps until all the top edges of the top edge group are screened to obtain the contour of the workpiece.

[0167] Traditional industrial camera vision systems collect planar images of workpieces, identify the information in the images through software algorithms, and judge the edge positions of the workpieces. However, the information that can be obtained from the planar images obtained by traditional industrial cameras is not comprehensive enough. Simple position information and dimension information can be obtained through positioning measurement based on the edge positions of the workpieces, but it is impossible to effectively judge the situations such as uneven surfaces and deformations of the workpieces (such as plates). How to ensure that the product processing quality meets the requirements and how to more accurately obtain and analyze the state of the steel plates are the key points for subsequent optimization. In industrial applications, 3D positioning algorithms are based on features such as surface normal vectors, curvatures, and edges and corners. Therefore, they are usually used for objects with a certain shape complexity. For workpieces such as stacked steel plates that have no surface feature changes and whose boundary transition shapes are not obvious due to their small thickness, the recognition effect is not ideal.

[0168] The method for identifying workpieces provided by the embodiments of the present application, for stacked steel plates, obtains depth images through a binocular camera, processes the data of the depth graphics, screens the edges of the steel plates, and finally determines the accurate position and area of the topmost steel plate for overall height measurement. It can extract relatively accurate edges, with a more ideal recognition effect, provides coordinate guidance for an external grasping mechanism, and can also detect the surface shape of the steel plate to judge whether there are defects such as deformation. It is particularly suitable for the positioning detection of stacked rectangular steel plates and is also a workpiece detection method.

[0169] Corresponding to the method described in the above embodiments, Figure 13 The structural block diagram of the device for identifying workpieces provided by the embodiments of the present application is shown. For the sake of convenience of description, only the parts related to the embodiments of the present application are shown.

[0170] Referring to Figure 13 , the device for identifying workpieces provided by the embodiments of the present application includes an image acquisition module 1A, a correction module 2A, a splicing module 3A, an extraction module 4A, a fitting module 5A, and a screening module 6A.

[0171] The image acquisition module 1A is used to: acquire multiple depth images of the workpiece.

[0172] The correction module 2A is used to: correct each depth image so that the workpieces in each depth image have a unified size ratio.

[0173] The splicing module 3A is used to: splice the corrected depth images.

[0174] The extraction module 4A is used to: extract the edges of the workpiece from the spliced depth image.

[0175] The fitting module 5A is used to: perform geometric shape fitting on the extracted edges.

[0176] The screening module 6A is used to: screen out the edge combinations from the fitted edges to form the contour of the workpiece.

[0177] Figure 14 is the structural schematic diagram of the device for identifying workpieces provided by another embodiment of the present application. Referring to Figure 14 , the above device for identifying workpieces may further include a contour processing module 7A.

[0178] The contour processing module 7A is used to: screen out the contour located at the topmost layer from the contours as the contour of the workpiece.

[0179] Figure 15 is the structural schematic diagram of the correction module of the device for identifying workpieces provided by an embodiment of the present application. Referring to Figure 15, the above correction module 2A may include a center determination sub-module 21A, a range determination sub-module 22A, an interval determination sub-module 23A, and a discretization sub-module 24A.

[0180] The center determination sub-module 21A is configured to: determine the center point coordinates of each depth image.

[0181] The range determination sub-module 22A is configured to: determine the coordinate range of each corrected depth image according to the center point coordinates.

[0182] The interval determination sub-module 23A is configured to: determine the sampling interval for discretization processing.

[0183] The discretization sub-module 24A is configured to: discretize the coordinate data of each depth image according to the coordinate range and the sampling interval.

[0184] Figure 16 It is a schematic structural diagram of the splicing module of the device for identifying workpieces provided by an embodiment of the present application. Refer to Figure 16 , the above splicing module 3A may include a specified coordinate acquisition sub-module 31A, a normal vector determination sub-module 32A, a matrix determination sub-module 33A, and a conversion sub-module 34A.

[0185] The specified coordinate acquisition sub-module 31A is configured to: acquire the specified point coordinates of the specified point of the workpiece in each depth image.

[0186] The normal vector determination sub-module 32A is configured to: determine the surface normal vector of the workpiece in each depth image according to the specified point coordinates.

[0187] The matrix determination sub-module 33A is configured to: determine the rotation matrix and the translation matrix according to the surface normal vector.

[0188] The conversion sub-module 34A is configured to: convert the coordinate data of each depth image to the same image according to the rotation matrix and the translation matrix to complete the splicing.

[0189] Figure 17 It is a schematic structural diagram of the extraction module of the device for identifying workpieces provided by an embodiment of the present application. Refer to Figure 17 , the above extraction module 4A may include a height difference acquisition sub-module 41A and an edge determination sub-module 42A.

[0190] The height difference acquisition sub-module 41A is configured to: acquire the height differences on both sides of each position of the spliced depth image.

[0191] The edge determination sub-module 42A is configured to: determine all edges where the height changes according to the height differences.

[0192] Figure 18It is a schematic structural diagram of the fitting module of the device for identifying workpieces provided by an embodiment of the present application. Refer to Figure 18 As shown above, the fitting module 5A may include a parameter extraction sub-module 51A and a merging sub-module 52A.

[0193] The parameter extraction sub-module 51A is configured to: extract the edge line parameters of each edge.

[0194] The merging sub-module 52A is configured to: merge multiple edges into one edge according to the edge line growth merging parameters and the edge line parameters.

[0195] Figure 19 It is a schematic structural diagram of the screening module of the device for identifying workpieces provided by an embodiment of the present application. Refer to Figure 19 As shown above, the screening module 6A may include a geometric operation sub-module 61A and a contour screening sub-module 62A.

[0196] The geometric operation sub-module 61A is configured to: determine the angular and positional relationships of the fitted edges according to the geometric constraint conditions of the contour of the workpiece.

[0197] The contour screening sub-module 62A is configured to: screen out the edges that meet the angular and positional relationship constraints from the fitted edges to form the contour of the workpiece.

[0198] It should be noted that the information interaction, execution process, etc. between the above-mentioned devices / units, due to being based on the same concept as the method embodiment of the present application, for their specific functions and the technical effects brought, please refer to the method embodiment part specifically, and will not be elaborated here.

[0199] Figure 20 It is a schematic structural diagram of the processing equipment provided by an embodiment of the present application. As Figure 20 shown, the processing equipment 20 of this embodiment includes: at least one processor 200 ( Figure 20 only one is shown in the figure), a memory 201, and a computer program 202 stored in the memory 201 and executable on at least one processor 200; when the processor 200 executes the computer program 202, the steps in the above-mentioned various method embodiments are implemented.

[0200] The processing equipment 20 may include, but is not limited to, a processor 200 and a memory 201. Those skilled in the art can understand that Figure 20 merely an example of the processing equipment, which does not constitute a limitation on the processing equipment, and may include more or fewer components than shown in the figure, or combine certain components, or different components. For example, it may also include input / output devices, network access devices, buses, etc.

[0201] The processor 200 can be a Central Processing Unit (CPU), and the processor 200 can also be other general-purpose processors, Digital Signal Processors (DSPs), Application Specific Integrated Circuits (ASICs), Field-Programmable Gate Arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc.

[0202] The memory 201 can be an internal storage unit of the processing device 20 in some embodiments, such as the hard disk or memory of the processing device. The memory 201 can also be an external storage device of the processing device in other embodiments, such as a plug-in hard disk equipped on the processing device, a Smart Media Card (SMC), a Secure Digital (SD) card, a Flash Card, etc. Further, the memory 201 can also include both the internal storage unit and the external storage device of the processing device. The memory 201 is used to store an operating system, application programs, a Boot Loader, data, and other programs, such as the program code of a computer program. The memory 201 can also be used to temporarily store the data that has been output or will be output.

[0203] Exemplarily, the computer program 202 can be divided into one or more modules / units, and one or more modules / units are stored in the memory 201 and executed by the processor 200 to complete this application. One or more modules / units can be a series of computer program instruction segments capable of performing specific functions, and the instruction segments are used to describe the execution process of the computer program 202 in the processing device 20.

[0204] It should be understood that the magnitudes of the sequence numbers of the steps in the above embodiments do not mean the order of execution is prior or subsequent. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation to the implementation process of the embodiments of this application.

[0205] Those skilled in the art can clearly understand that, for the convenience and simplicity of description, only the above-mentioned division of each functional unit and module is used as an example. In actual applications, the above-mentioned functions can be allocated to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. Each functional unit and module in the embodiments can be integrated into a processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of a software functional unit. In addition, the specific names of each functional unit and module are only for the convenience of mutual distinction and do not limit the protection scope of the present application. The specific working processes of the units and modules in the above system can refer to the corresponding processes in the foregoing method embodiments and will not be elaborated here.

[0206] If the foregoing integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such an understanding, to implement all or part of the processes in the above method embodiments of the present application, a computer program can be used to instruct relevant hardware to complete. The computer program can be stored in a computer-readable storage medium; when the computer program is executed by a processor, the steps of the above-mentioned various method embodiments can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in the form of source code, object code, executable file or some intermediate form, etc. The computer-readable medium includes: any entity or device that can carry the computer program code to the device / terminal device, recording medium, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electrical carrier signal, telecommunication signal, and software distribution medium. For example, a USB flash drive, a mobile hard disk, a magnetic disk or an optical disc, etc. In some jurisdictions, according to legislation and patent practice, the computer-readable medium cannot be an electrical carrier signal and a telecommunication signal.

[0207] The embodiments of the present application also provide a computer-readable storage medium that stores a computer program, and when the computer program is executed by a processor, the steps in the above-mentioned various method embodiments can be implemented.

[0208] The embodiments of the present application provide a computer program product, and when the computer program product runs on a terminal device, the terminal device can implement the steps in the above-mentioned various method embodiments.

[0209] In the above embodiments, the descriptions of the various embodiments each have their own emphasis. For parts not described in detail or recorded in a certain embodiment, reference may be made to the relevant descriptions of other embodiments.

[0210] Those of ordinary skill in the art can realize that the units and algorithm steps of the examples described in combination with the embodiments disclosed herein can be implemented by electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are executed in a hardware or software manner depends on the specific application and design constraints of the technical solution. A professional technician can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of this application.

[0211] In the embodiments provided in this application, it should be understood that the disclosed device / apparatus and method can be implemented in other ways. For example, the device / apparatus embodiments described above are merely illustrative. For example, the division of the modules or units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed couplings or direct couplings or communication connections to each other can be through some interfaces. The indirect couplings or communication connections of the device or unit can be in an electrical, mechanical or other form.

[0212] The units described above as separate components may or may not be physically separated. The components shown as units may or may not be physical units, that is, they may be located in one place, or may be distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0213] The above-described embodiments are only used to illustrate the technical solutions of this application, rather than to limit them; although this application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or perform equivalent replacements on some of the technical features; and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A method for identifying a workpiece, characterized in that, The method includes: Obtaining multiple depth images of the workpiece; Rectifying each of the depth images so that the workpieces in each of the depth images have a unified size ratio; Stitching the rectified depth images; Extracting each edge of the workpiece from the stitched depth image; Performing geometric shape fitting on the extracted edges; Selecting edge combinations from the fitted edges to form the contour of the workpiece.

2. The method according to claim 1, characterized in that, The method further includes: Selecting the contour located at the uppermost layer from the contours as the contour of the workpiece.

3. The method according to claim 1, wherein The rectifying each of the depth images so that the workpieces in each of the depth images have a unified size ratio includes: Determining the center point coordinates of each of the depth images; Determining the coordinate range of each of the rectified depth images according to the center point coordinates; Determining the sampling interval for discretization processing; Discretizing the coordinate data of each of the depth images according to the coordinate range and the sampling interval.

4. The method according to claim 1, characterized in that The stitching the rectified depth images includes: Obtaining the specified point coordinates of the specified point of the workpiece in each of the depth images; Determining the surface normal vectors of the workpiece in each of the depth images according to the specified point coordinates; Determining the rotation matrix and the translation matrix according to the surface normal vectors; Converting the coordinate data of each of the depth images to the same image according to the rotation matrix and the translation matrix to complete the stitching.

5. The method according to claim 1, characterized in that, The extracting the edges of the workpiece from the stitched depth image includes: Obtaining the height difference between both sides of each position of the stitched depth image; Determining all edges where the height changes according to the height difference.

6. The method according to claim 1, characterized in that, The performing geometric shape fitting on the extracted edges includes: Extracting the edge line parameters of each of the edges; Merging multiple edges into one edge according to the edge line growth merging parameters and the edge line parameters.

7. The method according to any one of claims 1 to 6, characterized in that, The selecting edge combinations from the fitted edges to form the contour of the workpiece includes: Determining the angular and positional relationships of the fitted edges according to the geometric constraint conditions of the contour of the workpiece; Selecting the edges that satisfy the angular and positional relationship constraints from the fitted edges to form the contour of the workpiece.

8. A device for identifying a workpiece, characterized in that, The device includes: An image acquisition module for obtaining multiple depth images of the workpiece; A rectification module for rectifying each of the depth images so that the workpieces in each of the depth images have a unified size ratio; A stitching module for stitching the rectified depth images; An extraction module for extracting each edge of the workpiece from the stitched depth image; A fitting module for performing geometric shape fitting on the extracted edges; A screening module for selecting edge combinations from the fitted edges to form the contour of the workpiece.

9. A processing device, characterized in that, It includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the method described in any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, and when the computer program is executed by the processor, the method described in any one of claims 1 to 7 is implemented.

Images