Method, device and apparatus for positioning a steel sheet

Abstract

The positioning method, device and equipment of the steel plate provided by the present disclosure, the method comprises: obtaining a preset template and a depth image obtained by a camera, the preset template comprising a plurality of first point cloud data; the first point cloud data has pose information; identifying a plurality of second point cloud data corresponding to a steel plate contour of the steel plate in the depth image; performing feature matching on the first point cloud data and the second point cloud data to determine the correspondence between the steel plate contour and the preset template, wherein the correspondence is used to indicate the first point cloud data and the second point cloud data with the same feature; and determining the positioning information of the steel plate according to the correspondence and the pose information of the first point cloud data. The steel plate positioning method provided by the present disclosure does not require manual adjustment of the position of the steel plate, which is beneficial to reduce the consumption of manpower, improve the positioning efficiency of the steel plate, and further improve the efficiency of subsequent grabbing of the workpiece in the steel plate.

Description

Technical Field

[0001] This disclosure relates to the field of electronics, and more particularly to a method, apparatus and equipment for positioning steel plates. Background Technology

[0002] Currently, in the industrial steel plate processing, a single steel plate is typically cut into multiple workpieces. These workpieces can then be welded into the devices required by the user. Furthermore, during workpiece handling, the positional information of each workpiece within the steel plate is determined based on the overall positional information of the steel plate.

[0003] In related technologies, users pre-set fixed position information on the workpiece gripping table. After manually moving the steel plate to the fixed position information, the position corresponding to the fixed position information can be used as the position of the steel plate. Then, based on the fixed position information, the position information of the workpiece in the steel plate can be determined.

[0004] However, manually aligning the steel plate with the fixed position information requires a significant amount of manpower. Summary of the Invention

[0005] This disclosure provides a method, apparatus, and equipment for positioning steel plates, which solves the problem that steel plate positioning in related technologies requires a lot of manpower and takes a long time.

[0006] In a first aspect, this disclosure provides a method for positioning a steel plate, comprising:

[0007] A preset template and a depth image captured by a camera are obtained, wherein the preset template includes multiple first point cloud data; the first point cloud data has pose information;

[0008] Identify multiple second point cloud data corresponding to the steel plate outline in the depth image;

[0009] Feature matching is performed on the first point cloud data and the second point cloud data to determine the correspondence between the steel plate outline and the preset template, wherein the correspondence is used to indicate the first point cloud data and the second point cloud data with the same features;

[0010] Based on the correspondence and the pose information of the first point cloud data, the positioning information of the steel plate is determined.

[0011] In one possible implementation, feature matching is performed on the first point cloud data and the second point cloud data to determine the correspondence between the steel plate outline and the preset template, including:

[0012] Acquire at least one first point pair feature from multiple first point cloud data, and at least one second point pair feature from multiple second point cloud data; the first point pair feature is used to characterize the relative distance and orientation between two first point cloud data; the second point pair feature is used to characterize the relative distance and orientation between two second point cloud data.

[0013] The first point-to-feature pair and the second point-to-feature pair are matched to determine the correspondence between the steel plate outline and the preset template.

[0014] In one possible implementation, identifying multiple second point cloud data corresponding to the steel plate contour in the depth image includes:

[0015] Based on the depth image, determine the mask information of the steel plate in the depth image;

[0016] The mask information is subjected to contour extraction processing to obtain the contour information of the steel plate, and multiple second point cloud data corresponding to the contour information are determined.

[0017] In one possible implementation, determining the mask information of the steel plate in the depth image based on the depth image includes:

[0018] Extract two-dimensional image information from the depth image;

[0019] The two-dimensional image information is input into the image recognition model to obtain the mask information of the steel plate. The image recognition model is used to identify the mask corresponding to the steel plate in the received image.

[0020] In one possible implementation, determining the positioning information of the steel plate based on the correspondence and the pose information of the first point cloud data includes:

[0021] Determine the relative positional relationship between the first point cloud data and the second point cloud data that have the aforementioned correspondence;

[0022] Based on the relative positional relationship, the pose information of the first point cloud data is subjected to coordinate transformation processing to obtain the pose information of the second point cloud data;

[0023] The positioning information of the steel plate is determined based on the pose information of the second point cloud data.

[0024] In one possible implementation, the graphic formed by the multiple first point cloud data corresponding to the preset template is the same as the graphic of a preset local area of ​​the steel plate outline.

[0025] In one possible implementation, the steel plate includes at least one workpiece to be gripped; the workpiece to be gripped has first relative position information; the first relative position information is used to indicate the relative positional relationship between the workpiece to be gripped and the outline of the steel plate; the method further includes:

[0026] The position information of the workpiece to be grasped is determined based on the positioning information of the steel plate and the first relative position information.

[0027] In one possible implementation, the method further includes:

[0028] Determine the second relative position information of the second point cloud data corresponding to the outline of the steel plate, which is located at the position information;

[0029] If it is determined that the second relative position information is different from the first relative position information, then a difference calculation is performed based on the second relative position information and the first relative position information to obtain the difference result; and based on the difference, the position information of the workpiece to be grasped is calibrated to obtain the calibrated position information.

[0030] In one possible implementation, the graphic formed by the multiple first point cloud data corresponding to the preset template is the lower right corner of the steel plate outline; the first relative position information is specifically used to indicate the vertical distance between the center point of the workpiece to be grasped and the horizontal side of the steel plate outline; the second relative position information is specifically used to indicate the vertical distance between the center point of the workpiece to be grasped at the position information and the second point cloud data of the horizontal side of the steel plate outline.

[0031] Secondly, this disclosure provides a positioning device for a steel plate, comprising:

[0032] The acquisition unit is used to acquire a preset template and a depth image captured by a camera. The preset template includes multiple first point cloud data. The first point cloud data has pose information.

[0033] The recognition unit is used to recognize multiple second point cloud data corresponding to the outline of the steel plate in the depth image;

[0034] The first determining unit is used to perform feature matching on the first point cloud data and the second point cloud data to determine the correspondence between the steel plate outline and the preset template, wherein the correspondence is used to indicate the first point cloud data and the second point cloud data having the same features.

[0035] The second determining unit is used to determine the positioning information of the steel plate based on the correspondence and the pose information of the first point cloud data.

[0036] In one possible implementation, the first determining unit includes:

[0037] The acquisition module is used to acquire at least one first point pair feature of multiple first point cloud data and at least one second point pair feature of multiple second point cloud data; the first point pair feature is used to characterize the relative distance and orientation between two first point cloud data; the second point pair feature is used to characterize the relative distance and orientation between two second point cloud data.

[0038] The first determining module is used to perform feature matching on the first point-to-point feature and the second point-to-point feature to determine the correspondence between the steel plate outline and the preset template.

[0039] In one possible implementation, the identification unit includes:

[0040] The second determining module is used to determine the mask information of the steel plate in the depth image based on the depth image;

[0041] The processing module is used to perform contour extraction processing on the mask information to obtain the contour information of the steel plate, and to determine multiple second point cloud data corresponding to the contour information.

[0042] In one possible implementation, the second determining module is specifically used to extract two-dimensional image information from the depth image;

[0043] The two-dimensional image information is input into the image recognition model to obtain the mask information of the steel plate. The image recognition model is used to identify the mask corresponding to the steel plate in the received image.

[0044] In one possible implementation, the second determining unit includes:

[0045] The third determining module is used to determine the relative positional relationship between the first point cloud data and the second point cloud data that have the corresponding relationship;

[0046] The processing module is used to perform coordinate transformation processing on the pose information of the first point cloud data according to the relative positional relationship to obtain the pose information of the second point cloud data.

[0047] The fourth determining module is used to determine the positioning information of the steel plate based on the pose information of the second point cloud data.

[0048] 0 In one possible implementation, the graph enclosed by the multiple first point cloud data corresponding to the preset template is...

[0049] The shape is the same as the graphic of a preset local area of ​​the outline of the steel plate.

[0050] In one possible implementation, the steel plate includes at least one workpiece to be gripped; the workpiece to be gripped has first relative position information; the first relative position information is used to indicate the relationship between the workpiece to be gripped and the steel plate.

[0051] The relative positional relationship of the plate outline; the device further includes: 5 a third determining unit, used to determine the relative positional relationship of the steel plate according to the positioning information of the steel plate and the first relative positional information.

[0052] Describe the position information of the workpiece to be grabbed.

[0053] In one possible implementation, the device further includes:

[0054] The fourth determining unit is used to determine whether the workpiece to be grasped at the location information corresponds to the outline of the steel plate.

[0055] The second relative position information of the second point cloud data;

[0056] A processing unit 0 is configured to, if it is determined that the second relative position information is different from the first relative position information, then

[0057] Based on the second relative position information and the first relative position information, a difference calculation is performed to obtain the difference result;

[0058] A calibration unit is used to calibrate the position information of the workpiece to be grasped based on the difference, so as to obtain calibrated position information.

[0059] In one possible implementation, the preset template is enclosed by multiple first point cloud data. Figure 5 The shape is the lower right corner of the steel plate outline; the first relative position information is specifically used to indicate the vertical distance between the center point of the workpiece to be gripped and the horizontal side of the steel plate outline; the second relative position information is specifically used to indicate the vertical distance between the center point of the workpiece to be gripped at the position information and the second point cloud data of the horizontal side of the steel plate outline.

[0060] Thirdly, this disclosure provides an electronic device, including: a memory and a processor;

[0061] 0. Memory; memory used to store the processor-executable instructions;

[0062] The processor is configured to execute the method as described in any of the first aspects according to the executable instructions.

[0063] Fourthly, this disclosure provides a computer-readable storage medium storing computer-executable instructions that, when executed by a processor, are used to implement the method as described in any of the first aspects.

[0064] Fifthly, this disclosure provides a computer program product comprising a computer program that, when executed by a processor, implements the method described in any one of the first aspects.

[0065] The present disclosure provides a method, apparatus, and device for positioning steel plates. The method includes: acquiring a depth image obtained from a preset template and a camera, wherein the preset template includes multiple first point cloud data; the first point cloud data has pose information; identifying multiple second point cloud data corresponding to the steel plate contour in the depth image; performing feature matching on the first point cloud data and the second point cloud data to determine the correspondence between the steel plate contour and the preset template, wherein the correspondence is used to indicate first point cloud data and second point cloud data having the same features; and determining the positioning information of the steel plate based on the correspondence and the pose information of the first point cloud data. The steel plate positioning method provided in this embodiment eliminates the need for manual adjustment of the steel plate position by the user, which helps reduce manual labor, improves steel plate positioning efficiency, and consequently improves the efficiency of subsequent workpiece gripping within the steel plate. Attached Figure Description

[0066] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0067] Figure 1 A schematic flowchart illustrating a method for positioning a steel plate according to an embodiment of this disclosure;

[0068] Figure 2 A schematic flowchart illustrating the second steel plate positioning method provided in this embodiment of the present disclosure;

[0069] Figure 3 A schematic diagram of a preset template provided in an embodiment of this disclosure;

[0070] Figure 4 A schematic flowchart illustrating the third steel plate positioning method provided in this embodiment of the present disclosure;

[0071] Figure 5 A schematic diagram illustrating a steel plate offset according to an embodiment of this disclosure;

[0072] Figure 6 A schematic diagram of a steel plate positioning device provided in an embodiment of this disclosure;

[0073] Figure 7 A schematic diagram of the structure of the positioning device for the second type of steel plate provided in the embodiments of this disclosure;

[0074] Figure 8 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure.

[0075] The accompanying drawings have illustrated specific embodiments of this disclosure, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concepts of this disclosure to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0076] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure.

[0077] Currently, in industrial steel plate processing, a single steel plate is typically cut into multiple steel plate workpieces. These workpieces can have different sizes and shapes to meet user requirements. After the steel plate is cut, the resulting workpieces are usually handled and sorted.

[0078] In order to improve the gripping efficiency of steel plate workpieces, related technologies typically employ pre-set gripping devices such as robots to grip each workpiece. To ensure that the robot can accurately grip each workpiece, fixed position frames can be set on a pre-set placement platform, and the workpieces to be gripped are transferred to the fixed position frames in the pre-set placement platform via a conveyor device.

[0079] However, when the steel plate workpiece is transferred to the designated position frame, the steel plate often shifts. When the steel plate shifts, the position of the workpiece within it also shifts, making it impossible to ensure that the robot can accurately grasp the workpiece. In one example, the steel plate can be manually aligned with the designated position frame; however, this method requires significant manpower and can easily lead to low steel plate grasping efficiency.

[0080] The steel plate positioning method, apparatus, and equipment disclosed herein are used to solve the above-mentioned technical problems.

[0081] The technical solutions of this disclosure and how they solve the aforementioned technical problems will be described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of this disclosure will now be described with reference to the accompanying drawings.

[0082] Figure 1 This is a flowchart illustrating a method for positioning a steel plate according to an embodiment of the present disclosure, as shown below. Figure 1 As shown, the method includes the following steps:

[0083] S101. Obtain a preset template and a depth image captured by a camera. The preset template includes multiple first point cloud data. The first point cloud data has pose information.

[0084] For example, in this embodiment, when gripping a workpiece in a steel plate, it is first necessary to determine the position information of the steel plate, that is, to position the steel plate. In practical applications, when the steel plate is transported to the gripping table, a depth image of the area captured by the camera installed on the gripping table is obtained. In addition, a preset template is also obtained, wherein the preset template is pre-established and consists of first point cloud data with multiple known pose information.

[0085] In one example, the template in this embodiment can be a template with the same size as the steel plate that needs to be positioned. For example, when the outline size of the steel plate that needs to be positioned is a rectangular outline of A*B, the size of the shape outline formed by the first point cloud data in the preset template can also be a rectangular outline of A*B.

[0086] In one example, there are many templates of different sizes in the preset template. When determining the style of the preset template, it can be selected according to the style specified by the user, or it can be selected according to the outline of the steel plate after obtaining the outline of the steel plate in the depth image. No specific restrictions are made in this embodiment.

[0087] S102. Identify multiple second point cloud data corresponding to the steel plate outline in the depth image.

[0088] For example, after acquiring the depth image captured by the camera, multiple second point cloud data corresponding to the steel plate outline contained in the depth image are determined based on the acquired depth image.

[0089] In one example, when identifying the point cloud of a steel plate's contour in a depth image, the steel plate's contour can be directly determined based on the 3D point cloud contained in the depth image. Specifically, the method for extracting the steel plate point cloud contour can be found in the contour recognition techniques provided in related technologies.

[0090] S103. Perform feature matching on the first point cloud data and the second point cloud data to determine the correspondence between the steel plate outline and the preset template, wherein the correspondence is used to indicate the first point cloud data and the second point cloud data with the same features.

[0091] For example, in this embodiment, after obtaining the first point cloud data and the second point cloud data in the preset template, feature matching can be performed on the first point cloud data and the second point cloud data to determine the point cloud that has the same features as the second point cloud data among multiple first point cloud data. That is, by using the above feature matching method, the first point cloud data and the second point cloud data with the same features are found among multiple first point cloud data and multiple second point cloud data.

[0092] In one example, when performing feature matching on the first and second point cloud data, one can match the features of a single first point cloud data point with the features of a single second point cloud data point, i.e., single-point feature matching; alternatively, one can match the features of a region composed of multiple first point cloud data points with the features of a region composed of multiple second point cloud data points, and then determine the correspondence between the first and second point cloud data points within the region. This embodiment does not impose specific restrictions on the feature matching method.

[0093] It should be noted that, in this embodiment, two identical features can be understood as two features whose similarity is greater than a preset threshold.

[0094] S104. Determine the positioning information of the steel plate based on the correspondence and the pose information of the first point cloud data.

[0095] For example, after determining the correspondence between the first point cloud data and the second point cloud data, since the pose information of the first point cloud data is known, the pose information of the second point cloud data with the same characteristics as the first point cloud data can be determined based on the pose information of the first point cloud data, and thus the pose information of the outline of the steel plate (i.e., the pose information of the steel plate) can be obtained.

[0096] In one example, the pose information of the first point cloud data is the pose information in the camera coordinate system. This is so that after the second point cloud data is determined by the depth image obtained by the camera, the pose information of the second point cloud data in the camera coordinate system can be determined based on the determined correspondence.

[0097] It is understood that, in this embodiment, to achieve the positioning of the steel plate, a preset template composed of multiple first point cloud data with known pose information can be pre-set. When the steel plate needs to be positioned, the camera can acquire the contour point cloud (i.e., second point cloud data) corresponding to the steel plate, and then the preset template and the second point cloud data are feature matched to obtain the correspondence between them. This allows the pose information of the second point cloud data to be determined based on the pose information of the first point cloud data in the preset template, thereby achieving steel plate positioning. The steel plate positioning method provided in this embodiment does not require the user to manually adjust the position of the steel plate, which helps to reduce manual labor, improve the steel plate positioning efficiency, and thus improve the efficiency of subsequent workpiece gripping in the steel plate.

[0098] Figure 2 This is a flowchart illustrating the second steel plate positioning method provided in this embodiment of the present disclosure, as shown below. Figure 2 As shown, the method includes the following steps:

[0099] S201. Obtain a preset template and a depth image captured by a camera. The preset template includes multiple first point cloud data. The first point cloud data has pose information.

[0100] In one example, the shape enclosed by multiple first point cloud data corresponding to the preset template is the same as the shape of the preset local area of ​​the steel plate outline.

[0101] For example, in this embodiment, when setting the preset template, the contour corresponding to the first point cloud data in the preset template can be the contour of a preset local area of ​​the steel plate contour.

[0102] For example, such as Figure 3 As shown, Figure 3 This is a schematic diagram of a preset template provided in an embodiment of this disclosure. As shown in the figure, Figure 3 Figure (a) shows the outline of the steel plate, which can be seen from the figure as a rectangular shape. Figure 3 Figure (b) shows the contour formed by the point cloud corresponding to the preset template. It can be seen from the figure that the contour corresponding to the preset template is the shape corresponding to the lower right corner of the steel plate contour. Therefore, when performing feature matching of the first and second point cloud data, it is necessary to select the second point cloud data at the lower right corner of the steel plate contour and perform contour matching with the first point cloud data in the preset template. The pose information of the remaining second point cloud data in the steel plate contour, excluding the second point cloud data that already has the same features as the first point cloud data in the preset template, can be determined based on the relative pose relationships between the second point cloud data in the steel plate contour.

[0103] It is understood that in this embodiment, the image enclosed by the first point cloud data in the preset template can be a preset local area of ​​the steel plate outline. Therefore, by setting the preset template as described above, compared to using a preset template identical to the steel plate outline, the method provided in this embodiment can reduce the amount of point cloud data that needs to be compared during feature comparison, which is beneficial for improving the steel plate positioning efficiency and reducing the processing resources consumed by the equipment. Furthermore, the above method can also reduce the storage space occupied by the preset template.

[0104] S202. Based on the depth image, determine the mask information of the steel plate in the depth image.

[0105] For example, in this embodiment, when determining the outline of the steel plate contained in the depth image, the mask information corresponding to the steel plate can first be identified in the depth image, that is, the area where the steel plate is located in the depth image can be determined.

[0106] In one example, step S202 includes the following steps:

[0107] The first step of step S202: Extract two-dimensional image information from the depth image.

[0108] The second step of step S202: Input the two-dimensional image information into the image recognition model to obtain the mask information of the steel plate. The image recognition model is used to identify the mask corresponding to the steel plate in the received image.

[0109] For example, in this embodiment, since the steel plate itself is relatively thin, it is difficult to extract the mask information corresponding to the steel plate in the depth image directly based on the three-dimensional depth image. Therefore, in this embodiment, the three-dimensional depth image can first be converted into two-dimensional image information. After obtaining the two-dimensional image information corresponding to the depth image, the obtained two-dimensional image information can be input into a pre-trained image recognition model, which can be used to identify the steel plate in the image. In practical applications, when training the image recognition model, the position information of the steel plate in the image can be pre-annotated, and the image recognition model can be trained based on the annotated two-dimensional image so that the obtained image recognition model can accurately identify the mask information of the region corresponding to the steel plate in the input two-dimensional image.

[0110] It is understood that in this embodiment, by converting the depth image into a two-dimensional image and determining the mask information corresponding to the steel plate in the depth image based on the two-dimensional image, the problem of inaccurate positioning of the steel plate contour is avoided when directly using the three-dimensional depth image to determine the contour point cloud corresponding to the steel plate, which is prone to occur due to the thinness of the steel plate.

[0111] S203. Perform contour extraction processing on the mask information to obtain the contour information of the steel plate, and determine the multiple second point cloud data corresponding to the contour information.

[0112] For example, after obtaining the mask information corresponding to the steel plate, contour recognition can be performed on the mask information of the steel plate to determine the contour information corresponding to the steel plate.

[0113] In one example, when identifying the outline of a steel plate, the pixel in the mask information can be continuously compared with the pixels of its surrounding pixels. If they are the same, the pixel is considered to be inside the outline; otherwise, the pixel can be considered to be the outline boundary.

[0114] After determining the contour information of the steel plate, the point cloud data indicated by the contour information in the depth image is determined as the second point cloud data corresponding to the contour information of the steel plate.

[0115] It is understood that in this embodiment, when extracting the second point cloud data corresponding to the steel plate contour in the depth image, the mask information corresponding to the steel plate in the depth image can be identified first. That is, the area where the entire steel plate is located in the depth image is determined first. Then, the contour information of the steel plate is accurately extracted according to the area where the depth image is located, so as to determine the second point cloud data corresponding to the contour information, so that the steel plate can be located based on the second point cloud data and the preset template.

[0116] S204. Obtain at least one first point pair feature of multiple first point cloud data, and at least one second point pair feature of multiple second point cloud data; the first point pair feature is used to characterize the relative distance and orientation between two first point cloud data; the second point pair feature is used to characterize the relative distance and orientation between two second point cloud data.

[0117] For example, in this embodiment, when performing feature matching on the first point cloud data and the second point cloud data, firstly, at least one first point pair feature is determined among the multiple first point cloud data, and at least one second point pair feature is determined among the multiple second point cloud data. The first point pair feature can be used to indicate the relative distance and orientation features between two different first point cloud data. Similarly, the second point pair feature can be used to indicate the relative distance and orientation features between two different second point cloud data.

[0118] In one example, when determining the first point pair features, two first point cloud data points can be randomly selected from multiple first point cloud data points to form a point pair. Then, the distance information between the two first point cloud data points in the point pair and the normal features (i.e., orientation features) of the two first point cloud data points are used as the first point pair features of the point pair. It should be noted that this embodiment does not impose a specific limit on the number of point pairs.

[0119] S205. Perform feature matching on the first point pair feature and the second point pair feature to determine the correspondence between the steel plate outline and the preset template; wherein, the correspondence is used to indicate the first point cloud data and the second point cloud data with the same features.

[0120] For example, after determining at least one first point pair feature and at least one second point pair feature, feature matching is performed on the first point pair feature and the second point pair feature, that is, the similarity between the first point pair feature and the second point pair feature is determined, and then the first point pair feature and the second point pair feature with the same feature are determined, so as to determine the first point cloud data and the second point cloud data with the same feature based on the first point pair feature and the second point pair feature with the same feature.

[0121] It is understood that in this embodiment, when matching features between the first point cloud data and the second point cloud data, the first point pair feature determined by the first point cloud data and the second point pair feature determined by the second point cloud data can be matched. In this way, feature matching can be performed on the two aspects of distance and orientation indicated by the point pair feature, which helps to improve the accuracy of feature matching, so as to improve the accuracy of subsequent steel plate positioning.

[0122] S206. Determine the relative positional relationship between the first point cloud data and the second point cloud data that have a corresponding relationship.

[0123] For example, in this embodiment, when determining the pose information of the second point cloud data according to the correspondence and the preset template, since the first point cloud data and the second point cloud data with the same features have been determined during the feature matching process, the relative position relationship when the first point cloud data moves to the second point cloud data with the same features as the first point cloud data can be determined.

[0124] S207. Based on the relative positional relationship, perform coordinate transformation on the pose information of the first point cloud data to obtain the pose information of the second point cloud data.

[0125] For example, when the relative positional relationship between the first point cloud data and the second point cloud data having the same characteristics is obtained, the pose information of the second point cloud data having the same characteristics as the first point cloud data can be determined based on the pose information of the first point cloud data and the determined relative positional relationship.

[0126] S208. Determine the positioning information of the steel plate based on the pose information of the second point cloud data.

[0127] For example, after obtaining the pose information of the second point cloud data, the positioning information of the steel plate can be determined based on the obtained pose information of the second point cloud data.

[0128] In one example, the pose information of the second point cloud data determined in step S207 is the pose in the camera coordinate system. In order to facilitate the robot's subsequent grasping of the steel plate, the pose information of the second point cloud data can be converted from the camera coordinate system to the robot coordinate system, and the converted pose information is determined as the positioning information of the steel plate.

[0129] Understandably, once the corresponding first point cloud data and second point cloud data are determined, the pose information corresponding to the second point cloud data can be determined based on the relative positional relationship between the corresponding first point cloud data and the pose information of the first point cloud data, so as to accurately determine the positioning information of the steel plate.

[0130] In this embodiment, by matching the first point pair features determined by the first point cloud data and the second point pair features determined by the second point cloud data, feature matching can be performed on both the distance and orientation indicated by the point pair features, thereby improving the accuracy of feature matching and thus improving the accuracy of subsequent steel plate positioning. Furthermore, when extracting the second point cloud data corresponding to the steel plate contour in the depth image, the mask information corresponding to the steel plate in the depth image can first be identified. That is, the region where the entire steel plate is located in the depth image is first determined. Then, the contour information of the steel plate is accurately extracted based on the region in the depth image to determine the second point cloud data corresponding to the contour information. This allows for subsequent positioning of the steel plate based on the second point cloud data and a preset template. Moreover, by converting the depth image into a two-dimensional image and determining the mask information corresponding to the steel plate in the depth image based on the two-dimensional image, the problem of inaccurate steel plate contour positioning due to the thinness of the steel plate when directly using a three-dimensional depth image to determine the contour point cloud is avoided.

[0131] Figure 4 A flowchart illustrating the third steel plate positioning method provided in this embodiment is shown below. Figure 4 As shown, the method includes the following steps:

[0132] S401. Obtain a preset template and a depth image captured by a camera. The preset template includes multiple first point cloud data. The first point cloud data has pose information.

[0133] S402. Identify multiple second point cloud data corresponding to the steel plate outline in the depth image.

[0134] S403. Perform feature matching on the first point cloud data and the second point cloud data to determine the correspondence between the steel plate outline and the preset template, wherein the correspondence is used to indicate the first point cloud data and the second point cloud data with the same features.

[0135] S404. Based on the correspondence and the pose information of the first point cloud data, determine the positioning information of the steel plate.

[0136] For example, the specific principles of steps S401-S404 in this embodiment can be found in steps S101-S104, and will not be repeated here.

[0137] S405. Based on the positioning information of the steel plate and the first relative position information, determine the position information of the workpiece to be gripped, wherein the steel plate includes at least one workpiece to be gripped; the workpiece to be gripped has the first relative position information; the first relative position information is used to indicate the relative positional relationship between the workpiece to be gripped and the outline of the steel plate.

[0138] For example, in this embodiment, when the steel plate includes one or more workpieces to be gripped, and the relative positional relationship (i.e., the first relative positional relationship) between each workpiece to be gripped and the outline of the steel plate can be obtained in advance, when the position of the steel plate outline is determined based on the above steps S401-S404, the position information corresponding to the workpiece to be gripped in the steel plate can be determined according to the positioning information of the steel plate obtained in the above steps and the known first relative positional relationship. The first relative positional relationship between the steel plate outline and the workpiece to be gripped in the steel plate can be determined by the workpiece position information when cutting the workpiece on the steel plate.

[0139] It is understood that in this embodiment, when the steel plate includes a pre-cut workpiece to be gripped, the position information of the workpiece to be gripped in the steel plate can be further determined based on the positioning information determined by the above-mentioned steel plate positioning method, so that the workpiece can be gripped according to the determined position information, thereby improving the efficiency of workpiece gripping.

[0140] S406. Determine the second relative position information of the second point cloud data corresponding to the outline of the steel plate and the workpiece to be grabbed at the position information location.

[0141] For example, when the preset template is Figure 3 In the case shown in Figure (b), where the graphic formed by the first point cloud data corresponding to the preset template is a preset local area of ​​the graphic corresponding to the steel plate outline, in order to avoid the problem of deviation between the second point cloud data of the steel plate outline and the actual position caused by feature matching error, this embodiment will further calibrate the obtained position information of the workpiece to be grasped. Figure 5 As shown, Figure 5This is a schematic diagram illustrating a steel plate offset according to an embodiment of this disclosure. Figure 5 As shown in the figure, the rectangular area enclosed by the solid lines represents the position of the steel plate determined according to steps S101-S104. The coordinate system corresponding to the lower right corner of the steel plate can be regarded as the shape formed by the preset template. The rectangular area enclosed by the dashed lines represents the actual position of the steel plate. It can be seen from the figure that when feature matching is used to determine the correspondence between the first and second point cloud data to determine the positioning information of the steel plate, there will still be a certain rotational deviation between the determined positioning information and the actual position of the steel plate. Therefore, further calibration of the position information is required.

[0142] When the position information corresponding to the workpiece to be grasped is determined based on the positioning information (i.e., the result obtained from feature matching) and the first relative positional relationship, a second relative positional relationship can be further determined between the determined position information and the second point cloud data corresponding to the steel plate contour (i.e., the point cloud information corresponding to the steel plate contour). For example, the second relative positional relationship can be the distance between the position information and the second point cloud data, such as the distance indicated by the double arrows in the figure.

[0143] S407. If it is determined that the second relative position information is different from the first relative position information, then the difference is calculated based on the second relative position information and the first relative position information to obtain the difference result; and based on the difference, the position information of the workpiece to be grasped is calibrated to obtain the calibrated position information.

[0144] For example, after determining the second relative position information, if the second relative position information is different from the first relative position information, it indicates that there is a deviation in the position information corresponding to the workpiece to be grasped. Therefore, the difference between the first relative position information and the second relative position information can be calculated, and the position information of the workpiece to be grasped can be adjusted according to the difference.

[0145] For example, such as Figure 5 As shown, the closed shape within the solid-line rectangle indicates the workpiece to be grasped in the steel plate. The position information corresponding to this solid-line closed shape is determined based on the positioning information and the first relative position information. There is a positional deviation between this position information and the dashed-line closed shape within the dashed-line box (i.e., the workpiece to be grasped at the actual position). To correct the position information, in this embodiment, a second relative positional relationship is further determined between the realized closed shape and the second point cloud data (wherein the area enclosed by the second point cloud data is the dashed-line rectangle). Furthermore, the position information can be corrected by the deviation between the first and second relative positions. That is, by determining the difference between the two, the determined position information is shifted downwards by the value corresponding to the difference to obtain the calibrated position information.

[0146] In one example, when the steel plate contains multiple workpieces to be gripped, the position information corresponding to each workpiece to be gripped can be corrected in the above manner.

[0147] It is understood that in this embodiment, in order to avoid the deviation between the determined position information of the workpiece to be grasped and the actual position of the workpiece to be grasped, the position is calibrated by calculating the difference between the second relative position information and the first relative position information, thereby improving the accuracy of the position determination of the workpiece to be grasped.

[0148] In some embodiments, to further improve the accuracy of workpiece positioning after step S407, step S408 may be included. In step S408, based on the calibrated position obtained in step S407 and the image previously captured by the camera, the workpiece contour corresponding to the workpiece can be identified in the image, and the workpiece contour corresponding to the workpiece can be matched with the workpiece template corresponding to the workpiece type. The workpiece template includes point cloud data with multiple known pose information, thereby determining the matching relationship between the workpiece contour point cloud data and the workpiece template point cloud data that have the same features. Further, based on the point cloud data in the workpiece template and the determined matching relationship, the accurate positioning information of the workpiece is obtained. The feature matching process here can refer to the matching process of the steel plate contour and the preset template in the above embodiments, and will not be repeated here.

[0149] In one example, the graphic formed by multiple first point cloud data corresponding to the preset template is the lower right corner of the steel plate outline; the first relative position information is specifically used to indicate the vertical distance between the center point of the workpiece to be grasped and the horizontal side of the steel plate outline; the second relative position information is specifically used to indicate the vertical distance between the center point of the workpiece to be grasped at the position information and the second point cloud data of the horizontal side of the steel plate outline.

[0150] For example, the preset template provided in this example can be referred to Figure 5 The pattern shown is the lower right corner of the steel plate outline formed by the multiple first point cloud data corresponding to the preset template. Furthermore, when the first relative position information is the vertical distance between the center point of the workpiece to be gripped and the boundary of the steel plate outline (indicated by the positioning information) along the X-axis, the second relative position information specifically indicates the vertical distance between the center point of the workpiece to be gripped at the position information location and the second point cloud data (actual steel plate outline) of the edge along the X-axis of the steel plate outline. Therefore, only the distance between the center point and the edge enclosed by the aforementioned second point cloud data needs to be determined to obtain the second relative position information.

[0151] Understandably, determining the second relative position information by selecting the center point as described above can improve the efficiency of position calibration.

[0152] Figure 6 This is a schematic diagram of the structure of a steel plate positioning device provided in an embodiment of the present disclosure, as shown below. Figure 6 As shown, the device includes:

[0153] The acquisition unit 601 is used to acquire a preset template and a depth image captured by a camera. The preset template includes multiple first point cloud data. The first point cloud data has pose information.

[0154] The recognition unit 602 is used to recognize multiple second point cloud data corresponding to the steel plate outline in the depth image;

[0155] The first determining unit 603 is used to perform feature matching on the first point cloud data and the second point cloud data to determine the correspondence between the steel plate outline and the preset template, wherein the correspondence is used to indicate the first point cloud data and the second point cloud data with the same features.

[0156] The second determining unit 604 is used to determine the positioning information of the steel plate based on the correspondence and the pose information of the first point cloud data.

[0157] The apparatus provided in this embodiment is used to implement the technical solution provided by the above method. Its implementation principle and technical effect are similar, and will not be described again.

[0158] Figure 7 This is a schematic diagram of the structure of the second type of steel plate positioning device provided in the embodiments of this disclosure, as shown below. Figure 7 As shown, in Figure 6 Based on the structural diagram shown, the first determining unit 603 includes:

[0159] The acquisition module 6031 is used to acquire at least one first point pair feature of multiple first point cloud data and at least one second point pair feature of multiple second point cloud data; the first point pair feature is used to characterize the relative distance and orientation between two first point cloud data; the second point pair feature is used to characterize the relative distance and orientation between two second point cloud data.

[0160] The first determining module 6032 is used to perform feature matching on the first point-to-feature pair and the second point-to-feature pair to determine the correspondence between the steel plate outline and the preset template.

[0161] In one possible implementation, the identification unit 602 includes:

[0162] The second determining module 6021 is used to determine the mask information of the steel plate in the depth image based on the depth image;

[0163] The processing module 6022 is used to perform contour extraction processing on the mask information to obtain the contour information of the steel plate and determine the multiple second point cloud data corresponding to the contour information.

[0164] In one possible implementation, the second determining module 6021 is specifically used to extract two-dimensional image information from the depth image;

[0165] Two-dimensional image information is input into the image recognition model to obtain the mask information of the steel plate. The image recognition model is used to identify the mask corresponding to the steel plate in the received image.

[0166] In one possible implementation, the second determining unit 604 includes:

[0167] The third determining module 6041 is used to determine the relative positional relationship between the first point cloud data and the second point cloud data that have a corresponding relationship;

[0168] The processing module 6042 is used to perform coordinate transformation processing on the pose information of the first point cloud data according to the relative position relationship to obtain the pose information of the second point cloud data.

[0169] The fourth determining module 6043 is used to determine the positioning information of the steel plate based on the pose information of the second point cloud data.

[0170] In one possible implementation, the graphic formed by multiple first point cloud data corresponding to the preset template is the same as the graphic of a preset local area of ​​the steel plate outline.

[0171] In one possible implementation, the steel plate includes at least one workpiece to be gripped; the workpiece to be gripped has first relative position information; the first relative position information is used to indicate the relative positional relationship between the workpiece to be gripped and the outline of the steel plate; the device further includes:

[0172] The third determining unit 605 is used to determine the position information of the workpiece to be grasped based on the positioning information of the steel plate and the first relative position information.

[0173] In one possible implementation, the device further includes:

[0174] The fourth determining unit 606 is used to determine the second relative position information of the second point cloud data corresponding to the outline of the steel plate of the workpiece to be grasped at the position information location;

[0175] The processing unit 607 is configured to, if it is determined that the second relative position information is different from the first relative position information, perform a difference calculation based on the second relative position information and the first relative position information to obtain a difference result;

[0176] The calibration unit 608 is used to calibrate the position information of the workpiece to be gripped based on the difference, so as to obtain the calibrated position information.

[0177] In one possible implementation, the graphic formed by multiple first point cloud data corresponding to the preset template is the lower right corner of the steel plate outline; the first relative position information is specifically used to indicate the vertical distance between the center point of the workpiece to be grasped and the horizontal side of the steel plate outline; the second relative position information is specifically used to indicate the vertical distance between the center point of the workpiece to be grasped at the position information and the second point cloud data of the horizontal side of the steel plate outline.

[0178] The apparatus provided in this embodiment is used to implement the technical solution provided by the above method. Its implementation principle and technical effect are similar, and will not be described again.

[0179] This disclosure provides an electronic device, including: a memory and a processor;

[0180] Memory; memory used to store processor-executable instructions;

[0181] The processor is used to execute methods according to executable instructions.

[0182] Figure 8 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure, such as... Figure 8 As shown, the electronic device includes:

[0183] The electronic device includes a processor 291 and a memory 292; it may also include a communication interface 293 and a bus 294. The processor 291, memory 292, and communication interface 293 can communicate with each other via the bus 294. The communication interface 293 can be used for information transmission. The processor 291 can invoke logical instructions stored in the memory 294 to execute the methods of the above embodiments.

[0184] Furthermore, the logic instructions in the aforementioned memory 292 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium.

[0185] The memory 292, as a computer-readable storage medium, can be used to store software programs and computer-executable programs, such as program instructions / modules corresponding to the methods in the embodiments of this disclosure. The processor 291 executes functional applications and data processing by running the software programs, instructions, and modules stored in the memory 292, thereby implementing the methods in the above-described method embodiments.

[0186] The memory 292 may include a program storage area and a data storage area. The program storage area may store the operating system and application programs required for at least one function; the data storage area may store data created based on the use of the terminal device. Furthermore, the memory 292 may include high-speed random access memory and may also include non-volatile memory.

[0187] This disclosure provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement any of the methods.

[0188] This disclosure provides a computer program product including a computer program that, when executed by a processor, implements any one of the methods.

[0189] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the appended claims.

[0190] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. A method for positioning a steel plate, characterized in that, include: Acquire a preset template and a depth image captured by a camera, wherein the preset template includes multiple first point cloud data; The first point cloud data has pose information; Identify multiple second point cloud data corresponding to the steel plate outline in the depth image; Feature matching is performed on the first point cloud data and the second point cloud data to determine the correspondence between the steel plate outline and the preset template, wherein the correspondence is used to indicate the first point cloud data and the second point cloud data with the same features; Based on the correspondence and the pose information of the first point cloud data, the positioning information of the steel plate is determined; The steel plate includes at least one workpiece to be gripped; the workpiece to be gripped has first relative position information; the first relative position information is used to indicate the relative positional relationship between the workpiece to be gripped and the outline of the steel plate; the method further includes: determining the position information of the workpiece to be gripped based on the positioning information of the steel plate and the first relative position information; Determine the second relative position information of the second point cloud data corresponding to the outline of the steel plate, which is located at the position information; If it is determined that the second relative position information is different from the first relative position information, then a difference calculation is performed based on the second relative position information and the first relative position information to obtain the difference result; and based on the difference, the position information of the workpiece to be grasped is calibrated to obtain the calibrated position information.

2. The method according to claim 1, characterized in that, Perform feature matching on the first point cloud data and the second point cloud data to determine the correspondence between the steel plate outline and the preset template, including: Acquire at least one first point pair feature from multiple first point cloud data, and at least one second point pair feature from multiple second point cloud data; the first point pair feature is used to characterize the relative distance and orientation between two first point cloud data; the second point pair feature is used to characterize the relative distance and orientation between two second point cloud data. The first point-to-feature pair and the second point-to-feature pair are matched to determine the correspondence between the steel plate outline and the preset template.

3. The method according to claim 1, characterized in that, Identify multiple second point cloud data corresponding to the steel plate contour in the depth image, including: Based on the depth image, determine the mask information of the steel plate in the depth image; The mask information is subjected to contour extraction processing to obtain the contour information of the steel plate, and multiple second point cloud data corresponding to the contour information are determined.

4. The method according to claim 3, characterized in that, Based on the depth image, determine the mask information of the steel plate in the depth image, including: Extract two-dimensional image information from the depth image; The two-dimensional image information is input into the image recognition model to obtain the mask information of the steel plate. The image recognition model is used to identify the mask corresponding to the steel plate in the received image.

5. The method according to claim 1, characterized in that, Based on the correspondence and the pose information of the first point cloud data, the positioning information of the steel plate is determined, including: Determine the relative positional relationship between the first point cloud data and the second point cloud data that have the aforementioned correspondence; Based on the relative positional relationship, the pose information of the first point cloud data is subjected to coordinate transformation processing to obtain the pose information of the second point cloud data; The positioning information of the steel plate is determined based on the pose information of the second point cloud data.

6. The method according to claim 1, characterized in that, The graphic formed by the multiple first point cloud data corresponding to the preset template is the same as the graphic of the preset local area of ​​the steel plate outline.

7. The method according to claim 6, characterized in that, The graphic formed by the multiple first point cloud data corresponding to the preset template is the lower right corner of the steel plate outline; the first relative position information is specifically used to indicate the vertical distance between the center point of the workpiece to be grasped and the horizontal side of the steel plate outline; the second relative position information is specifically used to indicate the vertical distance between the center point of the workpiece to be grasped at the position information and the second point cloud data of the horizontal side of the steel plate outline.

8. A positioning device for a steel plate, comprising: The acquisition unit is used to acquire a preset template and a depth image captured by a camera, wherein the preset template includes multiple first point cloud data. The first point cloud data has pose information; The recognition unit is used to recognize multiple second point cloud data corresponding to the outline of the steel plate in the depth image; The first determining unit is used to perform feature matching on the first point cloud data and the second point cloud data to determine the correspondence between the steel plate outline and the preset template, wherein the correspondence is used to indicate the first point cloud data and the second point cloud data having the same features. The second determining unit is used to determine the positioning information of the steel plate based on the correspondence and the pose information of the first point cloud data. The steel plate includes at least one workpiece to be gripped; the workpiece to be gripped has first relative position information; the first relative position information is used to indicate the relative positional relationship between the workpiece to be gripped and the outline of the steel plate; the device further includes: The third determining unit is used to determine the position information of the workpiece to be grasped based on the positioning information of the steel plate and the first relative position information. The fourth determining unit is used to determine the second relative position information of the second point cloud data corresponding to the outline of the steel plate of the workpiece to be grasped at the position information. The processing unit is configured to, if it is determined that the second relative position information is different from the first relative position information, perform a difference calculation based on the second relative position information and the first relative position information to obtain a difference result; A calibration unit is used to calibrate the position information of the workpiece to be grasped based on the difference, so as to obtain calibrated position information.

9. An electronic device, characterized in that, include: Memory, processor; Memory; Memory used to store the processor's executable instructions; The processor is configured to execute the method as described in any one of claims 1-7 according to the executable instructions.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of claims 1-7.

11. A computer program product comprising a computer program that, when executed by a processor, implements the method of any one of claims 1-7.

Images