Workpiece recognition method and device, electronic equipment and storage medium

Abstract

Embodiments of the present application provide a workpiece recognition method and device, electronic equipment and storage medium, and relate to the technical field of image processing. In the workpiece recognition method, a first recognition result of a to-be-recognized workpiece is obtained through an RGB image and a depth map of the to-be-recognized workpiece, a second recognition result of the to-be-recognized workpiece is obtained through three-dimensional point cloud data of the to-be-recognized workpiece, errors of the first recognition result and the second recognition result with respect to each type of workpiece are determined, a recognition result with the smallest error is taken as a third recognition result from the first recognition result and the second recognition result, and information of the to-be-recognized workpiece is output according to the third recognition result. Obviously, through mutual cooperation of recognition based on the RGB image and the depth map and recognition based on the three-dimensional point cloud data, the to-be-recognized workpiece is recognized, the to-be-recognized workpiece can be recognized based on mutual cooperation of the two recognition modes, the accuracy of workpiece recognition is improved, and diversified recognition requirements can also be met.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and more specifically, to a workpiece recognition method, apparatus, electronic device, and storage medium. Background Technology

[0002] In the heavy industry's production system, welding and grinding are extremely critical and frequent operations. Currently, the fully automated operation of teach-free welding and grinding technology encompasses multiple core steps, including material loading, vision system recognition, welding, grinding, and unloading. Among these, accurate identification of the workpiece type and spatial location is a crucial foundation for the efficient and stable progress of each step. Only by ensuring accurate identification in each process can the smooth operation of the entire teach-free welding and grinding system be guaranteed, thereby improving production efficiency and product quality.

[0003] Currently, most commonly used workpiece recognition solutions can only identify a single type of workpiece. However, in actual production scenarios, workpieces exhibit diverse characteristics due to their varying sizes. This situation makes it difficult for commonly used workpiece recognition solutions to meet diverse recognition needs, resulting in reduced workpiece recognition accuracy and hindering the efficiency and quality of automated welding and grinding operations. Summary of the Invention

[0004] This application provides a workpiece identification method, device, electronic device, and storage medium, which can meet diverse identification needs, improve the accuracy of workpiece identification, and thus promote the efficiency and quality of automated welding and grinding operations.

[0005] In a first aspect, embodiments of this application provide a workpiece recognition method, comprising: obtaining an RGB image, a depth map, and three-dimensional point cloud data of a workpiece to be recognized through an image sensor; recognizing the workpiece to be recognized in the RGB image using a recognition model to obtain the type of the workpiece and two-dimensional coordinates of its boundary points; converting the two-dimensional coordinates of the boundary points of the workpiece to be recognized into three-dimensional coordinates by combining the depth map, the focal length of the image sensor, and the image coordinate offset, and using the type of the workpiece to be recognized and the three-dimensional coordinates of its boundary points as a first recognition result; recognizing the workpiece to be recognized based on the three-dimensional point cloud data to obtain a second recognition result, wherein the second recognition result includes the type of the workpiece to be recognized and the three-dimensional coordinates of its boundary points; determining the errors of the first recognition result and the second recognition result with respect to each type of workpiece based on the three-dimensional coordinates of the boundary points in the first recognition result, the three-dimensional coordinates of the boundary points in the second recognition result, and a reference size for each type of workpiece; and outputting information about the workpiece to be recognized based on a third recognition result, wherein the third recognition result is the recognition result with the smallest error among the first recognition result and the second recognition result, and the information about the workpiece to be recognized includes at least the type of the workpiece to be recognized, the three-dimensional coordinates of its boundary points, its size, and its center point.

[0006] Optionally, the recognition model is obtained by training the YOLOv11m-OBB model.

[0007] Optionally, converting the two-dimensional coordinates of the boundary points of the workpiece to be identified into three-dimensional coordinates by combining the depth map, the focal length of the image sensor, and the image coordinate offset includes: performing an dilatational erosion operation on the depth map; determining whether the point corresponding to the boundary point in the depth map after dilatational erosion is in an abnormal region; if it is in the abnormal region, determining the pixel value of the point corresponding to the boundary point in the depth map after dilatational erosion using linear interpolation, and using the determined pixel value as the depth value of the boundary point; if it is not in the abnormal region, using the pixel value of the point corresponding to the boundary point in the depth map after dilatational erosion as the depth value of the boundary point; and determining the three-dimensional coordinates of the boundary point based on the two-dimensional coordinates of the boundary point, the depth value of the boundary point, and the focal length and image coordinate offset of the image sensor.

[0008] Optionally, determining the three-dimensional coordinates of the boundary point based on its two-dimensional coordinates and depth value, combined with the focal length and image coordinate offset of the image sensor, includes: determining the three-dimensional coordinates of the boundary point using the following formula:

[0009]

[0010]

[0011]

[0012] in, The two-dimensional coordinates of the boundary point are: The depth value of the boundary point. The focal length of the image sensor. The image coordinate offset, The coordinates of the boundary point are 3D coordinates.

[0013] Optionally, the step of identifying the workpiece to be identified based on the three-dimensional point cloud data to obtain a second identification result includes: voxelizing the three-dimensional point cloud data; clustering the voxelized three-dimensional point cloud data based on Euclidean distance to obtain multiple point cloud datasets; determining a target dataset among the multiple point cloud datasets; wherein the target dataset is the point cloud dataset corresponding to the workpiece to be identified; performing plane fitting on the target dataset based on the RANSAC algorithm to obtain multiple planes of the workpiece to be identified; determining the contour information of the top plane of the workpiece to be identified in the multiple planes according to the normal vector direction of each plane; and determining the second identification result according to the contour information of the top plane and the reference size of each type of workpiece.

[0014] Optionally, determining the errors between the first identification result and the second identification result and each type of workpiece based on the three-dimensional coordinates of the boundary points in the first identification result, the three-dimensional coordinates of the boundary points in the second identification result, and the reference size of each type of workpiece includes: determining a first size of the workpiece to be identified based on the three-dimensional coordinates of the boundary points in the first identification result; determining a second size of the workpiece to be identified based on the three-dimensional coordinates of the boundary points in the second identification result; determining the errors between the first identification result and each type of workpiece based on the first size and the reference size of each type of workpiece; and determining the errors between the second identification result and each type of workpiece based on the second size and the reference size of each type of workpiece.

[0015] Optionally, determining the error between the first identification result and each type of workpiece based on the first size and the reference size of each type of workpiece includes: determining the error between the first identification result and each type of workpiece using the following formula:

[0016]

[0017] in, The error between the first identification result and the j-th type of workpiece is... Let i be the first dimension of the workpiece to be identified. For the i-th reference dimension of the j-th type of workpiece, .

[0018] Secondly, embodiments of this application provide a workpiece recognition device, comprising: an acquisition module, configured to acquire an RGB image, a depth map, and three-dimensional point cloud data of a workpiece to be recognized via an image sensor; a first recognition module, configured to recognize the workpiece to be recognized in the RGB image using a recognition model to obtain the type of the workpiece and two-dimensional coordinates of its boundary points; a conversion module, configured to combine the depth map, the focal length of the image sensor, and the image coordinate offset to convert the two-dimensional coordinates of the boundary points of the workpiece to be recognized into three-dimensional coordinates, and to use the type of the workpiece to be recognized and the three-dimensional coordinates of its boundary points as a first recognition result; and a second recognition module, configured to recognize the workpiece to be recognized based on the three-dimensional point cloud data, to obtain the type of the workpiece to be recognized and the two-dimensional coordinates of its boundary points; and a second recognition module, configured to recognize the workpiece to be recognized based on the three-dimensional point cloud data. A second identification result is obtained, wherein the second identification result includes the type of the workpiece to be identified and the three-dimensional coordinates of the boundary points; a determination module is used to determine the errors between the first identification result and the second identification result and the workpiece of each type, respectively, based on the three-dimensional coordinates of the boundary points in the first identification result, the three-dimensional coordinates of the boundary points in the second identification result, and the reference size of each type of workpiece; an output module is used to output the information of the workpiece to be identified based on the third identification result, wherein the third identification result is the identification result with the smallest error between the first identification result and the second identification result, and the information of the workpiece to be identified includes at least the type of the workpiece to be identified, the three-dimensional coordinates of the boundary points, the size, and the center point.

[0019] Thirdly, embodiments of this application provide an electronic device, including: a memory and a processor; the memory is used to store a computer program; the processor is used to implement the workpiece identification method described in any one of the first aspects when the computer program is executed.

[0020] Fourthly, embodiments of this application provide a computer-readable storage medium, characterized in that the storage medium stores a computer program, which, when executed by a processor, implements the workpiece identification method described in any one of the first aspects.

[0021] The beneficial effects of the workpiece identification method in this application embodiment are:

[0022] 3D point cloud data can describe the geometry of a workpiece to be identified using spatial coordinates, providing complete 3D shape, size, and spatial location information. It offers more accurate global shape and spatial location perception for large workpieces. However, when the workpiece is small, the density of the acquired 3D point cloud data may be insufficient, making it difficult to reconstruct the true shape from the 3D point data.

[0023] In comparison, the high resolution of RGB images can capture detailed features such as surface texture and color markings of small workpieces to be identified. Combined with the two-dimensional spatial location of the depth map, it has a greater advantage in classifying and identifying smaller workpieces.

[0024] Based on this, in the workpiece recognition method provided in this application embodiment, a first recognition result of the workpiece to be recognized can be obtained through the RGB image and depth map of the workpiece to be recognized, and a second recognition result of the workpiece to be recognized can be obtained through the three-dimensional point cloud data of the workpiece to be recognized. The errors of the first and second recognition results with each type of workpiece are determined, and the recognition result with the smallest error between the first and second recognition results is taken as the third recognition result. Information about the workpiece to be recognized is then output based on the third recognition result. In other words, by the cooperation between recognition based on RGB images and depth maps and recognition based on three-dimensional point cloud data, the workpiece to be recognized can be identified. This collaboration of two recognition methods can address the recognition of workpieces of different sizes, improving the accuracy of workpiece recognition and meeting diverse recognition needs, thereby promoting the efficiency and quality of automated welding and grinding operations.

[0025] Furthermore, the combination of RGB images and depth maps focuses on representing the features of the workpiece to be identified from the perspective of the relationship between visual texture and two-dimensional spatial depth. This approach is highly sensitive to capturing visual information such as the color, texture, and shape contour of the workpiece's surface, making it suitable for identifying workpieces with obvious appearance differences. However, when the workpiece is occluded, there are changes in lighting, or the surface textures are similar, this type of vision-based recognition method is prone to misjudgment.

[0026] In contrast, 3D point cloud data can describe the geometry of the workpiece to be identified using spatial coordinates, providing complete 3D shape, size, and spatial location information. Even in occluded or unevenly lit environments, stable identification can be achieved based on the object's geometric features. However, 3D point cloud data has lower sensitivity to surface detail features, resulting in poor identification performance for workpieces that rely solely on surface texture for differentiation.

[0027] Based on this, an error compensation mechanism was constructed by calculating the similarity between the two recognition results and each type of workpiece separately, and selecting the result with the lowest similarity as the final output. Since the two information acquisition and processing dimensions are different, errors generated by a single recognition method are often difficult to simultaneously appear in the other method. Therefore, employing two methods in collaboration can reduce the limitations of a single information source, thereby improving the accuracy of recognition. Attached Figure Description

[0028] Figure 1 A schematic flowchart of a workpiece identification method provided in an embodiment of this application;

[0029] Figure 2 A schematic diagram of the coordinate transformation process provided in the embodiments of this application;

[0030] Figure 3 A schematic diagram of the process for determining the second identification result provided in an embodiment of this application;

[0031] Figure 4 A flowchart illustrating the process of determining errors provided in an embodiment of this application;

[0032] Figure 5 This is a schematic diagram of the frame of a workpiece identification device provided in an embodiment of this application;

[0033] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0034] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Although some embodiments of the present invention are shown in the drawings, it should be understood that the present invention can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of the present invention. It should be understood that the accompanying drawings and embodiments of the present invention are for illustrative purposes only and are not intended to limit the scope of protection of the present invention.

[0035] It should be understood that the various steps described in the method embodiments of the present invention may be performed in different orders and / or in parallel. Furthermore, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present invention is not limited in this respect.

[0036] The term "comprising" and its variations as used herein are open-ended, meaning "including but not limited to"; the term "based on" means "at least partially based on"; the term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments"; and the term "optionally" means "optional embodiments". Definitions of other terms will be given in the following description. It should be noted that the concepts of "first," "second," etc., mentioned in this invention are used only to distinguish different devices, modules, or units, and are not intended to limit the order of functions performed by these devices, modules, or units or their interdependencies.

[0037] It should be noted that the terms "a" and "a plurality of" used in this invention are illustrative rather than restrictive. Those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0038] The names of the messages or information exchanged between the multiple devices in the embodiments of the present invention are for illustrative purposes only and are not intended to limit the scope of these messages or information.

[0039] Figure 1 This is a flowchart illustrating a workpiece identification method provided in an embodiment of this application. This workpiece identification method can be executed by an electronic device, including but not limited to laptops, desktop computers, e-readers, mobile phones, and other devices with information processing capabilities. Figure 1 As shown, the workpiece identification method may include the following steps:

[0040] 110. Obtain RGB images, depth maps, and 3D point cloud data of the workpiece to be identified using an image sensor.

[0041] In this embodiment of the application, an image sensor can be set on the conveyor belt of the workpiece to be identified. When the workpiece to be identified moves under the image sensor driven by the conveyor belt, the image sensor can obtain the RGB image, depth map and three-dimensional point cloud data of the workpiece to be identified.

[0042] There are various types of workpieces, and the dimensions of different types of workpieces vary. For example, the workpiece type may include the body and lid of a small three-box container, wherein the body and lid have different dimensions.

[0043] 120. The workpiece to be identified in the RGB image is identified by the recognition model to obtain the type of the workpiece and the two-dimensional coordinates of the boundary points.

[0044] The boundary points of the workpiece to be identified may include points used to indicate the boundary extent of the top plane of the workpiece. For example, when the workpiece to be identified is rectangular, the boundary points of the workpiece to be identified include the four vertices of the top plane of the workpiece to be identified.

[0045] It should be noted that the processing plane of the workpiece to be identified can be defined as the top plane of the workpiece to be identified.

[0046] The recognition model is used to identify the type of workpiece to be identified and the two-dimensional coordinates of boundary points.

[0047] For example, the recognition model is obtained by training the YOLOv11m-OBB model.

[0048] YOLOv11m employs an improved backbone network and neck architecture, enhancing feature extraction capabilities for more accurate detection. YOLOv11m introduces a refined architectural design and optimized training process, providing faster processing speeds and maintaining an optimal balance between accuracy and performance. YOLOv11m can be seamlessly deployed in a variety of environments, ensuring maximum flexibility.

[0049] OBB (Oriented Bounding Box) models are a class of algorithms used to detect rotated or arbitrarily oriented targets. Unlike other detection methods, OBB can output bounding boxes with orientation and angle, making it more suitable for handling directional targets.

[0050] Thus, the recognition model trained based on the YOLOv11m-OBB model (i.e., the combination of the two models mentioned above) can improve the accuracy and efficiency of the recognition of the workpiece in the RGB image, and can also more accurately mark objects with tilted orientation.

[0051] The training process of the YOLOv11m-OBB model will be explained below.

[0052] First, multiple training samples are obtained, which include multiple RGB images obtained by taking pictures of various types of workpieces.

[0053] Then, the type of workpiece and the location of the boundary points in each of the multiple training samples are labeled.

[0054] Next, the YOLOv11m-OBB model was trained using multiple labeled training samples to adjust the parameters of the YOLOv11m-OBB model.

[0055] Finally, the trained YOLOv11m-OBB model was selected as the recognition model.

[0056] 130. Combining the depth map, the focal length of the image sensor, and the image coordinate offset, the two-dimensional coordinates of the boundary points of the workpiece to be identified are converted into three-dimensional coordinates, and the type of the workpiece to be identified and the three-dimensional coordinates of the boundary points are used as the first identification result.

[0057] For example, such as Figure 2 As shown, the specific implementation process of 130 can be described as follows:

[0058] 210. Perform dilatational erosion on the depth map.

[0059] Outliers in a depth map can be removed by dilatational erosion.

[0060] Outliers typically refer to points with all zeros. Based on this, the expansion corrosion process can be described as follows:

[0061] Perform the dilation operation: Iterate through each pixel in the depth map, checking if each pixel's value is 0. If not, continue iterating; if so, fill the pixel with the pixel value based on the non-zero pixel values ​​of its surrounding pixels. For example, take a 3x3 neighborhood centered on the pixel and use the average of the non-zero pixel values ​​in that neighborhood to fill the pixel. It should be noted that if all pixel values ​​in the neighborhood are zero, the global pixel value of the depth map can be used for filling.

[0062] Perform the erosion operation: Iterate through each pixel of the dilated depth map and replace the pixel value of each pixel with the minimum value of all pixels in its neighborhood.

[0063] 220. Determine whether the point corresponding to the boundary point in the depth map after expansion corrosion is in an abnormal region.

[0064] It should be noted that abnormal areas refer to areas with a pixel value of 0.

[0065] 230. In the case of an abnormal region, determine the pixel value of the point corresponding to the boundary point in the depth map after dilation and erosion by linear interpolation, and use the determined pixel value as the depth value of the boundary point.

[0066] It should be noted that in a depth map, pixel values ​​refer to depth values.

[0067] 240. In cases where the boundary point is not in an abnormal region, use the pixel value of the point corresponding to the boundary point in the depth map after dilation and erosion as the depth value of the boundary point.

[0068] 250. Based on the two-dimensional coordinates and depth values ​​of the boundary points, and combined with the focal length of the image sensor and the image coordinate offset, determine the three-dimensional coordinates of the boundary points.

[0069] For example, the three-dimensional coordinates of the boundary points can be determined using the following formula:

[0070]

[0071]

[0072]

[0073] in, The two-dimensional coordinates of the boundary points The depth value of the boundary point. The focal length of the image sensor. Image coordinate offset, These are the three-dimensional coordinates of the boundary points.

[0074] 140. The workpiece to be identified is identified based on the three-dimensional point cloud data to obtain a second identification result, wherein the second identification result includes the type of the workpiece to be identified and the three-dimensional coordinates of the boundary points.

[0075] For example, such as Figure 3 As shown, the specific implementation process of 140 can be described as follows:

[0076] 310. Voxelize the 3D point cloud data.

[0077] Voxelization is the process of discretizing a continuous three-dimensional space (the three-dimensional space indicated by three-dimensional point cloud data) into small cubes (voxels) for downsampling and uniform spatial representation.

[0078] 320. Cluster the voxelized 3D point cloud data based on Euclidean distance to obtain multiple point cloud datasets.

[0079] Clustering is the process of dividing a voxelized point cloud into different objects. Euclidean distance clustering is suitable for spatially separated objects.

[0080] The specific implementation process of 320 can be described as follows:

[0081] First, define the maximum distance between two points belonging to the same cluster. Then, calculate the Euclidean distance between any two points in the point cloud data. Finally, determine two points whose distance is less than the maximum distance as a cluster. In this way, multiple point cloud datasets are obtained.

[0082] 330. Among multiple point cloud datasets, determine the target dataset. The target dataset is the point cloud dataset corresponding to the workpiece to be identified.

[0083] Because 3D point cloud data includes not only the identifiable workpiece but also the background and other objects, clustering is necessary to determine the target dataset from multiple point cloud datasets.

[0084] Specifically, the methods for determining the target dataset from multiple point cloud datasets include, but are not limited to, the following two.

[0085] The first approach is that, since the workpiece to be identified is usually located within a preset working area, the point cloud dataset located within the working area can be determined as the target dataset.

[0086] The second approach involves adjusting the image sensor parameters to include only the workpiece to be identified within the field of view when acquiring 3D point cloud data, in order to obtain more accurate data. Based on this, the point cloud dataset with the largest area can be determined as the target dataset.

[0087] 340. Based on the RANSAC algorithm, perform plane fitting on the target dataset to obtain multiple planes of the workpiece to be identified.

[0088] The specific implementation process can be as follows: Randomly select 3 points from the target dataset to determine a plane. Calculate the distance from other points in the target dataset to this plane; points with a distance less than a threshold are considered interior points. Count the number of interior points, and the model with the most interior points is the best model. Repeat the above process multiple times to select the optimal plane model (i.e., complete the fitting of a plane). After fitting a plane, repeat the above process to fit other planes from the remaining point cloud until the remaining points are insufficient to fit a plane or the preset number of planes is reached.

[0089] 350. Determine the contour information of the top plane of the workpiece to be identified in multiple planes based on the normal vector direction of each plane.

[0090] The specific implementation process can be as follows: calculate the normal vector of each plane, select the plane whose normal vector direction is close to the vertical upward (such as the positive direction of the Z axis), select the plane with the highest Z coordinate as the top plane, perform boundary detection on the point cloud of the top plane, and obtain contour information.

[0091] 360. Determine the second identification result based on the contour information of the top plane and the reference dimensions of each type of workpiece.

[0092] The size of the workpiece to be identified can be determined based on the contour information of the top plane. The size of the workpiece to be identified is compared with the reference size of each type of workpiece, and the type of workpiece with the smallest error is determined as the type of workpiece to be identified.

[0093] The three-dimensional coordinates of the boundary points of the workpiece to be identified are determined based on the contour information of the top plane. For example, when the workpiece to be identified is rectangular, the three-dimensional coordinates of the four vertices of the top plane of the workpiece to be identified are determined from the contour information, so that the three-dimensional coordinates of the four vertices are determined as the three-dimensional coordinates of the boundary points of the workpiece to be identified.

[0094] Thus, the type of the workpiece to be identified and the three-dimensional coordinates of the boundary points are used as the second identification result.

[0095] 150. Based on the three-dimensional coordinates of the boundary points in the first identification result, the three-dimensional coordinates of the boundary points in the second identification result, and the reference dimensions of each type of workpiece, determine the errors between the first identification result and the second identification result and each type of workpiece.

[0096] For example, such as Figure 4 As shown, the implementation process of 150 can be described as follows:

[0097] 410. Determine the first dimension of the workpiece to be identified based on the three-dimensional coordinates of the boundary points in the first identification result.

[0098] The number of first dimensions of the workpiece to be identified can be determined based on the shape of the top surface of the workpiece. For example, when the top surface of the workpiece to be identified is rectangular, the first dimensions include the length and width of the top surface. When the top surface of the workpiece to be identified is circular, the first dimensions include the radius or diameter of the top surface.

[0099] 420. Determine the second dimension of the workpiece to be identified based on the three-dimensional coordinates of the boundary points in the second identification result.

[0100] The method for determining the number of second dimensions of the workpiece to be identified can be found in the method for determining the first dimension of the workpiece to be identified, and will not be repeated here.

[0101] It should be noted that, for the purpose of differentiation, the size of the workpiece to be identified obtained based on the first identification result is defined as the first size, and the size of the workpiece to be identified determined based on the second identification result is defined as the second size.

[0102] 430. Based on the first dimension and the reference dimension of each type of workpiece, determine the error between the first identification result and each type of workpiece.

[0103] For example, the error between the first identification result and each type of workpiece can be determined by the following formula.

[0104]

[0105] in, The error between the first identification result and the j-th type of workpiece is... Let i be the first dimension of the workpiece to be identified. For the i-th reference dimension of the j-th type of workpiece, .

[0106] 440. Based on the second dimension and the reference dimension of each type of workpiece, determine the error of the second identification result relative to each type of workpiece.

[0107] For details on the implementation of 440, please refer to the implementation of 430; it will not be repeated here.

[0108] 160. Output the information of the workpiece to be identified based on the third identification result, wherein the third identification result is the identification result with the smallest error among the first identification result and the second identification result, and the information of the workpiece to be identified includes at least the type of the workpiece to be identified, the three-dimensional coordinates of the boundary points, the size, and the center point.

[0109] For example, among the errors between the first identification result and each type of workpiece, a minimum error value (referred to as the first error) is determined. And among the errors between the second identification result and each type of workpiece, a minimum error value (referred to as the second error) is determined. The first error and the second error are compared. If the first error is less than the second error, the first identification result is used as the third identification result. If the first error is greater than the second error, the second identification result is used as the third identification result.

[0110] Since the third identification result includes the type of the workpiece to be identified and the three-dimensional coordinates of the boundary points, the size and center point of the workpiece to be identified can be calculated based on the third identification result. Thus, the type of the workpiece to be identified, the three-dimensional coordinates of the boundary points, the size, and the center point can be output based on the third identification result.

[0111] In summary, 3D point cloud data can describe the geometric structure of a workpiece through spatial coordinates, providing complete 3D shape, size, and spatial location information, and offering more accurate perception of the global shape and spatial location of large workpieces. However, when the size of the workpiece is small, the density of the acquired 3D point cloud data may be insufficient, making it difficult to reconstruct the true shape from the 3D point data.

[0112] In comparison, the high resolution of RGB images can capture detailed features such as surface texture and color markings of small workpieces to be identified. Combined with the two-dimensional spatial location of the depth map, it has a greater advantage in classifying and identifying smaller workpieces.

[0113] Based on this, in the workpiece recognition method provided in this application embodiment, a first recognition result of the workpiece to be recognized can be obtained through the RGB image and depth map of the workpiece to be recognized, and a second recognition result of the workpiece to be recognized can be obtained through the three-dimensional point cloud data of the workpiece to be recognized. The errors of the first and second recognition results with each type of workpiece are determined, and the recognition result with the smallest error between the first and second recognition results is taken as the third recognition result. Information about the workpiece to be recognized is then output based on the third recognition result. In other words, by the cooperation between recognition based on RGB images and depth maps and recognition based on three-dimensional point cloud data, the workpiece to be recognized can be identified. This collaboration of two recognition methods can address the recognition of workpieces of different sizes, improving the accuracy of workpiece recognition and meeting diverse recognition needs, thereby promoting the efficiency and quality of automated welding and grinding operations.

[0114] Furthermore, the combination of RGB images and depth maps focuses on representing the features of the workpiece to be identified from the perspective of the relationship between visual texture and two-dimensional spatial depth. This approach is highly sensitive to capturing visual information such as the color, texture, and shape contour of the workpiece's surface, making it suitable for identifying workpieces with obvious appearance differences. However, when the workpiece is occluded, there are changes in lighting, or the surface textures are similar, this type of vision-based recognition method is prone to misjudgment.

[0115] In contrast, 3D point cloud data can describe the geometry of the workpiece to be identified using spatial coordinates, providing complete 3D shape, size, and spatial location information. Even in occluded or unevenly lit environments, stable identification can be achieved based on the object's geometric features. However, 3D point cloud data has lower sensitivity to surface detail features, resulting in poor identification performance for workpieces that rely solely on surface texture for differentiation.

[0116] Based on this, an error compensation mechanism was constructed by calculating the similarity between the two recognition results and each type of workpiece separately, and selecting the result with the lowest similarity as the final output. Since the two information acquisition and processing dimensions are different, errors generated by a single recognition method are often difficult to simultaneously appear in the other method. Therefore, employing two methods in collaboration can reduce the limitations of a single information source, thereby improving the accuracy of recognition.

[0117] Figure 5 This is a schematic diagram of the frame of a workpiece identification device provided in an embodiment of this application, as shown below. Figure 5 As shown, the workpiece identification device 500 may include:

[0118] The acquisition module 510 is used to acquire RGB images, depth maps, and 3D point cloud data of the workpiece to be identified through an image sensor;

[0119] The first recognition module 520 is used to recognize the workpiece to be recognized in the RGB image through a recognition model, so as to obtain the type of the workpiece to be recognized and the two-dimensional coordinates of the boundary points.

[0120] The conversion module 530 is used to combine the depth map, the focal length of the image sensor and the image coordinate offset to convert the two-dimensional coordinates of the boundary points of the workpiece to be identified into three-dimensional coordinates, and to use the type of the workpiece to be identified and the three-dimensional coordinates of the boundary points as the first identification result.

[0121] The second identification module 540 is used to identify the workpiece to be identified based on the three-dimensional point cloud data to obtain a second identification result, wherein the second identification result includes the type of the workpiece to be identified and the three-dimensional coordinates of the boundary points;

[0122] The determination module 550 is used to determine the errors between the first identification result and the second identification result and each type of workpiece, respectively, based on the three-dimensional coordinates of the boundary points in the first identification result, the three-dimensional coordinates of the boundary points in the second identification result, and the reference dimensions of each type of workpiece.

[0123] The output module 560 is used to output information about the workpiece to be identified based on the third identification result, wherein the third identification result is the identification result with the smallest error between the first identification result and the second identification result, and the information about the workpiece to be identified includes at least the type of the workpiece to be identified, the three-dimensional coordinates of the boundary points, the size, and the center point.

[0124] Optionally, the recognition model is obtained by training the YOLOv11m-OBB model.

[0125] Optionally, the conversion module 530 is specifically used to perform dilation and erosion operations on the depth map; determine whether the point corresponding to the boundary point in the depth map after dilation and erosion is in an abnormal region; if it is in the abnormal region, determine the pixel value of the point corresponding to the boundary point in the depth map after dilation and erosion using linear interpolation, and use the determined pixel value as the depth value of the boundary point; if it is not in the abnormal region, use the pixel value of the point corresponding to the boundary point in the depth map after dilation and erosion as the depth value of the boundary point; and determine the three-dimensional coordinates of the boundary point based on the two-dimensional coordinates of the boundary point and the depth value of the boundary point, combined with the focal length of the image sensor and the image coordinate offset.

[0126] Optionally, the conversion module 530 is specifically used to determine the three-dimensional coordinates of the boundary points using the following formula:

[0127]

[0128]

[0129]

[0130] in, The two-dimensional coordinates of the boundary point are: The depth value of the boundary point. The focal length of the image sensor. The image coordinate offset, The coordinates of the boundary point are 3D coordinates.

[0131] Optionally, the second recognition module 540 is specifically used to voxelize the three-dimensional point cloud data; cluster the voxelized three-dimensional point cloud data based on Euclidean distance to obtain multiple point cloud datasets; determine a target dataset among the multiple point cloud datasets; wherein the target dataset is the point cloud dataset corresponding to the workpiece to be recognized; perform plane fitting on the target dataset based on the RANSAC algorithm to obtain multiple planes of the workpiece to be recognized; determine the contour information of the top plane of the workpiece to be recognized in the multiple planes according to the normal vector direction of each plane; and determine the second recognition result according to the contour information of the top plane and the reference size of each type of workpiece.

[0132] Optionally, the determining module 550 is specifically configured to: determine a first size of the workpiece to be identified based on the three-dimensional coordinates of the boundary points in the first identification result; determine a second size of the workpiece to be identified based on the three-dimensional coordinates of the boundary points in the second identification result; determine the error between the first identification result and each type of workpiece based on the first size and the reference size of each type of workpiece; and determine the error between the second identification result and each type of workpiece based on the second size and the reference size of each type of workpiece.

[0133] Optionally, the determining module 550 is specifically used to determine the error between the first identification result and each type of workpiece using the following formula:

[0134]

[0135] in, The error between the first identification result and the j-th type of workpiece is... Let i be the first dimension of the workpiece to be identified. For the i-th reference dimension of the j-th type of workpiece, .

[0136] like Figure 6As shown, an electronic device 600 provided in this embodiment of the invention includes a memory 610 and a processor 620; the memory 610 is used to store a computer program; the processor 620 is used to implement the workpiece identification as described above when the computer program is executed.

[0137] This invention provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it realizes the workpiece identification as described above.

[0138] Electronic device 600, which can serve as a server or client of the present invention, is described below as an example of a hardware device applicable to various aspects of the present invention. Electronic device 600 is intended to represent various forms of digital electronic computer devices, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. Electronic device 600 can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0139] Electronic device 600 includes a computing unit that can perform various appropriate actions and processes based on a computer program stored in read-only memory (ROM) or a computer program loaded from a storage unit into random access memory (RAM). The RAM may also store various programs and data required for device operation. The computing unit, ROM, and RAM are interconnected via a bus. Input / output (I / O) interfaces are also connected to the bus.

[0140] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc. In this application, the units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of the embodiments of the present invention according to actual needs. Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated units can be implemented in hardware or as software functional units.

[0141] While the present invention has been disclosed above, its scope of protection is not limited thereto. Those skilled in the art can make various changes and modifications without departing from the spirit and scope of the present invention, and all such changes and modifications will fall within the scope of protection of the present invention.

Claims

1. A workpiece identification method, characterized in that, include: The image sensor is used to obtain RGB images, depth maps, and 3D point cloud data of the workpiece to be identified. The identification model is used to identify the workpiece to be identified in the RGB image to obtain the type of the workpiece and the two-dimensional coordinates of the boundary points. By combining the depth map, the focal length of the image sensor, and the image coordinate offset, the two-dimensional coordinates of the boundary points of the workpiece to be identified are converted into three-dimensional coordinates, and the type of the workpiece to be identified and the three-dimensional coordinates of the boundary points are used as the first identification result. The workpiece to be identified is identified based on the three-dimensional point cloud data to obtain a second identification result, wherein the second identification result includes the type of the workpiece to be identified and the three-dimensional coordinates of the boundary points; Based on the three-dimensional coordinates of the boundary points in the first identification result, the three-dimensional coordinates of the boundary points in the second identification result, and the reference dimensions of each type of workpiece, the errors between the first identification result and the second identification result and each type of workpiece are determined. The information of the workpiece to be identified is output based on the third identification result, wherein the third identification result is the identification result with the smallest error between the first identification result and the second identification result, and the information of the workpiece to be identified includes at least the type of the workpiece to be identified, the three-dimensional coordinates of the boundary points, the size, and the center point; The recognition model is obtained by training the YOLOv11m-OBB model; The step of converting the two-dimensional coordinates of the boundary points of the workpiece to be identified into three-dimensional coordinates by combining the depth map, the focal length of the image sensor, and the image coordinate offset includes: Perform an expansion erosion operation on the depth map; Determine whether the point corresponding to the boundary point in the depth map after expansion corrosion is in an abnormal region; In the case of the abnormal region, the pixel value of the point corresponding to the boundary point in the depth map after dilation and erosion is determined by linear interpolation, and the determined pixel value is used as the depth value of the boundary point. If the boundary point is not in the abnormal region, the pixel value of the point corresponding to the boundary point in the depth map after dilation and erosion is used as the depth value of the boundary point. The three-dimensional coordinates of the boundary point are determined based on the two-dimensional coordinates and depth value of the boundary point, combined with the focal length of the image sensor and the image coordinate offset. The step of identifying the workpiece to be identified based on the three-dimensional point cloud data to obtain a second identification result includes: The three-dimensional point cloud data is voxelized; Clustering of the voxelized 3D point cloud data based on Euclidean distance yields multiple point cloud datasets. Among the plurality of point cloud datasets, a target dataset is determined; wherein, the target dataset is the point cloud dataset corresponding to the workpiece to be identified; The target dataset is fitted with planes based on the RANSAC algorithm to obtain multiple planes of the workpiece to be identified; Based on the normal vector direction of each of the plurality of planes, the contour information of the top plane of the workpiece to be identified is determined in the plurality of planes; The second identification result is determined based on the contour information of the top plane and the reference dimensions of each type of workpiece.

2. The workpiece identification method according to claim 1, characterized in that, The step of determining the three-dimensional coordinates of the boundary point based on its two-dimensional coordinates and depth value, combined with the focal length of the image sensor and the image coordinate offset, includes: The three-dimensional coordinates of the boundary point are determined using the following formula: in, The two-dimensional coordinates of the boundary point are: The depth value of the boundary point. The focal length of the image sensor. The image coordinate offset, The coordinates of the boundary point are 3D coordinates.

3. The workpiece identification method according to claim 1, characterized in that, The step of determining the errors between the first identification result and the second identification result and each type of workpiece, based on the three-dimensional coordinates of the boundary points in the first identification result, the three-dimensional coordinates of the boundary points in the second identification result, and the reference dimensions of each type of workpiece, includes: Based on the three-dimensional coordinates of the boundary points in the first identification result, determine the first dimension of the workpiece to be identified; The second dimension of the workpiece to be identified is determined based on the three-dimensional coordinates of the boundary points in the second identification result; Based on the first size and the reference size of each type of workpiece, determine the error between the first identification result and each type of workpiece; Based on the second dimension and the reference dimension of each type of workpiece, the error between the second identification result and each type of workpiece is determined.

4. The workpiece identification method according to claim 3, characterized in that, The step of determining the error between the first identification result and each type of workpiece, based on the first size and the reference size of each type of workpiece, includes: The error between the first identification result and each type of workpiece is determined by the following formula: in, The error between the first identification result and the j-th type of workpiece is... Let i be the first dimension of the workpiece to be identified. For the i-th reference dimension of the j-th type of workpiece, .

5. A workpiece identification device, characterized in that, include: The acquisition module is used to acquire RGB images, depth maps, and 3D point cloud data of the workpiece to be identified through an image sensor; The first recognition module is used to recognize the workpiece to be recognized in the RGB image through a recognition model, so as to obtain the type of the workpiece to be recognized and the two-dimensional coordinates of the boundary points; The conversion module is used to combine the depth map, the focal length of the image sensor and the image coordinate offset to convert the two-dimensional coordinates of the boundary points of the workpiece to be identified into three-dimensional coordinates, and to use the type of the workpiece to be identified and the three-dimensional coordinates of the boundary points as the first identification result. The second identification module is used to identify the workpiece to be identified based on the three-dimensional point cloud data to obtain a second identification result, wherein the second identification result includes the type of the workpiece to be identified and the three-dimensional coordinates of the boundary points; The determination module is used to determine the errors between the first identification result and the second identification result and each type of workpiece, respectively, based on the three-dimensional coordinates of the boundary points in the first identification result, the three-dimensional coordinates of the boundary points in the second identification result, and the reference dimensions of each type of workpiece. The output module is used to output the information of the workpiece to be identified based on the third identification result, wherein the third identification result is the identification result with the smallest error between the first identification result and the second identification result, and the information of the workpiece to be identified includes at least the type of the workpiece to be identified, the three-dimensional coordinates of the boundary points, the size, and the center point; The recognition model is obtained by training the YOLOv11m-OBB model; The step of converting the two-dimensional coordinates of the boundary points of the workpiece to be identified into three-dimensional coordinates by combining the depth map, the focal length of the image sensor, and the image coordinate offset includes: Perform an expansion erosion operation on the depth map; Determine whether the point corresponding to the boundary point in the depth map after expansion corrosion is in an abnormal region; In the case of the abnormal region, the pixel value of the point corresponding to the boundary point in the depth map after dilation and erosion is determined by linear interpolation, and the determined pixel value is used as the depth value of the boundary point. If the boundary point is not in the abnormal region, the pixel value of the point corresponding to the boundary point in the depth map after dilation and erosion is used as the depth value of the boundary point. The three-dimensional coordinates of the boundary point are determined based on the two-dimensional coordinates and depth value of the boundary point, combined with the focal length of the image sensor and the image coordinate offset. The step of identifying the workpiece to be identified based on the three-dimensional point cloud data to obtain a second identification result includes: The three-dimensional point cloud data is voxelized; Clustering of the voxelized 3D point cloud data based on Euclidean distance yields multiple point cloud datasets. Among the plurality of point cloud datasets, a target dataset is determined; wherein, the target dataset is the point cloud dataset corresponding to the workpiece to be identified; The target dataset is fitted with planes based on the RANSAC algorithm to obtain multiple planes of the workpiece to be identified; Based on the normal vector direction of each of the plurality of planes, the contour information of the top plane of the workpiece to be identified is determined in the plurality of planes; The second identification result is determined based on the contour information of the top plane and the reference dimensions of each type of workpiece.

6. An electronic device, characterized in that, include: Memory and processor; The memory is used to store computer programs; The processor is configured to implement the workpiece identification method as described in any one of claims 1 to 4 when executing the computer program.

7. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, which, when executed by a processor, implements the workpiece identification method as described in any one of claims 1 to 4.

Images