Point cloud welding seam identification method and device based on dimension transformation

Abstract

The invention discloses a point cloud welding seam recognition method and device based on dimension transformation. The method comprises the steps that three-dimensional point cloud data of a target welding area are obtained; projecting the three-dimensional point cloud data to a two-dimensional plane to obtain a depth map of the target welding area; inputting the depth map into a preset part segmentation model to obtain at least one welding workpiece mask; for each welding workpiece mask, inputting the welding workpiece mask and the depth map into a preset plane segmentation model to obtain a plurality of plane masks; based on the depth map, back-projecting all the plane masks into the three-dimensional point cloud data to obtain coordinate information of a plurality of planes; according to the coordinate information of the multiple planes, all adjacent planes are determined, and according to the relative position relation between every two adjacent planes, the types and coordinate information of all welding seams are determined. According to the method, the complexity of point cloud data processing can be greatly reduced, and the welding seam area can be accurately identified while the redundancy and noise influence in the point cloud data processing is reduced.

Description

Technical Field

[0001] The present invention relates to a method and device for identifying a weld seam in point cloud based on dimensional transformation. Background Art

[0002] With the continuous development of industrial automation and intelligent manufacturing technologies, automated welding technology, as a key production process, is widely used in industrial production processes, especially in fields such as automobile manufacturing, shipbuilding, and steel structure production. The quality of the weld seam directly affects the strength, safety, and service life of the structure. Therefore, accurate identification and welding of the weld seam have become an important link in improving product quality and production efficiency.

[0003] With the maturity of three-dimensional point cloud technology, the method for identifying weld seams based on point cloud data has become a research hotspot. Point cloud data can comprehensively and truly reflect the three-dimensional shape of an object, providing richer spatial information than traditional two-dimensional images. Therefore, the method for identifying weld seams based on three-dimensional point cloud data, with its richer spatial information and higher accuracy, has become an important research direction in the field of welding automation, providing a new technical path for improving welding quality and efficiency. However, the current 3D camera imaging technology is prone to missing some point clouds due to factors such as the smooth or reflective surface of the workpiece. The state of the surface of the welded workpiece, such as oil stains, oxides, etc., these factors will further affect the quality of the point cloud. When the point cloud processing algorithm processes missing data, it often relies on interpolation or prediction of surrounding points, which may lead to misjudgment when there are many missing points, greatly reducing the accuracy and robustness of weld seam identification. Summary of the Invention

[0004] In order to achieve efficient and accurate weld seam identification, an embodiment of the present invention provides a method and device for identifying a weld seam in point cloud based on dimensional transformation.

[0005] In a first aspect, an embodiment of the present invention provides a method for identifying a weld seam in point cloud based on dimensional transformation, which may include:

[0006] Obtain three-dimensional point cloud data of a target welding area;

[0007] Project the three-dimensional point cloud data onto a two-dimensional plane to obtain a depth map of the target welding area;

[0008] Input the depth map into a preset component segmentation model to obtain at least one welding workpiece mask;

[0009] For each welding workpiece mask, input the welding workpiece mask and the depth map into a preset plane segmentation model to obtain a plurality of plane masks;

[0010] Based on the depth map, back-project all the plane masks into the three-dimensional point cloud data to obtain coordinate information of a plurality of planes;

[0011] Determine all adjacent planes according to the coordinate information of the multiple planes, and determine the types and coordinate information of all welds according to the relative position relationship between every two adjacent planes.

[0012] In one or some alternative embodiments of the embodiments of the present application, the determining all adjacent planes according to the coordinate information of the multiple planes, and determining the types and coordinate information of all welds according to the relative position relationship between every two adjacent planes includes:

[0013] Determine all adjacent planes according to the coordinate information of the multiple planes;

[0014] For every two adjacent planes, determine whether there is an included angle relationship between the two adjacent planes:

[0015] If so, determine that the weld type of the weld between the two adjacent planes is a fillet weld, and calculate the coordinate information of the fillet weld based on the plane intersection algorithm;

[0016] If not, determine that the weld type of the weld between the two adjacent planes is a butt weld, and calculate the coordinate information of the butt weld based on the nearest neighbor boundary algorithm.

[0017] In one or some alternative embodiments of the embodiments of the present application, the back-projecting all the plane masks into the three-dimensional point cloud data based on the depth map to obtain the coordinate information of multiple planes includes:

[0018] For each pixel point in each plane mask, back-project to obtain the depth value of the pixel point based on the depth map and the coordinates of the pixel point;

[0019] Determine the coordinate information of the pixel point in the three-dimensional point cloud data according to the depth value.

[0020] In one or some alternative embodiments of the embodiments of the present application, the preset component segmentation model and the preset plane segmentation model are obtained through the following methods:

[0021] Collect multiple three-dimensional point cloud data including welding areas, and project the three-dimensional point cloud data onto a two-dimensional plane to obtain corresponding depth maps;

[0022] Label different components in all the depth maps to obtain a component segmentation data set;

[0023] Train a preset component segmentation model based on the component segmentation data set;

[0024] For each component in each depth map, label different planes on the component to obtain a plane segmentation data set;

[0025] Train a preset plane segmentation model based on the plane segmentation dataset.

[0026] In one or some alternative embodiments of the embodiments of the present application, the obtaining of the three-dimensional point cloud data of the target welding area includes:

[0027] Collect the initial point cloud data of the target welding area using a point cloud camera;

[0028] Based on the hand-eye matrix, convert the coordinates of the initial point cloud data to the robot base coordinate system to obtain the three-dimensional point cloud data of the target welding area.

[0029] In one or some alternative embodiments of the embodiments of the present application, after obtaining the three-dimensional point cloud data of the target welding area, it further includes:

[0030] Use the central cropping algorithm to reduce the size of the three-dimensional point cloud data;

[0031] Filter the redundant data in the three-dimensional point cloud data according to a preset depth threshold;

[0032] Perform normal vector calculation based on the three-dimensional point cloud data, and add normal vector geometric information to each point in the three-dimensional point cloud data;

[0033] Perform normalization processing on the three-dimensional point cloud data.

[0034] In one or some alternative embodiments of the embodiments of the present application, after determining the types and coordinate information of all welds, it further includes:

[0035] According to the internal parameters of the point cloud camera and the hand-eye matrix, convert the coordinate information of all welds from the three-dimensional point cloud coordinate system to the robot coordinate system to obtain the coordinate information of the welds in the robot coordinate system.

[0036] In a second aspect, an embodiment of the present invention provides a point cloud weld identification device based on dimensional transformation, which may include:

[0037] A first acquisition module, configured to acquire three-dimensional point cloud data of a target welding area;

[0038] A first projection module, configured to project the three-dimensional point cloud data onto a two-dimensional plane to obtain a depth map of the target welding area;

[0039] A first prediction module, configured to input the depth map into a preset component segmentation model to obtain at least one welding workpiece mask;

[0040] A second prediction module, configured to, for each of the welding workpiece masks, input the welding workpiece mask and the depth map into a preset plane segmentation model to obtain a plurality of plane masks;

[0041] A second projection module, configured to back-project all the plane masks into the three-dimensional point cloud data based on the depth map to obtain coordinate information of multiple planes;

[0042] A first determination module, configured to determine all adjacent planes according to the coordinate information of the multiple planes, and determine the types and coordinate information of all weld seams according to the relative position relationship between every two adjacent planes.

[0043] In a third aspect, an embodiment of the present invention provides a computer-readable storage medium, on which a computer program / instruction is stored, and when the computer program / instruction is executed by a processor, the method for identifying point cloud weld seams based on dimensional transformation as described above is implemented.

[0044] In a fourth aspect, an embodiment of the present invention provides a computer program product, including a computer program / instruction, and when the computer program / instruction is executed by a processor, the method for identifying point cloud weld seams based on dimensional transformation as described above is implemented.

[0045] In a fifth aspect, an embodiment of the present invention provides a computer device, including a memory, a processor, and a computer program stored on the memory, and when the processor executes the computer program, the method for identifying point cloud weld seams based on dimensional transformation as described above is implemented.

[0046] The beneficial effects of the above technical solutions provided by the embodiments of the present invention at least include:

[0047] An embodiment of the present invention provides a method for identifying point cloud welds based on dimensional transformation. This method obtains three-dimensional point cloud data of the target welding area to provide spatial information for subsequent weld identification. Then, the three-dimensional point cloud data is projected onto a two-dimensional plane to generate a depth map of the target welding area. Such a transformation makes subsequent processing simpler and is beneficial for the input of the model. Then, the depth map is input into a preset component segmentation model to obtain a welding workpiece mask for identifying different components in the welding area. Based on each welding workpiece mask, after combining it with the depth map, it is input into a plane segmentation model to obtain multiple plane masks. The inverse projection technology is used to reflect the plane masks from the two-dimensional depth map into the three-dimensional point cloud data to restore the coordinate information of multiple planes. Finally, based on the coordinate information between every two adjacent planes, the relative position relationship between adjacent planes is analyzed to accurately determine the type and coordinate information of the weld. This method can greatly reduce the complexity of point cloud data processing and effectively reduce the amount of calculation. Compared with traditional methods based on three-dimensional point cloud data, it not only improves the processing efficiency but also can accurately identify the weld area, avoiding redundancy and noise in point cloud data processing. At the same time, when performing image segmentation, this method uses a combination of a preset component segmentation model and a preset plane segmentation model, which can handle more complex environmental changes, improve the robustness and stability of the recognition results, and further help accurately identify the type and coordinate information of the weld, providing accurate positioning and decision-making support for subsequent automated welding processes.

[0048] Other features and advantages of the present invention will be described in the following specification, and, in part, will be obvious from the specification, or can be understood by implementing the present invention. The objectives and other advantages of the present invention can be achieved and obtained by the structures specifically pointed out in the written specification and the drawings.

[0049] The technical solution of the present invention will be further described in detail below through the drawings and embodiments. Description of the Drawings

[0050] The drawings are used to provide a further understanding of the present invention and constitute a part of the specification. They are used together with the embodiments of the present invention to explain the present invention and do not constitute a limitation to the present invention. In the drawings:

[0051] Figure 1 is a schematic flow chart of the method for identifying point cloud welds based on dimensional transformation provided by the embodiment of the present invention;

[0052] Figure 2 is a preprocessing flow chart of the three-dimensional point cloud data provided by the embodiment of the present invention;

[0053] Figure 3 is a flow chart of weld type identification extraction and post-processing provided by the embodiment of the present invention;

[0054] Figure 4 This is a schematic structural diagram of the point cloud weld recognition device based on dimensional transformation provided by the embodiments of the present application. Detailed implementation manners

[0055] Hereinafter, exemplary embodiments of the present disclosure will be described in more detail with reference to the accompanying drawings. Although the exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided so that the present disclosure can be more thoroughly understood and the scope of the present disclosure can be fully conveyed to those skilled in the art.

[0056] The inventors found that in the prior art, with the popularization of laser scanning technology, point cloud data has become an indispensable and important data source in weld detection. However, the high-dimensional, dense, and noisy characteristics of point cloud data make its processing and analysis extremely complex. Early weld recognition methods based on point cloud data often require a large amount of calculation and preprocessing, and have poor robustness.

[0057] Although in recent years, methods for processing point cloud data based on deep learning have made progress, most existing methods still focus on three-dimensional data processing, directly performing model prediction based on three-dimensional point cloud data, resulting in large computational amounts and low efficiency. At the same time, it is also difficult to cope with changes in complex environments, such as the density of point cloud data, the complexity of the background, and illumination changes, etc., resulting in the accuracy and robustness of weld recognition being unable to meet the requirements of industrial applications.

[0058] In addition, real-time performance and automation are another major challenge in the current welding process, especially in the dynamic welding process. In high-speed production lines, although there are already high-precision sensors that can acquire point cloud data, how to achieve real-time, efficient, and automated weld recognition and welding is still an urgent problem to be solved.

[0059] Based on this, the inventors have made further research and developed the present invention to provide a point cloud weld recognition method and device based on dimensional transformation.

[0060] Among them, a point cloud is a three-dimensional data set obtained by a three-dimensional scanning device, which represents the shape of an object as a set of discrete points in space. Three-dimensional point cloud data can not only provide the three-dimensional coordinate information of each point, but also provide the distance information between each point and the sensor through a depth map or other means. For the field of weld recognition, using three-dimensional point cloud data has two important advantages: providing the three-dimensional coordinates of the real world, and the position, shape, and other characteristics of the weld in the three-dimensional space can be directly reflected by the point cloud data; supplementing the missing information in the two-dimensional image. Since the two-dimensional image lacks depth information, the three-dimensional shape of the object cannot be accurately restored solely by the two-dimensional image, while the point cloud data can provide complete spatial information to ensure the three-dimensional positioning of the weld.

[0061] Example 1

[0062] In Example 1 of the present invention, a method for identifying a point cloud weld seam based on dimensional transformation is provided. Referring to Figure 1 as shown, the method may include the following steps S101 - S106:

[0063] S101: Obtain the three - dimensional point cloud data of the target welding area.

[0064] S102: Project the three - dimensional point cloud data onto a two - dimensional plane to obtain the depth map of the target welding area.

[0065] S103: Input the depth map into a preset component segmentation model to obtain at least one welding workpiece mask.

[0066] S104: For each welding workpiece mask, input the welding workpiece mask and the depth map into a preset plane segmentation model to obtain multiple plane masks.

[0067] S105: Based on the depth map, back - project all the plane masks into the three - dimensional point cloud data to obtain the coordinate information of multiple planes.

[0068] S106: According to the coordinate information of multiple planes, determine all adjacent planes, and according to the relative position relationship between every two adjacent planes, determine the types and coordinate information of all weld seams.

[0069] An embodiment of the present invention provides a method for identifying a point cloud weld seam based on dimensional transformation. This method obtains three-dimensional point cloud data of a target welding area, providing spatial information for subsequent weld seam identification. Then, the three-dimensional point cloud data is projected onto a two-dimensional plane to generate a depth map of the target welding area. Such a transformation makes subsequent processing simpler and is conducive to the input of the model. Then, the depth map is input into a preset component segmentation model to obtain a welding workpiece mask for identifying different components in the welding area. Based on each welding workpiece mask, after combining it with the depth map, it is input into a plane segmentation model to obtain multiple plane masks. The inverse projection technique is used to reflect the plane masks from the two-dimensional depth map into the three-dimensional point cloud data to restore the coordinate information of multiple planes. Finally, based on the coordinate information between every two adjacent planes, the relative position relationship between adjacent planes is analyzed to accurately determine the type and coordinate information of the weld seam. This method can greatly reduce the complexity of point cloud data processing and effectively reduce the computational amount. Compared with traditional methods based on three-dimensional point cloud data, it not only improves the processing efficiency but also can accurately identify the weld seam area, avoiding redundancy and noise in point cloud data processing. At the same time, when performing image segmentation, this method uses a combination of a preset component segmentation model and a preset plane segmentation model, which can cope with more complex environmental changes, improving the robustness and stability of the recognition result, and then helping to accurately identify the type and coordinate information of the weld seam, providing accurate positioning and decision-making support for subsequent automated welding processes.

[0070] In step S101 above, three-dimensional point cloud data of the target welding area is obtained. Specifically, it includes the following steps S1011 - S1012:

[0071] S1011: Use a point cloud camera to collect the initial point cloud data of the target welding area.

[0072] Specifically, it can be that the robot is taught through a predetermined path and position, and a point cloud camera is used in the target welding area to collect multiple frames of point cloud data from different perspectives to ensure coverage of the three-dimensional information of the entire welding area. Then, the multiple frames of point cloud data are stitched through point cloud stitching to obtain the initial point cloud data.

[0073] At the same time, defective data caused by exposure or shooting angle limitations during shooting is deleted.

[0074] S1012: Based on the hand-eye matrix, the coordinates of the initial point cloud data are converted to the robot base coordinate system to obtain the three-dimensional point cloud data of the target welding area.

[0075] Specifically, it can be that the initial point cloud data in the camera coordinate system is converted to the robot base coordinate system through the hand-eye matrix. This conversion ensures that the three-dimensional point cloud data can be aligned with the coordinate system of the robot operating environment, making subsequent welding positioning and control more accurate.

[0076] In the embodiments of the present application, after obtaining the three-dimensional point cloud data of the target welding area in the above step S101, it is also necessary to preprocess the three-dimensional point cloud data. Refer to Figure 2 As shown, the preprocessing specifically includes the following steps S10101-S10104:

[0077] S10101: Use the central cropping algorithm to reduce the size of the three-dimensional point cloud data.

[0078] Specifically, it can be to crop the size of the three-dimensional point cloud data through the central cropping algorithm, remove unnecessary areas, and retain the point cloud information related to the target welding area, which can reduce the computational complexity, improve the processing efficiency, enhance the correlation between the three-dimensional point cloud data and the weld seam, and enable the neural network to focus on the key areas related to the weld seam.

[0079] S10102: Filter redundant data in the three-dimensional point cloud data according to a preset depth threshold.

[0080] Specifically, it can be to set a depth threshold to shield the data in the three-dimensional point cloud data that is far from the camera. Points with a greater depth are usually irrelevant to the welding area. Filtering these redundant points can simplify the data set, reduce the noise during processing, and thus improve the accuracy of subsequent analysis.

[0081] S10103: Calculate the normal vector based on the three-dimensional point cloud data and add the normal vector geometric information to each point in the three-dimensional point cloud data.

[0082] Specifically, it can be to calculate the normal vector of each point in the three-dimensional point cloud data through the normal vector calculation algorithm, and attach the normal vector to each point in the three-dimensional point cloud data, further supplementing the geometric features of the three-dimensional point cloud data in space and providing more complete parameter information for the subsequent neural network learning.

[0083] S10104: Normalize the three-dimensional point cloud data.

[0084] Specifically, it can be to adjust the scale of the three-dimensional point cloud data through the normalization algorithm so that the range of each feature remains consistent. This not only helps to reduce the influence between different scales, but also improves the computational stability and computational speed of the neural network.

[0085] In the above step S102, project the three-dimensional point cloud data onto a two-dimensional plane to obtain the depth map of the target welding area.

[0086] Specifically, it can be achieved by projecting the 3D point cloud data onto a 2D plane to generate a depth map of the target welding area. In the specific operation, first, a suitable viewing angle is determined, and then the 3D point cloud data is converted into 2D data using perspective projection or orthographic projection. In this way, the point cloud information in the 3D space can be mapped into a 2D image, and the depth value of each point (i.e., the distance of the point from the camera) will be represented as the pixel value of the corresponding point in the 2D image.

[0087] Among them, in order to ensure that the spatial information of the target weld area is retained as much as possible, the suitable viewing angle selected during the projection can be the top, front, or side projection of the target welding area to highlight the shape and edge features of the weld and reduce the interference of background clutter. The depth map generated in this way can provide spatial information closely related to the weld area, thus helping the subsequent deep learning model to more accurately identify and segment the weld.

[0088] In a specific embodiment, the optimal shooting angle can usually be calculated based on the position and orientation of the weld relative to the camera, the layout of the workpiece, or the previous modeling information to ensure that the weld area is clearly visible and the occlusion is minimized.

[0089] In the embodiment of the present application, the above step S102 projects the 3D point cloud data onto a 2D plane to generate a depth map of the target welding area, which not only effectively simplifies the data dimension but also retains the spatial information of the weld area. During the projection process, selecting a suitable viewing angle can highlight the shape and edge features of the weld while reducing the interference of background clutter. This process ensures that the generation of the depth map can closely reflect the real spatial features of the weld area, providing more accurate and high-quality input data for the subsequent deep learning model, thereby improving the accuracy and robustness of weld recognition and segmentation.

[0090] In the above step S103, the depth map is input into a preset component segmentation model to obtain at least one welding workpiece mask.

[0091] Specifically, it can be that the depth map is input into a preset component segmentation model, and the preset component segmentation model will identify all the welding workpieces in the depth map according to the spatial information in the depth map, distinguish each welding workpiece from other non-welding areas or welding workpieces, and output the welding workpiece mask corresponding to each welding workpiece.

[0092] Among them, the preset component segmentation model is a 2D segmentation network model constructed based on the Segment Anything Model (SAM) network deep learning technology. Segment Anything Model is a general segmentation framework based on transformers that can handle various types of image segmentation tasks, including object segmentation, instance segmentation, and semantic segmentation.

[0093] Those skilled in the art can select a suitable neural network according to the detailed description of the prior art to train a preset component segmentation model. The training process may specifically include:

[0094] In the first step, collect a plurality of three-dimensional point cloud data containing welding areas. After preprocessing, project all the three-dimensional point cloud data onto a two-dimensional plane to obtain a depth map corresponding to each three-dimensional point cloud data.

[0095] In the second step, preprocess and perform data augmentation on all depth maps. The data augmentation includes rotation, flipping, scaling, etc. And use the data annotation tool Labelme to label each component in all depth maps, and save the label data as a.json file in the plan label format to obtain a component segmentation data set.

[0096] Among them, the way to preprocess the depth map is the same as the preprocessing method described in the above step S101, and will not be elaborated here.

[0097] In addition, it is also necessary to perform dimensional transformation on the data in the component segmentation data set. The purpose of performing dimensional transformation is to adapt to the input format of the deep learning framework (such as PyTorch). The deep learning framework usually expects the input data to contain a batch dimension (batch), so a batch dimension is added to accelerate the training process using batch processing and effectively utilize hardware resources (such as GPU) for subsequent network calculations.

[0098] In the third step, select a suitable neural network model as the initial component segmentation model, such as the Segment AnythingModel model, the U-Net model, etc.

[0099] Among them, in order to facilitate those skilled in the art to understand the preset component segmentation model, taking the Segment AnythingModel model as an example, the structure of the model is briefly introduced here. The Segment Anything Model network adopts a Transformer encoder-decoder architecture and improves the semantic expression ability of the image by fusing features at different scales. The specific structure is as follows:

[0100] Encoder: Capture the global features of the image through a series of convolutional layers and multi-layer self-attention mechanisms. This part converts the important information in the image into a high-dimensional feature map for the decoder to use.

[0101] Decoder: Convert the features extracted by the encoder into the segmentation results of the target area through the decoder. The decoder also combines Mask information to make the class prediction of each pixel more accurate.

[0102] Output: The final output is a binary image, in which the weld area is marked as the target area and the non-weld area is marked as the background.

[0103] In addition, based on the Segment Anything Model, in order to extract specific discriminative features and improve the recognition accuracy, this method will adopt the methods of feature fusion and multi-scale feature learning. Multi-scale feature learning: The model extracts features at multiple scales to capture different details of the weld area. By fusing features at different scales, the adaptability of the model to small welds, different welding morphologies, and complex backgrounds is enhanced. Local and global feature fusion: During the training process, the model not only learns the detailed features of local welds but also learns the global structural features so that the shape, position, and adjacent relationships of the welds can be comprehensively considered during recognition.

[0104] Fourth step, divide the component segmentation dataset into a training set, a validation set, and a test set. The division ratio of the training set, validation set, and test set should follow the following principles: Training set: 70%-80% of the data is used for training; Validation set: 20%-30% of the data is used to verify the performance of the model in real time to avoid overfitting; Test set: Independent of the training and validation processes, it is used to finally evaluate the generalization ability of the model.

[0105] Fifth step, define the training strategy and optimization strategy, including setting loss functions (such as cross-entropy loss function, IoU loss, etc.) and optimization algorithms (such as Adam, SGD, etc.). Use the learning rate decay strategy, set the initial learning rate to a relatively large value (such as 1e-4), and gradually decrease it as the training progresses to avoid oscillation phenomena in the later stage of training. Set the batch size to 16 to make full use of hardware acceleration and balance the training speed and memory consumption. Adopt the early stopping strategy during the training process to prevent overfitting.

[0106] Sixth step, use the training set and validation set of the component segmentation dataset to train the initial component segmentation model to obtain the trained component segmentation model.

[0107] Repeat the above training process of the component segmentation model until the preset conditions are met, stop the training, and obtain the preset component segmentation model. Among them, the preset conditions can be set to reach a fixed number of iterations, the accuracy reaches a threshold, the accuracy does not change within the preset number of iterations, etc. No specific limitations are made here.

[0108] Among them, after the preset conditions are completed, the trained component segmentation model can also be evaluated through a test set. Common evaluation metrics include: Accuracy: The ratio of the pixels correctly predicted by the model to the total pixels. Intersection over Union (IoU): Used to evaluate the discrimination effect between the target area and the background area. Dice coefficient: Used to measure the accuracy of segmentation, especially the performance of small areas in the weld area. Through the above evaluation metrics, the performance of the component segmentation model is finally determined and optimized to improve accuracy and robustness.

[0109] In the above step S104, for each welding workpiece mask, the welding workpiece mask and the depth map are input into a preset plane segmentation model to obtain multiple plane masks.

[0110] Specifically, it can be that by combining each welding workpiece mask with the depth map, the regional information related to the welding workpiece is extracted. The main purpose is to screen and extract the part of the depth map corresponding to the welding workpiece mask, so as to focus on the area where the welding workpiece is located. Then, these extracted areas are used as inputs and input into the preset plane segmentation model.

[0111] In this step, the preset plane segmentation model will identify the input image and identify multiple plane areas existing in the welding workpiece. In this way, the preset plane segmentation model can effectively separate different planes in the welding workpiece, providing more accurate information for subsequent weld identification and segmentation.

[0112] Among them, the preset plane segmentation model, like the preset component segmentation model, is a 2D segmentation network model constructed based on the Segment Anything Model network deep learning technology.

[0113] Those skilled in the art can select a suitable neural network according to the detailed description of the prior art to train the preset plane segmentation model. The training process of the preset plane segmentation model is basically the same as the training process of the preset component segmentation model in the above step S103. A brief introduction is given here. The training process can specifically include:

[0114] The first step is to collect a plurality of three-dimensional point cloud data containing welding areas. After preprocessing, all the three-dimensional point cloud data are projected onto a two-dimensional plane to obtain the depth map corresponding to each three-dimensional point cloud data.

[0115] The second step is to preprocess, perform data augmentation and dimensional transformation on all depth maps, and for each component in each depth map, use the data annotation tool Labelme to label different planes on each component, and save the label data as a.json file in the plan label format to obtain a component segmentation data set.

[0116] In the third step, select a suitable neural network model as the initial plane segmentation model, such as the Segment Anything Model, U-Net model, etc.

[0117] In the fourth step, divide the plane segmentation dataset into a training set, a validation set, and a test set.

[0118] In the fifth step, define a loss function (such as cross-entropy loss function, IoU loss, etc.), an optimization algorithm (such as Adam, SGD, etc.), etc.

[0119] In the sixth step, use the training set and validation set of the plane segmentation dataset to train the initial plane segmentation model to obtain the trained plane segmentation model.

[0120] Repeat the above training process of the plane segmentation model until the preset conditions are met, stop training, and obtain the preset plane segmentation model. Among them, the preset conditions can be set to reach a fixed number of iterations, the accuracy reaches a threshold, the accuracy does not change within the preset number of iterations, etc. No specific limitation is made here.

[0121] Summary: Through two trainings and 2D segmentations by this network, they are component segmentation respectively. Taking components as units, the workpiece structure is segmented, and then plane segmentation is carried out. Taking planes as units, component information is segmented, and all the segmented plane information is processed by a weld recognition algorithm.

[0122] In the embodiments of the present application, the above steps S103 - S104 complete the recognition of the planes on the weld workpiece through image segmentation in two stages, including:

[0123] Component Segmentation: Corresponding to the above step S103, the purpose of this step is to segment different welding workpieces and extract the structure in units of welding workpieces. Each welding workpiece will be individually recognized and segmented, providing clear segmentation information for the subsequent steps.

[0124] Plane Segmentation: After the segmentation of the welding workpiece is completed, plane segmentation processing is performed based on the preset plane segmentation model, and each welding workpiece is segmented into different planes, and each plane represents a sectional or surface information of the welding workpiece. In this way, the plane features where the welds are located can be more accurately extracted, further realizing the recognition of the weld area.

[0125] In summary, in the above steps S103 - S104, by inputting the depth map into a preset component segmentation model to extract the welding workpiece mask, and then combining the welding workpiece mask and the depth map to input into the plane segmentation model to extract the plane mask, each region of the welding workpiece can be refined and processed hierarchically. This process makes the identification of the weld seam region more accurate, avoiding misidentification caused by complex backgrounds or variable workpiece shapes. The two - stage segmentation strategy not only improves the accuracy of identification, but also provides clear and structured feature data for subsequent weld seam identification, helping to improve the overall identification effect and application efficiency.

[0126] In the above step S105, based on the depth map, all plane masks are back - projected into the three - dimensional point cloud data to obtain the coordinate information of multiple planes. Specifically, it includes the following steps S1051 - S1052:

[0127] S1051: For each pixel point in each plane mask, based on the depth map and the coordinates of the pixel point, back - project to obtain the depth value of the pixel point.

[0128] Specifically, it can be that, through the depth map or the point cloud data, the depth value corresponding to each pixel in the two - dimensional image (i.e., the distance from the camera) is obtained. In this step, by performing depth queries on each pixel point in the image and combining the camera calibration information, the depth data of each pixel point in the actual three - dimensional space is obtained. These depth values will provide the necessary information for subsequent three - dimensional coordinate recovery.

[0129] S1052: Determine the coordinate information of the pixel point in the three - dimensional point cloud data according to the depth value.

[0130] Specifically, it can be that, based on the internal parameter matrix of the point cloud camera, the pixel coordinates of each pixel point in the plane mask are transformed into three - dimensional coordinates in the camera coordinate system. Given the depth value, combined with the internal and external parameters of the camera, through the back - projection formula, the weld seam pixel points in the two - dimensional image are mapped into the three - dimensional space to obtain the actual coordinates of the weld seam in the three - dimensional space. This process is called the back - projection operation, through which the position of each pixel point in the three - dimensional space can be accurately restored, and then the three - dimensional positioning of the weld seam region is completed.

[0131] In the embodiment of the present application, although the coordinate information of the plane is obtained through the above step S104, this information is still limited to the two - dimensional plane. If the weld seam needs to be positioned in the actual three - dimensional space, the back - projection operation must be performed through step S105 to map the points on the two - dimensional image back to the three - dimensional point cloud space. Through the above back - projection operation, the two - dimensional plane coordinate information can be accurately mapped into the three - dimensional space, which provides reliable three - dimensional space data support for subsequent three - dimensional weld seam identification, precise positioning and automated operation, and improves the operability in practical applications.

[0132] In the above step S106, according to the coordinate information of multiple planes, all adjacent planes are determined, and according to the relative position relationship between every two adjacent planes, the types and coordinate information of all welds are determined. Specifically, it includes the following steps S1061 - S1064:

[0133] S1061: According to the coordinate information of multiple planes, all adjacent planes are determined.

[0134] Specifically, it can be by calculating the spatial distance and relative position relationship between planes to determine which planes are adjacent to each other and constructing a plane adjacency relationship graph. This step provides a basis for the subsequent judgment of weld types and coordinate extraction.

[0135] S1062: For every two adjacent planes, determine whether there is an included angle relationship between the two adjacent planes: if so, execute the following step S1063. If not, execute the following step S1064.

[0136] S1063: Determine that the weld type of the weld between two adjacent planes is a fillet weld, and based on the plane intersection algorithm, calculate the coordinate information of the fillet weld.

[0137] Specifically, it can be based on the plane intersection algorithm to calculate the intersection line of two adjacent planes in three - dimensional space. This intersection line is the trajectory of the fillet weld, and the coordinate information of the fillet weld can be extracted through this trajectory.

[0138] S1064: Determine that the weld type of the weld between two adjacent planes is a butt weld, and based on the nearest - neighbor boundary algorithm, calculate the coordinate information of the butt weld.

[0139] Specifically, it can be based on the nearest - neighbor boundary algorithm to calculate the adjacent intersection line between two planes. This intersection line is the trajectory of the butt weld, and the coordinate information of the butt weld can be extracted through this trajectory.

[0140] In the embodiment of the present application, after determining the types and coordinate information of all welds in the above step S106, further processing and optimization are required to improve the recognition accuracy and reliability. Refer to Figure 3 As shown, after contour simplification, judging the weld type based on the included angle and executing the corresponding algorithm to determine the weld coordinate information, the subsequent steps include: projecting a straight line, analyzing by mapping the geometric features of the weld area to different perspectives or planes, and then optimizing its shape; projecting point statistics, counting the distribution of the weld area in different projection planes to help accurately determine the geometric position of the weld; contour optimization, smoothing and adjusting the extracted weld contour to remove noise and fill in missing parts to ensure that its shape is as close to the real situation as possible.

[0141] In addition, according to the internal parameters of the point cloud camera and the hand-eye matrix, the coordinate information of all welds needs to be converted from the three-dimensional point cloud coordinate system to the robot coordinate system to obtain the coordinate information of the welds in the robot coordinate system.

[0142] Finally, the three-dimensional weld coordinate information is determined, and then this coordinate information can be used for welding path planning, precise positioning of welding points, and real-time monitoring during the welding process. The automated welding system can utilize this coordinate information to achieve precise control of the welding path, thereby ensuring the quality and accuracy of welding.

[0143] Embodiment 2

[0144] Based on the same inventive concept, an embodiment of the present invention further provides a point cloud weld recognition device based on dimensional transformation. Referring to Figure 4 as shown, the device includes:

[0145] A first acquisition module 101, configured to acquire three-dimensional point cloud data of a target welding area;

[0146] A first projection module 102, configured to project the three-dimensional point cloud data onto a two-dimensional plane to obtain a depth map of the target welding area;

[0147] A first prediction module 103, configured to input the depth map into a preset component segmentation model to obtain at least one welding workpiece mask;

[0148] A second prediction module 104, configured to, for each of the welding workpiece masks, input the welding workpiece mask and the depth map into a preset plane segmentation model to obtain a plurality of plane masks;

[0149] A second projection module 105, configured to, based on the depth map, back-project all the plane masks into the three-dimensional point cloud data to obtain coordinate information of a plurality of planes;

[0150] A first determination module 106, configured to determine all adjacent planes according to the coordinate information of the plurality of planes, and determine the types and coordinate information of all welds according to the relative position relationship between every two adjacent planes.

[0151] Embodiment 3

[0152] Based on the same inventive concept, an embodiment of the present invention further provides a computer-readable storage medium, on which a computer program / instructions are stored, and when the computer program / instructions are executed by a processor, the method for recognizing point cloud welds based on dimensional transformation described in Embodiment 1 above is implemented.

[0153] Embodiment 4

[0154] Based on the same inventive concept, an embodiment of the present invention further provides a computer program product, including computer programs / instructions, which when executed by a processor implement the method for identifying point cloud weld seams based on dimensional transformation as described in the first embodiment above.

[0155] Embodiment Five

[0156] Based on the same inventive concept, an embodiment of the present invention further provides a computer device, including a memory, a processor, and a computer program stored on the memory, which when executed by the processor implement the method for identifying point cloud weld seams based on dimensional transformation as described in the first embodiment above.

[0157] Those skilled in the art should understand that the embodiments of the present invention can be provided as a method, a system, or a computer program product. Therefore, the present invention can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present invention can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk memories and optical memories, etc.) containing computer-usable program codes.

[0158] The present invention is described with reference to the flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and the combination of flows and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to generate a machine, such that the instructions executed by the processor of the computer or other programmable data processing devices generate means for implementing the functions specified in Figure 1 one flow or multiple flows and / or blocks Figure 1 one block or multiple blocks.

[0159] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, such that the instructions stored in the computer-readable memory generate a manufactured article including instruction means, and the instruction means implement the functions specified in Figure 1 one flow or multiple flows and / or blocks Figure 1 one block or multiple blocks.

[0160] These computer program instructions can also be loaded onto a computer or other programmable data processing device, such that a series of operation steps are executed on the computer or other programmable device to generate a computer-implemented process, and thus the instructions executed on the computer or other programmable device provide for implementing the functions in the flowFigure 1 One process or multiple processes and / or boxes Figure 1 Steps of the functions specified in one box or multiple boxes.

[0161] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalent technologies, the present invention is also intended to include these modifications and variations.

Claims

1. A method for identifying point cloud weld seams based on dimensional transformation, characterized in that, Including: Obtain the three-dimensional point cloud data of the target welding area; Project the three-dimensional point cloud data onto a two-dimensional plane to obtain the depth map of the target welding area; Input the depth map into a preset component segmentation model to obtain at least one welding workpiece mask; For each welding workpiece mask, input the welding workpiece mask and the depth map into a preset plane segmentation model to obtain multiple plane masks; Based on the depth map, back-project all the plane masks into the three-dimensional point cloud data to obtain the coordinate information of multiple planes; According to the coordinate information of the multiple planes, determine all adjacent planes, and according to the relative position relationship between every two adjacent planes, determine the types and coordinate information of all welds.

2. The method according to claim 1, wherein The step of according to the coordinate information of the multiple planes, determining all adjacent planes, and according to the relative position relationship between every two adjacent planes, determining the types and coordinate information of all welds includes: According to the coordinate information of the multiple planes, determine all adjacent planes; For every two adjacent planes, judge whether there is an included angle relationship between the two adjacent planes: If so, determine that the weld type of the weld between the two adjacent planes is a fillet weld, and based on the plane intersection algorithm, calculate the coordinate information of the fillet weld; If not, determine that the weld type of the weld between the two adjacent planes is a butt weld, and based on the nearest neighbor boundary algorithm, calculate the coordinate information of the butt weld.

3. The method according to claim 1, wherein The step of based on the depth map, back-projecting all the plane masks into the three-dimensional point cloud data to obtain the coordinate information of multiple planes includes: For each pixel point in each plane mask, based on the depth map and the coordinates of the pixel point, back-project to obtain the depth value of the pixel point; According to the depth value, determine the coordinate information of the pixel point in the three-dimensional point cloud data.

4. The method according to claim 1, wherein Obtain the preset component segmentation model and the preset plane segmentation model through the following methods: Collect multiple three-dimensional point cloud data containing welding areas, and project the three-dimensional point cloud data onto a two-dimensional plane to obtain corresponding depth maps; Label different components in all the depth maps to obtain a component segmentation data set; Train a preset component segmentation model based on the component segmentation data set; For each component in each depth map, label different planes on the component to obtain a plane segmentation data set; Train a preset plane segmentation model based on the plane segmentation data set.

5. The method according to claim 1, characterized in that, The step of obtaining the three-dimensional point cloud data of the target welding area includes: Use a point cloud camera to collect the initial point cloud data of the target welding area; Based on the hand-eye matrix, convert the coordinates of the initial point cloud data to the robot base coordinate system to obtain the three-dimensional point cloud data of the target welding area.

6. The method according to claim 1, wherein After obtaining the three-dimensional point cloud data of the target welding area, it further includes: Use the central cropping algorithm to reduce the size of the three-dimensional point cloud data; Filter redundant data in the three-dimensional point cloud data according to a preset depth threshold; Perform normal vector calculation based on the three-dimensional point cloud data, and add normal vector geometric information to each point in the three-dimensional point cloud data; Perform normalization processing on the three-dimensional point cloud data.

7. The method according to claim 1, characterized in that, After determining the types and coordinate information of all welds, it further includes: According to the internal parameters of the point cloud camera and the hand-eye matrix, the coordinate information of all the welds is converted from the three-dimensional point cloud coordinate system to the robot coordinate system, and the coordinate information of the welds in the robot coordinate system is obtained.

8. A point cloud weld seam recognition device based on dimensional transformation, characterized in that It includes: A first acquisition module, configured to acquire three-dimensional point cloud data of a target welding area; A first projection module, configured to project the three-dimensional point cloud data onto a two-dimensional plane to obtain a depth map of the target welding area; A first prediction module, configured to input the depth map into a preset component segmentation model to obtain at least one welding workpiece mask; A second prediction module, configured to, for each of the welding workpiece masks, input the welding workpiece mask and the depth map into a preset plane segmentation model to obtain a plurality of plane masks; A second projection module, configured to, based on the depth map, back-project all the plane masks into the three-dimensional point cloud data to obtain the coordinate information of a plurality of planes; A first determination module, configured to determine all adjacent planes according to the coordinate information of the plurality of planes, and determine the types and coordinate information of all welds according to the relative position relationship between every two adjacent planes.

9. A computer-readable storage medium having computer programs / instructions stored thereon, characterized in that, When the computer program / instructions are executed by a processor, the method for identifying point cloud welds based on dimensional transformation according to any one of claims 1-7 is implemented.

10. A computer device, comprising a memory, a processor, and a computer program stored on the memory, characterized in that, The processor executes the computer program to implement the method for identifying point cloud welds based on dimensional transformation according to any one of claims 1-7.

Images